Commit Graph

48 Commits

Author SHA1 Message Date
Tyera c5368a3d35
ledger-tool: read bigtable entries (#34266)
* Add helper to get a protobuf cell without bothering with bincode

* Add conversion back to EntrySummaries

* Add LedgerStorage::get_block_entries

* Add ledger-tool bigtable entries subcommand

* Move CliEntries into ledger_tool::output for now

* Fetch entries with read-only token
2023-11-30 21:33:04 +00:00
Kirill Fomichev 873bef9fef
bigtable: fix AccessToken issues (#34213)
* bigtable: fix AccessToken issue

* remove inner

* less changes

* fmt + drop lock
2023-11-27 16:46:49 -07:00
Brooks 7d7d72e498
clippy: Allow enum variant names in generated google.bigtable.v2 (#34126) 2023-11-16 18:21:11 -05:00
Tyera 6fc6a49c05
storage-bigtable: Return error if attempting to write to table NotFound (#34098)
* Add helper fn to stop retrying if table not found

* Use helper in put_cells_with_retry methods

* Review suggestions
2023-11-16 06:15:57 +00:00
Yihau Chen 94cb88ffad
add solana-ledger-tool bigtable copy (#28122)
* init copy cmd

* extract creating emulator connection logic

* extract copy args as struct

* add new_for_emulator

* add tryFrom confirmed block to versioned block

* implement bigtable copy command

* use 'force' flag to force upload

* use unwrap_or

* remove redundant importing

* fix nightly lint

* explicit transactions missing error

* process ending_slot

* prevent start slot > end slot

* print skip slots in debug level

* fix destination bigtable should not be readonly

* combine is-emulator and endpoint to emulated source. conflict with crenditial path

* wording

* log some error messages with error level

* nightly lint

* add dry-run

* extract create bigtable instances logic

* use a lighter way to check block

* use the latest futures version which is used in the repo

* use futures = "0.3"

Co-authored-by: Tyera <teulberg@gmail.com>

* wording

Co-authored-by: Tyera <teulberg@gmail.com>

* wording

Co-authored-by: Tyera <teulberg@gmail.com>

Co-authored-by: Tyera <teulberg@gmail.com>
2023-01-09 11:23:35 +08:00
behzad nouri 9524c9dbff patches errors from clippy::uninlined_format_args
https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
2022-12-06 19:32:15 +00:00
Yihau Chen d949f4f42f
add a lighter way to check whether a block exists in the Bigtable (#28308)
* add does_row_key_exist

* add does_confirmed_block_exist

* lint

* Update storage-bigtable/src/bigtable.rs

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* rename does_row_key_exist -> row_key_exists

* rename does_confirmed_block_exist -> confirmed_block_exists

* Update storage-bigtable/src/lib.rs

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2022-10-18 06:34:26 +00:00
Tyera Eulberg 7a11571e7f
Update BigTable apis to respect a limit of zero (#27380)
Update apis to respect a limit of zero
2022-08-24 12:44:19 -06:00
apfitzge caca525695
ignore clippy derive_partial_eq_without_eq in prost generated code (#27337)
ignore clippy derive_partial_eq_without_eq in prost generated code
2022-08-23 14:21:24 -05:00
Richard Patel 270315a7f6
transaction-status, storage-proto: add compute_units_consumed (#26528)
* transaction-status, storage-proto: add compute_units_consumed

* fix bpf test

Co-authored-by: Justin Starry <justin@solana.com>
2022-08-06 17:14:31 +00:00
Kirill Fomichev 0c98b06249
bigtable: set_nodelay for http connector (#26398) 2022-07-25 15:16:55 -06:00
lwus d12e566a4c
bigtable: allow forward proxy in gRPC connection (#25918)
* bigtable: allow forward proxy in tonic connection

* bigtable: try fixing ci

* bigtable: cargo fmt

* bigtable: add docs for BIGTABLE_PROXY env var

* bigtable: fix readme header level

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2022-06-23 10:51:31 -06:00
Dan Coombs 32a58dd9e0
Add ability to use a non-default app profile id in bigtable requests (#25968)
* Add ability to use a non-default app profile id in bigtable requests

* Only run subcommand once when getting global configs

* Remove unneded scoping on option type
2022-06-16 00:58:16 +00:00
dependabot[bot] 4520c25330
chore: bump tonic-build from 0.7.1 to 0.7.2 (#25037)
* chore: bump tonic-build from 0.7.1 to 0.7.2

Bumps [tonic-build](https://github.com/hyperium/tonic) from 0.7.1 to 0.7.2.
- [Release notes](https://github.com/hyperium/tonic/releases)
- [Changelog](https://github.com/hyperium/tonic/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/tonic/compare/v0.7.1...v0.7.2)

---
updated-dependencies:
- dependency-name: tonic-build
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

* Regenerate proto

* Bump tonic and remove deprecated method

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2022-05-09 12:26:58 -06:00
Rachael Pai aea17c35ae
Add a stringified credential option for LedgerStorage (#24314)
* add a stringified credential option for LedgerStorage

* fix clippy::useless-format warning

* change CredentialOption to enum CredentialType

* rename credential_option to credential_type

* restore LedgerStorage new fn signature

* fmt

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2022-04-13 14:35:06 -06:00
Edgar Xi d8be0d9430 make get_protobuf_or_bincode_cells accept IntoIter on row_keys, make get_confirmed_blocks_with_data return an Iterator 2022-03-22 22:47:25 -06:00
Edgar Xi f717fda9a3 modify get_protobuf_or_bincode_cells to accept and return an iterator 2022-03-22 22:47:25 -06:00
Edgar Xi fbcf6a0802 use &[T] instead of Vec<T> where appropriate
clippy
2022-03-22 22:47:25 -06:00
Edgar Xi f3219fb695 add get_confirmed_blocks_with_data and get_protobuf_or_bincode_cells 2022-03-22 22:47:25 -06:00
Jon Cinque 7af48465fa
transaction-status: Add return data to meta (#23688)
* transaction-status: Add return data to meta

* Add return data to simulation results

* Use pretty-hex for printing return data

* Update arg name, make TransactionRecord struct

* Rename TransactionRecord -> ExecutionRecord
2022-03-22 23:17:05 +01:00
Justin Starry d5dec989b9
Enforce tx metadata upload with static types (#23028) 2022-02-10 13:28:18 +08:00
Justin Starry f804ccdece
Store address table lookups in blockstore and bigtable (#22402) 2022-01-14 15:24:41 +08:00
pieceofr d9220652ad
[ledger-tool]compare_blocks (#22229)
* 1.made load_credentials accept credential path as a parameter. 2.partial implement bigtable comparasion function

* finding missing blocks in bigtables in a specified range

* refactor compare-blocks,add unit test for missing_blocks and fmt

* compare-block fix last block bug

* refactor compare-block and improve wording

* Update ledger-tool/src/bigtable.rs

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* update compare-block command-line description

* style:improve wording/naming/code style

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2022-01-06 14:36:03 +08:00
Michael Vines b8837c04ec Reformat imports to a consistent style for imports
rustfmt.toml configuration:
  imports_granularity = "One"
  group_imports = "One"
2021-12-03 09:19:13 -08:00
Tyera Eulberg f0de3e9bf0
chore: bump tonic from 0.5.2 to 0.6.1 (#21035)
* Bump tonic and prost crates

* connect_lazy no longer infallible
2021-10-28 06:59:36 +00:00
Justin Starry c71fab6cb3
Add `delete` subcommand to `ledger-tool bigtable` (#19931)
* Add `delete` subcommand to `ledger-tool bigtable` command

* feedback
2021-09-16 23:37:45 +00:00
Tyera Eulberg c2e7d39154
Add storage-proto build.rs and readme (#18353)
* Use build.rs for storage-proto generation

* Add readme

* Single use statements
2021-07-09 20:06:06 +00:00
Tyera Eulberg 761de8b1a3
Bump prost, prost-types, and tonic (#18537)
* Bump prost+tonic and accommodate generated service changes

* Unignore advisory

* Fixup .proto error list
2021-07-09 17:31:26 +00:00
Michael Vines 4a12c715a3 Drop Error suffix from enum values to avoid the enum_variant_names clippy lint 2021-06-18 23:02:13 +00:00
Alexander Meißner 6514096a67 chore: cargo +nightly clippy --fix -Z unstable-options 2021-06-18 10:42:46 -07:00
Tyera Eulberg ab581dafc2
Add block height to ConfirmedBlock structs (#17523)
* Add BlockHeight CF to blockstore

* Rename CacheBlockTimeService to be more general

* Cache block-height using service

* Fixup previous proto mishandling

* Add block_height to block structs

* Add block-height to solana block

* Fallback to BankForks if block time or block height are not yet written to Blockstore

* Add docs

* Review comments
2021-05-26 22:16:16 -06:00
Michael Vines 9541411c15 Plumb transaction-level rewards (aka "rent debits") into the `getTransaction` RPC method 2021-05-27 03:05:05 +00:00
François Garillot b08cff9e77
Simplify some pattern-matches (#16402)
When those match an exact combinator on Option / Result.

Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust).
2021-04-08 12:40:37 -06:00
Tyera Eulberg d1563f0ccd
Bump tonic, prost, tarpc, tokio (#15013)
* Update tonic & prost, and regenerate proto

* Reignore doc code

* Revert pull #14367, but pin tokio to v0.2 for jsonrpc

* Bump backoff and goauth -> and therefore tokio

* Bump tokio in faucet, net-utils

* Bump remaining tokio, plus tarpc
2021-02-05 00:21:53 -07:00
Ryo Onodera dcaa025822
Configure Bigtable's timeout, enabling by default (#14657)
* Configure bigtable's timeout when read-only

* Review comments

* Apply nits (thanks!)

Co-authored-by: Michael Vines <mvines@gmail.com>

* Timeout in the streamed decoding as well

Co-authored-by: Michael Vines <mvines@gmail.com>
2021-01-19 13:57:16 +00:00
Tyera Eulberg 2eb19fa5e5
Improve docs around bigtable read limit (#14660) 2021-01-19 06:33:51 +00:00
Michael Vines 7143aaa89b Clippy 2020-12-14 08:03:29 -08:00
Josh 13db3eca9f
SPL token balance in transaction metadata (#13673)
* feat: store pre / post token balances

* move helper functions into separate include

* move token balance functionality to transaction-status crate

* fix blockstore processor test

* fix bigtable legacy test

* add caching to decimals
2020-12-10 19:25:07 -08:00
Michael Vines 959880db60 Remove unused pubkey::Pubkey imports 2020-10-21 19:08:13 -07:00
Michael Vines 7bc073defe Run `codemod --extensions rs Pubkey::new_rand solana_sdk::pubkey::new_rand` 2020-10-21 19:08:13 -07:00
Tyera Eulberg 359707c85e
Convert Blockstore Rewards cf to protobuf (#12860)
* Add Blockstore protobuf cf type

* Add Rewards message to proto and make generated pub

* Convert Rewards cf to ProtobufColumn

* Add bench

* Adjust tags

* Move solana proto definitions and conversion methods to new crate
2020-10-15 18:04:10 -06:00
Josh 8f5431551e
Store program logs in blockstore / bigtable (TransactionWithStatusMeta) (#12678)
* introduce store program logs in blockstore / bigtable

* fix test, transaction logs created for successful transactions

* fix test for legacy bincode implementation around log_messages

* only api nodes should record logs

* truncate transaction logs to 100KB

* refactor log truncate for improved coverage
2020-10-08 12:06:15 -07:00
Justin Starry ce598c5c98
Use protobufs to store confirmed blocks in BigTable (#12526)
* Use protobufs to store confirmed blocks in BigTable

* Cleanup

* Reorganize proto

* Clean up use statements

* Split out function for unit testing

* s/utils/convert

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-09-30 17:55:22 +00:00
Tyera Eulberg b041afe1be
Bigtable method to return a single row of data (#11999) 2020-09-02 13:41:25 -06:00
Tyera Eulberg 55ce2ebd53
Get index (#11694) 2020-08-18 17:24:25 +00:00
Tyera Eulberg 6c5b8f324a
Rpc: Add until parameter for getConfirmedSignaturesForAddress2 (#11644)
* Refactor bigtable apis to accept start and end keys

* Make helper fn to deserialize cell data

* Refactor get_confirmed_signatures_for_address to use get_row_data range

* Add until param to get_confirmed_signatures_for_address

* Add until param to blockstore api

* Plumb until through client/cli

* Simplify client params
2020-08-15 10:42:17 -06:00
Michael Vines 1f7af14386 Simplify access token refreshing 2020-08-05 14:27:12 -07:00
Michael Vines 6e0353965a Add bigtable 2020-08-05 14:27:12 -07:00