* init copy cmd
* extract creating emulator connection logic
* extract copy args as struct
* add new_for_emulator
* add tryFrom confirmed block to versioned block
* implement bigtable copy command
* use 'force' flag to force upload
* use unwrap_or
* remove redundant importing
* fix nightly lint
* explicit transactions missing error
* process ending_slot
* prevent start slot > end slot
* print skip slots in debug level
* fix destination bigtable should not be readonly
* combine is-emulator and endpoint to emulated source. conflict with crenditial path
* wording
* log some error messages with error level
* nightly lint
* add dry-run
* extract create bigtable instances logic
* use a lighter way to check block
* use the latest futures version which is used in the repo
* use futures = "0.3"
Co-authored-by: Tyera <teulberg@gmail.com>
* wording
Co-authored-by: Tyera <teulberg@gmail.com>
* wording
Co-authored-by: Tyera <teulberg@gmail.com>
Co-authored-by: Tyera <teulberg@gmail.com>
Currently, if starting-slot is unspecified, a value of 0 will be chosen.
In the common case where someone is operating on a much more recent
range, this would result in a ton of wasted operations & time.
Instead, choose a smarter default value for starting-slot based on what
we detect is currently in the blockstore.
RocksDB settings include an option to create_if_missing, which will
create missing columns or the entire rocksdb directory if starting from
scratch. However, create_if_missing functionality only works if the
session has Primary (read+write) access. Many ledger-tool commands only
need Secondary (read-only) access to the database, so these commands are
unable to open the Blockstore when a column must be added.
This change detects when Secondary access open fails due to missing
column(s) or files, opens the database temporarily with Primary access,
and then reattempts to open the database Secondary access.
#### Problem
Ledger-tool doesn't support shred-compaction-type other than the default rocksdb level compaction.
#### Summary of Changes
This PR enables ledger-tool to automatically detect the shred-compaction-type of the specified ledger.
#### Test Plan
New ledger-tool tests are added for both level and fifo compactions.
* Add ability to use a non-default app profile id in bigtable requests
* Only run subcommand once when getting global configs
* Remove unneded scoping on option type
#### Problem
blockstore_db.rs has a mutual dependency between blockstore_metrics.rs.
#### Summary of Changes
This PR removes the mutual dependency by moving the option-related stuff
out from blockstore_db.rs to its new home --- blockstore_options.rs.
By doing this, we address the mutual dependency and also make the code cleaner.
* Add ConfirmedBlockUploadConfig, no behavior changes
* Add comment
* A little DRY cleanup
* Add configurable limit to number of blocks to check in Blockstore and Bigtable before uploading
* Limit blockstore and bigtable look-ahead
* Exit iterator early when reach ending_slot
* Use rooted_slot_iterator instead of slot_meta_iterator
* Only check blocks in the ledger
* 1.made load_credentials accept credential path as a parameter. 2.partial implement bigtable comparasion function
* finding missing blocks in bigtables in a specified range
* refactor compare-blocks,add unit test for missing_blocks and fmt
* compare-block fix last block bug
* refactor compare-block and improve wording
* Update ledger-tool/src/bigtable.rs
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* update compare-block command-line description
* style:improve wording/naming/code style
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Update tonic & prost, and regenerate proto
* Reignore doc code
* Revert pull #14367, but pin tokio to v0.2 for jsonrpc
* Bump backoff and goauth -> and therefore tokio
* Bump tokio in faucet, net-utils
* Bump remaining tokio, plus tarpc
* Configure bigtable's timeout when read-only
* Review comments
* Apply nits (thanks!)
Co-authored-by: Michael Vines <mvines@gmail.com>
* Timeout in the streamed decoding as well
Co-authored-by: Michael Vines <mvines@gmail.com>
* Refactor bigtable apis to accept start and end keys
* Make helper fn to deserialize cell data
* Refactor get_confirmed_signatures_for_address to use get_row_data range
* Add until param to get_confirmed_signatures_for_address
* Add until param to blockstore api
* Plumb until through client/cli
* Simplify client params