This change makes solana-ledger-tool match solana-validator CLI; the old
flag --snapshot-archive-path is retained as an argument alias and can
still be used for the sake of backwards compatibility.
The accounts command currently dumps every single account in the
AccountsDb. This is obviously a lot of output, so a previous change
streamed the accounts instead of collecting and dumping at the end.
The streaming approach is much more performant, but the implementation
is non-trivial. This change
- Moves the accounts output code to output.rs
- Refactor the logic to several objects that implment the functionality
- Adjust the json output to also include the summary
This change lays the groundwork for cleanly adding several more flags
that will allow for querying different subsets of accounts.
This argument is a flag and doesn't take a value; however, it had the
.value_name() modifier set with "FORMAT". This could be confusing so
remove .value_name() and add .takes_value(false)
The function open_genesis_config() performs several operations that
could fail. If any of these fail, the process exits immediately.
Instead of exiting immediately, bubble up the error and let the caller
decide the appropriate action. solana-validator and solana-ledger-tool
will functionally be unchanged, but this consolidates startup failures
for both of these processes.
The code to parse process options was repeated in several (8) commands
that replay block processing. So, move the argument parsing into a
common helper that can be used by all of those commands.
* Update proto files with tonic-build v0.9.2
* Manually ignore invalid doc-tests
* Add new ReadRowsRequest fields
* Add LedgerStorageConfig::max_message_size and default value
* Add BigtableConnection::max_message_size and use on client creation
* Add max_message_size to RpcBigtableConfig and make const pub
* Add solana-validator cli arg
solana-ledger-tool is a bit of a kitchen sink and currently has
upwards of 30 commands at the top level of the CLI. UI aside, the
contents of ledger-tool/src/main.rs are somewhat cluttered.
To reduce clutter and introduce a more hierarchical structure, this
change introduces a new blockstore subcommand and moves all commands
that only touch the blockstore to a blockstore subcommand. For example,
solana-ledger-tool slot X
is now invoked like
solana-ledger-tool blockstore slot X
However, given how long the tool has existed, support is retained for
calling these old commands without the extra blockstore command. But,
the commands are not listed in the help in order to steer new users to
use the newer calling structure.
TransactionStatusService needs Primary access in order to write
transaction status into the Blockstore if enable_rpc_transaction_history
is set to True. The current logic attempts to get Primary access for the
service.
However, in the event that this function had been called with a
Blockstore that already had Primary access, this second attempt to get
Primary access would fail. So, only attempt to open with Primary access
when necessary AND when the current access level is not sufficient.
A future change will add more cases to this if-else if-...-else chain.
Using a match statement will be easier to follow then a very long
if-else if-... chain.
This change was broken out in order to have a higher signal to noise
ratio in the subsequent change.
The open_blockstore() helper currently takes multiple configurable
options. While the arguments are parsed at a high enough scope in
main.rs to avoid repeated calls, this parsing is duplicated in the
separate subcommand files (program.rs and bigtable.rs).
The repeated parsing is redundant, and also prone to having to missing
an arg (as was the case with bigtable not having wal_recovery_mode).
So, this PR consolidates the parsing to a single function and uses that
function across the previous callers.
warning: `flatten()` will run forever if the iterator repeatedly produces an `Err`
--> ledger-tool/src/main.rs:2649:39
|
2649 | for line in f.lines().flatten() {
| ^^^^^^^^^ help: replace with: `map_while(Result::ok)`
|
note: this expression returning a `std::io::Lines` may produce an infinite number of `Err` in case of a read error
--> ledger-tool/src/main.rs:2649:29
|
2649 | for line in f.lines().flatten() {
| ^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#lines_filter_map_ok
= note: `#[warn(clippy::lines_filter_map_ok)]` on by default
warning: `solana-ledger-tool` (bin "solana-ledger-tool" test) generated 1 warning
* Add cli flag to show entry data
* Add display structs
* Add writeln_entry helper fn
* Add entry conversion method
* Populate Display for CliBlockWithEntries
* Add ctor from flattened block and entries iterator
* Support show_entries
* Finalize unified scheduler plumbing with min impl
* Fix comment
* Rename leftover type name...
* Make logging text less ambiguous
* Make PhantomData simplyer without already used S
* Make TaskHandler stateless again
* Introduce HandlerContext to simplify TaskHandler
* Add comment for coexistence of Pool::{new,new_dyn}
* Fix grammar
* Remove confusing const for upcoming changes
* Demote InstalledScheduler::context() into dcou
* Delay drop of context up to return_to_pool()-ing
* Revert "Demote InstalledScheduler::context() into dcou"
This reverts commit 049a126c905df0ba8ad975c5cb1007ae90a21050.
* Revert "Delay drop of context up to return_to_pool()-ing"
This reverts commit 60b1bd2511a714690b0b2331e49bc3d0c72e3475.
* Make context handling really type-safe
* Update comment
* Fix grammar...
* Refine type aliases for boxed traits
* Swap the tuple order for readability & semantics
* Simplify PooledScheduler::result_with_timings type
* Restore .in_sequence()
* Use where for aesthetics
* Simplify if...
* Fix typo...
* Polish ::schedule_execution() a bit
* Fix rebase conflicts..
* Make test more readable
* Fix test failures after rebase...
* Return early if reference set is empty
* Remove unreachable case
* Update reference_last_block
* Limit reference set by last_block_checked
* Return superflous-block data too
* Sort missing/superfluous blocks
* Fix tests
* Move last_block_checked calculation into missing_blocks()
* Prevent missing_slots panic
* Fix test
Several commands call load_and_process_ledger() which can fail in a
number of ways. These callers currently all handle the result in the
same way by matching the return Result:
- The Ok(_) case uses the returned types as normal
- The Err(_) case prints an error message and exits
This error handling is redundant, and a helper could remove the
duplicate code. So, this PR adds a wrapper around that checks the
result and unwraps OR prints error messages and exits.
load_and_process_ledger() performs many checks and sub-operations that
can fail. The current error handling prints an error message and exits
immediately. The long error/help messages written inline add clutter
to the functions actual implementation.
This PR creates a new error enum for all of these previous error
conditions, and bubbles up the error to let the caller decide what to
do instead of exiting immediately.
Long string literals can cause rustfmt to fail, which results in rustfmt
failing to format entire functions. There are several instances of this
in ledger-tool, so format these files with wrapped strings so that
formatting will apply to functions again.
Note that this PR was created by adding format_strings = true to
rustfmt.toml; however, this change does NOT persist that rule as the
rule would format the entire repo.
There are operations in bank_fork_utils that may fail; we explicitly
call std::process::exit() on several of these. Granted we may end up
exiting the process higher up the callstack, bubbling the errors up
allow a caller that could handle the error to do so.
* Add helper to get a protobuf cell without bothering with bincode
* Add conversion back to EntrySummaries
* Add LedgerStorage::get_block_entries
* Add ledger-tool bigtable entries subcommand
* Move CliEntries into ledger_tool::output for now
* Fetch entries with read-only token
Currently, the file is generated when a node drops a block that was
produced by another node. However, it would also be beneficial to see
the account state when a node drops its' own block.
Output the file in this additional failure codepath
This helper simply called std::mem::size_of<Self::Index>(). However, all
of the underlying functions that create keys manually copy fields into a
byte array. The fields are copied in end-to-end whereas size_of() might
include alignment bytes.
For example, a (u64, u32) only has 12 bytes of "data", but it would
have size 16 due to the 4 alignment padding bytes that would be
added to get the u32 (size 4) aligned with the u64 (size 8).
The Blockstore currently maintains a RwLock<Slot> of the maximum root
it has seen inserted. The value is initialized during
Blockstore::open() and updated during calls to Blockstore::set_roots().
The max root is queried fairly often for several use cases, and caching
the value is cheaper than constructing an iterator to look it up every
time.
However, the access patterns of these RwLock match that of an atomic.
That is, there is no critical section of code that is run while the
lock is head. Rather, read/write locks are acquired in order to read/
update, respectively. So, change the RwLock<u64> to an AtomicU64.
* allow test feature to skip rewrites
* hook up cli arg for test skip rewrites, update tests
* fix sanity checker
* add account hash to abi to fix a test
* reviews
* use hashmap to collect skip_rewrites. exclude skip_rewrites from dirty
pubkey set
* accumulate skipped_rewrite in reduce
* mutex
* fmt
* skip hash verify for this test flag
* add skipped rewrites num stat
* skip bank hash verify not account hash verify
* reviews
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
Tyera
steviez
Pankaj Garg
Brooks
steviez
HaoranYi
Ryo Onodera
Alexander Meißner
Jeff Washington (jwash)
Justin Starry
Andrew Fitzgerald