* Return early if reference set is empty
* Remove unreachable case
* Update reference_last_block
* Limit reference set by last_block_checked
* Return superflous-block data too
* Sort missing/superfluous blocks
* Fix tests
* Move last_block_checked calculation into missing_blocks()
* Prevent missing_slots panic
* Fix test
Several commands call load_and_process_ledger() which can fail in a
number of ways. These callers currently all handle the result in the
same way by matching the return Result:
- The Ok(_) case uses the returned types as normal
- The Err(_) case prints an error message and exits
This error handling is redundant, and a helper could remove the
duplicate code. So, this PR adds a wrapper around that checks the
result and unwraps OR prints error messages and exits.
load_and_process_ledger() performs many checks and sub-operations that
can fail. The current error handling prints an error message and exits
immediately. The long error/help messages written inline add clutter
to the functions actual implementation.
This PR creates a new error enum for all of these previous error
conditions, and bubbles up the error to let the caller decide what to
do instead of exiting immediately.
Long string literals can cause rustfmt to fail, which results in rustfmt
failing to format entire functions. There are several instances of this
in ledger-tool, so format these files with wrapped strings so that
formatting will apply to functions again.
Note that this PR was created by adding format_strings = true to
rustfmt.toml; however, this change does NOT persist that rule as the
rule would format the entire repo.
There are operations in bank_fork_utils that may fail; we explicitly
call std::process::exit() on several of these. Granted we may end up
exiting the process higher up the callstack, bubbling the errors up
allow a caller that could handle the error to do so.
* Add helper to get a protobuf cell without bothering with bincode
* Add conversion back to EntrySummaries
* Add LedgerStorage::get_block_entries
* Add ledger-tool bigtable entries subcommand
* Move CliEntries into ledger_tool::output for now
* Fetch entries with read-only token
Currently, the file is generated when a node drops a block that was
produced by another node. However, it would also be beneficial to see
the account state when a node drops its' own block.
Output the file in this additional failure codepath
This helper simply called std::mem::size_of<Self::Index>(). However, all
of the underlying functions that create keys manually copy fields into a
byte array. The fields are copied in end-to-end whereas size_of() might
include alignment bytes.
For example, a (u64, u32) only has 12 bytes of "data", but it would
have size 16 due to the 4 alignment padding bytes that would be
added to get the u32 (size 4) aligned with the u64 (size 8).
The Blockstore currently maintains a RwLock<Slot> of the maximum root
it has seen inserted. The value is initialized during
Blockstore::open() and updated during calls to Blockstore::set_roots().
The max root is queried fairly often for several use cases, and caching
the value is cheaper than constructing an iterator to look it up every
time.
However, the access patterns of these RwLock match that of an atomic.
That is, there is no critical section of code that is run while the
lock is head. Rather, read/write locks are acquired in order to read/
update, respectively. So, change the RwLock<u64> to an AtomicU64.
* allow test feature to skip rewrites
* hook up cli arg for test skip rewrites, update tests
* fix sanity checker
* add account hash to abi to fix a test
* reviews
* use hashmap to collect skip_rewrites. exclude skip_rewrites from dirty
pubkey set
* accumulate skipped_rewrite in reduce
* mutex
* fmt
* skip hash verify for this test flag
* add skipped rewrites num stat
* skip bank hash verify not account hash verify
* reviews
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
* Do not unnecessarily re-verify unloaded program
* clippy fixes
* new unit test for extract
* fixes after rebase
* fixes after rebase
* address review comments
* fix clippy
* allow pedantic invalid cast lint
* allow lint with false-positive triggered by `test-case` crate
* nightly `fmt` correction
* adapt to rust layout changes
* remove dubious test
* Use transmute instead of pointer cast and de/ref when check_aligned is false.
* Renames clippy::integer_arithmetic to clippy::arithmetic_side_effects.
* bump rust nightly to 2023-08-25
* Upgrades Rust to 1.72.0
---------
Co-authored-by: Trent Nelson <trent@solana.com>
In most cases, either a &Bank or an Arc<Bank> is more proper.
- &Bank is used if the function only needs a momentary reference
- Arc<Bank> is used if the function needs its' own copy
This PR leaves several instances of &Arc<Bank> around; these instances
are situations where a clone may only happen conditionally.
* Integrate program loader-v4 with bank
* fix tests
* new struct for ProgramRuntimeEnvironments
* remove environment from program_runtime_environment_v
* move find_program_in_cache() to invoke_context
* cleanup
When ledger-tool runs, it may create secondary directories for things
like accounts, accounts-index, etc as not to potentially interfere with
solana-validator's directories. These would show as multiple directories
with ".ledger-tool" appended to the typical directory name.
To more clearly group these items, make the default directories for
snapshots, accounts and accounts-index all within a common "ledger-tool"
directory.
When a consensus divergance occurs, the current workflow involves a
handful of manual steps to hone in on the offending slot and
transaction. This process isn't overly difficult to execute; however, it
is tedious and currently involves creating and parsing logs.
This change introduces functionality to output a debug file that
contains the components go into the bank hash. The file can be generated
in two ways:
- Via solana-validator when the node realizes it has diverged
- Via solana-ledger-tool verify by passing a flag
When a divergance occurs now, the steps to debug would be:
- Grab the file from the node that diverged
- Generate a file for the same slot with ledger-tool with a known good
version
- Diff the files, they are pretty-printed json
Several of the program command subcommands do not require a ledger as
they act solely on the program object. So, defer checking the ledger
path until we know we need to load. Additionally, remove genesis arg
from these commands that do not load a genesis.bin
The current desired open file descriptor limit is 1,000,000. This is
quite a large number, and not needed for every command. Namely, commands
that do not unpack a snapshot and create an AccountsDB will likely not
use this many files.
There is already an option in BlockstoreOptions to ignore errors if the
desired value cannot be set; this PR just bubbles that option up to a
CLI flag in ledger-tool.
The --encoding flag only makes sense for the genesis command if the
genesis accounts will be printed. Hence, the encoding flag used the
Arg::requires() function on --accounts argument.
However, in what appears to be a clap bug, this made the --accounts flag
required even though it should not be. So, remove the .requires()
dependency between these two arguments.
Tyera
steviez
Brooks
Alexander Meißner
Jeff Washington (jwash)
Justin Starry
Ryo Onodera
Andrew Fitzgerald
Pankaj Garg
Alessandro Decina
Jon Cinque
HaoranYi
behzad nouri
Illia Bobyr