* ancient append vec append creates big enough ancient append vecs
* assert we write the correct # of bytes
* fix tests
* u64 align accounts bytes written
* fix build
* add tests
* we don't need to align the store to u64.
When we call `remaining_bytes` to calculate, for the last element, u64_align!(len) could be greater than capacity. However, because saturate_sub is used, we still get the correct answer of 0. Therefore, no need to align the storage to u64.
* add comments
* add shink test for min_size
* remove dead code
* fix build
* reviews
* add test for squash ancient vec overflow too much to fit into ideal av
* add test for get_bytes on AccountToStore
* clippy
* modify test to assert 0 size of opposite
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
#### Problem
After we have defined AccountOffset to be u32, it means the address space within
one HotAccountsFile is up to 4GB. However, since the Accounts Blocks in a
HotAccountsFile is 8-byte aligned, it has the opportunity to store more data by
internally multiplying the AccountOffset by 8.
#### Summary of Changes
This PR allows a HotAccountsFile to store up to 32GB accounts data by using
8 x AccountOffset as its actual offset.
#### Test Plan
Updated existing unit-tests.
Long string literals can cause rustfmt to fail, which results in rustfmt
failing to format entire functions. There are several instances of this
in ledger-tool, so format these files with wrapped strings so that
formatting will apply to functions again.
Note that this PR was created by adding format_strings = true to
rustfmt.toml; however, this change does NOT persist that rule as the
rule would format the entire repo.
There are operations in bank_fork_utils that may fail; we explicitly
call std::process::exit() on several of these. Granted we may end up
exiting the process higher up the callstack, bubbling the errors up
allow a caller that could handle the error to do so.
* Add helper to get a protobuf cell without bothering with bincode
* Add conversion back to EntrySummaries
* Add LedgerStorage::get_block_entries
* Add ledger-tool bigtable entries subcommand
* Move CliEntries into ledger_tool::output for now
* Fetch entries with read-only token
As we develop new features or modifications, we occassionally need to
introduce new columns to the Blockstore. Adding a new column introduces
a compatibility break given that opening the database in Primary mode
(R/W access) requires opening all columns. Reverting to an old software
version that is unaware of the new column is obviously problematic.
In the past, we have addressed by backporting minimal "stub" PR's to
older versions. This is annoying, and only allow compatibility for the
single version or two that we backport to.
This PR adds a change to automatically detect all columns, and create
default column descriptors for columns we were unaware of. As a result,
older software versions can open a Blockstore that was modified by a
newer software version, even if that new version added columns that the
old version is unaware of.
* ci: add clippy test for macos and windows
* ci: remove windows from clippy test
* ci: cancel cargo clippy on Github Actions when new commits comming
* ci: set -e for .github/scripts/cargo-clippy-before-script.sh
Currently, the file is generated when a node drops a block that was
produced by another node. However, it would also be beneficial to see
the account state when a node drops its' own block.
Output the file in this additional failure codepath
* Add entries table to bt init
* Add entries to storage-proto
* Use new Blockstore method in bigtable_upload
* Add LedgerStorage::upload_confirmed_block_with_entries and use in bigtable_upload
* Upload entries to bigtable
iter_memory_pair_chunks was iterating regions in reverse, but not memory
_within_ regions in reverse.
This commit fixes the issue and simplifies the implementation by removing
nested loops which made control flow hard to reason about.