#### Problem
To write the owners-block, it requires an in-memory struct that maintains
a set of unique owner addresses while providing a look-up function to
obtain the OwnerOffset with the specified owner address.
#### Summary of Changes
This PR adds OwnersTable, the in-memory struct that maintains
a set of unique owner addresses while providing a look-up function to
obtain the OwnerOffset with the specified owner address.
#### Test Plan
A new unit-test is added.
#### Problem
The OwnersBlockFormat is currently defined inside footer.rs
instead of inside owners.rs. In addition, the implementation of
OwnersBlock doesn't honor OwnersBlockFormat.
#### Summary of Changes
This PR moves OwnersBlockFormat from footer.rs to owners.rs
and repurpose OwnersBlock as OwnersBlockFormat (just like
the IndexBlockFormat inside index.rs)
#### Test Plan
Existing unit-tests.
The function open_genesis_config() performs several operations that
could fail. If any of these fail, the process exits immediately.
Instead of exiting immediately, bubble up the error and let the caller
decide the appropriate action. solana-validator and solana-ledger-tool
will functionally be unchanged, but this consolidates startup failures
for both of these processes.
#### Problem
Before we have fully switched to the new way to determine whether
an account is executable, we still need a bit for th executable flag at
this moment in the TieredStorage as well as for backward compatibility
in case we want to revert it back.
#### Summary of Changes
This PR adds the executable flag into AccountMetaFlags.
#### Test Plan
Updated existing tests for AccountMetaFlags to cover executable flag.
#### Problem
The implementation of HotAccountWriter is missing.
#### Summary of Changes
This PR kicks off the implementation of HotStorageWriter by
adding HotStorageWriter::new().
#### Test Plan
Add a new unit-test that verifies the correctness of HotStorageWriter
writing zero accounts using HotStorageReader.
#### Problem
In the previous refactoring, IndexBlockFormat::AddressAndBlockOffsetOnly
can now work independently with the AccountMetaFormat. As a result,
the naming of this format should be updated to reflect this.
#### Summary of Changes
As the format first persists the list of addresses followed by the list of offsets,
this PR renames AddressAndBlockOffsetOnly to AddressesThenOffsets.
#### Problem
All structs that implement AccountsFile are required to support get_account,
but the implementation for HotAccounts is currently missing.
#### Summary of Changes
This PR implements HotAccountsReader::get_account_from_index_offset.
This will allow it to support AccountsFile::get_account API.
#### Test Plan
Add new unit-test.
#### Problem
There're some typos in the comments and assert messages index.rs mentioned in #34529
#### Summary of Changes
Fix the typos (only in assert message and comments).
#### Problem
Each AccountsFile is required to implement account_matches_owners(),
a public API that checks whether the account located at the specified
offset matches any input owners. However, the implementation of
account_matches_owners() for HotAccountsStorage is missing.
#### Summary of Changes
This PR implements HotStorageReader::account_matches_owners().
#### Test Plan
A new unit-test is added to this PR.
#### Problem
TieredStorage doesn't perform boundary check in get_account_offset
when the input IndexOffset isn't valid.
#### Summary of Changes
This PR adds two checks. First, it checks whether the IndexOffset exceeds
the boundary of the index block. Second, when an index format that has the
same index entries as account entries is used, it also checks whether IndexOffset
is smaller than account_entry_count.
#### Test Plan
Two new tests are added to this PR.
#### Problem
get_account_address() does not check whether IndexOffset is valid.
#### Summary of Changes
This PR adds two checks. First, it checks whether the IndexOffset exceeds
the boundary of the index block. Second, when an index format that has the
same index entries as account entries is used, it also checks whether IndexOffset
is smaller than account_entry_count.
#### Test Plan
New unit-test is added.
#### Problem
Hot accounts are stored in accounts blocks, whose offset is smaller than
the index block offset. However, the current code doesn't perform
any boundary checks when accessing hot account meta.
#### Summary of Changes
Adds boundary check when accessing hot account meta.
#### Problem
HotAccountOffset::new() might return Err for invalid offset, and this
part needs some test coverage.
#### Summary of Changes
Add unit-tests for checking invalid HotAccountOffset.
#### Problem
The current naming and the code comments for HOT_ACCOUNT_OFFSET_ALIGNMENT
aren't really reflecting its role as pointed out in #34335.
#### Summary of Changes
This PR renames HOT_ACCOUNT_OFFSET_ALIGNMENT to HOT_ACCOUNT_ALIGNMENT
as it's the hot account instead of hot account offset needs to be aligned.
In addition, improve the comment block for HOT_ACCOUNT_ALIGNMENT.
#### Problem
Hot and cold accounts storage have different implementations of
their offsets. As a result, a single struct AccountOffset isn't suitable
to describe the offsets used by hot and cold accounts storage.
#### Summary of Changes
This PR makes AccountOffset a trait. On top of that, introduces
HotAccountOffset that implements AccountOffset.
#### Test Plan
Updated existing unit-tests.
* tune ancient append vec size to 130M
* fix a test and get rid of the assert since it is covered in the test
* use 128M
* assert max append vec size for ancient append vec
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
* ancient append vec append creates big enough ancient append vecs
* assert we write the correct # of bytes
* fix tests
* u64 align accounts bytes written
* fix build
* add tests
* we don't need to align the store to u64.
When we call `remaining_bytes` to calculate, for the last element, u64_align!(len) could be greater than capacity. However, because saturate_sub is used, we still get the correct answer of 0. Therefore, no need to align the storage to u64.
* add comments
* add shink test for min_size
* remove dead code
* fix build
* reviews
* add test for squash ancient vec overflow too much to fit into ideal av
* add test for get_bytes on AccountToStore
* clippy
* modify test to assert 0 size of opposite
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
#### Problem
After we have defined AccountOffset to be u32, it means the address space within
one HotAccountsFile is up to 4GB. However, since the Accounts Blocks in a
HotAccountsFile is 8-byte aligned, it has the opportunity to store more data by
internally multiplying the AccountOffset by 8.
#### Summary of Changes
This PR allows a HotAccountsFile to store up to 32GB accounts data by using
8 x AccountOffset as its actual offset.
#### Test Plan
Updated existing unit-tests.
#### Problem
AccountOffset currently uses `usize`, which size is platform dependent.
We want a fixed size type that is consist to what we persist in the tiered-storage file.
#### Summary of Changes
This PR makes AccountOffset use u32.
#### Problem
MatchAccountOwnerError currently belongs to append_vec.
However, it is a public error type that is also required by other AccountsFile
implementations such as TieredStorageFile.
#### Summary of Changes
Move MatchAccountOwnerError from append_vec to accounts_file.
* remaining_bytes aligns len since all writes will align first
* use remaining_bytes() to check for whether the new account can fit into the av storage
* Add test coverage for av remaining bytes alignment
* use great equal to check space available
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
Yueh-Hsuan Chiang
Brooks
steviez
Andrew Fitzgerald
Nick Frostbutter
GoodDaisy
Joe C
Ashwin Sekar
HaoranYi
Jeff Washington (jwash)