#### Problem
While the implementation of hot-storage reader and writer
are mostly done, it is not yet connected to TieredStorage.
#### Summary of Changes
This PR enables hot-storage in TieredStorage::write_accounts().
#### Test Plan
Completes the existing tests in TieredStorage to directly
write and read from a TieredStorage with the hot storage format.
#### Problem
append_accounts() only appends (len - skip) accounts.
However, AppendVec::append_accounts() reserves `len`
instead of `(len - skip)` for its vectors.
#### Summary of Changes
Use (len - skip) as the initial size of the Vectors.
#### Problem
HotStorageReader and TieredStorageReader haven't implemented
accounts() that is required by AcocuntsFile.
#### Summary of Changes
This PR implements accounts() for both HotStorageReader
and TieredStorageReader
#### Test Plan
Extend the existing test to cover accounts().
#### Problem
TieredStorageMeta and TieredStorageReader::get_account API uses
u32 to represent IndexOffset. However, within the TieredStorage scope,
IndexOffset should be used, it is not until working with AccountsFile API
when u32 representation of offset is needed.
#### Summary of Changes
Have TieredStorageMeta and TieredStorageReader to use IndexOffset.
#### Test Plan
Existing unit-tests.
#### Problem
To allow hot-storage to use HotStorageWriter::write_account() to
implement AccountsFile::append_accounts(), it is required to
provide a Vector of StoredAccountInfo to allow AccountsDB to
properly prepare the entry for each account.
#### Summary of Changes
This PR enables HotStorageWriter::write_account() to return
Vec<StoredAccountInfo>.
#### Test Plan
Extend existing tests for HotStorageWriter to verify the correctness
of the returned Vec<StoredAccountInfo>.
#### Problem
TieredStorageReader is a wrapper enum that works for
both Hot and Cold storage readers, but its get_account()
and account_matches_owner() API are missing.
#### Summary of Changes
Add get_account() and account_matches_owner() to
TieredStorageReader.
#### Test Plan
hot.rs offers similar coverage for HotStorageReader.
#### Problem
In HotStorageReader, the account_matches_owners takes
&[&Pubkey] as the address candidates. However, it should
be &[Pubkey] as defined in the accounts_file API.
#### Summary of Changes
Correct HotStorageReader::account_matches_owners() to
take &[Pubkey] instead.
#### Test Plan
Existing unit-tests
#### Problem
Using non-reference type of AccountHash in
AccountMetaOptionalFields causes an unnecessary copy
as mentioned in #34948.
#### Summary of Changes
Uses &AccountHash in AccountMetaOptionalFields to
avoid copying.
#### Test Plan
Existing unit tests.
Fixes#34948
#### Problem
So far the current HotStorageWriter::write_accounts() only writes
accounts blocks and index block.
#### Summary of Changes
The PR further writes owners block in HotStorageWriter::write_accounts().
#### Test Plan
Extended existing test for HotStorageWriter to cover the owners block.
#### Problem
In HotStorageWriter::write_accounts, it skips storing rent-epoch when
the rent-epoch equals Epoch::MAX. While the value is correct, it is
more suitable to use RENT_EXEMPT_RENT_EPOCH instead as the
goal here is to save bytes for rent-exempt accounts.
#### Summary of Changes
Replace Epoch::MAX by RENT_EXEMPT_RENT_EPOCH when checking
whether to skip storing rent-epoch in HotStorageWriter.
#### Problem
The implementation of write_accounts() for HotAccountStorage is missing.
It consists of the writing of account blocks, index block, and owners block.
#### Summary of Changes
This PR completes part of the HotStorageWriter::write_accounts().
Specifically, it finishes the writing of account blocks and index block.
#### Test Plan
A new unit-test is added to verify the correctness of the work-in-progress
HotStorageWriter::write_accounts().
#### Problem
To write the owners-block, it requires an in-memory struct that maintains
a set of unique owner addresses while providing a look-up function to
obtain the OwnerOffset with the specified owner address.
#### Summary of Changes
This PR adds OwnersTable, the in-memory struct that maintains
a set of unique owner addresses while providing a look-up function to
obtain the OwnerOffset with the specified owner address.
#### Test Plan
A new unit-test is added.
#### Problem
The OwnersBlockFormat is currently defined inside footer.rs
instead of inside owners.rs. In addition, the implementation of
OwnersBlock doesn't honor OwnersBlockFormat.
#### Summary of Changes
This PR moves OwnersBlockFormat from footer.rs to owners.rs
and repurpose OwnersBlock as OwnersBlockFormat (just like
the IndexBlockFormat inside index.rs)
#### Test Plan
Existing unit-tests.
The function open_genesis_config() performs several operations that
could fail. If any of these fail, the process exits immediately.
Instead of exiting immediately, bubble up the error and let the caller
decide the appropriate action. solana-validator and solana-ledger-tool
will functionally be unchanged, but this consolidates startup failures
for both of these processes.
#### Problem
Before we have fully switched to the new way to determine whether
an account is executable, we still need a bit for th executable flag at
this moment in the TieredStorage as well as for backward compatibility
in case we want to revert it back.
#### Summary of Changes
This PR adds the executable flag into AccountMetaFlags.
#### Test Plan
Updated existing tests for AccountMetaFlags to cover executable flag.
#### Problem
The implementation of HotAccountWriter is missing.
#### Summary of Changes
This PR kicks off the implementation of HotStorageWriter by
adding HotStorageWriter::new().
#### Test Plan
Add a new unit-test that verifies the correctness of HotStorageWriter
writing zero accounts using HotStorageReader.
#### Problem
In the previous refactoring, IndexBlockFormat::AddressAndBlockOffsetOnly
can now work independently with the AccountMetaFormat. As a result,
the naming of this format should be updated to reflect this.
#### Summary of Changes
As the format first persists the list of addresses followed by the list of offsets,
this PR renames AddressAndBlockOffsetOnly to AddressesThenOffsets.
#### Problem
All structs that implement AccountsFile are required to support get_account,
but the implementation for HotAccounts is currently missing.
#### Summary of Changes
This PR implements HotAccountsReader::get_account_from_index_offset.
This will allow it to support AccountsFile::get_account API.
#### Test Plan
Add new unit-test.
#### Problem
There're some typos in the comments and assert messages index.rs mentioned in #34529
#### Summary of Changes
Fix the typos (only in assert message and comments).
#### Problem
Each AccountsFile is required to implement account_matches_owners(),
a public API that checks whether the account located at the specified
offset matches any input owners. However, the implementation of
account_matches_owners() for HotAccountsStorage is missing.
#### Summary of Changes
This PR implements HotStorageReader::account_matches_owners().
#### Test Plan
A new unit-test is added to this PR.
#### Problem
TieredStorage doesn't perform boundary check in get_account_offset
when the input IndexOffset isn't valid.
#### Summary of Changes
This PR adds two checks. First, it checks whether the IndexOffset exceeds
the boundary of the index block. Second, when an index format that has the
same index entries as account entries is used, it also checks whether IndexOffset
is smaller than account_entry_count.
#### Test Plan
Two new tests are added to this PR.
#### Problem
get_account_address() does not check whether IndexOffset is valid.
#### Summary of Changes
This PR adds two checks. First, it checks whether the IndexOffset exceeds
the boundary of the index block. Second, when an index format that has the
same index entries as account entries is used, it also checks whether IndexOffset
is smaller than account_entry_count.
#### Test Plan
New unit-test is added.
#### Problem
Hot accounts are stored in accounts blocks, whose offset is smaller than
the index block offset. However, the current code doesn't perform
any boundary checks when accessing hot account meta.
#### Summary of Changes
Adds boundary check when accessing hot account meta.
Pankaj Garg
Dmitri Makarov
Yueh-Hsuan Chiang
HaoranYi
Brooks
steviez
Andrew Fitzgerald
Nick Frostbutter
GoodDaisy