#### Problem
While accounts-db might not invoke appends_account twice
for the same AccountsFile, TieredStorage::write_accounts()
itself isn't thread-safe, and it depends on the above accounts-db
assumption.
#### Summary of Changes
This PR makes TieredStorage::write_accounts() thread-safe.
So only the first thread that successfully updates the already_written
flag can proceed and write the input accounts. All subsequent
calls to write_accounts() will be a no-op and return AttemptToUpdateReadOnly
Error.
#### Problem
There're some test functions that have been used in different
mod in TieredStorage. It's better to have one same place for
all tiere-storage related test functions.
#### Summary of Changes
Created test_utils.rs under /tiered_storage and move test-related
functions into it.
#### Test Plan
Existing tests.
#### Problem
While the implementation of hot-storage reader and writer
are mostly done, it is not yet connected to TieredStorage.
#### Summary of Changes
This PR enables hot-storage in TieredStorage::write_accounts().
#### Test Plan
Completes the existing tests in TieredStorage to directly
write and read from a TieredStorage with the hot storage format.
#### Problem
append_accounts() only appends (len - skip) accounts.
However, AppendVec::append_accounts() reserves `len`
instead of `(len - skip)` for its vectors.
#### Summary of Changes
Use (len - skip) as the initial size of the Vectors.
#### Problem
HotStorageReader and TieredStorageReader haven't implemented
accounts() that is required by AcocuntsFile.
#### Summary of Changes
This PR implements accounts() for both HotStorageReader
and TieredStorageReader
#### Test Plan
Extend the existing test to cover accounts().
#### Problem
TieredStorageMeta and TieredStorageReader::get_account API uses
u32 to represent IndexOffset. However, within the TieredStorage scope,
IndexOffset should be used, it is not until working with AccountsFile API
when u32 representation of offset is needed.
#### Summary of Changes
Have TieredStorageMeta and TieredStorageReader to use IndexOffset.
#### Test Plan
Existing unit-tests.
#### Problem
To allow hot-storage to use HotStorageWriter::write_account() to
implement AccountsFile::append_accounts(), it is required to
provide a Vector of StoredAccountInfo to allow AccountsDB to
properly prepare the entry for each account.
#### Summary of Changes
This PR enables HotStorageWriter::write_account() to return
Vec<StoredAccountInfo>.
#### Test Plan
Extend existing tests for HotStorageWriter to verify the correctness
of the returned Vec<StoredAccountInfo>.
#### Problem
TieredStorageReader is a wrapper enum that works for
both Hot and Cold storage readers, but its get_account()
and account_matches_owner() API are missing.
#### Summary of Changes
Add get_account() and account_matches_owner() to
TieredStorageReader.
#### Test Plan
hot.rs offers similar coverage for HotStorageReader.
#### Problem
In HotStorageReader, the account_matches_owners takes
&[&Pubkey] as the address candidates. However, it should
be &[Pubkey] as defined in the accounts_file API.
#### Summary of Changes
Correct HotStorageReader::account_matches_owners() to
take &[Pubkey] instead.
#### Test Plan
Existing unit-tests
#### Problem
Using non-reference type of AccountHash in
AccountMetaOptionalFields causes an unnecessary copy
as mentioned in #34948.
#### Summary of Changes
Uses &AccountHash in AccountMetaOptionalFields to
avoid copying.
#### Test Plan
Existing unit tests.
Fixes#34948
#### Problem
So far the current HotStorageWriter::write_accounts() only writes
accounts blocks and index block.
#### Summary of Changes
The PR further writes owners block in HotStorageWriter::write_accounts().
#### Test Plan
Extended existing test for HotStorageWriter to cover the owners block.
#### Problem
In HotStorageWriter::write_accounts, it skips storing rent-epoch when
the rent-epoch equals Epoch::MAX. While the value is correct, it is
more suitable to use RENT_EXEMPT_RENT_EPOCH instead as the
goal here is to save bytes for rent-exempt accounts.
#### Summary of Changes
Replace Epoch::MAX by RENT_EXEMPT_RENT_EPOCH when checking
whether to skip storing rent-epoch in HotStorageWriter.
#### Problem
The implementation of write_accounts() for HotAccountStorage is missing.
It consists of the writing of account blocks, index block, and owners block.
#### Summary of Changes
This PR completes part of the HotStorageWriter::write_accounts().
Specifically, it finishes the writing of account blocks and index block.
#### Test Plan
A new unit-test is added to verify the correctness of the work-in-progress
HotStorageWriter::write_accounts().
#### Problem
To write the owners-block, it requires an in-memory struct that maintains
a set of unique owner addresses while providing a look-up function to
obtain the OwnerOffset with the specified owner address.
#### Summary of Changes
This PR adds OwnersTable, the in-memory struct that maintains
a set of unique owner addresses while providing a look-up function to
obtain the OwnerOffset with the specified owner address.
#### Test Plan
A new unit-test is added.
#### Problem
The OwnersBlockFormat is currently defined inside footer.rs
instead of inside owners.rs. In addition, the implementation of
OwnersBlock doesn't honor OwnersBlockFormat.
#### Summary of Changes
This PR moves OwnersBlockFormat from footer.rs to owners.rs
and repurpose OwnersBlock as OwnersBlockFormat (just like
the IndexBlockFormat inside index.rs)
#### Test Plan
Existing unit-tests.
The function open_genesis_config() performs several operations that
could fail. If any of these fail, the process exits immediately.
Instead of exiting immediately, bubble up the error and let the caller
decide the appropriate action. solana-validator and solana-ledger-tool
will functionally be unchanged, but this consolidates startup failures
for both of these processes.
#### Problem
Before we have fully switched to the new way to determine whether
an account is executable, we still need a bit for th executable flag at
this moment in the TieredStorage as well as for backward compatibility
in case we want to revert it back.
#### Summary of Changes
This PR adds the executable flag into AccountMetaFlags.
#### Test Plan
Updated existing tests for AccountMetaFlags to cover executable flag.
#### Problem
The implementation of HotAccountWriter is missing.
#### Summary of Changes
This PR kicks off the implementation of HotStorageWriter by
adding HotStorageWriter::new().
#### Test Plan
Add a new unit-test that verifies the correctness of HotStorageWriter
writing zero accounts using HotStorageReader.
#### Problem
In the previous refactoring, IndexBlockFormat::AddressAndBlockOffsetOnly
can now work independently with the AccountMetaFormat. As a result,
the naming of this format should be updated to reflect this.
#### Summary of Changes
As the format first persists the list of addresses followed by the list of offsets,
this PR renames AddressAndBlockOffsetOnly to AddressesThenOffsets.
Brooks
Yueh-Hsuan Chiang
Dmitri Makarov
Pankaj Garg
Jeff Washington (jwash)
HaoranYi
steviez
Andrew Fitzgerald