* ancient append vec append creates big enough ancient append vecs
* assert we write the correct # of bytes
* fix tests
* u64 align accounts bytes written
* fix build
* add tests
* we don't need to align the store to u64.
When we call `remaining_bytes` to calculate, for the last element, u64_align!(len) could be greater than capacity. However, because saturate_sub is used, we still get the correct answer of 0. Therefore, no need to align the storage to u64.
* add comments
* add shink test for min_size
* remove dead code
* fix build
* reviews
* add test for squash ancient vec overflow too much to fit into ideal av
* add test for get_bytes on AccountToStore
* clippy
* modify test to assert 0 size of opposite
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
#### Problem
After we have defined AccountOffset to be u32, it means the address space within
one HotAccountsFile is up to 4GB. However, since the Accounts Blocks in a
HotAccountsFile is 8-byte aligned, it has the opportunity to store more data by
internally multiplying the AccountOffset by 8.
#### Summary of Changes
This PR allows a HotAccountsFile to store up to 32GB accounts data by using
8 x AccountOffset as its actual offset.
#### Test Plan
Updated existing unit-tests.
#### Problem
AccountOffset currently uses `usize`, which size is platform dependent.
We want a fixed size type that is consist to what we persist in the tiered-storage file.
#### Summary of Changes
This PR makes AccountOffset use u32.
#### Problem
MatchAccountOwnerError currently belongs to append_vec.
However, it is a public error type that is also required by other AccountsFile
implementations such as TieredStorageFile.
#### Summary of Changes
Move MatchAccountOwnerError from append_vec to accounts_file.
* remaining_bytes aligns len since all writes will align first
* use remaining_bytes() to check for whether the new account can fit into the av storage
* Add test coverage for av remaining bytes alignment
* use great equal to check space available
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
#### Problem
IndexOffset currently uses `usize`, which size is platform dependent.
We want a fixed size type that is consist to what we persist in the tiered-storage file.
#### Summary of Changes
This PR makes IndexOffset use u32.
#### Problem
In TieredStorage, we want to make AccountOffset and IndexOffset u32 instead
of usize just like OwnerOffset. However, we need to first change what we persist
in the storage. Currently, IndexBlock persists offsets as u64 instead of u32.
#### Summary of Changes
This PR makes IndexBlock persist u32 offsets.
#### Test Plan
Existing test cases.
#### Problem
TieredAccountMeta currently uses `u32` for owner_offset, while
we already have a dedicate type for that --- OwnerOffset.
#### Summary of Changes
This PR makes TieredAccountMeta use OwnerOffset for its
owner_offset field.
#### Test Plan
Existing unit-tests.
#### Problem
The current OwnerOffset is defined as usize, which actual size could be u32 or u64.
On the other hand, the OwnerOffset is used to access the ith owner in one
TieredStorageFile. As a result, u32 is more than enough to represent the number of
owners inside one TieredStorageFile while u16 might not be always enough.
#### Summary of Changes
This PR defines OwnerOffset as u32.
#### Test Plan
Existing test cases.
* add a feature to disable rent collection
* fix a test
* fix a test
* rekey
* should collect rent
* Update runtime/src/bank/fee_distribution.rs
Co-authored-by: Brooks <brooks@prumo.org>
* expand tests to cover both rent collection disabled and enabled
* feedbacks
* reviews - move should collect rent check out of rent collector into bank
* enforce rent_epoch to u64:max when rent collection is disabled
* review feedbacks and fix a test
When rent fee collection is disabled, we won't collect rent for any account. If there are any rent paying accounts, their `rent_epoch` won't change too.
* revise comments
* update rent_epoch for rent exempted account
* rebase
* set rent_epoch in rent collection for rent exempted account
* revert test change
* don't assert
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
Co-authored-by: Brooks <brooks@prumo.org>
#### Problem
The HotStorageReader does not have an API to obtain owners_address.
#### Summary of Changes
This PR adds HotStorageReader::get_owner_address().
#### Test Plan
A new unit-test is added to this PR.
#### Problem
#34032 mentioned nice-to-have a formatting improvement that
can improve the readability.
#### Summary of Changes
Improve the code formatting in test_hot_storage_get_account_offset_and_address().
#### Problem
A TieredStorage file has three types of block: accounts block, index block,
and owner block, but implementation of the TieredStorage OwnersBlock
is missing in the current master.
#### Summary of Changes
This PR implements OwnersBlock which stores a set of owners' addresses
in a compact but efficient way.
#### Test Plan
A new unit-test is included in this PR.
#### Problem
HotStorageReader currently not yet has an API to obtain account_address
#### Summary of Changes
This PR adds HotStorageReader::get_account_address() which returns
the Pubkey of the account associated with the specified IndexOffset.
#### Test Plan
Augmented an existing unit-test to cover get_account_address() case.
#### Problem
TieredStorageFooter::owners_offset refers to the offset to the
owners block, which looks very similar to OwnerOffset that
has a completely different concept -- the offset to access the
address of an owner in the owners-block.
#### Summary of Changes
Similar to the previous renaming index_offset to index_block_offset,
this PR renames owners_offset to owners_block_offset to
avoid confusion.
#### Problem
HotStorageReader currently not yet has an API to obtain account_offset
#### Summary of Changes
This PR adds HotStorageReader::get_account_offset() which takes
IndexOffset and returns AccountOffset.
#### Test Plan
A new unit-test is included in this PR.
#### Problem
In IndexBlockFormat, both `IndexOffset` and `AccountOffset` parameters
are named `offset` in some functions which could be confusing.
#### Summary of Changes
Renamed `offset` to `index_offset` and `account_offset` to improve
readability.
#### Problem
#33927 introduced a new type AccountOffset, but HotStorageReader
still uses `usize` to access accounts.
#### Summary of Changes
This PR makes HotStorageReader use the new AccountOffset type.
#### Problem
TieredStorage conceptually has different offsets. However, the current code directly
uses the same primitive type for accessing offsets, which is error-prone as one could
easily use one offset to access data that should be accessed with a different offset
type.
#### Summary of Changes
This PR adds IndexOffset type -- a struct for obtaining the ith entry inside the
index-block to obtain account's offset and address.
#### Problem
TieredStorage conceptually has different offsets. However, the current code directly
uses the same primitive type for accessing offsets, which is error-prone as one could
easily use one offset to access data that should be accessed with a different offset
type.
#### Summary of Changes
This PR introduces the AccountOffset type, which allows static-check to on different
type of TieredStorage offsets.
#### Problem
The current tiered-storage code uses "account-index" to call index-block.
This could lead to confusion especially as we start giving each offset/position/index a specific type.
#### Summary of Changes
This PR renames all structs/variables that use account-index to refer to index-block.
* allow test feature to skip rewrites
* hook up cli arg for test skip rewrites, update tests
* fix sanity checker
* add account hash to abi to fix a test
* reviews
* use hashmap to collect skip_rewrites. exclude skip_rewrites from dirty
pubkey set
* accumulate skipped_rewrite in reduce
* mutex
* fmt
* skip hash verify for this test flag
* add skipped rewrites num stat
* skip bank hash verify not account hash verify
* reviews
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
* Split compute budget instruction processing from ComputeBudget struct itself, so CB instructions can be processed elsewhere without involving ComputeBudget
* updated tests
* avoid built ComputeBudget from dated ComputeBudgetLimits in this refactoring PR
* Clean-up program-runtime/src/compute_budget_processor.rs
* Add test for a corner case that deprecated instruction is used to request units greater than max limit;
* Update code to handle the corner case.
* buffer accounts field for hash
* use smallvec to allocate hash buffer on stack
* sort deps
* more opt
* clippy
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
Revert "Split compute budget instructions process from struct itself (#33513)"
This reverts commit c73bebe984. This
was found to be a consensus breaking change.
* Split compute budget instruction processing from ComputeBudget struct itself, allow compute_budget_instructions be processed elsewhere without having to instantiate ComputeBudget
* updated tests
#### Problem
HotStorageReader currently only implements get_footer(). It does not
have a function to obtain the account meta.
#### Summary of Changes
This PR implements HotStorageReader::get_account_meta_from_offset().
A function that returns the account meta located at the specified offset.
This will be the helper function that will be later used to obtain the account
meta when the offset is available from the index block of a hot storage file.
#### Test Plan
A new test is included in this PR.
#### Problem
HotStorageReader currently doesn't have a test that covers its footer.
#### Summary of Changes
This PR includes a test for HotStorageReader that verifies the footer.
#### Problem
All account storage formats are required to implement both StoredAccountMeta
and ReadableAccount, but the implementation for the hot account format is missing.
#### Summary of Changes
This PR includes hot account format into StoredAccountMeta and ReadableAccount
enum. This will allow the TieredStorageReader in the future PRs to return hot account
format in its `get_account` implementation.
* hash_dedup vec algo
* reviews
* reviews
* more reviews
* simplify working_set init with add_next_item
* refactor to remove special case "new" from add_item.
The new change is that, even the new item is the new min, it will still be added to working_set.
This change will make init working_set code simpler and the loop loop check simpler.
Since the item is inserted in at the end of the vector, the cost of push into and pop from the working will be O(1), shouldn't affect performance much.
* comments
* refactor unnamed tuple in working set to SlotGroupPointer type
* use SlotGroupPointer in ItemLocation
* Add Copy traits to avoid explicty call of clone on SlotGroupPointer
* consume next in add_next_item fn (credit to jeff).
note that the old code is still correct, since before call to
add_next_item, we will have already overwritten `next` to correct value.
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
#### Problem
The TieredStorageFormat field in the TieredStorage is only used in the write path.
#### Summary of Changes
This PR simplifies the handling of TieredStorageFormat by removing its field from
TieredStorage struct but passing via write_accounts().
The program owners pubkeys are constant, no need to reconstruct the
Vec<Pubkey> and Vec<&Pubkey> each time this function runs (every time we
execute transactions).
Brooks
Jeff Washington (jwash)
Yueh-Hsuan Chiang
Andrew Fitzgerald
HaoranYi
Lucas Steuernagel
Alexander Meißner
Pankaj Garg
Lijun Wang
Tao Zhu
Kevin Heavey
steviez
Yihau Chen
Trent Nelson