Commit Graph

558 Commits

Author SHA1 Message Date
HaoranYi 1ac523c121
Move delta hash test function to dev-context-utils (#151)
move delta hash test function to dev-context-utils

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-03-08 12:14:40 -06:00
Yueh-Hsuan Chiang 0bf9ec861a
[TieredStorage] Deprecate the use of account-hash in HotStorage (#93)
#### Problem
TieredStorage stores account hash as an optional field inside its HotStorage.
However, the field isn't used and we have already decided to deprecate
the account hash.

#### Summary of Changes
Remove account-hash from the tiered-storage.

#### Test Plan
Existing tiered-storage tests.
Running validators w/ tiered-storage in mainnet-beta w/o storing account-hash.
2024-03-06 18:51:50 -08:00
Justin Starry 9cc55349f7
Refactor transaction account unlocking (#103)
refactor: unlock accounts
2024-03-07 09:52:23 +08:00
Brooks 1e133bc067
Increases account hash's stack buffer to hold 200 bytes of data (#56) 2024-03-05 12:02:47 -05:00
Brooks 93f5b514fa
Adds StartingSnapshotStorages to AccountsHashVerifier (#58) 2024-03-04 16:32:51 -05:00
Yueh-Hsuan Chiang 608329b974
[TieredStorage] rent_epoch() returns 0 for zero-lamport accounts (#35344)
#### Problem
In TieredAccountMeta, RENT_EXEMPT_RENT_EPOCH will be used when
its optional field rent_epoch is None.  However, for legacy reasons, 0
should be used for zero-lamport accounts.

#### Summary of Changes
Return 0 for TieredAccountMeta::rent_epoch() for zero-lamport accounts.

#### Test Plan
accounts_db::tests::test_clean_zero_lamport_and_dead_slot
2024-03-01 15:18:12 -08:00
Brooks 564a9f78a0
Casts executable bool to integer when computing account hash (#35373) 2024-03-01 14:28:28 -05:00
Brooks 59ed049dc5
Gives back 8 bytes to stack buffer for account data when hashing (#35374) 2024-03-01 14:27:48 -05:00
Brooks 83de6a5930
Moves in_mem_accounts_index.rs into accounts_index directory (#35360) 2024-02-29 06:51:18 -05:00
Brooks 2e10b3b64f
Removes InMemAccountsIndex::get() (#35354) 2024-02-28 17:57:55 -05:00
Brooks 140c21f8a9
Removes ReadAccountMapEntry (#35351) 2024-02-28 16:08:00 -05:00
Brooks 6aaaf858c9
Adds more info to panic message in AccountsHashVerifier (#35353) 2024-02-28 15:55:05 -05:00
Brooks 7c48cbb7aa
Replaces InMemAccountsIndex::get() with AccountsIndex::get_cloned() (#35352) 2024-02-28 14:45:08 -05:00
Brooks 6402198902
Replaces ReadAccountMapEntry in calculate_accounts_hash_from_index() (#35349) 2024-02-28 14:38:19 -05:00
Brooks f340c1c181
Replaces ReadAccountMapEntry in do_scan_secondary_index() (#35219) 2024-02-28 11:43:33 -05:00
Brooks a4e1a9ac98
Adds AccountsIndex::get_account_info_with_and_then() (#35336) 2024-02-27 20:01:29 -05:00
Brooks da088681ba
Adds safer alternatives to get_internal() (#35325) 2024-02-27 18:08:25 -05:00
Brooks bf2e8ee32f
AccountsIndex::get_cloned() *must* add entry to in-mem cache (#35322) 2024-02-26 18:20:21 -05:00
Brooks 8143fc3f4a
Replaces ReadAccountMapEntry in read_index_for_accessor_or_load_slow() (#35220) 2024-02-26 14:19:18 -05:00
Brooks 8235feefc4
Removes get_for_tests() (#35311) 2024-02-26 14:17:33 -05:00
Brooks 54706a885b
Adds get_with_and_then() to AccountsIndex (#35307) 2024-02-23 18:48:36 -05:00
Brooks 1e47aacd0d
Removes get_account_read_entry() (#35309) 2024-02-23 22:25:30 +00:00
Brooks 58c1b7aefb
Replaces ReadAccountMapEntry in more tests (#35308) 2024-02-23 21:20:42 +00:00
Brooks 923cac8d7f
Replaces ReadAccountMapEntry in exhaustively_verify_refcounts() (#35243) 2024-02-23 11:13:57 -05:00
Brooks b5ffc24a38
Replaces ReadAccountMapEntry in retry_to_get_account_accessor() (#35244) 2024-02-23 11:12:24 -05:00
Brooks db2071a225
Removes get_account_read_entry_with_lock() (#35240) 2024-02-21 11:25:38 -05:00
Dmitri Makarov 0acee67891
SVM: move transaction_results from accounts-db to SVM (#35183)
SVM: Remove accounts-db deps in accounts_loader tests
2024-02-20 12:54:56 -08:00
Brooks 0b5cc03aa3
Replaces ReadAccountMapEntry in ancient append vecs (#35238) 2024-02-20 13:52:11 -05:00
Brooks ce72c22b3a
Replaces ReadAccountMapEntry in accounts index (#35239) 2024-02-20 13:51:54 -05:00
Brooks 2ec136a1ea
Adds get_and_then() & family to AccountsIndex (#35218) 2024-02-19 10:22:00 -05:00
Yueh-Hsuan Chiang 69345899f3
[TieredStorage] Make TieredStorage::write_accounts() thread-safe (#35143)
#### Problem
While accounts-db might not invoke appends_account twice
for the same AccountsFile, TieredStorage::write_accounts()
itself isn't thread-safe, and it depends on the above accounts-db
assumption.

#### Summary of Changes
This PR makes TieredStorage::write_accounts() thread-safe.
So only the first thread that successfully updates the already_written
flag can proceed and write the input accounts.  All subsequent
calls to write_accounts() will be a no-op and return AttemptToUpdateReadOnly
Error.
2024-02-17 16:10:58 -08:00
Yueh-Hsuan Chiang cb61ce435e
[TieredStorage] Put commonly used test functions into test_utils.rs (#35065)
#### Problem
There're some test functions that have been used in different
mod in TieredStorage.  It's better to have one same place for
all tiere-storage related test functions.

#### Summary of Changes
Created test_utils.rs under /tiered_storage and move test-related
functions into it.

#### Test Plan
Existing tests.
2024-02-09 16:35:40 -08:00
Brooks b5e903d1d7
Refactors AccountsIndex::get() (#35163) 2024-02-09 18:26:51 -05:00
Dmitri Makarov 245d1c4087
SVM: Move TransactionCheckResult definition from accounts-db to SVM (#35153) 2024-02-08 21:13:00 -05:00
Dmitri Makarov eeb0cf1ea8
SVM: Move nonce_info from accounts-db to Solana SDK (#35138) 2024-02-08 15:01:15 -05:00
Pankaj Garg 28a320d815
SVM: Move `RentCollector` to sdk (#35122) 2024-02-07 16:38:42 -08:00
Dmitri Makarov b9ee3b475b
SVM: Move RentDebits from accounts-db to Solana SDK (#35135) 2024-02-07 15:10:17 -08:00
Dmitri Makarov 2c0001b530
SVM: Move RewardInfo from accounts-db to Solana SDK (#35120) 2024-02-07 10:55:39 -08:00
Pankaj Garg 56391f655d
Remove unnecessary usage of RentCollector (#35121) 2024-02-07 07:40:17 -08:00
Jeff Washington (jwash) 2aa8b82990
remove activated feature set_exempt_rent_epoch_max (#35082)
* remove activated feature set_exempt_rent_epoch_max

* fix test_rent_eager_collect_rent_in_partition test

* update hash values for test_bank_hash_consistency

* clean up commas
2024-02-07 09:20:31 -06:00
Pankaj Garg 10defb161f
SVM: Move TransactionErrorMetrics to SVM folder (#35112) 2024-02-06 11:15:48 -08:00
Pankaj Garg 99760e519a
SVM: move `inner_instructions_list_from_instruction_trace` to SVM (#35099)
SVM: move inner_instructions_list_from_instruction_trace to SVM
2024-02-06 08:38:24 -08:00
Dmitri Makarov ab30fb5f05
SVM: Move AccountOverrides from accounts-db to SVM (#35091) 2024-02-05 19:16:47 -05:00
Yueh-Hsuan Chiang 785dd2132e
[TieredStorage] Enable hot-storage in TieredStorage::write_accounts() (#35049)
#### Problem
While the implementation of hot-storage reader and writer
are mostly done, it is not yet connected to TieredStorage. 

#### Summary of Changes
This PR enables hot-storage in TieredStorage::write_accounts().

#### Test Plan
Completes the existing tests in TieredStorage to directly
write and read from a TieredStorage with the hot storage format.
2024-02-05 10:23:30 -08:00
HaoranYi 440c3bb156
Avoid account index entry Arc clone in shrinking (#35010)
* avoid account index entry Arc clone in shrink

* use scan to addref

* update code comments for scan fn

* expect

* warn

* update log message

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-02-05 16:47:00 +00:00
Brooks a16f982169
Removes redundant check on STORE_META_OVERHEAD (#35069) 2024-02-04 16:56:12 +00:00
Yueh-Hsuan Chiang 9935c2b5e7
[AppendVec] Use proper Vec initial size in append_accounts() (#35047)
#### Problem
append_accounts() only appends (len - skip) accounts.
However, AppendVec::append_accounts() reserves `len`
instead of `(len - skip)` for its vectors.

#### Summary of Changes
Use (len - skip) as the initial size of the Vectors.
2024-02-03 23:02:09 -08:00
Yueh-Hsuan Chiang dd30175e55
[TieredStorage] TieredStorageReader:: and HotStorageReader:: accounts() (#35031)
#### Problem
HotStorageReader and TieredStorageReader haven't implemented
accounts() that is required by AcocuntsFile.

#### Summary of Changes
This PR implements accounts() for both HotStorageReader
and TieredStorageReader

#### Test Plan
Extend the existing test to cover accounts().
2024-02-02 11:18:52 -08:00
Brooks 4b528e890c
Ensures STORE_META_OVERHEAD remains accurate (#35053) 2024-02-02 13:00:54 -05:00
Yueh-Hsuan Chiang 97d994ee6f
[TieredStorage] Use IndexOffset in TieredStorageMeta and get_account() (#35046)
#### Problem
TieredStorageMeta and TieredStorageReader::get_account API uses
u32 to represent IndexOffset.  However, within the TieredStorage scope,
IndexOffset should be used, it is not until working with AccountsFile API
when u32 representation of offset is needed.

#### Summary of Changes
Have TieredStorageMeta and TieredStorageReader to use IndexOffset.

#### Test Plan
Existing unit-tests.
2024-02-02 09:53:13 -08:00
Brooks f62293918d
Moves the async deleter code to accounts-db (#35040) 2024-02-02 09:21:26 -05:00
Yueh-Hsuan Chiang be9f17f053
[TieredStorage] Have HotStorageWriter::write_account() return Vec<StoredAccountInfo> (#34929)
#### Problem
To allow hot-storage to use HotStorageWriter::write_account() to
implement AccountsFile::append_accounts(), it is required to
provide a Vector of StoredAccountInfo to allow AccountsDB to
properly prepare the entry for each account.

#### Summary of Changes
This PR enables HotStorageWriter::write_account() to return
Vec<StoredAccountInfo>.

#### Test Plan
Extend existing tests for HotStorageWriter to verify the correctness
of the returned Vec<StoredAccountInfo>.
2024-02-01 13:33:42 -08:00
Brooks daa2449ad4
Removes RwLock on AccountsDb::shrink_paths (#35027) 2024-02-01 09:35:34 -05:00
Brooks 0569304835
Replaces bare Arc::default() in CacheHashData::new() (#35017) 2024-01-31 10:52:59 -05:00
Yueh-Hsuan Chiang 16a2f1bd3d
[TS] Add get_account() and account_matches_owner() to TieredStorageReader (#34968)
#### Problem
TieredStorageReader is a wrapper enum that works for
both Hot and Cold storage readers, but its get_account()
and account_matches_owner() API are missing.

#### Summary of Changes
Add get_account() and account_matches_owner() to
TieredStorageReader.

#### Test Plan
hot.rs offers similar coverage for HotStorageReader.
2024-01-29 11:05:47 -08:00
Brooks b9815da6cc
Fixes typo (#34990) 2024-01-28 17:32:09 -05:00
Pankaj Garg 0d117d420c
Remove BlockhashQueue dependency from SVM related code (#34974) 2024-01-26 13:46:44 -08:00
HaoranYi e38848e519
Remove unused get_append_vec_id function (#34949)
remove unused get_append_vec_id fn

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-01-26 21:41:09 +00:00
Yueh-Hsuan Chiang 3380c14044
[TieredStorage] Correct the HotStorage API for account_matches_owners (#34967)
#### Problem
In HotStorageReader, the account_matches_owners takes
&[&Pubkey] as the address candidates.  However, it should
be &[Pubkey] as defined in the accounts_file API.

#### Summary of Changes
Correct HotStorageReader::account_matches_owners() to
take &[Pubkey] instead.

#### Test Plan
Existing unit-tests
2024-01-26 09:40:05 -08:00
Yueh-Hsuan Chiang 7138f8767e
[TieredStorage] Avoid AccountHash copy in AccountMetaOptionalFields (#34969)
#### Problem
Using non-reference type of AccountHash in 
AccountMetaOptionalFields causes an unnecessary copy
as mentioned in #34948.

#### Summary of Changes
Uses &AccountHash in AccountMetaOptionalFields to
avoid copying.

#### Test Plan
Existing unit tests.

Fixes #34948
2024-01-26 09:13:09 -08:00
Yueh-Hsuan Chiang 663a1bb8f4
[TieredStorage] Write owners block for HotAccountStorage (#34927)
#### Problem
So far the current HotStorageWriter::write_accounts() only writes
accounts blocks and index block.

#### Summary of Changes
The PR further writes owners block in HotStorageWriter::write_accounts().

#### Test Plan
Extended existing test for HotStorageWriter to cover the owners block.
2024-01-26 06:48:28 -08:00
Yueh-Hsuan Chiang 51c0649af8
[TieredStorage] Use RENT_EXEMPT_RENT_EPOCH in HotStorageWriter (#34950)
#### Problem
In HotStorageWriter::write_accounts, it skips storing rent-epoch when
the rent-epoch equals Epoch::MAX.  While the value is correct, it is
more suitable to use RENT_EXEMPT_RENT_EPOCH instead as the
goal here is to save bytes for rent-exempt accounts.

#### Summary of Changes
Replace Epoch::MAX by RENT_EXEMPT_RENT_EPOCH when checking
whether to skip storing rent-epoch in HotStorageWriter.
2024-01-25 17:17:24 -08:00
Brooks f0d67d7f28
Adds hit/miss stats for cache hash data (#34954) 2024-01-25 17:27:45 -05:00
Brooks e155d9c445
Adds cache hash data deletion policy enum (#34956) 2024-01-25 16:58:56 -05:00
Yueh-Hsuan Chiang 26d62b9516
[TieredStorage] writing hot account blocks and index blocks (#34828)
#### Problem
The implementation of write_accounts() for HotAccountStorage is missing.
It consists of the writing of account blocks, index block, and owners block.

#### Summary of Changes
This PR completes part of the HotStorageWriter::write_accounts().
Specifically, it finishes the writing of account blocks and index block.

#### Test Plan
A new unit-test is added to verify the correctness of the work-in-progress
HotStorageWriter::write_accounts().
2024-01-25 09:38:06 -08:00
Brooks 5898b9a2f7
Cleans up stale accounts hash cache files (#34933) 2024-01-24 15:31:50 -05:00
Dmitri Makarov ef233eaaa7
Refactor load_accounts to take a reference to a slice instead of vec (#34919)
This is in preparation for further refactoring of
Bank::load_and_execute_transactions in a separate commit.
2024-01-24 09:43:18 -05:00
Yueh-Hsuan Chiang 1810feadc2
[TieredStorage] In-memory struct for writing OwnersBlock (#34853)
#### Problem
To write the owners-block, it requires an in-memory struct that maintains
a set of unique owner addresses while providing a look-up function to
obtain the OwnerOffset with the specified owner address. 

#### Summary of Changes
This PR adds OwnersTable, the in-memory struct that maintains
a set of unique owner addresses while providing a look-up function to
obtain the OwnerOffset with the specified owner address.

#### Test Plan
A new unit-test is added.
2024-01-23 14:57:53 -08:00
Brooks 8ff511e8fa
Moves create_and_canonicalize_directories() into accounts-db utils (#34882) 2024-01-23 06:46:27 -05:00
Brooks 2f744f1639
Moves create_all_accounts_run_and_snapshot_dirs() into accounts-db utils (#34877) 2024-01-22 18:18:43 -05:00
Brooks 3916c31cb4
Replaces fs-err in tiered storage (#34859) 2024-01-19 17:16:56 -05:00
Brooks 3eb06b4b37
Replaces fs-err in accounts-db (#34860) 2024-01-19 16:35:02 -05:00
Yueh-Hsuan Chiang 81b3210eec
[TieredStorage] Code refactoring for OwnersBlock (#34854)
#### Problem
The OwnersBlockFormat is currently defined inside footer.rs
instead of inside owners.rs.  In addition, the implementation of
OwnersBlock doesn't honor OwnersBlockFormat.

#### Summary of Changes
This PR moves OwnersBlockFormat from footer.rs to owners.rs
and repurpose OwnersBlock as OwnersBlockFormat (just like
the IndexBlockFormat inside index.rs)

#### Test Plan
Existing unit-tests.
2024-01-19 08:15:25 -08:00
steviez 3dd348802f
Bubble up genesis load errors instead of exiting (#34851)
The function open_genesis_config() performs several operations that
could fail. If any of these fail, the process exits immediately.

Instead of exiting immediately, bubble up the error and let the caller
decide the appropriate action. solana-validator and solana-ledger-tool
will functionally be unchanged, but this consolidates startup failures
for both of these processes.
2024-01-19 10:25:46 -05:00
Yueh-Hsuan Chiang 3fa44e6fbe
[TieredStorage] Include executable field into AccountMetaFlags (#34724)
#### Problem
Before we have fully switched to the new way to determine whether
an account is executable, we still need a bit for th executable flag at
this moment in the TieredStorage as well as for backward compatibility
in case we want to revert it back.

#### Summary of Changes
This PR adds the executable flag into AccountMetaFlags.

#### Test Plan
Updated existing tests for AccountMetaFlags to cover executable flag.
2024-01-16 11:25:57 -08:00
Andrew Fitzgerald 95f888ab54
Unlock only if lock_result was Ok (#34652) 2024-01-09 08:19:25 -08:00
Yueh-Hsuan Chiang 2082f8b3ca
[TieredStorage] HotStorageWriter::new() (#34659)
#### Problem
The implementation of HotAccountWriter is missing.

#### Summary of Changes
This PR kicks off the implementation of HotStorageWriter by
adding HotStorageWriter::new().

#### Test Plan
Add a new unit-test that verifies the correctness of HotStorageWriter
writing zero accounts using HotStorageReader.
2024-01-08 14:14:30 -08:00
Yueh-Hsuan Chiang 3a2f1320c5
[TieredStorage] Rename AddressAndBlockOffsetOnly to AddressesThenOffsets (#34658)
#### Problem
In the previous refactoring, IndexBlockFormat::AddressAndBlockOffsetOnly
can now work independently with the AccountMetaFormat.  As a result,
the naming of this format should be updated to reflect this.

#### Summary of Changes
As the format first persists the list of addresses followed by the list of offsets,
this PR renames AddressAndBlockOffsetOnly to AddressesThenOffsets.
2024-01-08 13:10:37 -08:00
Brooks e4ec68d704
clippy: accounts-db lints (#34628)
```
warning: this `let...else` may be rewritten with the `?` operator
  --> accounts-db/src/rolling_bit_field/iterators.rs:36:13
   |
36 | /             let Some(bit) = self.bit_range.next() else {
37 | |                 return None;
38 | |             };
   | |______________^ help: replace it with: `let bit = self.bit_range.next()?;`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#question_mark
   = note: `#[warn(clippy::question_mark)]` on by default

    Checking solana-rpc-client-api v1.18.0 (/Users/brooks/src/solana/rpc-client-api)
warning: `solana-accounts-db` (lib) generated 1 warning
```
2024-01-03 14:17:12 +00:00
Nick Frostbutter fc2a8794be
[docs] updated readme and fix links (#34565)
* feat: updated readme

* fix: updated links

* fix: proposal links

* fix: more links

* fix: json-rpc links

* fix: more links

* fix: zk links

* fix: managing forks

* fix: links for deprecated methods
2024-01-03 09:06:06 -05:00
Yueh-Hsuan Chiang d500eb6510
[TieredStorage] HotAccountsReader::get_account (#34499)
#### Problem
All structs that implement AccountsFile are required to support get_account,
but the implementation for HotAccounts is currently missing.

#### Summary of Changes
This PR implements HotAccountsReader::get_account_from_index_offset.
This will allow it to support AccountsFile::get_account API.

#### Test Plan
Add new unit-test.
2023-12-25 06:44:00 -08:00
Yueh-Hsuan Chiang 5b26d2737b
[TieredStorage] Fix typos in index.rs (#34546)
#### Problem
There're some typos in the comments and assert messages index.rs mentioned in #34529

#### Summary of Changes
Fix the typos (only in assert message and comments).
2023-12-21 22:36:38 -08:00
Yueh-Hsuan Chiang f73fd9dd3c
[TieredStorage] HotStorageReader::account_matches_owners (#34350)
#### Problem
Each AccountsFile is required to implement account_matches_owners(),
a public API that checks whether the account located at the specified
offset matches any input owners.  However, the implementation of
account_matches_owners() for HotAccountsStorage is missing.

#### Summary of Changes
This PR implements HotStorageReader::account_matches_owners().

#### Test Plan
A new unit-test is added to this PR.
2023-12-21 19:20:03 -08:00
Yueh-Hsuan Chiang a583035812
[TieredStorage] Boundary check for get_account_offset() (#34531)
#### Problem
TieredStorage doesn't perform boundary check in get_account_offset
when the input IndexOffset isn't valid.

#### Summary of Changes
This PR adds two checks.  First, it checks whether the IndexOffset exceeds
the boundary of the index block.  Second, when an index format that has the
same index entries as account entries is used, it also checks whether IndexOffset
is smaller than account_entry_count.

#### Test Plan
Two new tests are added to this PR.
2023-12-21 12:52:15 -08:00
GoodDaisy 03386cc7b9
Fix typos (#34459)
* Fix typos

* Fix typos

* fix typo
2023-12-21 13:06:00 -07:00
Yueh-Hsuan Chiang 09efd704eb
[TieredStorage] boundary check for get_account_address() (#34529)
#### Problem
get_account_address() does not check whether IndexOffset is valid.

#### Summary of Changes
This PR adds two checks.  First, it checks whether the IndexOffset exceeds
the boundary of the index block.  Second, when an index format that has the
same index entries as account entries is used, it also checks whether IndexOffset
is smaller than account_entry_count.

#### Test Plan
New unit-test is added.
2023-12-20 10:44:34 -08:00
Yueh-Hsuan Chiang eb948b1ddc
[TieredStorage] Boundary check for accessing hot account meta (#34349)
#### Problem
Hot accounts are stored in accounts blocks, whose offset is smaller than
the index block offset.  However, the current code doesn't perform
any boundary checks when accessing hot account meta.

#### Summary of Changes
Adds boundary check when accessing hot account meta.
2023-12-18 14:32:41 -08:00
Brooks e858a5a7c0
Removes unnecessary new_sized() AccountsDb ctor (#34504) 2023-12-18 16:39:57 -05:00
Brooks 4181ea4677
Removes unnecessary AccountsDb::new_with_config_for_benches() (#34484) 2023-12-18 06:52:41 -05:00
Brooks cbe8a02029
Removes unnecessary AccountsDb::new_with_config_for_tests() (#34485) 2023-12-18 00:46:46 -05:00
Joe C 171c58c5c0
RPC: Add inner instructions to simulate transaction response (#34313)
* rpc: add optional `innerInstructions: bool` arg to `simulateTransaction`

* bank: enable cpi recording in simulate

* sdk: move `InnerInstructions` into SDK from accounts DB

* bank: return inner instructions from simulate tx

* rpc: return inner instructions from simulate tx

* rpc: simulate tx: add `jsonParsed` support for inner instructions

* accounts db: add deprecated attribute to re-exported inner instructions

* rpc: de-dupe inner instruction mapping

* update deprecated comment

Co-authored-by: Tyera <teulberg@gmail.com>

---------

Co-authored-by: Tyera <teulberg@gmail.com>
2023-12-16 12:49:22 +00:00
Brooks 45eaa4c1a6
Removes unnecessary Accounts constructors (#34471) 2023-12-14 22:50:15 -05:00
Brooks e79c09bdc4
Refactors Accounts constructors (#34466) 2023-12-14 16:36:47 -05:00
Brooks c7a57672d7
Adds read/write/get_pod() fns to tiered storage (#34415) 2023-12-14 13:25:25 -05:00
Brooks d1cfd26abb
Refactors AccountsDb constructors (#34456) 2023-12-14 12:34:15 -05:00
Ashwin Sekar 337c233b5f
fix rename of HOT_ACCOUNT_ALIGNMENT (#34422) 2023-12-12 16:18:10 -05:00
Yueh-Hsuan Chiang 383aa0451e
[TieredStorage] Unit-tests for checking invalid HotAccountOffset (#34376)
#### Problem
HotAccountOffset::new() might return Err for invalid offset, and this
part needs some test coverage.

#### Summary of Changes
Add unit-tests for checking invalid HotAccountOffset.
2023-12-12 12:31:16 -08:00
Brooks a2d7be0d07
Derives Pod for tiered storage types (#34414) 2023-12-12 14:59:51 -05:00
Yueh-Hsuan Chiang 376b1430e7
[TieredStorage] Improve comments for HOT_ACCOUNT_ALIGNMENT (#34404)
#### Problem
The current naming and the code comments for HOT_ACCOUNT_OFFSET_ALIGNMENT
aren't really reflecting its role as pointed out in #34335.

#### Summary of Changes
This PR renames HOT_ACCOUNT_OFFSET_ALIGNMENT to HOT_ACCOUNT_ALIGNMENT
as it's the hot account instead of hot account offset needs to be aligned.

In addition, improve the comment block for HOT_ACCOUNT_ALIGNMENT.
2023-12-12 11:48:29 -08:00
Yueh-Hsuan Chiang 39a3566438
[TieredStorage] Make AccountOffset a trait, introduce HotAccountOffset (#34335)
#### Problem
Hot and cold accounts storage have different implementations of
their offsets.  As a result, a single struct AccountOffset isn't suitable
to describe the offsets used by hot and cold accounts storage.

#### Summary of Changes
This PR makes AccountOffset a trait.  On top of that, introduces
HotAccountOffset that implements AccountOffset.

#### Test Plan
Updated existing unit-tests.
2023-12-11 15:38:38 -08:00
HaoranYi aab12c31c7
tune ancient append vec size to 128M (#34067)
* tune ancient append vec size to 130M

* fix a test and get rid of the assert since it is covered in the test

* use 128M

* assert max append vec size for ancient append vec

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-12-11 15:45:35 -06:00
HaoranYi c43edd2fef
use seqlock for AccountStorage count and status tracking (#34356)
use seqlock for appendvec count and status tracking

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-12-08 08:49:54 -06:00
Brooks b32903a534
Removes glob import of AccountsHashVerificationError (#34332) 2023-12-06 13:46:00 -05:00
Brooks 5c00a33252
Removes unnecessary #[allow(dead_code)] in tiered storage (#34320) 2023-12-06 00:23:01 +00:00
Brooks c971ad2944
Removes unnecessary #[allow(dead_code)] in partitioned rewards (#34321) 2023-12-05 15:58:11 +00:00
Jeff Washington (jwash) 091fc820ba
ancient append vec append creates big enough ancient append vecs (#34154)
* ancient append vec append creates big enough ancient append vecs

* assert we write the correct # of bytes

* fix tests

* u64 align accounts bytes written

* fix build

* add tests

* we don't need to align the store to u64.

When we call `remaining_bytes` to calculate, for the last element, u64_align!(len) could be greater than capacity.  However, because saturate_sub is used, we still get the correct answer of 0. Therefore, no need to align the storage to u64.

* add comments

* add shink test for min_size

* remove dead code

* fix build

* reviews

* add test for squash ancient vec overflow too much to fit into ideal av

* add test for get_bytes on AccountToStore

* clippy

* modify test to assert 0 size of opposite

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-12-05 09:25:11 -06:00
Brooks 2c033e05e5
Switches order of tiered storage footer size and version (#34310) 2023-12-04 14:23:56 -05:00
Brooks 4f65f7dc8d
Sanitizes tiered storage footer after reading from disk (#34200) 2023-12-01 13:58:10 -05:00
Yueh-Hsuan Chiang fa0930f254
[TieredStorage] Allow HotStorage to handle more account data (#34155)
#### Problem
After we have defined AccountOffset to be u32, it means the address space within
one HotAccountsFile is up to 4GB.  However, since the Accounts Blocks in a
HotAccountsFile is 8-byte aligned, it has the opportunity to store more data by
internally multiplying the AccountOffset by 8.

#### Summary of Changes
This PR allows a HotAccountsFile to store up to 32GB accounts data by using
8 x AccountOffset as its actual offset.

#### Test Plan
Updated existing unit-tests.
2023-12-01 10:39:15 -08:00
Andrew Fitzgerald 2294801954
Do not derive Copy for EpochSchedule and Rent (#32767) 2023-12-01 07:57:25 -08:00
Jeff Washington (jwash) 32c7acbf74
add slot to assert message (#34225) 2023-11-28 10:29:39 -06:00
Brooks deee5f833c
Uses Display for error in VerifyAccountsHashInBackground's panic message (#34219) 2023-11-27 09:51:39 -05:00
Yueh-Hsuan Chiang ecc067f7ad
[TieredStorage] Make AccountOffset use u32 (#34151)
#### Problem
AccountOffset currently uses `usize`, which size is platform dependent.
We want a fixed size type that is consist to what we persist in the tiered-storage file.

#### Summary of Changes
This PR makes AccountOffset use u32.
2023-11-21 10:40:10 -08:00
Yueh-Hsuan Chiang db2444bd11
Move MatchAccountOwnerError from append_vec to accounts_file (#34187)
#### Problem
MatchAccountOwnerError currently belongs to append_vec.

However, it is a public error type that is also required by other AccountsFile
implementations such as TieredStorageFile.

#### Summary of Changes
Move MatchAccountOwnerError from append_vec to accounts_file.
2023-11-21 10:39:56 -08:00
Jeff Washington (jwash) 481c357543
remaining_bytes aligns len since all writes will align first (#34171)
* remaining_bytes aligns len since all writes will align first

* use remaining_bytes() to check for whether the new account can fit into the av storage

* Add test coverage for av remaining bytes alignment

* use great equal to check space available

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-11-21 09:34:52 -06:00
HaoranYi 90b11a608b
refactor AccountsToStore (#34185)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-11-21 09:27:31 -06:00
Yueh-Hsuan Chiang c73f226958
[TieredStorage] Make IndexOffset use u32 (#34152)
#### Problem
IndexOffset currently uses `usize`, which size is platform dependent.
We want a fixed size type that is consist to what we persist in the tiered-storage file.

#### Summary of Changes
This PR makes IndexOffset use u32.
2023-11-20 14:57:15 -08:00
Lucas Steuernagel a3670054be
Move `RewardInterval` to runtime (#34146)
Move RewardInterval to runtime
2023-11-20 10:17:21 -03:00
Brooks e02f25d5a2
Removes filler accounts (#34115) 2023-11-19 20:36:57 -05:00
Jeff Washington (jwash) 202463849a
log when appending exceeds available space (#34170) 2023-11-19 18:50:37 -06:00
Brooks 574b8b5bfc
Refactor assert in scan_snapshot_stores() (#34162) 2023-11-18 12:05:39 -05:00
Brooks 8c8cd66095
Uses Display in TieredStorageFile::new_readonly() panic message (#34148) 2023-11-17 15:24:05 -05:00
Yueh-Hsuan Chiang 625e898c63
[TieredStorage] Make IndexBlock persist u32 offsets (#34133)
#### Problem
In TieredStorage, we want to make AccountOffset and IndexOffset u32 instead
of usize just like OwnerOffset.  However, we need to first change what we persist
in the storage.  Currently, IndexBlock persists offsets as u64 instead of u32. 

#### Summary of Changes
This PR makes IndexBlock persist u32 offsets.

#### Test Plan
Existing test cases.
2023-11-17 11:35:32 -08:00
Brooks e23bfb5167
Refactors TieredStorageFile read/write methods (#34147) 2023-11-17 14:17:49 -05:00
Yueh-Hsuan Chiang 1e9f6d9d83
[TieredStorage] Use OwnerOffset type in TieredAccountMeta (#34106)
#### Problem
TieredAccountMeta currently uses `u32` for owner_offset, while
we already have a dedicate type for that --- OwnerOffset.

#### Summary of Changes
This PR makes TieredAccountMeta use OwnerOffset for its
owner_offset field.

#### Test Plan
Existing unit-tests.
2023-11-17 09:26:52 -08:00
Brooks 07b0b9f9e9
Uses `mmap` for names in tiered storage (#34127) 2023-11-16 23:59:08 +00:00
Lucas Steuernagel 2c71d21fad
Move load_accounts to runtime (#34017)
* Move load_accounts to runtime

---------

Signed-off-by: Lucas Steuernagel <lucas.tnagel@gmail.com>
2023-11-16 18:45:58 -03:00
Brooks 7e3b09defe
Uses consistent error types in tiered storage (#34110) 2023-11-16 15:51:57 -05:00
Yueh-Hsuan Chiang f59887077a
[TieredStorage] Define OwnerOffset as u32 (#34105)
#### Problem
The current OwnerOffset is defined as usize, which actual size could be u32 or u64.
On the other hand, the OwnerOffset is used to access the ith owner in one
TieredStorageFile.  As a result, u32 is more than enough to represent the number of
owners inside one TieredStorageFile while u16 might not be always enough.

#### Summary of Changes
This PR defines OwnerOffset as u32.

#### Test Plan
Existing test cases.
2023-11-16 12:48:33 -08:00
HaoranYi 60fdd85aed
Add a feature to disable rent collection (#33945)
* add a feature to disable rent collection

* fix a test

* fix a test

* rekey

* should collect rent

* Update runtime/src/bank/fee_distribution.rs

Co-authored-by: Brooks <brooks@prumo.org>

* expand tests to cover both rent collection disabled and enabled

* feedbacks

* reviews - move should collect rent check out of rent collector into bank

* enforce rent_epoch to u64:max when rent collection is disabled

* review feedbacks and fix a test
When rent fee collection is disabled, we won't collect rent for any account. If there are any rent paying accounts, their `rent_epoch` won't change too.

* revise comments

* update rent_epoch for rent exempted account

* rebase

* set rent_epoch in rent collection for rent exempted account

* revert test change

* don't assert

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
Co-authored-by: Brooks <brooks@prumo.org>
2023-11-16 08:57:49 -06:00
Yueh-Hsuan Chiang b7f839ea18
[TieredStorage] HotStorageReader::get_owner_address (#34053)
#### Problem
The HotStorageReader does not have an API to obtain owners_address.

#### Summary of Changes
This PR adds HotStorageReader::get_owner_address().

#### Test Plan
A new unit-test is added to this PR.
2023-11-15 15:54:53 -08:00
Yueh-Hsuan Chiang d58db6e255
Improve formatting in test_hot_storage_get_account_offset_and_address() (#34071)
#### Problem
#34032 mentioned nice-to-have a formatting improvement that
can improve the readability. 

#### Summary of Changes
Improve the code formatting in test_hot_storage_get_account_offset_and_address().
2023-11-15 11:25:18 -08:00
Brooks 67dab71e82
Uses `RENT_EXEMPT_RENT_EPOCH` to mark account as rent exempt (#34090) 2023-11-15 18:13:18 +00:00
Alexander Meißner ffa462dd3a
Cleanup - Feature gate of `delay_visibility_of_program_deployment` (#34087)
* Cleans up feature gate of delay_visibility_of_program_deployment.

* Removes programs_updated_only_for_global_cache.
2023-11-15 18:29:31 +01:00
Alexander Meißner 48ed70a8d8
Cleanup - Feature gate of `disable_builtin_loader_ownership_chains` (#34083)
* Cleans up feature gate of disable_builtin_loader_ownership_chains.
2023-11-15 17:17:31 +01:00
Yueh-Hsuan Chiang 7fd13c0cd2
[TieredStorage] OwnersBlock (#34052)
#### Problem
A TieredStorage file has three types of block:  accounts block, index block,
and owner block, but implementation of the TieredStorage OwnersBlock
is missing in the current master.

#### Summary of Changes
This PR implements OwnersBlock which stores a set of owners' addresses
in a compact but efficient way.

#### Test Plan
A new unit-test is included in this PR.
2023-11-15 01:51:13 -08:00
Yueh-Hsuan Chiang eb7e68f029
[TieredStorage] HotStorageReader::get_account_address (#34032)
#### Problem
HotStorageReader currently not yet has an API to obtain account_address

#### Summary of Changes
This PR adds HotStorageReader::get_account_address() which returns
the Pubkey of the account associated with the specified IndexOffset.

#### Test Plan
Augmented an existing unit-test to cover get_account_address() case.
2023-11-14 20:44:03 -08:00
Yueh-Hsuan Chiang 2877b8182c
[TieredStorage] Rename owners_offset to owners_block_offset (#34047)
#### Problem
TieredStorageFooter::owners_offset refers to the offset to the
owners block, which looks very similar to OwnerOffset that
has a completely different concept -- the offset to access the
address of an owner in the owners-block.

#### Summary of Changes
Similar to the previous renaming index_offset to index_block_offset,
this PR renames owners_offset to owners_block_offset to
avoid confusion.
2023-11-13 18:42:51 -08:00
Brooks 725ab37bf4
clippy: Replaces .get(0) with .first() (#34048) 2023-11-13 17:22:17 -05:00
Jeff Washington (jwash) a25eae41fc
stop page aligning shrunk append vecs (#34016)
* stop page aligning shrunk append vecs

* fix build

* fix test by removing aligned alive page bytes asserts

* rm dup assert

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-11-13 14:22:26 -06:00
Yueh-Hsuan Chiang 2b9054a60e
[TieredStorage] HotStorageReader::get_account_offset (#34031)
#### Problem
HotStorageReader currently not yet has an API to obtain account_offset

#### Summary of Changes
This PR adds HotStorageReader::get_account_offset() which takes
IndexOffset and returns AccountOffset.

#### Test Plan
A new unit-test is included in this PR.
2023-11-13 12:07:09 -08:00
Yueh-Hsuan Chiang 3db7eaf1fc
[TieredStorage] Improve param naming of IndexBlockFormat (#34033)
#### Problem
In IndexBlockFormat, both `IndexOffset` and `AccountOffset` parameters
are named `offset` in some functions which could be confusing.

#### Summary of Changes
Renamed `offset` to `index_offset` and `account_offset` to improve
readability.
2023-11-13 12:05:25 -08:00
Brooks 3c71f859e1
Uses fold+reduce for handling duplicate pubkeys during index generation (#34011) 2023-11-10 14:32:24 -05:00
Brooks 69ab8a8234
Uses IntSet for uncleaned slots from visit_duplicate_pubkeys_during_startup() (#34009) 2023-11-10 11:29:13 -05:00
Pankaj Garg 59eb55990c
Move filter_executable_program_accounts to bank.rs (#34004) 2023-11-09 14:40:04 -08:00
Brooks 9f25f67e60
Simplifies adding uncleaned pubkeys during index generation (#34007) 2023-11-09 16:06:15 -05:00
Brooks a96be5d2f0
Uses IntSet for uncleaned roots during index generation (#34008) 2023-11-09 15:04:19 -05:00
Yueh-Hsuan Chiang 874fae507f
[TieredStorage] Make HotStorageReader use AccountOffset type (#33964)
#### Problem
#33927 introduced a new type AccountOffset, but HotStorageReader
still uses `usize` to access accounts.

#### Summary of Changes
This PR makes HotStorageReader use the new AccountOffset type.
2023-11-08 21:58:44 -08:00
Brooks bba6ea2d69
Returns IntMap from select_candidates_by_total_usage() (#33976) 2023-11-08 16:15:51 -05:00
Brooks 7cb83bc491
Adds `S` to HashMap/HashSet impls of Contains (#33973) 2023-11-08 15:08:33 -05:00
Brooks 6f213c38aa
Removes Default from RollingBitField (#33969) 2023-11-07 15:58:52 -05:00
Yueh-Hsuan Chiang b013c03afa
[TieredStorage] Add IndexOffset type (#33929)
#### Problem
TieredStorage conceptually has different offsets. However, the current code directly
uses the same primitive type for accessing offsets, which is error-prone as one could
easily use one offset to access data that should be accessed with a different offset
type.

#### Summary of Changes
This PR adds IndexOffset type -- a struct for obtaining the ith entry inside the
index-block to obtain account's offset and address.
2023-11-07 10:26:21 -08:00
Jeff Washington (jwash) b8115b4303
chunk all ancient append vecs (#33909)
* chunk all ancient append vecs

* fix a test comments

* remove unneeded dead_code attr

* do full chunking when pack is used to create ancient storage

* refacotr and fix tests

* clippy

* add cache hash file stats

* comments

* fix test

* Update accounts-db/src/accounts_db.rs

Co-authored-by: Brooks <brooks@prumo.org>

* Update accounts-db/src/accounts_db.rs

Co-authored-by: Brooks <brooks@prumo.org>

* test_case

* remove commented out code

* remove hash cache data stats

* typo

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
Co-authored-by: HaoranYi <haoran.yi@gmail.com>
Co-authored-by: Brooks <brooks@prumo.org>
2023-11-07 09:10:59 -06:00
Lijun Wang 8c69a0ec38
Remove RwLock on AccountsUpdateNotifier (#33960) 2023-11-06 14:03:25 -08:00
Yueh-Hsuan Chiang da130b87d3
[TieredStorage] Add AccountOffset type (#33927)
#### Problem
TieredStorage conceptually has different offsets.  However, the current code directly
uses the same primitive type for accessing offsets, which is error-prone as one could
easily use one offset to access data that should be accessed with a different offset
type.

#### Summary of Changes
This PR introduces the AccountOffset type, which allows static-check to on different
type of TieredStorage offsets.
2023-11-06 12:21:08 -08:00
Brooks 70d97d3261
Adds `iter_ones()` to RollingBitField (#33956) 2023-11-06 14:06:43 -05:00
Yueh-Hsuan Chiang 6624a09d38
[TieredStorage] Rename account-index to index-block (#33928)
#### Problem
The current tiered-storage code uses "account-index" to call index-block.
This could lead to confusion especially as we start giving each offset/position/index a specific type.

#### Summary of Changes
This PR renames all structs/variables that use account-index to refer to index-block.
2023-11-06 10:32:19 -08:00
Brooks cdc284189a
Refactors RollingBitField::min() (#33911) 2023-10-28 12:47:29 -04:00
Brooks 1814b2bc81
Adds logs for starting/stopping of the background account hasher (#33903) 2023-10-27 13:03:33 -04:00
Jeff Washington (jwash) a18debc34a
allow test feature to skip rewrites (#33851)
* allow test feature to skip rewrites

* hook up cli arg for test skip rewrites, update tests

* fix sanity checker

* add account hash to abi to fix a test

* reviews

* use hashmap to collect skip_rewrites. exclude skip_rewrites from dirty
pubkey set

* accumulate skipped_rewrite in reduce

* mutex

* fmt

* skip hash verify for this test flag

* add skipped rewrites num stat

* skip bank hash verify not account hash verify

* reviews

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-27 09:14:05 -05:00
Tao Zhu 510b6b949f
Split compute budget instructions process from struct (#33852)
* Split compute budget instruction processing from ComputeBudget struct itself, so CB instructions can be processed elsewhere without involving ComputeBudget

* updated tests

* avoid built ComputeBudget from dated ComputeBudgetLimits in this refactoring PR

* Clean-up program-runtime/src/compute_budget_processor.rs

* Add test for a corner case that deprecated instruction is used to request units greater than max limit;
* Update code to handle the corner case.
2023-10-26 22:12:56 -05:00
Brooks bd1080b26f
Adds AtomicAge to bucket map holder (#33841) 2023-10-25 10:47:21 -04:00
Kevin Heavey af7fd32f4c
chore: fix some typos (#33833)
* fix spelling of "retrieved"
* fix spelling of "should"
* fix spelling of "comparisons"
2023-10-25 16:21:53 +02:00
Brooks 612e8e8457
Removes Default and pub from CumulativeOffset (#33840) 2023-10-24 19:22:34 +00:00
Brooks abf51864b4
Removes `#[allow(dead_code)]` in StorableAccounts impl (#33821) 2023-10-24 18:31:27 +00:00
HaoranYi 9064b8b2a2
Optimize account hash CumulativeOffset index from vec to 2-element array (#33839)
* optimize account hash merkle tree index from vec to 2-element array

* typo

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-24 18:26:16 +00:00
Brooks 2f024f0cc1
Uses AccountHash in CalculateHashIntermediate (#33822) 2023-10-24 08:39:32 -04:00
Jeff Washington (jwash) b0b4e1f0c0
remove IncludeSlotInHash after feature activation on mnb (#33816)
* remove IncludeSlotInHash after feature activation on mnb

* fix compile errors

* compile errors

* fix tests

* fix test results
2023-10-23 15:12:02 -07:00
Jeff Washington (jwash) 54b796f5a1
ancient pack: add low water mark (#33785) 2023-10-23 10:56:18 -07:00
HaoranYi 669bc43bcd
Buffer account's fields for hash (#33788)
* buffer accounts field for hash

* use smallvec to allocate hash buffer on stack

* sort deps

* more opt

* clippy

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-23 09:48:17 -05:00
Jeff Washington (jwash) a41b24f185
fix some ancient stats (#33791) 2023-10-23 07:23:47 -07:00
Jeff Washington (jwash) 34103e0913
typo in variable name (#33801) 2023-10-23 07:23:11 -07:00
Jeff Washington (jwash) e13756133a
ancient: add many_refs_this_is_newest_alive (#33741)
* add many_refs_this_is_newest_alive

* fix pathological case, add comments

* add log

* update comments

* add log when we fail to pack

* use with_capacity

* fix log comment

* clippy
2023-10-20 09:55:37 -07:00
steviez c98c24bd6d
Revert "Split compute budget instructions process from struct itself … (#33784)
Revert "Split compute budget instructions process from struct itself (#33513)"

This reverts commit c73bebe984. This
was found to be a consensus breaking change.
2023-10-20 15:53:45 +02:00
Brooks ce8ad77373
Uses AccountHash in AppendVec (#33764) 2023-10-19 17:38:09 -04:00
Jeff Washington (jwash) f13c78b7c8
add ancient append vec test (#33762)
* add ancient append vec test

* Update accounts-db/src/ancient_append_vecs.rs

Co-authored-by: Brooks <brooks@prumo.org>

* add some comments

---------

Co-authored-by: Brooks <brooks@prumo.org>
2023-10-19 09:50:38 -07:00
Tao Zhu c73bebe984
Split compute budget instructions process from struct itself (#33513)
* Split compute budget instruction processing from ComputeBudget struct itself, allow compute_budget_instructions be processed elsewhere without having to instantiate ComputeBudget

* updated tests
2023-10-19 11:10:42 -05:00
Brooks 4e5c545e23
Uses AccountHash in tiered storage (#33763) 2023-10-19 11:52:36 -04:00
Brooks 0fcc0a0941
Returns AccountHash from get_filler_account() (#33758) 2023-10-18 20:29:40 -04:00
Brooks 1045548606
Uses AccountHash in StorableAccountsWithHashesAndWriteVersions (#33751) 2023-10-18 18:58:19 -04:00
Brooks 3a580f4df6
StorableAccounts::hash() returns &AccountHash (#33748) 2023-10-18 15:41:26 -04:00
Yueh-Hsuan Chiang 94273434ce
[TieredStorage] HotStorageReader::get_account_meta_from_offset() (#33724)
#### Problem
HotStorageReader currently only implements get_footer().  It does not
have a function to obtain the account meta.

#### Summary of Changes
This PR implements HotStorageReader::get_account_meta_from_offset().
A function that returns the account meta located at the specified offset.
This will be the helper function that will be later used to obtain the account
meta when the offset is available from the index block of a hot storage file. 

#### Test Plan
A new test is included in this PR.
2023-10-18 12:27:39 -07:00
Brooks e96678b302
Uses SeqLock for CachedAccountInner::hash (#33696) 2023-10-18 10:43:35 -04:00
Yueh-Hsuan Chiang 0b05e8db11
[TieredStorage] Footer test for HotStorageReader (#33718)
#### Problem
HotStorageReader currently doesn't have a test that covers its footer.

#### Summary of Changes
This PR includes a test for HotStorageReader that verifies the footer.
2023-10-17 15:24:38 -07:00
Jeff Washington (jwash) c09cbbb778
sort ancient append vec target_slots_sorted (#33729) 2023-10-17 14:49:46 -07:00
Jeff Washington (jwash) 56a7485998
log ancient stats each time we run (#33730) 2023-10-17 14:41:23 -07:00
HaoranYi 673a38c892
move timer after early exit (#33732)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-17 15:20:29 -05:00
HaoranYi f178975185
typo (#33734)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-17 14:45:17 -05:00
Jeff Washington (jwash) 5de9163625
add metric for ancient can't move slots (#33713)
* add metric for ancient can't move slots

* rename

* fix erors in replacing text

* rename
2023-10-17 08:34:18 -07:00
Jeff Washington (jwash) d948e5bf69
ancient shrink on its own cadence (#33712) 2023-10-16 10:06:20 -07:00
HaoranYi 167dac204f
Retry hash file allocation (#33565)
* retry hash file allocation

* add sleep

* submit a datapoint for retry

* typo

* more typos

* Update accounts-db/src/accounts_hash.rs

Co-authored-by: Brooks <brooks@prumo.org>

* fmt

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
Co-authored-by: Brooks <brooks@prumo.org>
2023-10-16 09:21:08 -05:00
Jeff Washington (jwash) c354879a3f
stop padding new append vecs to page size (#33658)
* stop padding new append vecs to page size

* fix test

* fix another test

* for creating test accounts, allocate larger like we used to
2023-10-12 07:11:20 -07:00
HaoranYi 1a2c7f106e
fix typo in comments (#33665)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-12 09:05:10 -05:00
steviez 982d29cf60
Revert "stop padding new append vecs to page size (#33607)" (#33634)
This reverts commit b7962a3610.
2023-10-10 14:43:52 -05:00
Alexander Meißner cb695c7b32
Cleanup - Feature Gate of `enable_early_verification_of_account_modifications` (#33579)
Cleans up the feature gate of enable_early_verification_of_account_modifications:
- Removes PreAccount
- Removes InvokeContext::pre_accounts and InvokeContext::rent
- Removes InvokeContext::verify() and InvokeContext::verify_and_update()
- Removes TransactionContext::is_early_verification_of_account_modifications_enabled()
- Removes TransactionAccounts::is_early_verification_of_account_modifications_enabled
- No longer optional: TransactionContext::rent
2023-10-10 17:26:17 +02:00
Jeff Washington (jwash) b7962a3610
stop padding new append vecs to page size (#33607)
* stop padding new append vecs to page size

* for creating test accounts, allocate larger like we used to
2023-10-10 07:03:37 -07:00
Jeff Washington (jwash) 6c7d3c8aeb
remove redundant page_align (#33608) 2023-10-10 07:02:47 -07:00
Brooks fc73813db2
Adds AccountHash newtype (#33597) 2023-10-09 16:00:52 -04:00
Jeff Washington (jwash) 052677595c
in hash calc, delete old cache files that will not be used earlier (#33432)
* in hash calc, delete old cache files that will not be used earlier

* only delete if supposed to

* fmt
2023-10-09 11:47:39 -07:00
Brooks c924719040
Removes hash param from AccountsCache::store() (#33598) 2023-10-09 18:16:15 +00:00
HaoranYi 72574dac02
Assert acount hash mmap file capacity > 0 (#33575)
assert mmap capacity > 0

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-09 09:38:00 -05:00
Brooks bb27bd88d4
Removes write version from tiered storage (#33566) 2023-10-06 16:19:35 -04:00
Yueh-Hsuan Chiang ecb1f8a9d7
[TieredStorage] Include Hot Account in StoredAccountMeta and ReadableAccount (#33544)
#### Problem
All account storage formats are required to implement both StoredAccountMeta
and ReadableAccount, but the implementation for the hot account format is missing.

#### Summary of Changes
This PR includes hot account format into StoredAccountMeta and ReadableAccount
enum.  This will allow the TieredStorageReader in the future PRs to return hot account
format in its `get_account` implementation.
2023-10-06 11:54:02 -07:00
Brooks c66af12bdb
Uses `IntSet` for `RemoveUnrootedSlotsSynchronization::slots_under_contention` (#33530) 2023-10-04 16:34:18 -04:00
Brooks 3b93041401
Uses `IntSet` for dead slots (#33529) 2023-10-04 16:21:44 -04:00
Brooks befc903993
Uses `IntSet` in AccountsDb::calc_delete_dependencies() (#33528) 2023-10-04 16:21:06 -04:00
Brooks f714a44c2a
Uses `IntSet` for `RollingBitField::excess` (#33523) 2023-10-04 15:04:28 -04:00
Brooks 5a9956824f
Uses `IntSet` for `RootsTracker::uncleaned_roots` (#33524) 2023-10-04 13:26:31 -04:00
Brooks d41fa346cc
Uses `IntSet` for `ShrinkCandidates` (#33522) 2023-10-04 13:07:35 -04:00
HaoranYi bb19ebed49
Convert tuple into dedup result struct (#33450)
convert tupe into dedup result struct

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-29 21:12:50 -05:00
Brooks 0e9e91c65e
Drops the shrink_candidate_slots lock after done inserting (#33459) 2023-09-29 16:58:55 +00:00
Brooks b81ff5d654
Fixup the metrics for remove_dead_accounts_shrink_us (#33458) 2023-09-29 16:04:33 +00:00
HaoranYi 6ea51280dd
Move sort timer out of loop (#33448)
move sort timer out of loop

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-28 20:10:41 -05:00
HaoranYi 25c27d452c
hash dedup vec (#33246)
* hash_dedup vec algo

* reviews

* reviews

* more reviews

* simplify working_set init with add_next_item

* refactor to remove special case "new" from add_item.

The new change is that, even the new item is the new min, it will still be added to working_set.
This change will make init working_set code simpler and the loop loop check simpler.
Since the item is inserted in at the end of the vector, the cost of push into and pop from the working will be O(1), shouldn't affect performance much.

* comments

* refactor unnamed tuple in working set to SlotGroupPointer type

* use SlotGroupPointer in ItemLocation

* Add Copy traits to avoid explicty call of clone on SlotGroupPointer

* consume next in add_next_item fn (credit to jeff).

note that the old code is still correct, since before call to
add_next_item, we will have already overwritten `next` to correct value.

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-28 08:58:08 -05:00
HaoranYi e088eb2be0
Code clean up (#33417)
clean up

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-26 22:35:25 +00:00
Yueh-Hsuan Chiang 746f69772a
[TieredStorage] Streamline the handling of TieredStorageFormat (#33396)
#### Problem
The TieredStorageFormat field in the TieredStorage is only used in the write path.

#### Summary of Changes
This PR simplifies the handling of TieredStorageFormat by removing its field from
TieredStorage struct but passing via write_accounts().
2023-09-26 14:05:36 -07:00
Brooks 9f6f532535
`flush_slot_cache_with_clean()` takes a single Slot (#33413) 2023-09-26 15:38:37 -04:00
Jeff Washington (jwash) a9b0fb492b
split hash calc ancient slot boundary to allow for ancient shrinking … (#33216)
split hash calc ancient slot boundary to allow for ancient shrinking to be behind
2023-09-26 07:27:50 -07:00
Brooks 23ad476ffb
Removes unused `Versioned` trait (#33360) 2023-09-25 17:18:38 -04:00
Jeff Washington (jwash) 40f536010f
visit_duplicate_pubkeys_during_startup uses scan (#33397) 2023-09-25 12:48:29 -07:00
Jeff Washington (jwash) 18231e9a5a
dump final startup index stats only after startup is complete (#33400) 2023-09-25 12:19:52 -07:00
Jeff Washington (jwash) 1b15464414
diskidx: stats for created vs reused (#33385) 2023-09-23 06:22:12 -07:00
Jeff Washington (jwash) 7cf71011fd
disk bucket: init restart path (#33375) 2023-09-22 11:40:46 -07:00
Trent Nelson 7c545b0ae9
bump rust stable to 1.72.1 (#33333)
* bump rust stable to 1.72.1

* bump rust nightly to 2023-09-20

* fix nightly lint -- unused doc comment

-- rustdoc does not generate documentation for expression fields

* fix nightly lint -- unnecessarily eager cloning of iterator items

* fix nightly lint -- loop never actually loops
2023-09-20 23:53:36 -06:00
HaoranYi c85eb73300
O(n) dedup (#33297)
O(N) dedup

Co-authored-by: jeff washington <jeff.washington@solana.com>
2023-09-18 17:55:28 -05:00
Tao Zhu 8b8a21a52f
cleanup feature: enable request heap frame instruction #30076 (#33243)
* cleanup feature: enable request heap frame instruction #30076

* update sbf tests

* removed out dated comments and test
2023-09-18 16:06:24 -05:00
Jeff Washington (jwash) 17c3930bc8
in gen index, stop using big hashmap (#33252)
* in gen index, stop using big hashmap

* update accounts_data_len

* remove approx_stored_count
2023-09-18 11:13:18 -07:00
Jeff Washington (jwash) 402981e3c1
cleanup test (#33291) 2023-09-18 10:59:42 -07:00
Jeff Washington (jwash) 27caf4d1d2
add test for duplicates in generate_index and fix approx stored count (#33290)
add test
2023-09-18 09:46:52 -07:00
steviez 9e11ae6275
Make program owners a const array instead of Vec<_> (#33275)
The program owners pubkeys are constant, no need to reconstruct the
Vec<Pubkey> and Vec<&Pubkey> each time this function runs (every time we
execute transactions).
2023-09-18 10:59:03 -05:00
Brooks 6283c1d568
Refactors out `unsafe` from cache_hash_data.rs (#33271) 2023-09-15 14:25:43 -04:00
Brooks a47f65d882
Refactors `unsafe` out of CacheHashDataFile's header (#33270) 2023-09-15 16:49:35 +00:00
Brooks f77b3d9389
Derives Pod for CalculateHashIntermediate (#33267) 2023-09-15 12:47:38 -04:00
Brooks 4c42413c1f
Refactors out `unsafe` from MmapAccountHashesFile::read() (#33266) 2023-09-15 15:09:01 +00:00
Jeff Washington (jwash) 3ad8394047
stop removing empty bins during index generation (#33242) 2023-09-14 07:53:33 -07:00
Jeff Washington (jwash) 886eabd74d
gen idx: refactor StorageSizeAndCount population (#33244) 2023-09-14 07:53:05 -07:00
Jeff Washington (jwash) 602f0f6ecb
generate index: fn arg: &Slot -> Slot (#33229)
fn arg: &Slot -> Slot
2023-09-13 14:14:30 -07:00
Jeff Washington (jwash) c2bf2a969d
remove unused metrics (#33241) 2023-09-13 14:14:10 -07:00
Jeff Washington (jwash) cb310a310a
set approx store count at generate index time (#33240) 2023-09-13 14:13:46 -07:00
Jeff Washington (jwash) 093ae61e85
single pass through each storage to populate info (#33227) 2023-09-13 12:37:01 -07:00
Jeff Washington (jwash) 3ffd78f2d7
add comments (#33226) 2023-09-12 13:51:27 -07:00
Brooks 21e7f98b3a
Uses hex for the hash in accounts hash cache file names (#33196) 2023-09-12 11:26:08 -04:00
Brooks c61ee20b44
Transient accounts hash cache dir is unnecessary (#33181) 2023-09-12 11:23:07 -04:00
HaoranYi 3cd3994939
clean up pub crate (#33214)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-11 19:28:26 -05:00
Brooks 09936aac0e
Removes `new()` from POD CalculateHashIntermediate and CumulativeOffset (#33211) 2023-09-11 22:33:25 +00:00
HaoranYi cf35799b2a
remove unused stats (#33213)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-11 21:03:37 +00:00
Brooks 659fc6837b
Removes `pub` from accounts_hash.rs items (#33209) 2023-09-11 20:49:51 +00:00
Jeff Washington (jwash) d724af863c
add a few accounts hash dedup tests (#33208) 2023-09-11 13:20:28 -07:00
Brooks 6298c6c31e
Shares accounts hash cache data between full and incremental (#33164) 2023-09-11 15:55:48 -04:00
Alexander Meißner bbb57be0a5
Refactor - Move interfaces of address-lookup-table into the program SDK (#33165)
* Adds a module `address_lookup_table` to the SDK.

* Adds a module `address_lookup_table::instruction` to the SDK.

* Adds a module `address_lookup_table::error` to the SDK.

* Adds a module `address_lookup_table::state` to the SDK.

* Moves AddressLookupTable into SDK as well.

* Moves AddressLookupTableAccount into address_lookup_table.

* Adds deprecation messages.

* Disentangles dependencies across cargo files.
2023-09-11 21:10:40 +02:00
Jeff Washington (jwash) 4dfe62a2f0
rework accounts hash calc dedup to avoid kernel file error (#33195)
* in hash calc, calculate max_inclusive_num_pubkeys

* in hash calc, dedup uses mmap files to avoid os panic

* as_mut_ptr

* remove unsafe code

* refactor count in hash files

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-11 10:57:21 -07:00
Jeff Washington (jwash) a145ade564
stat for time spent copying generate index contents (#33187)
* stat for time spent copying generate index contents

* rework to move stat to generate_index

* fix fmt
2023-09-08 12:55:54 -07:00
Jeff Washington (jwash) dc6b1eb653
in shrink, get stores prior to shrink starting (#33194) 2023-09-08 08:45:32 -07:00
Jeff Washington (jwash) 0083e426b2
ShrinkCandidates only holds slot (#33173) 2023-09-07 12:28:05 -07:00
Jeff Washington (jwash) e331275178
remove coupling of ShrinkCandidates to HashMap (#33176) 2023-09-07 09:07:37 -07:00
Brooks f4816dc0cf
Moves CacheHashData test-only methods into tests module (#33170) 2023-09-06 15:25:58 -04:00
Brooks a3dc3eb37c
Takes inner HashSet when dropping CacheHashData (#33169) 2023-09-06 18:19:01 +00:00
behzad nouri 88ee8f5820
replaces once_cell::sync::OnceCell with std::sync::OnceLock (#33140)
std::sync::OnceLock has become stable since rust 1.70.0 and there is no
longer a need for an external crate dependency.
2023-09-06 16:46:51 +00:00
HaoranYi a8bc6ebe38
Use `write_all` for binned account hash file writes (#33095)
write_all

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-06 11:03:05 -05:00
Illia Bobyr d077b13efa
accounts-db: test_hash_stored_account: Avoid UB. (#33083)
unsafe { transmute }` in the test is generating undefined behavior, as it
assigns a value into `bool` that is neither 0 nor 1.

We see discrepancy between release and debug builds due to this.

It is better to avoid `unsafe` blocks if there alternatives that let the
compiler check everything.
2023-09-05 20:30:17 -07:00
Jeff Washington (jwash) fb1ba216f5
Revert "Add an RPC API that can query the list of Top N secondary index keys and their sizes (#28887)" (#33121)
* Revert "Add an RPC API that can query the list of Top N secondary index keys and their sizes (#28887)"

This reverts commit 1e3d6349aa.

* Revert "Add Admin RPC Front End for Top N Secondary Index Key Sizes Query. (#29352)"

This reverts commit aa353e4b83.

* fix test uses

* fmt
2023-09-01 14:36:06 -07:00
Brooks d1b849972f
Support a custom path for the accounts hash cache (#33115) 2023-09-01 15:29:56 -04:00
Jeff Washington (jwash) 255029f61a
add rate info to generate index log (#33108) 2023-09-01 11:40:00 -07:00
Alessandro Decina 16ea141402
sdk: put AccountSharedData::set_data() behind feature="dev-context-only-utils" (#33086)
set_data() is deprecated in favor of set_data_from_slice().
2023-09-01 21:02:14 +07:00
Alexander Meißner 9e703f85de
Upgrades Rust to 1.72.0 & nightly-2023-08-25 (#32961)
* allow pedantic invalid cast lint

* allow lint with false-positive triggered by `test-case` crate

* nightly `fmt` correction

* adapt to rust layout changes

* remove dubious test

* Use transmute instead of pointer cast and de/ref when check_aligned is false.

* Renames clippy::integer_arithmetic to clippy::arithmetic_side_effects.

* bump rust nightly to 2023-08-25

* Upgrades Rust to 1.72.0

---------

Co-authored-by: Trent Nelson <trent@solana.com>
2023-09-01 07:26:13 +00:00
Ryo Onodera 4fe0812f3c
Make should_panic tests compatible with 1.73+ panic format (#33099)
* Adjust should_panic tests for 1.73+ panic format

* Update more panic format change affected tests...
2023-09-01 02:04:22 +00:00
HaoranYi c87f9cdfc9
use shift for pubkey to bin calculation (#33090)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-08-31 15:04:25 -05:00
Jeff Washington (jwash) 9a7d503fe4
rework sorting during generate_index (#33076)
* drag data len around during generate_index

* fix tests

* update comments
2023-08-31 12:51:56 -07:00
Brooks 5e26b05597
Removes the "unpacked entries so far" log at startup (#33093) 2023-08-31 15:29:12 -04:00
Ryo Onodera 92a0b310c1
Add #[repr(C)] for more future-proof byte mangling (#33085)
* Add #[repr(C)] for more future-proof byte manglng

* Improve comments after code-review
2023-08-31 22:48:24 +09:00
Jeff Washington (jwash) 1b9c9a313c
reorder generate_index info (#33077)
reorder generateindex info
2023-08-30 16:16:40 -07:00
behzad nouri 4ec5ea6f7b
replaces assert!(matches!(...)) with assert_matches!(...) (#33068)
assert_matches!(...) provides more informative error message when it
fails and it is part of nightly rust:
https://doc.rust-lang.org/std/assert_matches/macro.assert_matches.html
2023-08-30 13:48:27 -04:00
Jeff Washington (jwash) 48f27d5e73
add stats for generate_index (#33061) 2023-08-30 06:43:27 -07:00
Trent Nelson b8dc5daedb
preliminaries for bumping nightly to 2023-08-25 (#33047)
* remove unnecessary hashes around raw string literals

* remove unncessary literal `unwrap()`s

* remove panicking `unwrap()`

* remove unnecessary `unwrap()`

* use `[]` instead of `vec![]` where applicable

* remove (more) unnecessary explicit `into_iter()` calls

* remove redundant pattern matching

* don't cast to same type and constness

* do not `cfg(any(...` a single item

* remove needless pass by `&mut`

* prefer `or_default()` to `or_insert_with(T::default())`

* `filter_map()` better written as `filter()`

* incorrect `PartialOrd` impl on `Ord` type

* replace "slow zero-filled `Vec` initializations"

* remove redundant local bindings

* add required lifetime to associated constant
2023-08-29 23:05:35 +00:00
Jeff Washington (jwash) 00bfc6abd3
remove unused previous_uncleaned_roots (#33033) 2023-08-29 14:54:17 -07:00
Jeff Washington (jwash) 4d452fc5e9
update lru for read only cache if it has been long enough since last access (#32560)
* update lru for read only cache if it has been long enough since last access

* update comment

* use acq/rel
2023-08-29 10:37:55 -07:00
HaoranYi 555741e4d6
Refactor total_items, bin_min/max size calcualtion (#33034)
refactor total_items, bin_min/max size calcuation

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-08-29 10:37:25 -05:00
Jeff Washington (jwash) 9bc09c9610
get unique keys while finding dups (#33039)
* get unique keys while finding dups

* pr feedback
2023-08-29 15:06:35 +00:00
Jeff Washington (jwash) 97d6e6f738
move uncleaned roots code to test mod (#33029) 2023-08-28 14:01:27 -07:00
Jeff Washington (jwash) ce71461bad
remove unused return value from reset_uncleaned_roots (#33030) 2023-08-28 13:17:52 -07:00
Jeff Washington (jwash) 35bd2df0a6
speedup populate_and_retrieve_duplicate_keys_from_startup (#33013) 2023-08-28 10:20:29 -07:00
Jeff Washington (jwash) f4dcb20d2e
add overall measurement for generating index (#33008) 2023-08-28 08:40:54 -07:00
Alexander Meißner a8be70fa7a
General cleanup (#32980)
* Fixes `cargo::` in build.rs

* Fixes overshadowing reexports.

* Removes unnecessary `as *const u8`.

* Removes unnecessary `.into_iter()`.

* cargo clippy
2023-08-24 21:44:19 +02:00
Jon Cinque 0fe902ced7
Bump rand to 0.8, rand_chacha to 0.3, getrandom to 0.2 (#32871)
* sdk: Add concurrent support for rand 0.7 and 0.8

* Update rand, rand_chacha, and getrandom versions

* Run command to replace `gen_range`

Run `git grep -l gen_range | xargs sed -i'' -e 's/gen_range(\(\S*\), /gen_range(\1../'

* sdk: Fix users of older `gen_range`

* Replace `hash::new_rand` with `hash::new_with_thread_rng`

Run:
```
git grep -l hash::new_rand | xargs sed -i'' -e 's/hash::new_rand([^)]*/hash::new_with_thread_rng(/'
```

* perf: Use `Keypair::new()` instead of `generate`

* Use older rand version in zk-token-sdk

* program-runtime: Inline random key generation

* bloom: Fix clippy warnings in tests

* streamer: Scope rng usage correctly

* perf: Fix clippy warning

* accounts-db: Map to char to generate a random string

* Remove `from_secret_key_bytes`, it's just `keypair_from_seed`

* ledger: Generate keypairs by hand

* ed25519-tests: Use new rand

* runtime: Use new rand in all tests

* gossip: Clean up clippy and inline keypair generators

* core: Inline keypair generation for tests

* Push sbf lockfile change

* sdk: Sort dependencies correctly

* Remove `hash::new_with_thread_rng`, use `Hash::new_unique()`

* Use Keypair::new where chacha isn't used

* sdk: Fix build by marking rand 0.7 optional

* Hardcode secret key length, add static assertion

* Unify `getrandom` crate usage to fix linking errors

* bloom: Fix tests that require a random hash

* Remove some dependencies, try to unify others

* Remove unnecessary uses of rand and rand_core

* Update lockfiles

* Add back some dependencies to reduce rebuilds

* Increase max rebuilds from 14 to 15

* frozen-abi: Remove `getrandom`

* Bump rebuilds to 17

* Remove getrandom from zk-token-proof
2023-08-21 19:11:21 +02:00
Jeff Washington (jwash) 524274d8b5
update comment (#32919) 2023-08-21 10:04:37 -07:00
Brooks c6989189ef
Renames CalcAccountsHashFlavor to CalcAccountsHashKind (#32905) 2023-08-21 15:41:07 +00:00
Brooks a563b4cccb
Renames AccountsHashEnum to AccountsHashKind (#32904) 2023-08-21 10:34:56 -04:00
Jeff Washington (jwash) a7d340538e
cache hash files: separate get_file_reference_to_map_later and map (#32891) 2023-08-21 07:30:29 -07:00
Brooks e28c819819
Add docs to load_by_program_slot() (#32896) 2023-08-18 19:11:42 -04:00
Yueh-Hsuan Chiang 66b72a61be
TieredStorage struct (5/N) -- tests for write_accounts (#32850)
#### Problem
Having a PR that is self-contained with a big picture while having it testable
for TieredStorage::write_accounts is challenging, as it requires all the write-side
code, read-side code, and test code.

#### Summary of Changes
This PR solves part of the problem by having one dedicated PR for tests.
Specifically, it includes test utils that allow us to generate test accounts,
invoke write_accounts, and verify the written tiered-storage instance.

With this PR, it will be easier to write future PRs. 

#### Test Plan
A new set of unit-tests and test utils is included in this PR.
2023-08-18 11:29:53 -07:00
Jeff Washington (jwash) 48e51134d7
refactor cache hash data stats (#32884) 2023-08-18 09:24:59 -07:00
Jeff Washington (jwash) 7d203722cb
remove clone() (#32885) 2023-08-18 08:18:32 -07:00
Jeff Washington (jwash) 9499d3d2f8
move `map` to `CacheHashDataFileReference` (#32883)
refactor cache hash data map
2023-08-18 08:12:08 -07:00
Jeff Washington (jwash) 1693af8e68
minor cleanup to read only accounts cache (#32876) 2023-08-18 07:07:40 -07:00
Jeff Washington (jwash) ac6523c373
refactor cache hash data files (#32877) 2023-08-18 07:01:01 -07:00
Jeff Washington (jwash) 9260c0f645
move invariant code outside loop (#32837) 2023-08-17 15:28:04 -07:00
Ryo Onodera e700dde617
dcou: Update qualifier_attr to 0.2.2 (#32838)
Update qualifier_attr to 0.2.2
2023-08-15 13:10:47 +00:00
Brooks e316db2ab4
dcou: set_accounts_hash() (#32829) 2023-08-15 07:18:59 -04:00
steviez 6bbf514e78
Add ability to output components that go into Bank hash (#32632)
When a consensus divergance occurs, the current workflow involves a
handful of manual steps to hone in on the offending slot and
transaction. This process isn't overly difficult to execute; however, it
is tedious and currently involves creating and parsing logs.

This change introduces functionality to output a debug file that
contains the components go into the bank hash. The file can be generated
in two ways:
- Via solana-validator when the node realizes it has diverged
- Via solana-ledger-tool verify by passing a flag

When a divergance occurs now, the steps to debug would be:
- Grab the file from the node that diverged
- Generate a file for the same slot with ledger-tool with a known good
  version
- Diff the files, they are pretty-printed json
2023-08-15 00:12:05 -05:00
Brooks ce57cac370
Use update_accounts_hash() in AccountsHashVerifier (#32830) 2023-08-14 13:11:23 -04:00