Commit Graph

558 Commits

Author SHA1 Message Date
Jeff Washington (jwash) f610e7a06a
rework handle_reclaims ReclaimResult (#563) 2024-04-03 13:47:58 -05:00
Jeff Washington (jwash) 51c51ddf6d
cleanup `handle_reclaims` (#552)
get rid of `Option` on clean path
2024-04-03 11:46:33 -05:00
Brooks afa65c6690
Removes write version from StorableAccountsWithHashesAndWriteVersions (#561) 2024-04-03 12:41:20 -04:00
Brooks ce1f41e547
Removes write version from StorableAccounts (#542) 2024-04-03 10:21:11 -04:00
Jeff Washington (jwash) 57572d59c8
add scan_index for improving index generation (#524)
* add scan_index for improving index generation

* pr feedback

* rework some stuff from pr feedback

* get rid of redundant if

* deal with rent correctly
2024-04-03 09:19:01 -05:00
Brooks 4247a8a546
Archives storages directly (#503) 2024-04-02 18:43:38 +00:00
Jeff Washington (jwash) ccb09866b5
stop requiring data allocation to check for rent (#543) 2024-04-02 13:25:43 -05:00
HaoranYi 9ea627c16d
Recompute hash on load if default hash is stored for the account (#519)
recompute hash on load if default hash is stored for the account

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-02 13:10:18 -05:00
Jeff Washington (jwash) c59143b980
add get_stored_account to append vec (#508)
* add get_stored_account to append vec

* Update accounts-db/src/append_vec.rs

Co-authored-by: Brooks <brooks@prumo.org>

* renames

* accountshash -> accounthash

---------

Co-authored-by: Brooks <brooks@prumo.org>
2024-04-02 11:39:11 -05:00
Jeff Washington (jwash) 90999017a4
no need to call handle_reclaims if reclaims was not populated (#527) 2024-04-02 11:22:49 -05:00
Brooks 01855eda43
Sets write version to 0 when storing accounts (#476) 2024-04-02 11:27:23 -04:00
Yueh-Hsuan Chiang e8159251e9
Enable accountsdb_scan_account_storage_no_bank tests for hot storage (#344)
Enable accountsdb_scan_account_storage_no_bank tests for hot storage.
2024-04-02 08:58:13 -05:00
Yueh-Hsuan Chiang b47a4ec74d
[TieredStorage] Make TieredStorage's Offset compatible with AccountsDB's offset (#74)
* [TieredStorage] Handles reduced-offset and offset conversion

* fix comment

---------

Co-authored-by: jeff washington <jeff.washington@solana.com>
2024-04-01 16:26:41 -05:00
Yueh-Hsuan Chiang 8153c52938
[TieredStorage] enum-based AccountsFileProvider (#457)
* [TieredStorage] enum-based AccountsFileProvider

* derive traits

* remove refs

---------

Co-authored-by: jeff washington <jeff.washington@solana.com>
2024-04-01 20:54:45 +00:00
Yueh-Hsuan Chiang a5aee48722
Rename append vec to accounts file for CurrentAncientAppendVec (#447) 2024-04-01 15:54:14 -05:00
Jeff Washington (jwash) d9dfe0f8ca
rework account load with check_and_get_loaded_account_shared_data (#506) 2024-04-01 19:48:17 +00:00
Jeff Washington (jwash) 620f5658da
introduce scan_pubkeys for clean storage iteration (#507)
* introduce pubkey_iter for clean storage iteration

* rename scan_pubkeys

* pr feedback

* pr feedback
2024-04-01 12:07:41 -05:00
Brooks 288d5ba121
Moves where modules are included in accounts_db.rs (#515) 2024-04-01 09:51:59 -04:00
sakridge e0e659680a
Remove duplicated token ids and use a shared inline-spl crate (#456) 2024-04-01 14:31:04 +02:00
Jeff Washington (jwash) e56d314df5
determine if account is in write cache earlier (#504) 2024-03-29 17:10:02 -05:00
Jeff Washington (jwash) 18c32aba35
disable read cache while populating stakes cache on load (#482)
* dsiable read cache while populating stakes cache on load

* use struct with drop as api

* use LoadHint

* remove disable_read_cache_updates_count

* add comment

* fmt
2024-03-29 15:20:38 -05:00
Jeff Washington (jwash) e261e2704a
store only removes from read cache if slot is possibly present (#452)
* store only removes from read cache if slot is possibly present

* remove_assume_not_present
2024-03-28 17:00:48 -05:00
Brooks d5c0c0b1c2
Removes support for loading snapshots with > 1 append vec per slot (#474) 2024-03-28 16:32:17 -04:00
Yueh-Hsuan Chiang 4b9e1e0ab3
[TieredStorage] Exclude NotFound in reporting storage leakage on drop() (#446)
#### Problem
TieredStorage::drop() currently panic when it fails to delete the
underlying file to raise awareness of possible storage resource
leakage, including io::ErrorKind::NotFound.  But sometimes the
TieredStorage (or AccountsFile in general) instance is created
then dropped without any file being created.  This causes some
false-alarms including unit-tests.

#### Summary of Changes
This PR excludes NotFound in reporting storage leakage on
TieredStorage::drop().
2024-03-27 09:43:38 -07:00
Brooks 21fbde741a
Uses Into<PathBuf> for path in AccountsFile::new_from_file() (#434) 2024-03-26 16:19:06 -04:00
Brooks 21b6821885
Uses Into<PathBuf> for path in AppendVec::new() (#433) 2024-03-26 14:44:17 -04:00
Yueh-Hsuan Chiang c7dba30f4f
[TieredStorage] Disable accounts-file type check before enabling tiered-storage (#418)
#### Problem
As #72 introduced AccountsFile::TieredStorage, it also performs
file-type check when opening an accounts-file to determine whether
it is a tiered-storage or an append-vec.  But before tiered-storage is
enabled, this opening check is unnecessary.

#### Summary of Changes
Remove the accounts-file type check code and simply assume everything
is append-vec on AccountsFile::new_from_file().
2024-03-25 12:08:48 -07:00
Yueh-Hsuan Chiang a916edb7a2
[TieredStorage] Add AccountsFile::TieredStorage (#72)
#### Problem
AccountsFile currently doesn't have an implementation for TieredStorage.
To enable AccountsDB tests for the TieredStorage, we need AccountsFile
to support TieredStorage.

#### Summary of Changes
This PR implements a AccountsFile::TieredStorage, a thin wrapper between
AccountsFile and TieredStorage.
2024-03-24 16:41:36 -07:00
Yueh-Hsuan Chiang 602471257e
[TieredStorage] Add capacity() API and limit file size to 16GB (#401)
#### Problem
The TieredStorage has not yet implemented the AccountsFile::capacity()
API.

#### Summary of Changes
Implement capacity() API for TieredStorage and limit file size to 16GB,
same as the append-vec file.
2024-03-23 22:14:19 -07:00
Yueh-Hsuan Chiang e9cc9f8379
[TieredStorage] Refactor file_size() code path (#400)
#### Problem
TieredStorage::file_size() essentially supports AccountsFile::len(),
but its API is inconsistent with AccountsFile's.

#### Summary of Changes
Refactor TieredStorage::file_size() to ::len() and share the same API
as AccountsFile's.

#### Test Plan
Build
Existing unit-tests.
2024-03-22 14:21:24 -07:00
Yueh-Hsuan Chiang f1a82cb666
[TieredStorage] Use mmap.len() in TieredStorage::file_size() for HotStorage (#381)
#### Problem
The current implementation of TieredStorage::file_size() requires
a sys-call to provide the file size.

#### Summary of Changes
Add len() API to TieredStorageReader, and have HotStorageReader()
implement the API using Mmap::len().

#### Test Plan
Update existing unit-test to also verify HotStorageReader::len().
2024-03-22 11:56:30 -07:00
Yueh-Hsuan Chiang 977b1b836f
Rename AppendVecId to AccountsFileId (#383)
#### Problem
The current AppendVecId actually refers to an accounts file id.

#### Summary of Changes
Rename AppendVecId to AccountsFileId.

#### Test Plan
Build
2024-03-22 11:25:30 -07:00
Brooks 24fe473b46
clippy: Automated fixes for Rust 1.77.0 (#390) 2024-03-22 13:48:46 -04:00
Brooks cbd0369da1
Uses AppendVecId in AccountsFIle::file_name() (#372) 2024-03-21 21:27:03 -04:00
Brooks 8b66a670b7
Removes AccountsFile::is_recyclable() (#359) 2024-03-21 15:09:26 -04:00
Yueh-Hsuan Chiang 3038d47f1c
[TieredStorage] Store account address range (#172)
#### Problem
The TieredStorageFooter has the min_account_address and
max_account_address fields to describe the account address
range in its file.  But the current implementation hasn't updated
the fields yet.

#### Summary of Changes
This PR enables the TieredStorage to persist address range
information into its footer via min_account_address and
max_account_address.

#### Test Plan
Updated tiered-storage test to verify persisted account address range.
2024-03-20 12:17:12 -07:00
Yueh-Hsuan Chiang 0e932c7308
[TieredStorage] Refactor TieredStorage::new_readonly() code path (#195)
#### Problem
The TieredStorage::new_readonly() function currently has the following
problems:

* It opens the file without checking the magic number before checking and loading the footer.
* It opens the file twice: first to load the footer, then open again by the reader.

#### Summary of Changes
This PR refactors TieredStorage::new_readonly() so that it first performs all
checks inside the constructor of TieredReadableFile.  The TieredReadableFile
instance is then passed to the proper reader (currently HotStorageReader)
when all checks are passed.

#### Test Plan
* Added a new test to check MagicNumberMismatch.
* Existing tiered-storage tests
2024-03-20 10:39:25 -07:00
Jon C 261b3e9ee7
CI: Add windows clippy job and fix clippy errors (#330)
* CI: Run clippy on windows

* Update cargo-clippy-before-script.sh for Windows

* Pacify clippy
2024-03-20 13:21:00 +01:00
Alessandro Decina 8df80d9c12
accounts-db: unpack_archive: unpack accounts straight into their final destination (#289)
* accounts-db: unpack_archive: avoid extra iteration on each path

We used to do a iterator.clone().any(...) followed by
iterator.collect(). Merge the two and avoid an extra iteration and
re-parsing of the path.

* accounts-db: unpack_archive: unpack accounts straight into their final destination

We used to unpack accounts into account_path/accounts/<account> then
rename to account_path/<account>. We now unpack them into their final
destination directly and avoid the rename syscall.
2024-03-20 09:39:33 +11:00
Yueh-Hsuan Chiang 7c49b9c59e
[TieredStorage] Use BufWriter in TieredWritableFile (#261)
#### Problem
TieredWritableFile currently uses File instead of BufWriter.
This will introduce more syscall when doing file writes.

#### Summary of Changes
This PR makes TieredWritableFile uses BufWriter to allow the
write-call to be more optimized to reduce the number of syscalls.

#### Test Plan
Existing tiered-storage test.
Will run experiments to verify its performance improvement.

#### Dependency
https://github.com/anza-xyz/agave/pull/260
2024-03-18 14:27:55 -07:00
Yueh-Hsuan Chiang 6441209682
[TieredStorage] TieredStorageFile -> TieredReadonlyFile and TieredWritableFIle (#260)
#### Problem
TieredStorageFile struct currently offers new_readonly() and new_writable()
to allow both read and write work-load to share the same struct.  However,
as we need the writer to use BufWriter to improve performance as well as
enable Hasher on writes.  There is a need to refactor TieredStorageFile to
split its usage for read-only and writable.

#### Summary of Changes
Refactor TieredStorageFile to TieredReadonlyFIle and TieredWritableFile.

#### Test Plan
Existing tiered-storage tests.
2024-03-18 12:24:19 -07:00
Jeff Washington (jwash) 1fc4e38a4f
add stats for write cache flushing (#233)
* add stats for write cache flushing

* some renames
2024-03-18 09:53:44 -05:00
HaoranYi dcc6f1eb3a
fix typos (#247)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-03-15 09:15:23 -05:00
Jeff Washington (jwash) 794cb2f856
allow FlushStats to accumulate (#215) 2024-03-13 18:47:55 -05:00
Yueh-Hsuan Chiang 69b6d5a376
[TieredStorage] Remove the general-purposed TieredStorageWriter (#196)
#### Problem
tiered_storage/writer.rs was added when we planned to support multiple
tiers in the tiered-storage (i.e., at least hot and cold).  However, as we
changed our plan to handle cold accounts as state-compressed accounts,
we don't need a general purposed tiered-storage writer at this moment.


#### Summary of Changes
Remove tiered_storage/writer.rs as we currently don't have plans to develop cold storage.

#### Test Plan
Existing tiered-storage tests.
2024-03-13 10:26:37 -07:00
Yueh-Hsuan Chiang e13fbeb198
[TieredStorage] Repurpose TieredReadableAccount to HotAccount (#218)
#### Problem
As we further optimize the HotStorageMeta in #146, there is a need
for a HotAccount struct that contains all the hot account information.
Meanwhile, we currently don't have plans to develop a cold account
format at this moment.  As a result, this makes it desirable to repurpose
TieredReadableAccount to HotAccount.

#### Summary of Changes
Repurpose TieredReadableAccount to HotAccount.

#### Test Plan
Existing tiered-storage tests.
2024-03-13 10:07:11 -07:00
Brooks 88f6a7a459
Removes holding storages in AccountsHashVerifier for fastboot (#120) 2024-03-11 17:09:26 -04:00
Brooks 1a085c8d46
Removes atomic-ness from AccountStorageEntry `id` and `slot` fields (#119) 2024-03-11 17:09:08 -04:00
Brooks 5c1df15e92
Removes the storage recycler (#118) 2024-03-11 15:38:34 -04:00
Jeff Washington (jwash) 158c4e05d5
remove dead code (#176) 2024-03-11 12:21:51 -05:00
HaoranYi 1ac523c121
Move delta hash test function to dev-context-utils (#151)
move delta hash test function to dev-context-utils

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-03-08 12:14:40 -06:00
Yueh-Hsuan Chiang 0bf9ec861a
[TieredStorage] Deprecate the use of account-hash in HotStorage (#93)
#### Problem
TieredStorage stores account hash as an optional field inside its HotStorage.
However, the field isn't used and we have already decided to deprecate
the account hash.

#### Summary of Changes
Remove account-hash from the tiered-storage.

#### Test Plan
Existing tiered-storage tests.
Running validators w/ tiered-storage in mainnet-beta w/o storing account-hash.
2024-03-06 18:51:50 -08:00
Justin Starry 9cc55349f7
Refactor transaction account unlocking (#103)
refactor: unlock accounts
2024-03-07 09:52:23 +08:00
Brooks 1e133bc067
Increases account hash's stack buffer to hold 200 bytes of data (#56) 2024-03-05 12:02:47 -05:00
Brooks 93f5b514fa
Adds StartingSnapshotStorages to AccountsHashVerifier (#58) 2024-03-04 16:32:51 -05:00
Yueh-Hsuan Chiang 608329b974
[TieredStorage] rent_epoch() returns 0 for zero-lamport accounts (#35344)
#### Problem
In TieredAccountMeta, RENT_EXEMPT_RENT_EPOCH will be used when
its optional field rent_epoch is None.  However, for legacy reasons, 0
should be used for zero-lamport accounts.

#### Summary of Changes
Return 0 for TieredAccountMeta::rent_epoch() for zero-lamport accounts.

#### Test Plan
accounts_db::tests::test_clean_zero_lamport_and_dead_slot
2024-03-01 15:18:12 -08:00
Brooks 564a9f78a0
Casts executable bool to integer when computing account hash (#35373) 2024-03-01 14:28:28 -05:00
Brooks 59ed049dc5
Gives back 8 bytes to stack buffer for account data when hashing (#35374) 2024-03-01 14:27:48 -05:00
Brooks 83de6a5930
Moves in_mem_accounts_index.rs into accounts_index directory (#35360) 2024-02-29 06:51:18 -05:00
Brooks 2e10b3b64f
Removes InMemAccountsIndex::get() (#35354) 2024-02-28 17:57:55 -05:00
Brooks 140c21f8a9
Removes ReadAccountMapEntry (#35351) 2024-02-28 16:08:00 -05:00
Brooks 6aaaf858c9
Adds more info to panic message in AccountsHashVerifier (#35353) 2024-02-28 15:55:05 -05:00
Brooks 7c48cbb7aa
Replaces InMemAccountsIndex::get() with AccountsIndex::get_cloned() (#35352) 2024-02-28 14:45:08 -05:00
Brooks 6402198902
Replaces ReadAccountMapEntry in calculate_accounts_hash_from_index() (#35349) 2024-02-28 14:38:19 -05:00
Brooks f340c1c181
Replaces ReadAccountMapEntry in do_scan_secondary_index() (#35219) 2024-02-28 11:43:33 -05:00
Brooks a4e1a9ac98
Adds AccountsIndex::get_account_info_with_and_then() (#35336) 2024-02-27 20:01:29 -05:00
Brooks da088681ba
Adds safer alternatives to get_internal() (#35325) 2024-02-27 18:08:25 -05:00
Brooks bf2e8ee32f
AccountsIndex::get_cloned() *must* add entry to in-mem cache (#35322) 2024-02-26 18:20:21 -05:00
Brooks 8143fc3f4a
Replaces ReadAccountMapEntry in read_index_for_accessor_or_load_slow() (#35220) 2024-02-26 14:19:18 -05:00
Brooks 8235feefc4
Removes get_for_tests() (#35311) 2024-02-26 14:17:33 -05:00
Brooks 54706a885b
Adds get_with_and_then() to AccountsIndex (#35307) 2024-02-23 18:48:36 -05:00
Brooks 1e47aacd0d
Removes get_account_read_entry() (#35309) 2024-02-23 22:25:30 +00:00
Brooks 58c1b7aefb
Replaces ReadAccountMapEntry in more tests (#35308) 2024-02-23 21:20:42 +00:00
Brooks 923cac8d7f
Replaces ReadAccountMapEntry in exhaustively_verify_refcounts() (#35243) 2024-02-23 11:13:57 -05:00
Brooks b5ffc24a38
Replaces ReadAccountMapEntry in retry_to_get_account_accessor() (#35244) 2024-02-23 11:12:24 -05:00
Brooks db2071a225
Removes get_account_read_entry_with_lock() (#35240) 2024-02-21 11:25:38 -05:00
Dmitri Makarov 0acee67891
SVM: move transaction_results from accounts-db to SVM (#35183)
SVM: Remove accounts-db deps in accounts_loader tests
2024-02-20 12:54:56 -08:00
Brooks 0b5cc03aa3
Replaces ReadAccountMapEntry in ancient append vecs (#35238) 2024-02-20 13:52:11 -05:00
Brooks ce72c22b3a
Replaces ReadAccountMapEntry in accounts index (#35239) 2024-02-20 13:51:54 -05:00
Brooks 2ec136a1ea
Adds get_and_then() & family to AccountsIndex (#35218) 2024-02-19 10:22:00 -05:00
Yueh-Hsuan Chiang 69345899f3
[TieredStorage] Make TieredStorage::write_accounts() thread-safe (#35143)
#### Problem
While accounts-db might not invoke appends_account twice
for the same AccountsFile, TieredStorage::write_accounts()
itself isn't thread-safe, and it depends on the above accounts-db
assumption.

#### Summary of Changes
This PR makes TieredStorage::write_accounts() thread-safe.
So only the first thread that successfully updates the already_written
flag can proceed and write the input accounts.  All subsequent
calls to write_accounts() will be a no-op and return AttemptToUpdateReadOnly
Error.
2024-02-17 16:10:58 -08:00
Yueh-Hsuan Chiang cb61ce435e
[TieredStorage] Put commonly used test functions into test_utils.rs (#35065)
#### Problem
There're some test functions that have been used in different
mod in TieredStorage.  It's better to have one same place for
all tiere-storage related test functions.

#### Summary of Changes
Created test_utils.rs under /tiered_storage and move test-related
functions into it.

#### Test Plan
Existing tests.
2024-02-09 16:35:40 -08:00
Brooks b5e903d1d7
Refactors AccountsIndex::get() (#35163) 2024-02-09 18:26:51 -05:00
Dmitri Makarov 245d1c4087
SVM: Move TransactionCheckResult definition from accounts-db to SVM (#35153) 2024-02-08 21:13:00 -05:00
Dmitri Makarov eeb0cf1ea8
SVM: Move nonce_info from accounts-db to Solana SDK (#35138) 2024-02-08 15:01:15 -05:00
Pankaj Garg 28a320d815
SVM: Move `RentCollector` to sdk (#35122) 2024-02-07 16:38:42 -08:00
Dmitri Makarov b9ee3b475b
SVM: Move RentDebits from accounts-db to Solana SDK (#35135) 2024-02-07 15:10:17 -08:00
Dmitri Makarov 2c0001b530
SVM: Move RewardInfo from accounts-db to Solana SDK (#35120) 2024-02-07 10:55:39 -08:00
Pankaj Garg 56391f655d
Remove unnecessary usage of RentCollector (#35121) 2024-02-07 07:40:17 -08:00
Jeff Washington (jwash) 2aa8b82990
remove activated feature set_exempt_rent_epoch_max (#35082)
* remove activated feature set_exempt_rent_epoch_max

* fix test_rent_eager_collect_rent_in_partition test

* update hash values for test_bank_hash_consistency

* clean up commas
2024-02-07 09:20:31 -06:00
Pankaj Garg 10defb161f
SVM: Move TransactionErrorMetrics to SVM folder (#35112) 2024-02-06 11:15:48 -08:00
Pankaj Garg 99760e519a
SVM: move `inner_instructions_list_from_instruction_trace` to SVM (#35099)
SVM: move inner_instructions_list_from_instruction_trace to SVM
2024-02-06 08:38:24 -08:00
Dmitri Makarov ab30fb5f05
SVM: Move AccountOverrides from accounts-db to SVM (#35091) 2024-02-05 19:16:47 -05:00
Yueh-Hsuan Chiang 785dd2132e
[TieredStorage] Enable hot-storage in TieredStorage::write_accounts() (#35049)
#### Problem
While the implementation of hot-storage reader and writer
are mostly done, it is not yet connected to TieredStorage. 

#### Summary of Changes
This PR enables hot-storage in TieredStorage::write_accounts().

#### Test Plan
Completes the existing tests in TieredStorage to directly
write and read from a TieredStorage with the hot storage format.
2024-02-05 10:23:30 -08:00
HaoranYi 440c3bb156
Avoid account index entry Arc clone in shrinking (#35010)
* avoid account index entry Arc clone in shrink

* use scan to addref

* update code comments for scan fn

* expect

* warn

* update log message

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-02-05 16:47:00 +00:00
Brooks a16f982169
Removes redundant check on STORE_META_OVERHEAD (#35069) 2024-02-04 16:56:12 +00:00
Yueh-Hsuan Chiang 9935c2b5e7
[AppendVec] Use proper Vec initial size in append_accounts() (#35047)
#### Problem
append_accounts() only appends (len - skip) accounts.
However, AppendVec::append_accounts() reserves `len`
instead of `(len - skip)` for its vectors.

#### Summary of Changes
Use (len - skip) as the initial size of the Vectors.
2024-02-03 23:02:09 -08:00
Yueh-Hsuan Chiang dd30175e55
[TieredStorage] TieredStorageReader:: and HotStorageReader:: accounts() (#35031)
#### Problem
HotStorageReader and TieredStorageReader haven't implemented
accounts() that is required by AcocuntsFile.

#### Summary of Changes
This PR implements accounts() for both HotStorageReader
and TieredStorageReader

#### Test Plan
Extend the existing test to cover accounts().
2024-02-02 11:18:52 -08:00
Brooks 4b528e890c
Ensures STORE_META_OVERHEAD remains accurate (#35053) 2024-02-02 13:00:54 -05:00
Yueh-Hsuan Chiang 97d994ee6f
[TieredStorage] Use IndexOffset in TieredStorageMeta and get_account() (#35046)
#### Problem
TieredStorageMeta and TieredStorageReader::get_account API uses
u32 to represent IndexOffset.  However, within the TieredStorage scope,
IndexOffset should be used, it is not until working with AccountsFile API
when u32 representation of offset is needed.

#### Summary of Changes
Have TieredStorageMeta and TieredStorageReader to use IndexOffset.

#### Test Plan
Existing unit-tests.
2024-02-02 09:53:13 -08:00
Brooks f62293918d
Moves the async deleter code to accounts-db (#35040) 2024-02-02 09:21:26 -05:00
Yueh-Hsuan Chiang be9f17f053
[TieredStorage] Have HotStorageWriter::write_account() return Vec<StoredAccountInfo> (#34929)
#### Problem
To allow hot-storage to use HotStorageWriter::write_account() to
implement AccountsFile::append_accounts(), it is required to
provide a Vector of StoredAccountInfo to allow AccountsDB to
properly prepare the entry for each account.

#### Summary of Changes
This PR enables HotStorageWriter::write_account() to return
Vec<StoredAccountInfo>.

#### Test Plan
Extend existing tests for HotStorageWriter to verify the correctness
of the returned Vec<StoredAccountInfo>.
2024-02-01 13:33:42 -08:00
Brooks daa2449ad4
Removes RwLock on AccountsDb::shrink_paths (#35027) 2024-02-01 09:35:34 -05:00
Brooks 0569304835
Replaces bare Arc::default() in CacheHashData::new() (#35017) 2024-01-31 10:52:59 -05:00
Yueh-Hsuan Chiang 16a2f1bd3d
[TS] Add get_account() and account_matches_owner() to TieredStorageReader (#34968)
#### Problem
TieredStorageReader is a wrapper enum that works for
both Hot and Cold storage readers, but its get_account()
and account_matches_owner() API are missing.

#### Summary of Changes
Add get_account() and account_matches_owner() to
TieredStorageReader.

#### Test Plan
hot.rs offers similar coverage for HotStorageReader.
2024-01-29 11:05:47 -08:00
Brooks b9815da6cc
Fixes typo (#34990) 2024-01-28 17:32:09 -05:00
Pankaj Garg 0d117d420c
Remove BlockhashQueue dependency from SVM related code (#34974) 2024-01-26 13:46:44 -08:00
HaoranYi e38848e519
Remove unused get_append_vec_id function (#34949)
remove unused get_append_vec_id fn

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-01-26 21:41:09 +00:00
Yueh-Hsuan Chiang 3380c14044
[TieredStorage] Correct the HotStorage API for account_matches_owners (#34967)
#### Problem
In HotStorageReader, the account_matches_owners takes
&[&Pubkey] as the address candidates.  However, it should
be &[Pubkey] as defined in the accounts_file API.

#### Summary of Changes
Correct HotStorageReader::account_matches_owners() to
take &[Pubkey] instead.

#### Test Plan
Existing unit-tests
2024-01-26 09:40:05 -08:00
Yueh-Hsuan Chiang 7138f8767e
[TieredStorage] Avoid AccountHash copy in AccountMetaOptionalFields (#34969)
#### Problem
Using non-reference type of AccountHash in 
AccountMetaOptionalFields causes an unnecessary copy
as mentioned in #34948.

#### Summary of Changes
Uses &AccountHash in AccountMetaOptionalFields to
avoid copying.

#### Test Plan
Existing unit tests.

Fixes #34948
2024-01-26 09:13:09 -08:00
Yueh-Hsuan Chiang 663a1bb8f4
[TieredStorage] Write owners block for HotAccountStorage (#34927)
#### Problem
So far the current HotStorageWriter::write_accounts() only writes
accounts blocks and index block.

#### Summary of Changes
The PR further writes owners block in HotStorageWriter::write_accounts().

#### Test Plan
Extended existing test for HotStorageWriter to cover the owners block.
2024-01-26 06:48:28 -08:00
Yueh-Hsuan Chiang 51c0649af8
[TieredStorage] Use RENT_EXEMPT_RENT_EPOCH in HotStorageWriter (#34950)
#### Problem
In HotStorageWriter::write_accounts, it skips storing rent-epoch when
the rent-epoch equals Epoch::MAX.  While the value is correct, it is
more suitable to use RENT_EXEMPT_RENT_EPOCH instead as the
goal here is to save bytes for rent-exempt accounts.

#### Summary of Changes
Replace Epoch::MAX by RENT_EXEMPT_RENT_EPOCH when checking
whether to skip storing rent-epoch in HotStorageWriter.
2024-01-25 17:17:24 -08:00
Brooks f0d67d7f28
Adds hit/miss stats for cache hash data (#34954) 2024-01-25 17:27:45 -05:00
Brooks e155d9c445
Adds cache hash data deletion policy enum (#34956) 2024-01-25 16:58:56 -05:00
Yueh-Hsuan Chiang 26d62b9516
[TieredStorage] writing hot account blocks and index blocks (#34828)
#### Problem
The implementation of write_accounts() for HotAccountStorage is missing.
It consists of the writing of account blocks, index block, and owners block.

#### Summary of Changes
This PR completes part of the HotStorageWriter::write_accounts().
Specifically, it finishes the writing of account blocks and index block.

#### Test Plan
A new unit-test is added to verify the correctness of the work-in-progress
HotStorageWriter::write_accounts().
2024-01-25 09:38:06 -08:00
Brooks 5898b9a2f7
Cleans up stale accounts hash cache files (#34933) 2024-01-24 15:31:50 -05:00
Dmitri Makarov ef233eaaa7
Refactor load_accounts to take a reference to a slice instead of vec (#34919)
This is in preparation for further refactoring of
Bank::load_and_execute_transactions in a separate commit.
2024-01-24 09:43:18 -05:00
Yueh-Hsuan Chiang 1810feadc2
[TieredStorage] In-memory struct for writing OwnersBlock (#34853)
#### Problem
To write the owners-block, it requires an in-memory struct that maintains
a set of unique owner addresses while providing a look-up function to
obtain the OwnerOffset with the specified owner address. 

#### Summary of Changes
This PR adds OwnersTable, the in-memory struct that maintains
a set of unique owner addresses while providing a look-up function to
obtain the OwnerOffset with the specified owner address.

#### Test Plan
A new unit-test is added.
2024-01-23 14:57:53 -08:00
Brooks 8ff511e8fa
Moves create_and_canonicalize_directories() into accounts-db utils (#34882) 2024-01-23 06:46:27 -05:00
Brooks 2f744f1639
Moves create_all_accounts_run_and_snapshot_dirs() into accounts-db utils (#34877) 2024-01-22 18:18:43 -05:00
Brooks 3916c31cb4
Replaces fs-err in tiered storage (#34859) 2024-01-19 17:16:56 -05:00
Brooks 3eb06b4b37
Replaces fs-err in accounts-db (#34860) 2024-01-19 16:35:02 -05:00
Yueh-Hsuan Chiang 81b3210eec
[TieredStorage] Code refactoring for OwnersBlock (#34854)
#### Problem
The OwnersBlockFormat is currently defined inside footer.rs
instead of inside owners.rs.  In addition, the implementation of
OwnersBlock doesn't honor OwnersBlockFormat.

#### Summary of Changes
This PR moves OwnersBlockFormat from footer.rs to owners.rs
and repurpose OwnersBlock as OwnersBlockFormat (just like
the IndexBlockFormat inside index.rs)

#### Test Plan
Existing unit-tests.
2024-01-19 08:15:25 -08:00
steviez 3dd348802f
Bubble up genesis load errors instead of exiting (#34851)
The function open_genesis_config() performs several operations that
could fail. If any of these fail, the process exits immediately.

Instead of exiting immediately, bubble up the error and let the caller
decide the appropriate action. solana-validator and solana-ledger-tool
will functionally be unchanged, but this consolidates startup failures
for both of these processes.
2024-01-19 10:25:46 -05:00
Yueh-Hsuan Chiang 3fa44e6fbe
[TieredStorage] Include executable field into AccountMetaFlags (#34724)
#### Problem
Before we have fully switched to the new way to determine whether
an account is executable, we still need a bit for th executable flag at
this moment in the TieredStorage as well as for backward compatibility
in case we want to revert it back.

#### Summary of Changes
This PR adds the executable flag into AccountMetaFlags.

#### Test Plan
Updated existing tests for AccountMetaFlags to cover executable flag.
2024-01-16 11:25:57 -08:00
Andrew Fitzgerald 95f888ab54
Unlock only if lock_result was Ok (#34652) 2024-01-09 08:19:25 -08:00
Yueh-Hsuan Chiang 2082f8b3ca
[TieredStorage] HotStorageWriter::new() (#34659)
#### Problem
The implementation of HotAccountWriter is missing.

#### Summary of Changes
This PR kicks off the implementation of HotStorageWriter by
adding HotStorageWriter::new().

#### Test Plan
Add a new unit-test that verifies the correctness of HotStorageWriter
writing zero accounts using HotStorageReader.
2024-01-08 14:14:30 -08:00
Yueh-Hsuan Chiang 3a2f1320c5
[TieredStorage] Rename AddressAndBlockOffsetOnly to AddressesThenOffsets (#34658)
#### Problem
In the previous refactoring, IndexBlockFormat::AddressAndBlockOffsetOnly
can now work independently with the AccountMetaFormat.  As a result,
the naming of this format should be updated to reflect this.

#### Summary of Changes
As the format first persists the list of addresses followed by the list of offsets,
this PR renames AddressAndBlockOffsetOnly to AddressesThenOffsets.
2024-01-08 13:10:37 -08:00
Brooks e4ec68d704
clippy: accounts-db lints (#34628)
```
warning: this `let...else` may be rewritten with the `?` operator
  --> accounts-db/src/rolling_bit_field/iterators.rs:36:13
   |
36 | /             let Some(bit) = self.bit_range.next() else {
37 | |                 return None;
38 | |             };
   | |______________^ help: replace it with: `let bit = self.bit_range.next()?;`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#question_mark
   = note: `#[warn(clippy::question_mark)]` on by default

    Checking solana-rpc-client-api v1.18.0 (/Users/brooks/src/solana/rpc-client-api)
warning: `solana-accounts-db` (lib) generated 1 warning
```
2024-01-03 14:17:12 +00:00
Nick Frostbutter fc2a8794be
[docs] updated readme and fix links (#34565)
* feat: updated readme

* fix: updated links

* fix: proposal links

* fix: more links

* fix: json-rpc links

* fix: more links

* fix: zk links

* fix: managing forks

* fix: links for deprecated methods
2024-01-03 09:06:06 -05:00
Yueh-Hsuan Chiang d500eb6510
[TieredStorage] HotAccountsReader::get_account (#34499)
#### Problem
All structs that implement AccountsFile are required to support get_account,
but the implementation for HotAccounts is currently missing.

#### Summary of Changes
This PR implements HotAccountsReader::get_account_from_index_offset.
This will allow it to support AccountsFile::get_account API.

#### Test Plan
Add new unit-test.
2023-12-25 06:44:00 -08:00
Yueh-Hsuan Chiang 5b26d2737b
[TieredStorage] Fix typos in index.rs (#34546)
#### Problem
There're some typos in the comments and assert messages index.rs mentioned in #34529

#### Summary of Changes
Fix the typos (only in assert message and comments).
2023-12-21 22:36:38 -08:00
Yueh-Hsuan Chiang f73fd9dd3c
[TieredStorage] HotStorageReader::account_matches_owners (#34350)
#### Problem
Each AccountsFile is required to implement account_matches_owners(),
a public API that checks whether the account located at the specified
offset matches any input owners.  However, the implementation of
account_matches_owners() for HotAccountsStorage is missing.

#### Summary of Changes
This PR implements HotStorageReader::account_matches_owners().

#### Test Plan
A new unit-test is added to this PR.
2023-12-21 19:20:03 -08:00
Yueh-Hsuan Chiang a583035812
[TieredStorage] Boundary check for get_account_offset() (#34531)
#### Problem
TieredStorage doesn't perform boundary check in get_account_offset
when the input IndexOffset isn't valid.

#### Summary of Changes
This PR adds two checks.  First, it checks whether the IndexOffset exceeds
the boundary of the index block.  Second, when an index format that has the
same index entries as account entries is used, it also checks whether IndexOffset
is smaller than account_entry_count.

#### Test Plan
Two new tests are added to this PR.
2023-12-21 12:52:15 -08:00
GoodDaisy 03386cc7b9
Fix typos (#34459)
* Fix typos

* Fix typos

* fix typo
2023-12-21 13:06:00 -07:00
Yueh-Hsuan Chiang 09efd704eb
[TieredStorage] boundary check for get_account_address() (#34529)
#### Problem
get_account_address() does not check whether IndexOffset is valid.

#### Summary of Changes
This PR adds two checks.  First, it checks whether the IndexOffset exceeds
the boundary of the index block.  Second, when an index format that has the
same index entries as account entries is used, it also checks whether IndexOffset
is smaller than account_entry_count.

#### Test Plan
New unit-test is added.
2023-12-20 10:44:34 -08:00
Yueh-Hsuan Chiang eb948b1ddc
[TieredStorage] Boundary check for accessing hot account meta (#34349)
#### Problem
Hot accounts are stored in accounts blocks, whose offset is smaller than
the index block offset.  However, the current code doesn't perform
any boundary checks when accessing hot account meta.

#### Summary of Changes
Adds boundary check when accessing hot account meta.
2023-12-18 14:32:41 -08:00
Brooks e858a5a7c0
Removes unnecessary new_sized() AccountsDb ctor (#34504) 2023-12-18 16:39:57 -05:00
Brooks 4181ea4677
Removes unnecessary AccountsDb::new_with_config_for_benches() (#34484) 2023-12-18 06:52:41 -05:00
Brooks cbe8a02029
Removes unnecessary AccountsDb::new_with_config_for_tests() (#34485) 2023-12-18 00:46:46 -05:00
Joe C 171c58c5c0
RPC: Add inner instructions to simulate transaction response (#34313)
* rpc: add optional `innerInstructions: bool` arg to `simulateTransaction`

* bank: enable cpi recording in simulate

* sdk: move `InnerInstructions` into SDK from accounts DB

* bank: return inner instructions from simulate tx

* rpc: return inner instructions from simulate tx

* rpc: simulate tx: add `jsonParsed` support for inner instructions

* accounts db: add deprecated attribute to re-exported inner instructions

* rpc: de-dupe inner instruction mapping

* update deprecated comment

Co-authored-by: Tyera <teulberg@gmail.com>

---------

Co-authored-by: Tyera <teulberg@gmail.com>
2023-12-16 12:49:22 +00:00
Brooks 45eaa4c1a6
Removes unnecessary Accounts constructors (#34471) 2023-12-14 22:50:15 -05:00
Brooks e79c09bdc4
Refactors Accounts constructors (#34466) 2023-12-14 16:36:47 -05:00
Brooks c7a57672d7
Adds read/write/get_pod() fns to tiered storage (#34415) 2023-12-14 13:25:25 -05:00
Brooks d1cfd26abb
Refactors AccountsDb constructors (#34456) 2023-12-14 12:34:15 -05:00
Ashwin Sekar 337c233b5f
fix rename of HOT_ACCOUNT_ALIGNMENT (#34422) 2023-12-12 16:18:10 -05:00
Yueh-Hsuan Chiang 383aa0451e
[TieredStorage] Unit-tests for checking invalid HotAccountOffset (#34376)
#### Problem
HotAccountOffset::new() might return Err for invalid offset, and this
part needs some test coverage.

#### Summary of Changes
Add unit-tests for checking invalid HotAccountOffset.
2023-12-12 12:31:16 -08:00
Brooks a2d7be0d07
Derives Pod for tiered storage types (#34414) 2023-12-12 14:59:51 -05:00
Yueh-Hsuan Chiang 376b1430e7
[TieredStorage] Improve comments for HOT_ACCOUNT_ALIGNMENT (#34404)
#### Problem
The current naming and the code comments for HOT_ACCOUNT_OFFSET_ALIGNMENT
aren't really reflecting its role as pointed out in #34335.

#### Summary of Changes
This PR renames HOT_ACCOUNT_OFFSET_ALIGNMENT to HOT_ACCOUNT_ALIGNMENT
as it's the hot account instead of hot account offset needs to be aligned.

In addition, improve the comment block for HOT_ACCOUNT_ALIGNMENT.
2023-12-12 11:48:29 -08:00
Yueh-Hsuan Chiang 39a3566438
[TieredStorage] Make AccountOffset a trait, introduce HotAccountOffset (#34335)
#### Problem
Hot and cold accounts storage have different implementations of
their offsets.  As a result, a single struct AccountOffset isn't suitable
to describe the offsets used by hot and cold accounts storage.

#### Summary of Changes
This PR makes AccountOffset a trait.  On top of that, introduces
HotAccountOffset that implements AccountOffset.

#### Test Plan
Updated existing unit-tests.
2023-12-11 15:38:38 -08:00
HaoranYi aab12c31c7
tune ancient append vec size to 128M (#34067)
* tune ancient append vec size to 130M

* fix a test and get rid of the assert since it is covered in the test

* use 128M

* assert max append vec size for ancient append vec

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-12-11 15:45:35 -06:00
HaoranYi c43edd2fef
use seqlock for AccountStorage count and status tracking (#34356)
use seqlock for appendvec count and status tracking

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-12-08 08:49:54 -06:00
Brooks b32903a534
Removes glob import of AccountsHashVerificationError (#34332) 2023-12-06 13:46:00 -05:00
Brooks 5c00a33252
Removes unnecessary #[allow(dead_code)] in tiered storage (#34320) 2023-12-06 00:23:01 +00:00
Brooks c971ad2944
Removes unnecessary #[allow(dead_code)] in partitioned rewards (#34321) 2023-12-05 15:58:11 +00:00
Jeff Washington (jwash) 091fc820ba
ancient append vec append creates big enough ancient append vecs (#34154)
* ancient append vec append creates big enough ancient append vecs

* assert we write the correct # of bytes

* fix tests

* u64 align accounts bytes written

* fix build

* add tests

* we don't need to align the store to u64.

When we call `remaining_bytes` to calculate, for the last element, u64_align!(len) could be greater than capacity.  However, because saturate_sub is used, we still get the correct answer of 0. Therefore, no need to align the storage to u64.

* add comments

* add shink test for min_size

* remove dead code

* fix build

* reviews

* add test for squash ancient vec overflow too much to fit into ideal av

* add test for get_bytes on AccountToStore

* clippy

* modify test to assert 0 size of opposite

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-12-05 09:25:11 -06:00
Brooks 2c033e05e5
Switches order of tiered storage footer size and version (#34310) 2023-12-04 14:23:56 -05:00
Brooks 4f65f7dc8d
Sanitizes tiered storage footer after reading from disk (#34200) 2023-12-01 13:58:10 -05:00
Yueh-Hsuan Chiang fa0930f254
[TieredStorage] Allow HotStorage to handle more account data (#34155)
#### Problem
After we have defined AccountOffset to be u32, it means the address space within
one HotAccountsFile is up to 4GB.  However, since the Accounts Blocks in a
HotAccountsFile is 8-byte aligned, it has the opportunity to store more data by
internally multiplying the AccountOffset by 8.

#### Summary of Changes
This PR allows a HotAccountsFile to store up to 32GB accounts data by using
8 x AccountOffset as its actual offset.

#### Test Plan
Updated existing unit-tests.
2023-12-01 10:39:15 -08:00
Andrew Fitzgerald 2294801954
Do not derive Copy for EpochSchedule and Rent (#32767) 2023-12-01 07:57:25 -08:00
Jeff Washington (jwash) 32c7acbf74
add slot to assert message (#34225) 2023-11-28 10:29:39 -06:00
Brooks deee5f833c
Uses Display for error in VerifyAccountsHashInBackground's panic message (#34219) 2023-11-27 09:51:39 -05:00
Yueh-Hsuan Chiang ecc067f7ad
[TieredStorage] Make AccountOffset use u32 (#34151)
#### Problem
AccountOffset currently uses `usize`, which size is platform dependent.
We want a fixed size type that is consist to what we persist in the tiered-storage file.

#### Summary of Changes
This PR makes AccountOffset use u32.
2023-11-21 10:40:10 -08:00
Yueh-Hsuan Chiang db2444bd11
Move MatchAccountOwnerError from append_vec to accounts_file (#34187)
#### Problem
MatchAccountOwnerError currently belongs to append_vec.

However, it is a public error type that is also required by other AccountsFile
implementations such as TieredStorageFile.

#### Summary of Changes
Move MatchAccountOwnerError from append_vec to accounts_file.
2023-11-21 10:39:56 -08:00
Jeff Washington (jwash) 481c357543
remaining_bytes aligns len since all writes will align first (#34171)
* remaining_bytes aligns len since all writes will align first

* use remaining_bytes() to check for whether the new account can fit into the av storage

* Add test coverage for av remaining bytes alignment

* use great equal to check space available

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-11-21 09:34:52 -06:00
HaoranYi 90b11a608b
refactor AccountsToStore (#34185)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-11-21 09:27:31 -06:00
Yueh-Hsuan Chiang c73f226958
[TieredStorage] Make IndexOffset use u32 (#34152)
#### Problem
IndexOffset currently uses `usize`, which size is platform dependent.
We want a fixed size type that is consist to what we persist in the tiered-storage file.

#### Summary of Changes
This PR makes IndexOffset use u32.
2023-11-20 14:57:15 -08:00
Lucas Steuernagel a3670054be
Move `RewardInterval` to runtime (#34146)
Move RewardInterval to runtime
2023-11-20 10:17:21 -03:00
Brooks e02f25d5a2
Removes filler accounts (#34115) 2023-11-19 20:36:57 -05:00
Jeff Washington (jwash) 202463849a
log when appending exceeds available space (#34170) 2023-11-19 18:50:37 -06:00
Brooks 574b8b5bfc
Refactor assert in scan_snapshot_stores() (#34162) 2023-11-18 12:05:39 -05:00
Brooks 8c8cd66095
Uses Display in TieredStorageFile::new_readonly() panic message (#34148) 2023-11-17 15:24:05 -05:00
Yueh-Hsuan Chiang 625e898c63
[TieredStorage] Make IndexBlock persist u32 offsets (#34133)
#### Problem
In TieredStorage, we want to make AccountOffset and IndexOffset u32 instead
of usize just like OwnerOffset.  However, we need to first change what we persist
in the storage.  Currently, IndexBlock persists offsets as u64 instead of u32. 

#### Summary of Changes
This PR makes IndexBlock persist u32 offsets.

#### Test Plan
Existing test cases.
2023-11-17 11:35:32 -08:00
Brooks e23bfb5167
Refactors TieredStorageFile read/write methods (#34147) 2023-11-17 14:17:49 -05:00
Yueh-Hsuan Chiang 1e9f6d9d83
[TieredStorage] Use OwnerOffset type in TieredAccountMeta (#34106)
#### Problem
TieredAccountMeta currently uses `u32` for owner_offset, while
we already have a dedicate type for that --- OwnerOffset.

#### Summary of Changes
This PR makes TieredAccountMeta use OwnerOffset for its
owner_offset field.

#### Test Plan
Existing unit-tests.
2023-11-17 09:26:52 -08:00
Brooks 07b0b9f9e9
Uses `mmap` for names in tiered storage (#34127) 2023-11-16 23:59:08 +00:00
Lucas Steuernagel 2c71d21fad
Move load_accounts to runtime (#34017)
* Move load_accounts to runtime

---------

Signed-off-by: Lucas Steuernagel <lucas.tnagel@gmail.com>
2023-11-16 18:45:58 -03:00
Brooks 7e3b09defe
Uses consistent error types in tiered storage (#34110) 2023-11-16 15:51:57 -05:00
Yueh-Hsuan Chiang f59887077a
[TieredStorage] Define OwnerOffset as u32 (#34105)
#### Problem
The current OwnerOffset is defined as usize, which actual size could be u32 or u64.
On the other hand, the OwnerOffset is used to access the ith owner in one
TieredStorageFile.  As a result, u32 is more than enough to represent the number of
owners inside one TieredStorageFile while u16 might not be always enough.

#### Summary of Changes
This PR defines OwnerOffset as u32.

#### Test Plan
Existing test cases.
2023-11-16 12:48:33 -08:00
HaoranYi 60fdd85aed
Add a feature to disable rent collection (#33945)
* add a feature to disable rent collection

* fix a test

* fix a test

* rekey

* should collect rent

* Update runtime/src/bank/fee_distribution.rs

Co-authored-by: Brooks <brooks@prumo.org>

* expand tests to cover both rent collection disabled and enabled

* feedbacks

* reviews - move should collect rent check out of rent collector into bank

* enforce rent_epoch to u64:max when rent collection is disabled

* review feedbacks and fix a test
When rent fee collection is disabled, we won't collect rent for any account. If there are any rent paying accounts, their `rent_epoch` won't change too.

* revise comments

* update rent_epoch for rent exempted account

* rebase

* set rent_epoch in rent collection for rent exempted account

* revert test change

* don't assert

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
Co-authored-by: Brooks <brooks@prumo.org>
2023-11-16 08:57:49 -06:00
Yueh-Hsuan Chiang b7f839ea18
[TieredStorage] HotStorageReader::get_owner_address (#34053)
#### Problem
The HotStorageReader does not have an API to obtain owners_address.

#### Summary of Changes
This PR adds HotStorageReader::get_owner_address().

#### Test Plan
A new unit-test is added to this PR.
2023-11-15 15:54:53 -08:00
Yueh-Hsuan Chiang d58db6e255
Improve formatting in test_hot_storage_get_account_offset_and_address() (#34071)
#### Problem
#34032 mentioned nice-to-have a formatting improvement that
can improve the readability. 

#### Summary of Changes
Improve the code formatting in test_hot_storage_get_account_offset_and_address().
2023-11-15 11:25:18 -08:00
Brooks 67dab71e82
Uses `RENT_EXEMPT_RENT_EPOCH` to mark account as rent exempt (#34090) 2023-11-15 18:13:18 +00:00
Alexander Meißner ffa462dd3a
Cleanup - Feature gate of `delay_visibility_of_program_deployment` (#34087)
* Cleans up feature gate of delay_visibility_of_program_deployment.

* Removes programs_updated_only_for_global_cache.
2023-11-15 18:29:31 +01:00
Alexander Meißner 48ed70a8d8
Cleanup - Feature gate of `disable_builtin_loader_ownership_chains` (#34083)
* Cleans up feature gate of disable_builtin_loader_ownership_chains.
2023-11-15 17:17:31 +01:00
Yueh-Hsuan Chiang 7fd13c0cd2
[TieredStorage] OwnersBlock (#34052)
#### Problem
A TieredStorage file has three types of block:  accounts block, index block,
and owner block, but implementation of the TieredStorage OwnersBlock
is missing in the current master.

#### Summary of Changes
This PR implements OwnersBlock which stores a set of owners' addresses
in a compact but efficient way.

#### Test Plan
A new unit-test is included in this PR.
2023-11-15 01:51:13 -08:00
Yueh-Hsuan Chiang eb7e68f029
[TieredStorage] HotStorageReader::get_account_address (#34032)
#### Problem
HotStorageReader currently not yet has an API to obtain account_address

#### Summary of Changes
This PR adds HotStorageReader::get_account_address() which returns
the Pubkey of the account associated with the specified IndexOffset.

#### Test Plan
Augmented an existing unit-test to cover get_account_address() case.
2023-11-14 20:44:03 -08:00
Yueh-Hsuan Chiang 2877b8182c
[TieredStorage] Rename owners_offset to owners_block_offset (#34047)
#### Problem
TieredStorageFooter::owners_offset refers to the offset to the
owners block, which looks very similar to OwnerOffset that
has a completely different concept -- the offset to access the
address of an owner in the owners-block.

#### Summary of Changes
Similar to the previous renaming index_offset to index_block_offset,
this PR renames owners_offset to owners_block_offset to
avoid confusion.
2023-11-13 18:42:51 -08:00
Brooks 725ab37bf4
clippy: Replaces .get(0) with .first() (#34048) 2023-11-13 17:22:17 -05:00
Jeff Washington (jwash) a25eae41fc
stop page aligning shrunk append vecs (#34016)
* stop page aligning shrunk append vecs

* fix build

* fix test by removing aligned alive page bytes asserts

* rm dup assert

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-11-13 14:22:26 -06:00
Yueh-Hsuan Chiang 2b9054a60e
[TieredStorage] HotStorageReader::get_account_offset (#34031)
#### Problem
HotStorageReader currently not yet has an API to obtain account_offset

#### Summary of Changes
This PR adds HotStorageReader::get_account_offset() which takes
IndexOffset and returns AccountOffset.

#### Test Plan
A new unit-test is included in this PR.
2023-11-13 12:07:09 -08:00
Yueh-Hsuan Chiang 3db7eaf1fc
[TieredStorage] Improve param naming of IndexBlockFormat (#34033)
#### Problem
In IndexBlockFormat, both `IndexOffset` and `AccountOffset` parameters
are named `offset` in some functions which could be confusing.

#### Summary of Changes
Renamed `offset` to `index_offset` and `account_offset` to improve
readability.
2023-11-13 12:05:25 -08:00
Brooks 3c71f859e1
Uses fold+reduce for handling duplicate pubkeys during index generation (#34011) 2023-11-10 14:32:24 -05:00
Brooks 69ab8a8234
Uses IntSet for uncleaned slots from visit_duplicate_pubkeys_during_startup() (#34009) 2023-11-10 11:29:13 -05:00
Pankaj Garg 59eb55990c
Move filter_executable_program_accounts to bank.rs (#34004) 2023-11-09 14:40:04 -08:00
Brooks 9f25f67e60
Simplifies adding uncleaned pubkeys during index generation (#34007) 2023-11-09 16:06:15 -05:00
Brooks a96be5d2f0
Uses IntSet for uncleaned roots during index generation (#34008) 2023-11-09 15:04:19 -05:00
Yueh-Hsuan Chiang 874fae507f
[TieredStorage] Make HotStorageReader use AccountOffset type (#33964)
#### Problem
#33927 introduced a new type AccountOffset, but HotStorageReader
still uses `usize` to access accounts.

#### Summary of Changes
This PR makes HotStorageReader use the new AccountOffset type.
2023-11-08 21:58:44 -08:00
Brooks bba6ea2d69
Returns IntMap from select_candidates_by_total_usage() (#33976) 2023-11-08 16:15:51 -05:00
Brooks 7cb83bc491
Adds `S` to HashMap/HashSet impls of Contains (#33973) 2023-11-08 15:08:33 -05:00
Brooks 6f213c38aa
Removes Default from RollingBitField (#33969) 2023-11-07 15:58:52 -05:00
Yueh-Hsuan Chiang b013c03afa
[TieredStorage] Add IndexOffset type (#33929)
#### Problem
TieredStorage conceptually has different offsets. However, the current code directly
uses the same primitive type for accessing offsets, which is error-prone as one could
easily use one offset to access data that should be accessed with a different offset
type.

#### Summary of Changes
This PR adds IndexOffset type -- a struct for obtaining the ith entry inside the
index-block to obtain account's offset and address.
2023-11-07 10:26:21 -08:00
Jeff Washington (jwash) b8115b4303
chunk all ancient append vecs (#33909)
* chunk all ancient append vecs

* fix a test comments

* remove unneeded dead_code attr

* do full chunking when pack is used to create ancient storage

* refacotr and fix tests

* clippy

* add cache hash file stats

* comments

* fix test

* Update accounts-db/src/accounts_db.rs

Co-authored-by: Brooks <brooks@prumo.org>

* Update accounts-db/src/accounts_db.rs

Co-authored-by: Brooks <brooks@prumo.org>

* test_case

* remove commented out code

* remove hash cache data stats

* typo

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
Co-authored-by: HaoranYi <haoran.yi@gmail.com>
Co-authored-by: Brooks <brooks@prumo.org>
2023-11-07 09:10:59 -06:00
Lijun Wang 8c69a0ec38
Remove RwLock on AccountsUpdateNotifier (#33960) 2023-11-06 14:03:25 -08:00
Yueh-Hsuan Chiang da130b87d3
[TieredStorage] Add AccountOffset type (#33927)
#### Problem
TieredStorage conceptually has different offsets.  However, the current code directly
uses the same primitive type for accessing offsets, which is error-prone as one could
easily use one offset to access data that should be accessed with a different offset
type.

#### Summary of Changes
This PR introduces the AccountOffset type, which allows static-check to on different
type of TieredStorage offsets.
2023-11-06 12:21:08 -08:00
Brooks 70d97d3261
Adds `iter_ones()` to RollingBitField (#33956) 2023-11-06 14:06:43 -05:00
Yueh-Hsuan Chiang 6624a09d38
[TieredStorage] Rename account-index to index-block (#33928)
#### Problem
The current tiered-storage code uses "account-index" to call index-block.
This could lead to confusion especially as we start giving each offset/position/index a specific type.

#### Summary of Changes
This PR renames all structs/variables that use account-index to refer to index-block.
2023-11-06 10:32:19 -08:00
Brooks cdc284189a
Refactors RollingBitField::min() (#33911) 2023-10-28 12:47:29 -04:00
Brooks 1814b2bc81
Adds logs for starting/stopping of the background account hasher (#33903) 2023-10-27 13:03:33 -04:00
Jeff Washington (jwash) a18debc34a
allow test feature to skip rewrites (#33851)
* allow test feature to skip rewrites

* hook up cli arg for test skip rewrites, update tests

* fix sanity checker

* add account hash to abi to fix a test

* reviews

* use hashmap to collect skip_rewrites. exclude skip_rewrites from dirty
pubkey set

* accumulate skipped_rewrite in reduce

* mutex

* fmt

* skip hash verify for this test flag

* add skipped rewrites num stat

* skip bank hash verify not account hash verify

* reviews

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-27 09:14:05 -05:00
Tao Zhu 510b6b949f
Split compute budget instructions process from struct (#33852)
* Split compute budget instruction processing from ComputeBudget struct itself, so CB instructions can be processed elsewhere without involving ComputeBudget

* updated tests

* avoid built ComputeBudget from dated ComputeBudgetLimits in this refactoring PR

* Clean-up program-runtime/src/compute_budget_processor.rs

* Add test for a corner case that deprecated instruction is used to request units greater than max limit;
* Update code to handle the corner case.
2023-10-26 22:12:56 -05:00
Brooks bd1080b26f
Adds AtomicAge to bucket map holder (#33841) 2023-10-25 10:47:21 -04:00
Kevin Heavey af7fd32f4c
chore: fix some typos (#33833)
* fix spelling of "retrieved"
* fix spelling of "should"
* fix spelling of "comparisons"
2023-10-25 16:21:53 +02:00
Brooks 612e8e8457
Removes Default and pub from CumulativeOffset (#33840) 2023-10-24 19:22:34 +00:00
Brooks abf51864b4
Removes `#[allow(dead_code)]` in StorableAccounts impl (#33821) 2023-10-24 18:31:27 +00:00
HaoranYi 9064b8b2a2
Optimize account hash CumulativeOffset index from vec to 2-element array (#33839)
* optimize account hash merkle tree index from vec to 2-element array

* typo

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-24 18:26:16 +00:00
Brooks 2f024f0cc1
Uses AccountHash in CalculateHashIntermediate (#33822) 2023-10-24 08:39:32 -04:00
Jeff Washington (jwash) b0b4e1f0c0
remove IncludeSlotInHash after feature activation on mnb (#33816)
* remove IncludeSlotInHash after feature activation on mnb

* fix compile errors

* compile errors

* fix tests

* fix test results
2023-10-23 15:12:02 -07:00
Jeff Washington (jwash) 54b796f5a1
ancient pack: add low water mark (#33785) 2023-10-23 10:56:18 -07:00
HaoranYi 669bc43bcd
Buffer account's fields for hash (#33788)
* buffer accounts field for hash

* use smallvec to allocate hash buffer on stack

* sort deps

* more opt

* clippy

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-23 09:48:17 -05:00
Jeff Washington (jwash) a41b24f185
fix some ancient stats (#33791) 2023-10-23 07:23:47 -07:00
Jeff Washington (jwash) 34103e0913
typo in variable name (#33801) 2023-10-23 07:23:11 -07:00
Jeff Washington (jwash) e13756133a
ancient: add many_refs_this_is_newest_alive (#33741)
* add many_refs_this_is_newest_alive

* fix pathological case, add comments

* add log

* update comments

* add log when we fail to pack

* use with_capacity

* fix log comment

* clippy
2023-10-20 09:55:37 -07:00
steviez c98c24bd6d
Revert "Split compute budget instructions process from struct itself … (#33784)
Revert "Split compute budget instructions process from struct itself (#33513)"

This reverts commit c73bebe984. This
was found to be a consensus breaking change.
2023-10-20 15:53:45 +02:00
Brooks ce8ad77373
Uses AccountHash in AppendVec (#33764) 2023-10-19 17:38:09 -04:00
Jeff Washington (jwash) f13c78b7c8
add ancient append vec test (#33762)
* add ancient append vec test

* Update accounts-db/src/ancient_append_vecs.rs

Co-authored-by: Brooks <brooks@prumo.org>

* add some comments

---------

Co-authored-by: Brooks <brooks@prumo.org>
2023-10-19 09:50:38 -07:00
Tao Zhu c73bebe984
Split compute budget instructions process from struct itself (#33513)
* Split compute budget instruction processing from ComputeBudget struct itself, allow compute_budget_instructions be processed elsewhere without having to instantiate ComputeBudget

* updated tests
2023-10-19 11:10:42 -05:00
Brooks 4e5c545e23
Uses AccountHash in tiered storage (#33763) 2023-10-19 11:52:36 -04:00
Brooks 0fcc0a0941
Returns AccountHash from get_filler_account() (#33758) 2023-10-18 20:29:40 -04:00
Brooks 1045548606
Uses AccountHash in StorableAccountsWithHashesAndWriteVersions (#33751) 2023-10-18 18:58:19 -04:00
Brooks 3a580f4df6
StorableAccounts::hash() returns &AccountHash (#33748) 2023-10-18 15:41:26 -04:00
Yueh-Hsuan Chiang 94273434ce
[TieredStorage] HotStorageReader::get_account_meta_from_offset() (#33724)
#### Problem
HotStorageReader currently only implements get_footer().  It does not
have a function to obtain the account meta.

#### Summary of Changes
This PR implements HotStorageReader::get_account_meta_from_offset().
A function that returns the account meta located at the specified offset.
This will be the helper function that will be later used to obtain the account
meta when the offset is available from the index block of a hot storage file. 

#### Test Plan
A new test is included in this PR.
2023-10-18 12:27:39 -07:00
Brooks e96678b302
Uses SeqLock for CachedAccountInner::hash (#33696) 2023-10-18 10:43:35 -04:00
Yueh-Hsuan Chiang 0b05e8db11
[TieredStorage] Footer test for HotStorageReader (#33718)
#### Problem
HotStorageReader currently doesn't have a test that covers its footer.

#### Summary of Changes
This PR includes a test for HotStorageReader that verifies the footer.
2023-10-17 15:24:38 -07:00
Jeff Washington (jwash) c09cbbb778
sort ancient append vec target_slots_sorted (#33729) 2023-10-17 14:49:46 -07:00
Jeff Washington (jwash) 56a7485998
log ancient stats each time we run (#33730) 2023-10-17 14:41:23 -07:00
HaoranYi 673a38c892
move timer after early exit (#33732)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-17 15:20:29 -05:00
HaoranYi f178975185
typo (#33734)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-17 14:45:17 -05:00
Jeff Washington (jwash) 5de9163625
add metric for ancient can't move slots (#33713)
* add metric for ancient can't move slots

* rename

* fix erors in replacing text

* rename
2023-10-17 08:34:18 -07:00
Jeff Washington (jwash) d948e5bf69
ancient shrink on its own cadence (#33712) 2023-10-16 10:06:20 -07:00
HaoranYi 167dac204f
Retry hash file allocation (#33565)
* retry hash file allocation

* add sleep

* submit a datapoint for retry

* typo

* more typos

* Update accounts-db/src/accounts_hash.rs

Co-authored-by: Brooks <brooks@prumo.org>

* fmt

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
Co-authored-by: Brooks <brooks@prumo.org>
2023-10-16 09:21:08 -05:00
Jeff Washington (jwash) c354879a3f
stop padding new append vecs to page size (#33658)
* stop padding new append vecs to page size

* fix test

* fix another test

* for creating test accounts, allocate larger like we used to
2023-10-12 07:11:20 -07:00
HaoranYi 1a2c7f106e
fix typo in comments (#33665)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-12 09:05:10 -05:00
steviez 982d29cf60
Revert "stop padding new append vecs to page size (#33607)" (#33634)
This reverts commit b7962a3610.
2023-10-10 14:43:52 -05:00
Alexander Meißner cb695c7b32
Cleanup - Feature Gate of `enable_early_verification_of_account_modifications` (#33579)
Cleans up the feature gate of enable_early_verification_of_account_modifications:
- Removes PreAccount
- Removes InvokeContext::pre_accounts and InvokeContext::rent
- Removes InvokeContext::verify() and InvokeContext::verify_and_update()
- Removes TransactionContext::is_early_verification_of_account_modifications_enabled()
- Removes TransactionAccounts::is_early_verification_of_account_modifications_enabled
- No longer optional: TransactionContext::rent
2023-10-10 17:26:17 +02:00
Jeff Washington (jwash) b7962a3610
stop padding new append vecs to page size (#33607)
* stop padding new append vecs to page size

* for creating test accounts, allocate larger like we used to
2023-10-10 07:03:37 -07:00
Jeff Washington (jwash) 6c7d3c8aeb
remove redundant page_align (#33608) 2023-10-10 07:02:47 -07:00
Brooks fc73813db2
Adds AccountHash newtype (#33597) 2023-10-09 16:00:52 -04:00
Jeff Washington (jwash) 052677595c
in hash calc, delete old cache files that will not be used earlier (#33432)
* in hash calc, delete old cache files that will not be used earlier

* only delete if supposed to

* fmt
2023-10-09 11:47:39 -07:00
Brooks c924719040
Removes hash param from AccountsCache::store() (#33598) 2023-10-09 18:16:15 +00:00