Commit Graph

558 Commits

Author SHA1 Message Date
Brooks d81ace8ff7
Removes AppendVec::file_name() (#976) 2024-04-24 10:09:04 -04:00
Jeff Washington (jwash) 45c49cedf2
get_unique_accounts_from_storage uses scan_index (#969)
* get_unique_accounts_from_storage uses scan_index

* avoid returning default pubkey=default accounts
2024-04-24 08:56:14 -05:00
Jeff Washington (jwash) 16f1f25dbb
cache storage lookup during shrink and ancient (#973)
* cache storage lookup during shrink and ancient

* rename and remove pub

* pr feedback
2024-04-23 11:51:37 -05:00
Jeff Washington (jwash) ca8f92e001
fix references on get_stored_account_meta_callback (#996)
* fix references on get_stored_account_meta_callback

* fmt
2024-04-23 11:51:03 -05:00
Jeff Washington (jwash) 57ab9449c7
remove multi slot test only StorableAccounts impl (#972) 2024-04-23 08:35:45 -05:00
Jeff Washington (jwash) 862c79e132
modify geyser account iter at snapshot load (#960) 2024-04-22 13:06:25 -05:00
Jeff Washington (jwash) 7389371112
rework secondary index generation (#959) 2024-04-22 12:59:47 -05:00
Jeff Washington (jwash) d4b86923ef
introduce AccountFromStorage for shrink (#881)
* introduce AccountFromStorage for shrink

* add comments and cleanup

* more comments

* unwrap to expect

* add comments
2024-04-22 12:20:18 -05:00
Brooks 422387982f
Removes unnecessary &Arc (#958) 2024-04-22 11:46:40 -04:00
Jeff Washington (jwash) 6239ece50d
remove unused StorableAccounts (#956) 2024-04-22 09:54:24 -05:00
Jeff Washington (jwash) 127faa3a07
shrinking and appending ancients use StorableAccountsBySlot (#935)
* shrinking and appending ancients use StorableAccountsBySlot

* snapshot minimizer uses StorableAccountsBySlot
2024-04-22 08:28:36 -05:00
Jeff Washington (jwash) 14d6c79c78
remove unused StorableAccountsMovingSlots (#952) 2024-04-22 05:05:05 -05:00
Jeff Washington (jwash) 8d4f2e6221
use scan_accounts to replace account_iter (#921) 2024-04-21 20:07:05 -05:00
Jeff Washington (jwash) 43e47bea28
remove a StorableAccounts impl (#936) 2024-04-21 20:03:55 -05:00
Alessandro Decina c2936ebb4a
accounts-db: send batches of accounts to the background hasher (#810)
Instead of sending accounts individually, send batches of accounts for
background hashing.

Before this change we used to send accounts individually, and the
background thread used to do:

    loop {
        let account = receiver.recv();
        account.hash();
        // go back to sleep in recv()
    }

Because most accounts are small and hashing them is very fast, the
background thread used to sleep a lot, and required many syscalls from
the sender in order to be woken up.

Batching reduces the number of syscalls.
2024-04-20 08:19:49 +10:00
Jeff Washington (jwash) fc2b05d50d
fix a few storage iter lifetimes (#917) 2024-04-19 17:10:43 -05:00
Brooks bbc0be2cb3
Removes returned Result from infallible fns (#914) 2024-04-19 20:44:15 +00:00
Jeff Washington (jwash) 1189055c98
load account lifetime improvements (#903)
* load account lifetime improvements

* restore 2 check_and_
2024-04-19 15:33:29 -05:00
Jeff Washington (jwash) e3d3cd6486
add appendvec.scan_accounts (#902) 2024-04-19 11:38:31 -05:00
Brooks 246dc481ca
Removes check_hash from CalcAccountsHashConfig (#913) 2024-04-19 15:43:04 +00:00
Jeff Washington (jwash) fb3bf55340
accounts_iter -> scan_pubkeys (#894) 2024-04-19 09:36:53 -05:00
Jeff Washington (jwash) f8a8d40ac7
test fixes to remove refs (#901) 2024-04-19 09:36:27 -05:00
HaoranYi 992a398fe8
accounts db/refactor accounts db test - test_partial_clean (#818)
* add hot storage test for test_partial_clean

* fix test_partial_clean for hot storage

* reviews

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-19 08:48:06 -05:00
Jeff Washington (jwash) f0e87f6210
replace one accounts iter with scan_pubkeys (#888) 2024-04-18 20:43:09 -05:00
Brooks 86667f5a6f
Removes check_hash from scan_snapshot_stores_with_cache() (#892) 2024-04-18 23:24:04 +00:00
Brooks 16770cce88
Removes tests that use check_hash (#893) 2024-04-18 22:32:03 +00:00
Brooks 4b9d18a361
Removes check_hash from ScanState (#891) 2024-04-18 22:05:10 +00:00
Brooks 54b816a338
Removes `check_hash` from calculate_accounts_hash_from_index() (#890) 2024-04-18 18:01:53 -04:00
Brooks ee113de4ce
Refactors stored accounts info (#887) 2024-04-18 15:45:44 -04:00
Alexander Meißner e43338f3d6
Refactor - Rename `LoadedProgram` to `ProgramCacheEntry` (#880)
LoadedProgramOwner => ProgramCacheEntryOwner.
LoadedProgramType => ProgramCacheEntryType.
LoadedProgram => ProgramCacheEntry.
LoadedProgramStats => ProgramCacheStats.
LoadedProgramMatchCriteria => ProgramCacheMatchCriteria.
LoadedProgramsForTxBatch => ProgramCacheForTxBatch.
2024-04-18 21:38:08 +02:00
Jeff Washington (jwash) 994ab254a2
some test tweaks (#882)
* some test tweaks

* get rid of some redundancy
2024-04-18 14:01:02 -05:00
Jeff Washington (jwash) d77118ac65
tests had invalid storage offset (#874) 2024-04-18 12:09:14 -05:00
HaoranYi 033bcfcab5
accounts db/refactor accounts db test - convert test_full_clean_refcount (#814)
* add full_clean_refcount tests for both account storage formats

* keep code comments

* fix test to match with code comments

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-17 16:42:44 -05:00
Brooks da0d78c504
Removes hash() and has_hash() from StorableAccounts (#867) 2024-04-17 16:41:41 -04:00
Brooks ee2c29ca2f
Removes unused hashes params from store functions (#863) 2024-04-17 16:37:45 -04:00
Brooks 613256e18f
Removes StorableAccountsWithHashes (#862) 2024-04-17 18:52:45 +00:00
HaoranYi 283f433a3b
accounts db/refactor accounts db test - add macro for accounts-db panic test (#786)
* add macro for accounts-db panic test
convert double remove test for both account file provider

* refactor to share accounts_db_test and accounts_db_panic_test macro

* rebase

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-17 12:52:00 -05:00
Brooks ad5b71dda8
Replaces StorableAccountsWithHashes in AccountsDb (#861) 2024-04-17 17:31:44 +00:00
Brooks 349eb7f925
Replaces StorableAccountsWithHashes in AccountsFile (#860) 2024-04-17 16:02:48 +00:00
Brooks 2aeac5b468
Replaces StorableAccountsWithHashes in TieredStorage (#859) 2024-04-17 14:49:24 +00:00
Brooks 97013978fe
Uses same lifetime name for local AccountForStorage in callback (#848) 2024-04-17 09:39:44 -04:00
Brooks 4d2d95fe14
Replaces StorableAccountsWithHashes in AppendVec (#853) 2024-04-17 09:34:03 -04:00
Jeff Washington (jwash) 63d4278cae
get_stored_account_meta_callback returns Option (#847)
* get_stored_account_meta_callback returns Option

* Update accounts-db/src/accounts_file.rs

Co-authored-by: Brooks <brooks@prumo.org>

---------

Co-authored-by: Brooks <brooks@prumo.org>
2024-04-17 08:12:32 -05:00
HaoranYi 03b596586c
Write default hash to account storage (#469)
* write default hash to apendvec

* fix get_pubkey_hash_for_slot

* fix tests

* fix rebase error

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-16 16:04:52 -05:00
Jeff Washington (jwash) a9548a6dc5
add `for<>` to callback on get fns (#838) 2024-04-16 14:55:50 -05:00
Jeff Washington (jwash) 0976e9b222
add get_stored_account_meta_callback to storage (#836) 2024-04-16 13:58:42 -05:00
HaoranYi c5a669ea34
accounts db/refactor accounts db test - fix account storage count and alive bytes tests for hot storage (#784)
fix account storage count and alive bytes tests for hot storage

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-16 12:11:44 -05:00
Jeff Washington (jwash) 877591ea7d
remove StorableAccounts.pubkey() (#829)
remove `StorableAccounts`.`pubkey()`
2024-04-16 11:39:19 -05:00
Jeff Washington (jwash) cb2fd2b632
AccountForStorage supports pubkey() (#825)
* account_default_if_zero_lamport always returns a value

* rename
2024-04-16 10:06:33 -05:00
Brooks 731d36ab26
Fixes pedantic clippy ptr cast lints in append_vec.rs (#650) 2024-04-16 13:41:00 +00:00
Jeff Washington (jwash) c75fe56354
account_default_if_zero_lamport always returns a value (#822) 2024-04-16 08:11:46 -05:00
Brooks ec6f525ea6
Fixes pedantic clippy ptr cast lints in tiered storage (#651) 2024-04-16 08:51:38 -04:00
Jeff Washington (jwash) 7f16e84237
hot storage tests don't use Option (#821) 2024-04-15 19:17:02 -05:00
Jeff Washington (jwash) ee5fb5102d
use callback for account and account_default_if_zero_lamport (#819)
use callback for `account` and `account_default_if_zero_lamport`
2024-04-15 19:09:57 -05:00
Brooks f16b5cf1e6
Marks append vec test utils as DCOU (#820) 2024-04-15 18:52:56 -04:00
HaoranYi d721f76084
accounts db/refactor accounts db test - Update accounts-db test to cover different account file providers. (#777)
* add define_accounts_db_test macro and refactor tests of accounts db for different AccountFileProvider

* use test macro for test_alive_bytes

* implement review feedbacks

* typo

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-15 17:40:20 -05:00
Jeff Washington (jwash) 9217cd476a
StorableAccountsWithHashes uses callback for lifetime (#813)
* StorableAccountsWithHashes uses callback for lifetime

* cleanup trait type
2024-04-15 15:47:00 -05:00
Brooks 9ba0d6fabb
Fixes race in RwLockSecondaryIndexEntry::insert_if_not_exists() (#815) 2024-04-15 14:52:20 -04:00
Jeff Washington (jwash) 6bd0bb611c
introduce AccountForStorage for StorableAccounts (#795)
introduce `AccountForStorage` for `StorableAccounts`
2024-04-15 13:21:07 -05:00
Jeff Washington (jwash) aed1a5e452
AccountAddressRange is byval (#807) 2024-04-15 10:09:10 -05:00
Jeff Washington (jwash) e0b0bcc803
hot storage AccountIndexWriterEntry needs values (#808) 2024-04-15 09:43:52 -05:00
Brooks 2ffc7f64b7
Renames fns to get_account_shared_data() (#782) 2024-04-14 18:25:39 -04:00
Jeff Washington (jwash) 970d925a2e
hot owners accumulation needs to be byval (#779) 2024-04-14 17:12:31 -05:00
Brooks 8c5a33a81a
Renames fns to get_stored_account_meta() (#783) 2024-04-13 14:01:51 -04:00
Brooks 864e044ea3
Refactors tiered storage tests (#776) 2024-04-13 10:11:34 -04:00
Jeff Washington (jwash) 1221dcf480
use `is_zero_lamport()` trait (#781) 2024-04-13 08:45:54 -05:00
Jeff Washington (jwash) 4e43820a0c
decouple StorableAccounts pubkey & hash from account (#780) 2024-04-13 08:45:35 -05:00
Brooks 45e09396b9
Renames tiered storage get_account() to get_stored_account_meta() (#774) 2024-04-12 15:46:27 -04:00
Brooks 3715830bcc
Adds get_account_shared_data() to tiered storage (#773) 2024-04-12 12:42:42 -04:00
HaoranYi 518a5b7df8
Change read-only-cache to be keyed on pubkey only (#576)
* change read-only-cache to keyed on pubkey only

* implement review feedbacks

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-12 09:25:19 -05:00
Jeff Washington (jwash) e9f8fcb4e6
use previously calculated `stored_size` (#757) 2024-04-11 14:36:53 -05:00
Jeff Washington (jwash) baed522d2b
call StorableAccounts.pubkey once (#752) 2024-04-11 11:22:50 -05:00
Brooks 8aa32172d6
Stops copying new PathBuf when querying storage files (#751) 2024-04-11 11:00:16 -04:00
Jeff Washington (jwash) 9778e489b0
add ScanAccountStorageData (#694) 2024-04-10 09:22:37 -05:00
Jeff Washington (jwash) 4535ea60a9
remove ReadableAccount trait from LoadedAccount (#692) 2024-04-09 22:14:52 -05:00
Jeff Washington (jwash) 0c5d56bd33
pass &LoadedAccount scan_account_storage (#693) 2024-04-09 22:13:10 -05:00
Jeff Washington (jwash) d9d11cda4d
break out parameters to hash_account (#691) 2024-04-09 17:03:37 -05:00
Jeff Washington (jwash) f62d79b8be
add `data_len()` to `LoadedAccount` (#689) 2024-04-09 16:32:07 -05:00
Jeff Washington (jwash) 9d700b8749
in get_pubkey_hash_account_for_slot use take_account earlier (#688) 2024-04-09 16:05:46 -05:00
Jeff Washington (jwash) 374705293c
implement get_account_sizes for hot storages (#676) 2024-04-09 15:21:09 -05:00
Jeff Washington (jwash) 5eab6ae2b5
`purge_slot_storage` doesn't use `scan_account_storage` (#684) 2024-04-09 13:55:00 -05:00
Brooks 87314a5549
Reduces what accounts-db exports publicly (#667) 2024-04-09 14:53:56 -04:00
Jeff Washington (jwash) a2544e434d
implement scan_index for hot storage (#675)
* implement scan-index for

* impl `stored_size` for hot, fix test
2024-04-09 13:29:54 -05:00
Jeff Washington (jwash) 4ed5b4c083
removed LoadedAccount::compute_hash (#683) 2024-04-09 13:29:03 -05:00
Jeff Washington (jwash) c0be86d0e1
implement scan_pubkeys for hot storage (#666) 2024-04-09 07:47:26 -05:00
Jeff Washington (jwash) 8e0a55fe64
add file_provider to accounts_db (#665) 2024-04-08 17:58:01 -05:00
Brooks 6fedfe5a4d
Evicts from the accounts read cache in the background (#575) 2024-04-08 17:10:15 -04:00
Jeff Washington (jwash) b9487f5263
add get_account_sizes (#661) 2024-04-08 15:41:38 -05:00
Jeff Washington (jwash) 294d9d7e50
intset for SlotOffsets (#656) 2024-04-08 15:10:46 -05:00
HaoranYi 287d0d2582
Remove reclaim param for store_accounts_frozen (#618)
remove reclaim param for store_accounts_frozen

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-08 14:28:13 -05:00
Jeff Washington (jwash) 0af9aaa573
handle reclaimed offsets as a batch (#652) 2024-04-08 12:10:38 -05:00
Jeff Washington (jwash) 0025ecd5af
append vec: remove_account->remove_accounts (#607) 2024-04-08 09:30:03 -05:00
Brooks ca24f6a44d
Adds metrics for how long it takes to store into the accounts read cache (#610) 2024-04-05 16:25:31 -04:00
Brooks e09541a762
Adds metrics for how long it takes to evict from the accounts read cache (#585) 2024-04-05 13:36:36 -04:00
Jeff Washington (jwash) de9e999335
group remove_dead_accounts by slot (#578) 2024-04-05 10:38:11 -05:00
Jeff Washington (jwash) dcc195e060
when flushing write cache, ignore reclaims (#581) 2024-04-04 16:13:00 -05:00
Jeff Washington (jwash) ccdfd9a4d2
rework remove_dead_accounts.reclaimed_offsets (#571) 2024-04-04 12:28:07 -05:00
steviez a088364eb0
Use std::num::Saturating over saturating_add_assign!() (#523)
std::num::Saturating allows us to create integers that will override
the standard arithmetic operators to use saturating math. This removes
the need for a custom macro as well as reduces mental load as someone
only needs to remember that they want saturating math once.

This PR introduces std::num::Saturating integers to replace all
use of saturating_add_assign!() in the accounts-db crate
2024-04-04 10:14:53 -05:00
Jeff Washington (jwash) 527cad2b22
don't try to ancient pack already large storages (#548) 2024-04-04 10:05:13 -05:00
Jeff Washington (jwash) 854a5b4929
stop once we need 10 storages (#549) 2024-04-03 14:56:02 -05:00
Jeff Washington (jwash) f610e7a06a
rework handle_reclaims ReclaimResult (#563) 2024-04-03 13:47:58 -05:00
Jeff Washington (jwash) 51c51ddf6d
cleanup `handle_reclaims` (#552)
get rid of `Option` on clean path
2024-04-03 11:46:33 -05:00
Brooks afa65c6690
Removes write version from StorableAccountsWithHashesAndWriteVersions (#561) 2024-04-03 12:41:20 -04:00
Brooks ce1f41e547
Removes write version from StorableAccounts (#542) 2024-04-03 10:21:11 -04:00
Jeff Washington (jwash) 57572d59c8
add scan_index for improving index generation (#524)
* add scan_index for improving index generation

* pr feedback

* rework some stuff from pr feedback

* get rid of redundant if

* deal with rent correctly
2024-04-03 09:19:01 -05:00
Brooks 4247a8a546
Archives storages directly (#503) 2024-04-02 18:43:38 +00:00
Jeff Washington (jwash) ccb09866b5
stop requiring data allocation to check for rent (#543) 2024-04-02 13:25:43 -05:00
HaoranYi 9ea627c16d
Recompute hash on load if default hash is stored for the account (#519)
recompute hash on load if default hash is stored for the account

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-02 13:10:18 -05:00
Jeff Washington (jwash) c59143b980
add get_stored_account to append vec (#508)
* add get_stored_account to append vec

* Update accounts-db/src/append_vec.rs

Co-authored-by: Brooks <brooks@prumo.org>

* renames

* accountshash -> accounthash

---------

Co-authored-by: Brooks <brooks@prumo.org>
2024-04-02 11:39:11 -05:00
Jeff Washington (jwash) 90999017a4
no need to call handle_reclaims if reclaims was not populated (#527) 2024-04-02 11:22:49 -05:00
Brooks 01855eda43
Sets write version to 0 when storing accounts (#476) 2024-04-02 11:27:23 -04:00
Yueh-Hsuan Chiang e8159251e9
Enable accountsdb_scan_account_storage_no_bank tests for hot storage (#344)
Enable accountsdb_scan_account_storage_no_bank tests for hot storage.
2024-04-02 08:58:13 -05:00
Yueh-Hsuan Chiang b47a4ec74d
[TieredStorage] Make TieredStorage's Offset compatible with AccountsDB's offset (#74)
* [TieredStorage] Handles reduced-offset and offset conversion

* fix comment

---------

Co-authored-by: jeff washington <jeff.washington@solana.com>
2024-04-01 16:26:41 -05:00
Yueh-Hsuan Chiang 8153c52938
[TieredStorage] enum-based AccountsFileProvider (#457)
* [TieredStorage] enum-based AccountsFileProvider

* derive traits

* remove refs

---------

Co-authored-by: jeff washington <jeff.washington@solana.com>
2024-04-01 20:54:45 +00:00
Yueh-Hsuan Chiang a5aee48722
Rename append vec to accounts file for CurrentAncientAppendVec (#447) 2024-04-01 15:54:14 -05:00
Jeff Washington (jwash) d9dfe0f8ca
rework account load with check_and_get_loaded_account_shared_data (#506) 2024-04-01 19:48:17 +00:00
Jeff Washington (jwash) 620f5658da
introduce scan_pubkeys for clean storage iteration (#507)
* introduce pubkey_iter for clean storage iteration

* rename scan_pubkeys

* pr feedback

* pr feedback
2024-04-01 12:07:41 -05:00
Brooks 288d5ba121
Moves where modules are included in accounts_db.rs (#515) 2024-04-01 09:51:59 -04:00
sakridge e0e659680a
Remove duplicated token ids and use a shared inline-spl crate (#456) 2024-04-01 14:31:04 +02:00
Jeff Washington (jwash) e56d314df5
determine if account is in write cache earlier (#504) 2024-03-29 17:10:02 -05:00
Jeff Washington (jwash) 18c32aba35
disable read cache while populating stakes cache on load (#482)
* dsiable read cache while populating stakes cache on load

* use struct with drop as api

* use LoadHint

* remove disable_read_cache_updates_count

* add comment

* fmt
2024-03-29 15:20:38 -05:00
Jeff Washington (jwash) e261e2704a
store only removes from read cache if slot is possibly present (#452)
* store only removes from read cache if slot is possibly present

* remove_assume_not_present
2024-03-28 17:00:48 -05:00
Brooks d5c0c0b1c2
Removes support for loading snapshots with > 1 append vec per slot (#474) 2024-03-28 16:32:17 -04:00
Yueh-Hsuan Chiang 4b9e1e0ab3
[TieredStorage] Exclude NotFound in reporting storage leakage on drop() (#446)
#### Problem
TieredStorage::drop() currently panic when it fails to delete the
underlying file to raise awareness of possible storage resource
leakage, including io::ErrorKind::NotFound.  But sometimes the
TieredStorage (or AccountsFile in general) instance is created
then dropped without any file being created.  This causes some
false-alarms including unit-tests.

#### Summary of Changes
This PR excludes NotFound in reporting storage leakage on
TieredStorage::drop().
2024-03-27 09:43:38 -07:00
Brooks 21fbde741a
Uses Into<PathBuf> for path in AccountsFile::new_from_file() (#434) 2024-03-26 16:19:06 -04:00
Brooks 21b6821885
Uses Into<PathBuf> for path in AppendVec::new() (#433) 2024-03-26 14:44:17 -04:00
Yueh-Hsuan Chiang c7dba30f4f
[TieredStorage] Disable accounts-file type check before enabling tiered-storage (#418)
#### Problem
As #72 introduced AccountsFile::TieredStorage, it also performs
file-type check when opening an accounts-file to determine whether
it is a tiered-storage or an append-vec.  But before tiered-storage is
enabled, this opening check is unnecessary.

#### Summary of Changes
Remove the accounts-file type check code and simply assume everything
is append-vec on AccountsFile::new_from_file().
2024-03-25 12:08:48 -07:00
Yueh-Hsuan Chiang a916edb7a2
[TieredStorage] Add AccountsFile::TieredStorage (#72)
#### Problem
AccountsFile currently doesn't have an implementation for TieredStorage.
To enable AccountsDB tests for the TieredStorage, we need AccountsFile
to support TieredStorage.

#### Summary of Changes
This PR implements a AccountsFile::TieredStorage, a thin wrapper between
AccountsFile and TieredStorage.
2024-03-24 16:41:36 -07:00
Yueh-Hsuan Chiang 602471257e
[TieredStorage] Add capacity() API and limit file size to 16GB (#401)
#### Problem
The TieredStorage has not yet implemented the AccountsFile::capacity()
API.

#### Summary of Changes
Implement capacity() API for TieredStorage and limit file size to 16GB,
same as the append-vec file.
2024-03-23 22:14:19 -07:00
Yueh-Hsuan Chiang e9cc9f8379
[TieredStorage] Refactor file_size() code path (#400)
#### Problem
TieredStorage::file_size() essentially supports AccountsFile::len(),
but its API is inconsistent with AccountsFile's.

#### Summary of Changes
Refactor TieredStorage::file_size() to ::len() and share the same API
as AccountsFile's.

#### Test Plan
Build
Existing unit-tests.
2024-03-22 14:21:24 -07:00
Yueh-Hsuan Chiang f1a82cb666
[TieredStorage] Use mmap.len() in TieredStorage::file_size() for HotStorage (#381)
#### Problem
The current implementation of TieredStorage::file_size() requires
a sys-call to provide the file size.

#### Summary of Changes
Add len() API to TieredStorageReader, and have HotStorageReader()
implement the API using Mmap::len().

#### Test Plan
Update existing unit-test to also verify HotStorageReader::len().
2024-03-22 11:56:30 -07:00
Yueh-Hsuan Chiang 977b1b836f
Rename AppendVecId to AccountsFileId (#383)
#### Problem
The current AppendVecId actually refers to an accounts file id.

#### Summary of Changes
Rename AppendVecId to AccountsFileId.

#### Test Plan
Build
2024-03-22 11:25:30 -07:00
Brooks 24fe473b46
clippy: Automated fixes for Rust 1.77.0 (#390) 2024-03-22 13:48:46 -04:00
Brooks cbd0369da1
Uses AppendVecId in AccountsFIle::file_name() (#372) 2024-03-21 21:27:03 -04:00
Brooks 8b66a670b7
Removes AccountsFile::is_recyclable() (#359) 2024-03-21 15:09:26 -04:00
Yueh-Hsuan Chiang 3038d47f1c
[TieredStorage] Store account address range (#172)
#### Problem
The TieredStorageFooter has the min_account_address and
max_account_address fields to describe the account address
range in its file.  But the current implementation hasn't updated
the fields yet.

#### Summary of Changes
This PR enables the TieredStorage to persist address range
information into its footer via min_account_address and
max_account_address.

#### Test Plan
Updated tiered-storage test to verify persisted account address range.
2024-03-20 12:17:12 -07:00
Yueh-Hsuan Chiang 0e932c7308
[TieredStorage] Refactor TieredStorage::new_readonly() code path (#195)
#### Problem
The TieredStorage::new_readonly() function currently has the following
problems:

* It opens the file without checking the magic number before checking and loading the footer.
* It opens the file twice: first to load the footer, then open again by the reader.

#### Summary of Changes
This PR refactors TieredStorage::new_readonly() so that it first performs all
checks inside the constructor of TieredReadableFile.  The TieredReadableFile
instance is then passed to the proper reader (currently HotStorageReader)
when all checks are passed.

#### Test Plan
* Added a new test to check MagicNumberMismatch.
* Existing tiered-storage tests
2024-03-20 10:39:25 -07:00
Jon C 261b3e9ee7
CI: Add windows clippy job and fix clippy errors (#330)
* CI: Run clippy on windows

* Update cargo-clippy-before-script.sh for Windows

* Pacify clippy
2024-03-20 13:21:00 +01:00
Alessandro Decina 8df80d9c12
accounts-db: unpack_archive: unpack accounts straight into their final destination (#289)
* accounts-db: unpack_archive: avoid extra iteration on each path

We used to do a iterator.clone().any(...) followed by
iterator.collect(). Merge the two and avoid an extra iteration and
re-parsing of the path.

* accounts-db: unpack_archive: unpack accounts straight into their final destination

We used to unpack accounts into account_path/accounts/<account> then
rename to account_path/<account>. We now unpack them into their final
destination directly and avoid the rename syscall.
2024-03-20 09:39:33 +11:00
Yueh-Hsuan Chiang 7c49b9c59e
[TieredStorage] Use BufWriter in TieredWritableFile (#261)
#### Problem
TieredWritableFile currently uses File instead of BufWriter.
This will introduce more syscall when doing file writes.

#### Summary of Changes
This PR makes TieredWritableFile uses BufWriter to allow the
write-call to be more optimized to reduce the number of syscalls.

#### Test Plan
Existing tiered-storage test.
Will run experiments to verify its performance improvement.

#### Dependency
https://github.com/anza-xyz/agave/pull/260
2024-03-18 14:27:55 -07:00
Yueh-Hsuan Chiang 6441209682
[TieredStorage] TieredStorageFile -> TieredReadonlyFile and TieredWritableFIle (#260)
#### Problem
TieredStorageFile struct currently offers new_readonly() and new_writable()
to allow both read and write work-load to share the same struct.  However,
as we need the writer to use BufWriter to improve performance as well as
enable Hasher on writes.  There is a need to refactor TieredStorageFile to
split its usage for read-only and writable.

#### Summary of Changes
Refactor TieredStorageFile to TieredReadonlyFIle and TieredWritableFile.

#### Test Plan
Existing tiered-storage tests.
2024-03-18 12:24:19 -07:00
Jeff Washington (jwash) 1fc4e38a4f
add stats for write cache flushing (#233)
* add stats for write cache flushing

* some renames
2024-03-18 09:53:44 -05:00
HaoranYi dcc6f1eb3a
fix typos (#247)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-03-15 09:15:23 -05:00
Jeff Washington (jwash) 794cb2f856
allow FlushStats to accumulate (#215) 2024-03-13 18:47:55 -05:00
Yueh-Hsuan Chiang 69b6d5a376
[TieredStorage] Remove the general-purposed TieredStorageWriter (#196)
#### Problem
tiered_storage/writer.rs was added when we planned to support multiple
tiers in the tiered-storage (i.e., at least hot and cold).  However, as we
changed our plan to handle cold accounts as state-compressed accounts,
we don't need a general purposed tiered-storage writer at this moment.


#### Summary of Changes
Remove tiered_storage/writer.rs as we currently don't have plans to develop cold storage.

#### Test Plan
Existing tiered-storage tests.
2024-03-13 10:26:37 -07:00
Yueh-Hsuan Chiang e13fbeb198
[TieredStorage] Repurpose TieredReadableAccount to HotAccount (#218)
#### Problem
As we further optimize the HotStorageMeta in #146, there is a need
for a HotAccount struct that contains all the hot account information.
Meanwhile, we currently don't have plans to develop a cold account
format at this moment.  As a result, this makes it desirable to repurpose
TieredReadableAccount to HotAccount.

#### Summary of Changes
Repurpose TieredReadableAccount to HotAccount.

#### Test Plan
Existing tiered-storage tests.
2024-03-13 10:07:11 -07:00
Brooks 88f6a7a459
Removes holding storages in AccountsHashVerifier for fastboot (#120) 2024-03-11 17:09:26 -04:00
Brooks 1a085c8d46
Removes atomic-ness from AccountStorageEntry `id` and `slot` fields (#119) 2024-03-11 17:09:08 -04:00
Brooks 5c1df15e92
Removes the storage recycler (#118) 2024-03-11 15:38:34 -04:00
Jeff Washington (jwash) 158c4e05d5
remove dead code (#176) 2024-03-11 12:21:51 -05:00