20ee70cd18
Supports archiving account storage files that are backed by Mmap or File ( #1393 )
2024-05-17 00:09:42 -04:00
ae8825e717
Add load_account_with() API to allow a callback to control whether to insert the loaded account into read_cache. ( #1350 )
...
* add load_account_with() API
* pr reviews
* typo
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-05-16 16:10:03 +00:00
7474e4722b
Deprecate `is_key_passed_to_program` in favor of `is_instruction_account` ( #1374 )
...
Deprecate is_key_passed_to_program in favor of is_instruction_account
2024-05-16 00:20:54 +00:00
537a3b75f3
Deprecate `is_non_loader_key` message methods ( #1219 )
...
deprecate is_non_loader_key methods
2024-05-15 17:42:19 -05:00
ec9bd79849
clippy: fix legacy_numeric_constants ( #1314 )
...
clippy: legacy_numeric_constants
2024-05-15 11:29:19 +08:00
6fb4338977
clippy: allow dead_code for ParsedFilename ( #1345 )
...
* clippy: allow dead_code for ParsedFilename
* use underscore to denote never read fields
2024-05-14 15:29:12 +00:00
3dbb8ff937
clippy: zero_repeat_side_effects ( #1341 )
2024-05-14 19:40:14 +08:00
fc99a39ec1
fix test oddity ( #1319 )
2024-05-13 11:59:55 -05:00
d7c9083457
Flushes tiered storage files explicitly after writing ( #1276 )
2024-05-13 12:55:17 -04:00
6147420142
add `can_append` ( #1273 )
2024-05-13 10:34:49 -05:00
2c71685b94
accounts-db: fix 8G+ memory spike during hash calculation ( #1308 )
...
We were accidentally doing several thousands 4MB allocations - even
during incremental hash - which added up to a 8G+ memory spikes over ~2s
every ~30s.
Fix by using Vec::new() in the identity function. Empirically 98%+
reduces join arrays with less than 128 elements, and only the last few
reduces join large vecs. Because realloc does exponential growth we
don't see pathological reallocation but reduces do at most one realloc
(and often 0 because of exp growth).
2024-05-13 22:44:50 +08:00
772604ebd2
Sets default accounts read cache watermarks to 400-410 MiB ( #1292 )
2024-05-10 15:11:43 -04:00
835eec02ba
add StorageAccess to specify how to access storages ( #1270 )
...
* add StorageAccess to specify how to access storages
* pr feedback
2024-05-10 12:55:22 -05:00
317f817f13
introduce enums for non-mmap based append vecs ( #1267 )
...
* introduce enums for non-mmap based append vecs
* renames
* renames
2024-05-10 10:34:46 -05:00
fadfa6140c
Simd 118: replace calculation-blocks config value with constant ( #1251 )
...
* Add const for num calculation blocks
* Use const in partitioned_epoch_rewards module
* Remove unused config-struct field
* Unpub test-only fn
* Preserve single-slot testing framework
* Fix incorrect type alias
* Use const instead of magic number in tests
* Cleanup get_reward_total_num_blocks
2024-05-09 19:59:36 -06:00
5bf7af9435
remove unused allow_shrink_ancient ( #1262 )
2024-05-09 12:55:16 -05:00
f4f4be261a
clippy: suspicious_open_options ( #1240 )
...
* clippy: suspicious_open_options
* accounts-db/src/append_vec.rs create_new
* install/src/command.rs create_new
* bucket_storage truncate true
* bucket_map restart create_new
* accounts-db/src/cache_hash_data create_new
2024-05-10 01:36:32 +08:00
f050705f98
clippy: unnecessary_get_then_check ( #1242 )
2024-05-09 00:17:18 +08:00
6725fe3454
make ancient appending default to packing ( #1224 )
...
* make ancient appending default to packing
* appending ancient packing doesn't re-write packed ancient append vecs from previous runs
* Update accounts-db/src/accounts_db.rs
Co-authored-by: Brooks <brooks@prumo.org>
* use default
---------
Co-authored-by: Brooks <brooks@prumo.org>
2024-05-08 08:01:52 -05:00
b284eb0402
SVM: Move nonce_info from SDK to SVM ( #1228 )
...
because it's not something we want to be subjected to the backwards compatibility policy.
2024-05-07 21:27:42 -04:00
7a50c57c08
Tidy up accounts read cache stats ( #1231 )
2024-05-07 18:40:03 +00:00
21aa128c57
Adds stats for how often the read cache evictor wakes up ( #1223 )
2024-05-07 12:55:32 -04:00
9cefe0c65d
Adds CLI arg for setting accounts read cache size limits ( #1218 )
2024-05-07 12:54:38 -04:00
9c8b621cad
Gets snapshot storages before calling verify_accounts_hash() ( #1202 )
2024-05-07 10:25:16 -04:00
b78d2550d2
refactor: simplify fee payer determination ( #1198 )
...
* refactor: simplify fee payer determination
* feedback
2024-05-07 12:31:29 +08:00
aa6c69a3b5
Refactors full accounts hash calculation during startup verify ( #1200 )
2024-05-06 13:25:17 -04:00
f46e3cab2d
Adds verify_accounts_hash_and_lamports_for_tests() ( #1201 )
2024-05-06 13:25:08 -04:00
b0cfffb290
Renames accounts hash calc fns ( #1183 )
2024-05-03 19:23:12 -04:00
b7bbe36918
clippy: multiple bound locations ( #1179 )
2024-05-03 11:33:09 -04:00
9606a1395d
Replaces DashMap with HashMap in secondary indexes ( #1137 )
2024-05-02 16:13:09 -04:00
eeda398197
remove get_stored_account_meta() ( #1127 )
...
remove `get_stored_account_meta()`
2024-05-01 08:15:52 -05:00
65c365542e
remove accounts() and all_accounts() ( #1117 )
...
remove `accounts()` and `all_accounts()`
2024-04-30 14:52:37 -05:00
016fe8af0c
add append vec test method for # accounts ( #1116 )
2024-04-30 10:03:53 -05:00
5c7ae92a2f
remove test uses of `all_accounts` ( #1110 )
2024-04-30 08:35:19 -05:00
7f629c5666
introduce accounts_count() ( #1099 )
...
replace use of `all_accounts`
2024-04-30 08:34:57 -05:00
3c0f263327
hot storage test uses `scan_accounts` ( #1100 )
2024-04-29 22:44:17 -05:00
ad88e90141
remove use of `accounts()` in tests ( #1111 )
2024-04-29 18:07:33 -05:00
bc2d06980c
use scan_accounts to avoid lifetime issues ( #1098 )
2024-04-29 17:55:22 -05:00
f8067ea788
Removes StoredAccountMeta::set_meta() ( #703 )
2024-04-29 17:58:37 -04:00
9f10f09613
hot storage tests use callback ( #1095 )
...
hot storage tests us callback
2024-04-29 13:20:41 -05:00
d61c3a3aed
test code uses `scan_accounts` ( #1097 )
2024-04-29 11:52:40 -05:00
cea03e3121
hot storage uses scan_accounts ( #1033 )
2024-04-29 10:56:29 -05:00
35bcf74300
Moves `0` for write_version in geyser when restoring from a snapshot ( #979 )
2024-04-29 11:15:31 -04:00
2d722719a2
add testing for get_stored_account_meta_callback ( #1031 )
2024-04-29 09:23:35 -05:00
e5bf340d33
sanitize_layout_and_length uses `scan_accounts` ( #1030 )
2024-04-29 09:22:59 -05:00
a4609bdab8
Simd 118: load stake accounts on distribution ( #845 )
...
* Use type alias for consistency
* Add PartitionedStakeReward(s) struct and type, duplicates StakeReward(s)
* Remove distribution balance check
* Add helper
* Add local error enum
* Add method to update stake state in distribution
* Add unit test for build_updated_stake_reward
* Update store_stake_accounts_in_partition to operate on PartitionedStakeReward
* Fix tests
* Log burned_rewards
* Update RewardInfo::post_balance during distribution
* Update update_reward_history_in_partition to use updated_stake_rewards
* Fix tests
* Improve coverage
* Fix typo
* Hoist log to error
* Remove unneeded dead_code allow
* Add assert and fixup lazy tests
2024-04-26 10:12:16 -06:00
196626dfcf
rpc: Return accurate last valid block height ( #1034 )
...
* blockhash-queue: Deprecate `is_hash_valid`
* blockhash-queue: Deprecate `get_max_age`
* bank: Add a test
* rpc: Fix tests, and limit nonce retries to MAX_PROCESSING_AGE
2024-04-26 13:01:53 +02:00
a0c256136c
fix hot_storage test for test_accountsdb_scan_account_storage_no_bank_one_slot ( #1032 )
...
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-25 08:46:58 -05:00
552559d056
get rid of AccountIter ( #900 )
2024-04-24 17:16:44 -05:00
e7452b7680
Replaces u128s when summing lamports in accounts hash calculation ( #993 )
2024-04-24 13:03:20 -04:00
d81ace8ff7
Removes AppendVec::file_name() ( #976 )
2024-04-24 10:09:04 -04:00
45c49cedf2
get_unique_accounts_from_storage uses scan_index ( #969 )
...
* get_unique_accounts_from_storage uses scan_index
* avoid returning default pubkey=default accounts
2024-04-24 08:56:14 -05:00
16f1f25dbb
cache storage lookup during shrink and ancient ( #973 )
...
* cache storage lookup during shrink and ancient
* rename and remove pub
* pr feedback
2024-04-23 11:51:37 -05:00
ca8f92e001
fix references on get_stored_account_meta_callback ( #996 )
...
* fix references on get_stored_account_meta_callback
* fmt
2024-04-23 11:51:03 -05:00
57ab9449c7
remove multi slot test only StorableAccounts impl ( #972 )
2024-04-23 08:35:45 -05:00
862c79e132
modify geyser account iter at snapshot load ( #960 )
2024-04-22 13:06:25 -05:00
7389371112
rework secondary index generation ( #959 )
2024-04-22 12:59:47 -05:00
d4b86923ef
introduce AccountFromStorage for shrink ( #881 )
...
* introduce AccountFromStorage for shrink
* add comments and cleanup
* more comments
* unwrap to expect
* add comments
2024-04-22 12:20:18 -05:00
422387982f
Removes unnecessary &Arc ( #958 )
2024-04-22 11:46:40 -04:00
6239ece50d
remove unused StorableAccounts ( #956 )
2024-04-22 09:54:24 -05:00
127faa3a07
shrinking and appending ancients use StorableAccountsBySlot ( #935 )
...
* shrinking and appending ancients use StorableAccountsBySlot
* snapshot minimizer uses StorableAccountsBySlot
2024-04-22 08:28:36 -05:00
14d6c79c78
remove unused StorableAccountsMovingSlots ( #952 )
2024-04-22 05:05:05 -05:00
8d4f2e6221
use scan_accounts to replace account_iter ( #921 )
2024-04-21 20:07:05 -05:00
43e47bea28
remove a StorableAccounts impl ( #936 )
2024-04-21 20:03:55 -05:00
c2936ebb4a
accounts-db: send batches of accounts to the background hasher ( #810 )
...
Instead of sending accounts individually, send batches of accounts for
background hashing.
Before this change we used to send accounts individually, and the
background thread used to do:
loop {
let account = receiver.recv();
account.hash();
// go back to sleep in recv()
}
Because most accounts are small and hashing them is very fast, the
background thread used to sleep a lot, and required many syscalls from
the sender in order to be woken up.
Batching reduces the number of syscalls.
2024-04-20 08:19:49 +10:00
fc2b05d50d
fix a few storage iter lifetimes ( #917 )
2024-04-19 17:10:43 -05:00
bbc0be2cb3
Removes returned Result from infallible fns ( #914 )
2024-04-19 20:44:15 +00:00
1189055c98
load account lifetime improvements ( #903 )
...
* load account lifetime improvements
* restore 2 check_and_
2024-04-19 15:33:29 -05:00
e3d3cd6486
add appendvec.scan_accounts ( #902 )
2024-04-19 11:38:31 -05:00
246dc481ca
Removes check_hash from CalcAccountsHashConfig ( #913 )
2024-04-19 15:43:04 +00:00
fb3bf55340
accounts_iter -> scan_pubkeys ( #894 )
2024-04-19 09:36:53 -05:00
f8a8d40ac7
test fixes to remove refs ( #901 )
2024-04-19 09:36:27 -05:00
992a398fe8
accounts db/refactor accounts db test - test_partial_clean ( #818 )
...
* add hot storage test for test_partial_clean
* fix test_partial_clean for hot storage
* reviews
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-19 08:48:06 -05:00
f0e87f6210
replace one accounts iter with scan_pubkeys ( #888 )
2024-04-18 20:43:09 -05:00
86667f5a6f
Removes check_hash from scan_snapshot_stores_with_cache() ( #892 )
2024-04-18 23:24:04 +00:00
16770cce88
Removes tests that use check_hash ( #893 )
2024-04-18 22:32:03 +00:00
4b9d18a361
Removes check_hash from ScanState ( #891 )
2024-04-18 22:05:10 +00:00
54b816a338
Removes `check_hash` from calculate_accounts_hash_from_index() ( #890 )
2024-04-18 18:01:53 -04:00
ee113de4ce
Refactors stored accounts info ( #887 )
2024-04-18 15:45:44 -04:00
e43338f3d6
Refactor - Rename `LoadedProgram` to `ProgramCacheEntry` ( #880 )
...
LoadedProgramOwner => ProgramCacheEntryOwner.
LoadedProgramType => ProgramCacheEntryType.
LoadedProgram => ProgramCacheEntry.
LoadedProgramStats => ProgramCacheStats.
LoadedProgramMatchCriteria => ProgramCacheMatchCriteria.
LoadedProgramsForTxBatch => ProgramCacheForTxBatch.
2024-04-18 21:38:08 +02:00
994ab254a2
some test tweaks ( #882 )
...
* some test tweaks
* get rid of some redundancy
2024-04-18 14:01:02 -05:00
d77118ac65
tests had invalid storage offset ( #874 )
2024-04-18 12:09:14 -05:00
033bcfcab5
accounts db/refactor accounts db test - convert test_full_clean_refcount ( #814 )
...
* add full_clean_refcount tests for both account storage formats
* keep code comments
* fix test to match with code comments
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-17 16:42:44 -05:00
da0d78c504
Removes hash() and has_hash() from StorableAccounts ( #867 )
2024-04-17 16:41:41 -04:00
ee2c29ca2f
Removes unused hashes params from store functions ( #863 )
2024-04-17 16:37:45 -04:00
613256e18f
Removes StorableAccountsWithHashes ( #862 )
2024-04-17 18:52:45 +00:00
283f433a3b
accounts db/refactor accounts db test - add macro for accounts-db panic test ( #786 )
...
* add macro for accounts-db panic test
convert double remove test for both account file provider
* refactor to share accounts_db_test and accounts_db_panic_test macro
* rebase
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-17 12:52:00 -05:00
ad5b71dda8
Replaces StorableAccountsWithHashes in AccountsDb ( #861 )
2024-04-17 17:31:44 +00:00
349eb7f925
Replaces StorableAccountsWithHashes in AccountsFile ( #860 )
2024-04-17 16:02:48 +00:00
2aeac5b468
Replaces StorableAccountsWithHashes in TieredStorage ( #859 )
2024-04-17 14:49:24 +00:00
97013978fe
Uses same lifetime name for local AccountForStorage in callback ( #848 )
2024-04-17 09:39:44 -04:00
4d2d95fe14
Replaces StorableAccountsWithHashes in AppendVec ( #853 )
2024-04-17 09:34:03 -04:00
63d4278cae
get_stored_account_meta_callback returns Option ( #847 )
...
* get_stored_account_meta_callback returns Option
* Update accounts-db/src/accounts_file.rs
Co-authored-by: Brooks <brooks@prumo.org>
---------
Co-authored-by: Brooks <brooks@prumo.org>
2024-04-17 08:12:32 -05:00
03b596586c
Write default hash to account storage ( #469 )
...
* write default hash to apendvec
* fix get_pubkey_hash_for_slot
* fix tests
* fix rebase error
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-16 16:04:52 -05:00
a9548a6dc5
add `for<>` to callback on get fns ( #838 )
2024-04-16 14:55:50 -05:00
0976e9b222
add get_stored_account_meta_callback to storage ( #836 )
2024-04-16 13:58:42 -05:00
c5a669ea34
accounts db/refactor accounts db test - fix account storage count and alive bytes tests for hot storage ( #784 )
...
fix account storage count and alive bytes tests for hot storage
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-16 12:11:44 -05:00
877591ea7d
remove StorableAccounts.pubkey() ( #829 )
...
remove `StorableAccounts`.`pubkey()`
2024-04-16 11:39:19 -05:00
cb2fd2b632
AccountForStorage supports pubkey() ( #825 )
...
* account_default_if_zero_lamport always returns a value
* rename
2024-04-16 10:06:33 -05:00
731d36ab26
Fixes pedantic clippy ptr cast lints in append_vec.rs ( #650 )
2024-04-16 13:41:00 +00:00
Brooks
HaoranYi
Justin Starry
Yihau Chen
Jeff Washington (jwash)
Alessandro Decina
Tyera
dmakarov
Jon C
Alexander Meißner