Commit Graph

578 Commits

Author SHA1 Message Date
Jeff Washington (jwash) 9a7bada228
add `reopen_storage_as_readonly_shrinking_in_progress_ok` (#1298)
add `reopen_as_readonly`
2024-05-17 11:32:37 -05:00
Kevin Heavey cadba689cb
Make solana-frozen-abi optional in all remaining crates (#1278)
* put most AbiExample derivations behind a cfg_attr

* feature gate all `extern crate solana_frozen_abi_macro;`

* use cfg_attr wherever we were deriving both AbiExample and AbiEnumVisitor

* fix cases where AbiEnumVisitor was still being derived unconditionally

* fix a case where AbiExample was derived unconditionally

* fix more cases where both AbiEnumVisitor and AbiExample were derived unconditionally

* two more cases where AbiExample and AbiEnumVisitor were unconditionally derived

* fix remaining unconditional derivations of AbiEnumVisitor

* fix cases where AbiExample is the first thing derived

* fix most remaining unconditional derivations of AbiExample

* move all `frozen_abi(digest =` behind cfg_attr

* replace incorrect cfg with cfg_attr

* fix one more unconditionally derived AbiExample

* feature gate AbiExample impls

* add frozen-abi feature to required Cargo.toml files

* make frozen-abi features activate recursively

* fmt

* add missing feature gating

* fix accidentally changed digest

* activate frozen-abi in relevant test scripts

* don't activate solana-program's frozen-abi in sdk dev-dependencies

* update to handle AbiExample derivation on new AppendVecFileBacking enum

* revert toml formatting

* remove unused frozen-abi entries from address-lookup-table Cargo.toml

* remove toml references to solana-address-lookup-table-program/frozen-abi

* update lock file

* remove no-longer-used generic param
2024-05-17 14:42:58 +02:00
Brooks 20ee70cd18
Supports archiving account storage files that are backed by Mmap or File (#1393) 2024-05-17 00:09:42 -04:00
HaoranYi ae8825e717
Add load_account_with() API to allow a callback to control whether to insert the loaded account into read_cache. (#1350)
* add load_account_with() API

* pr reviews

* typo

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-05-16 16:10:03 +00:00
Justin Starry 7474e4722b
Deprecate `is_key_passed_to_program` in favor of `is_instruction_account` (#1374)
Deprecate is_key_passed_to_program in favor of is_instruction_account
2024-05-16 00:20:54 +00:00
Justin Starry 537a3b75f3
Deprecate `is_non_loader_key` message methods (#1219)
deprecate is_non_loader_key methods
2024-05-15 17:42:19 -05:00
Yihau Chen ec9bd79849
clippy: fix legacy_numeric_constants (#1314)
clippy: legacy_numeric_constants
2024-05-15 11:29:19 +08:00
Yihau Chen 6fb4338977
clippy: allow dead_code for ParsedFilename (#1345)
* clippy: allow dead_code for ParsedFilename

* use underscore to denote never read fields
2024-05-14 15:29:12 +00:00
Yihau Chen d1afd78319
clippy: multiple_bound_locations (#1315) 2024-05-14 19:40:48 +08:00
Yihau Chen 3dbb8ff937
clippy: zero_repeat_side_effects (#1341) 2024-05-14 19:40:14 +08:00
Jeff Washington (jwash) fc99a39ec1
fix test oddity (#1319) 2024-05-13 11:59:55 -05:00
Brooks d7c9083457
Flushes tiered storage files explicitly after writing (#1276) 2024-05-13 12:55:17 -04:00
Jeff Washington (jwash) 6147420142
add `can_append` (#1273) 2024-05-13 10:34:49 -05:00
Alessandro Decina 2c71685b94
accounts-db: fix 8G+ memory spike during hash calculation (#1308)
We were accidentally doing several thousands 4MB allocations - even
during incremental hash - which added up to a 8G+ memory spikes over ~2s
every ~30s.

Fix by using Vec::new() in the identity function. Empirically 98%+
reduces join arrays with less than 128 elements, and only the last few
reduces join large vecs. Because realloc does exponential growth we
don't see pathological reallocation but reduces do at most one realloc
(and often 0 because of exp growth).
2024-05-13 22:44:50 +08:00
Brooks 772604ebd2
Sets default accounts read cache watermarks to 400-410 MiB (#1292) 2024-05-10 15:11:43 -04:00
Jeff Washington (jwash) 835eec02ba
add StorageAccess to specify how to access storages (#1270)
* add StorageAccess to specify how to access storages

* pr feedback
2024-05-10 12:55:22 -05:00
Jeff Washington (jwash) 317f817f13
introduce enums for non-mmap based append vecs (#1267)
* introduce enums for non-mmap based append vecs

* renames

* renames
2024-05-10 10:34:46 -05:00
Tyera fadfa6140c
Simd 118: replace calculation-blocks config value with constant (#1251)
* Add const for num calculation blocks

* Use const in partitioned_epoch_rewards module

* Remove unused config-struct field

* Unpub test-only fn

* Preserve single-slot testing framework

* Fix incorrect type alias

* Use const instead of magic number in tests

* Cleanup get_reward_total_num_blocks
2024-05-09 19:59:36 -06:00
Jeff Washington (jwash) 5bf7af9435
remove unused allow_shrink_ancient (#1262) 2024-05-09 12:55:16 -05:00
Yihau Chen f4f4be261a
clippy: suspicious_open_options (#1240)
* clippy: suspicious_open_options

* accounts-db/src/append_vec.rs create_new

* install/src/command.rs create_new

* bucket_storage truncate true

* bucket_map restart create_new

* accounts-db/src/cache_hash_data create_new
2024-05-10 01:36:32 +08:00
Yihau Chen f050705f98
clippy: unnecessary_get_then_check (#1242) 2024-05-09 00:17:18 +08:00
Jeff Washington (jwash) 6725fe3454
make ancient appending default to packing (#1224)
* make ancient appending default to packing

* appending ancient packing doesn't re-write packed ancient append vecs from previous runs

* Update accounts-db/src/accounts_db.rs

Co-authored-by: Brooks <brooks@prumo.org>

* use default

---------

Co-authored-by: Brooks <brooks@prumo.org>
2024-05-08 08:01:52 -05:00
dmakarov b284eb0402
SVM: Move nonce_info from SDK to SVM (#1228)
because it's not something we want to be subjected to the backwards compatibility policy.
2024-05-07 21:27:42 -04:00
Brooks 7a50c57c08
Tidy up accounts read cache stats (#1231) 2024-05-07 18:40:03 +00:00
Brooks 21aa128c57
Adds stats for how often the read cache evictor wakes up (#1223) 2024-05-07 12:55:32 -04:00
Brooks 9cefe0c65d
Adds CLI arg for setting accounts read cache size limits (#1218) 2024-05-07 12:54:38 -04:00
Brooks 9c8b621cad
Gets snapshot storages before calling verify_accounts_hash() (#1202) 2024-05-07 10:25:16 -04:00
Justin Starry b78d2550d2
refactor: simplify fee payer determination (#1198)
* refactor: simplify fee payer determination

* feedback
2024-05-07 12:31:29 +08:00
Brooks aa6c69a3b5
Refactors full accounts hash calculation during startup verify (#1200) 2024-05-06 13:25:17 -04:00
Brooks f46e3cab2d
Adds verify_accounts_hash_and_lamports_for_tests() (#1201) 2024-05-06 13:25:08 -04:00
Brooks b0cfffb290
Renames accounts hash calc fns (#1183) 2024-05-03 19:23:12 -04:00
Brooks b7bbe36918
clippy: multiple bound locations (#1179) 2024-05-03 11:33:09 -04:00
Brooks 9606a1395d
Replaces DashMap with HashMap in secondary indexes (#1137) 2024-05-02 16:13:09 -04:00
Jeff Washington (jwash) eeda398197
remove get_stored_account_meta() (#1127)
remove `get_stored_account_meta()`
2024-05-01 08:15:52 -05:00
Jeff Washington (jwash) 65c365542e
remove accounts() and all_accounts() (#1117)
remove `accounts()` and `all_accounts()`
2024-04-30 14:52:37 -05:00
Jeff Washington (jwash) 016fe8af0c
add append vec test method for # accounts (#1116) 2024-04-30 10:03:53 -05:00
Jeff Washington (jwash) 5c7ae92a2f
remove test uses of `all_accounts` (#1110) 2024-04-30 08:35:19 -05:00
Jeff Washington (jwash) 7f629c5666
introduce accounts_count() (#1099)
replace use of `all_accounts`
2024-04-30 08:34:57 -05:00
Jeff Washington (jwash) 3c0f263327
hot storage test uses `scan_accounts` (#1100) 2024-04-29 22:44:17 -05:00
Jeff Washington (jwash) ad88e90141
remove use of `accounts()` in tests (#1111) 2024-04-29 18:07:33 -05:00
Jeff Washington (jwash) bc2d06980c
use scan_accounts to avoid lifetime issues (#1098) 2024-04-29 17:55:22 -05:00
Brooks f8067ea788
Removes StoredAccountMeta::set_meta() (#703) 2024-04-29 17:58:37 -04:00
Jeff Washington (jwash) 9f10f09613
hot storage tests use callback (#1095)
hot storage tests us callback
2024-04-29 13:20:41 -05:00
Jeff Washington (jwash) d61c3a3aed
test code uses `scan_accounts` (#1097) 2024-04-29 11:52:40 -05:00
Jeff Washington (jwash) 37f6cedee3
use get_stored_account_meta_callback in bench (#1094) 2024-04-29 11:52:27 -05:00
Jeff Washington (jwash) cea03e3121
hot storage uses scan_accounts (#1033) 2024-04-29 10:56:29 -05:00
Brooks 35bcf74300
Moves `0` for write_version in geyser when restoring from a snapshot (#979) 2024-04-29 11:15:31 -04:00
Jeff Washington (jwash) 2d722719a2
add testing for get_stored_account_meta_callback (#1031) 2024-04-29 09:23:35 -05:00
Jeff Washington (jwash) e5bf340d33
sanitize_layout_and_length uses `scan_accounts` (#1030) 2024-04-29 09:22:59 -05:00
Tyera a4609bdab8
Simd 118: load stake accounts on distribution (#845)
* Use type alias for consistency

* Add PartitionedStakeReward(s) struct and type, duplicates StakeReward(s)

* Remove distribution balance check

* Add helper

* Add local error enum

* Add method to update stake state in distribution

* Add unit test for build_updated_stake_reward

* Update store_stake_accounts_in_partition to operate on PartitionedStakeReward

* Fix tests

* Log burned_rewards

* Update RewardInfo::post_balance during distribution

* Update update_reward_history_in_partition to use updated_stake_rewards

* Fix tests

* Improve coverage

* Fix typo

* Hoist log to error

* Remove unneeded dead_code allow

* Add assert and fixup lazy tests
2024-04-26 10:12:16 -06:00
Brooks c2417bcc73
Adds bench for account serde (#1052) 2024-04-26 10:51:09 -04:00
Jon C 196626dfcf
rpc: Return accurate last valid block height (#1034)
* blockhash-queue: Deprecate `is_hash_valid`

* blockhash-queue: Deprecate `get_max_age`

* bank: Add a test

* rpc: Fix tests, and limit nonce retries to MAX_PROCESSING_AGE
2024-04-26 13:01:53 +02:00
HaoranYi a0c256136c
fix hot_storage test for test_accountsdb_scan_account_storage_no_bank_one_slot (#1032)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-25 08:46:58 -05:00
Jeff Washington (jwash) 552559d056
get rid of AccountIter (#900) 2024-04-24 17:16:44 -05:00
Brooks e7452b7680
Replaces u128s when summing lamports in accounts hash calculation (#993) 2024-04-24 13:03:20 -04:00
Brooks d81ace8ff7
Removes AppendVec::file_name() (#976) 2024-04-24 10:09:04 -04:00
Jeff Washington (jwash) 45c49cedf2
get_unique_accounts_from_storage uses scan_index (#969)
* get_unique_accounts_from_storage uses scan_index

* avoid returning default pubkey=default accounts
2024-04-24 08:56:14 -05:00
Jeff Washington (jwash) 16f1f25dbb
cache storage lookup during shrink and ancient (#973)
* cache storage lookup during shrink and ancient

* rename and remove pub

* pr feedback
2024-04-23 11:51:37 -05:00
Jeff Washington (jwash) ca8f92e001
fix references on get_stored_account_meta_callback (#996)
* fix references on get_stored_account_meta_callback

* fmt
2024-04-23 11:51:03 -05:00
Jeff Washington (jwash) 57ab9449c7
remove multi slot test only StorableAccounts impl (#972) 2024-04-23 08:35:45 -05:00
Jeff Washington (jwash) 862c79e132
modify geyser account iter at snapshot load (#960) 2024-04-22 13:06:25 -05:00
Jeff Washington (jwash) 7389371112
rework secondary index generation (#959) 2024-04-22 12:59:47 -05:00
Jeff Washington (jwash) d4b86923ef
introduce AccountFromStorage for shrink (#881)
* introduce AccountFromStorage for shrink

* add comments and cleanup

* more comments

* unwrap to expect

* add comments
2024-04-22 12:20:18 -05:00
Brooks 422387982f
Removes unnecessary &Arc (#958) 2024-04-22 11:46:40 -04:00
Jeff Washington (jwash) 6239ece50d
remove unused StorableAccounts (#956) 2024-04-22 09:54:24 -05:00
Jeff Washington (jwash) 127faa3a07
shrinking and appending ancients use StorableAccountsBySlot (#935)
* shrinking and appending ancients use StorableAccountsBySlot

* snapshot minimizer uses StorableAccountsBySlot
2024-04-22 08:28:36 -05:00
Jeff Washington (jwash) 14d6c79c78
remove unused StorableAccountsMovingSlots (#952) 2024-04-22 05:05:05 -05:00
Jeff Washington (jwash) 8d4f2e6221
use scan_accounts to replace account_iter (#921) 2024-04-21 20:07:05 -05:00
Jeff Washington (jwash) 43e47bea28
remove a StorableAccounts impl (#936) 2024-04-21 20:03:55 -05:00
Alessandro Decina c2936ebb4a
accounts-db: send batches of accounts to the background hasher (#810)
Instead of sending accounts individually, send batches of accounts for
background hashing.

Before this change we used to send accounts individually, and the
background thread used to do:

    loop {
        let account = receiver.recv();
        account.hash();
        // go back to sleep in recv()
    }

Because most accounts are small and hashing them is very fast, the
background thread used to sleep a lot, and required many syscalls from
the sender in order to be woken up.

Batching reduces the number of syscalls.
2024-04-20 08:19:49 +10:00
Jeff Washington (jwash) fc2b05d50d
fix a few storage iter lifetimes (#917) 2024-04-19 17:10:43 -05:00
Brooks bbc0be2cb3
Removes returned Result from infallible fns (#914) 2024-04-19 20:44:15 +00:00
Jeff Washington (jwash) 1189055c98
load account lifetime improvements (#903)
* load account lifetime improvements

* restore 2 check_and_
2024-04-19 15:33:29 -05:00
Jeff Washington (jwash) e3d3cd6486
add appendvec.scan_accounts (#902) 2024-04-19 11:38:31 -05:00
Brooks 246dc481ca
Removes check_hash from CalcAccountsHashConfig (#913) 2024-04-19 15:43:04 +00:00
Jeff Washington (jwash) fb3bf55340
accounts_iter -> scan_pubkeys (#894) 2024-04-19 09:36:53 -05:00
Jeff Washington (jwash) f8a8d40ac7
test fixes to remove refs (#901) 2024-04-19 09:36:27 -05:00
HaoranYi 992a398fe8
accounts db/refactor accounts db test - test_partial_clean (#818)
* add hot storage test for test_partial_clean

* fix test_partial_clean for hot storage

* reviews

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-19 08:48:06 -05:00
Jeff Washington (jwash) f0e87f6210
replace one accounts iter with scan_pubkeys (#888) 2024-04-18 20:43:09 -05:00
Brooks 86667f5a6f
Removes check_hash from scan_snapshot_stores_with_cache() (#892) 2024-04-18 23:24:04 +00:00
Brooks 16770cce88
Removes tests that use check_hash (#893) 2024-04-18 22:32:03 +00:00
Brooks 4b9d18a361
Removes check_hash from ScanState (#891) 2024-04-18 22:05:10 +00:00
Brooks 54b816a338
Removes `check_hash` from calculate_accounts_hash_from_index() (#890) 2024-04-18 18:01:53 -04:00
Brooks ee113de4ce
Refactors stored accounts info (#887) 2024-04-18 15:45:44 -04:00
Alexander Meißner e43338f3d6
Refactor - Rename `LoadedProgram` to `ProgramCacheEntry` (#880)
LoadedProgramOwner => ProgramCacheEntryOwner.
LoadedProgramType => ProgramCacheEntryType.
LoadedProgram => ProgramCacheEntry.
LoadedProgramStats => ProgramCacheStats.
LoadedProgramMatchCriteria => ProgramCacheMatchCriteria.
LoadedProgramsForTxBatch => ProgramCacheForTxBatch.
2024-04-18 21:38:08 +02:00
Jeff Washington (jwash) 994ab254a2
some test tweaks (#882)
* some test tweaks

* get rid of some redundancy
2024-04-18 14:01:02 -05:00
Jeff Washington (jwash) d77118ac65
tests had invalid storage offset (#874) 2024-04-18 12:09:14 -05:00
HaoranYi 033bcfcab5
accounts db/refactor accounts db test - convert test_full_clean_refcount (#814)
* add full_clean_refcount tests for both account storage formats

* keep code comments

* fix test to match with code comments

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-17 16:42:44 -05:00
Brooks da0d78c504
Removes hash() and has_hash() from StorableAccounts (#867) 2024-04-17 16:41:41 -04:00
Brooks ee2c29ca2f
Removes unused hashes params from store functions (#863) 2024-04-17 16:37:45 -04:00
Brooks 613256e18f
Removes StorableAccountsWithHashes (#862) 2024-04-17 18:52:45 +00:00
HaoranYi 283f433a3b
accounts db/refactor accounts db test - add macro for accounts-db panic test (#786)
* add macro for accounts-db panic test
convert double remove test for both account file provider

* refactor to share accounts_db_test and accounts_db_panic_test macro

* rebase

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-17 12:52:00 -05:00
Brooks ad5b71dda8
Replaces StorableAccountsWithHashes in AccountsDb (#861) 2024-04-17 17:31:44 +00:00
Brooks 349eb7f925
Replaces StorableAccountsWithHashes in AccountsFile (#860) 2024-04-17 16:02:48 +00:00
Brooks 2aeac5b468
Replaces StorableAccountsWithHashes in TieredStorage (#859) 2024-04-17 14:49:24 +00:00
Brooks 97013978fe
Uses same lifetime name for local AccountForStorage in callback (#848) 2024-04-17 09:39:44 -04:00
Brooks 4d2d95fe14
Replaces StorableAccountsWithHashes in AppendVec (#853) 2024-04-17 09:34:03 -04:00
Jeff Washington (jwash) 63d4278cae
get_stored_account_meta_callback returns Option (#847)
* get_stored_account_meta_callback returns Option

* Update accounts-db/src/accounts_file.rs

Co-authored-by: Brooks <brooks@prumo.org>

---------

Co-authored-by: Brooks <brooks@prumo.org>
2024-04-17 08:12:32 -05:00
HaoranYi 03b596586c
Write default hash to account storage (#469)
* write default hash to apendvec

* fix get_pubkey_hash_for_slot

* fix tests

* fix rebase error

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-04-16 16:04:52 -05:00
Jeff Washington (jwash) a9548a6dc5
add `for<>` to callback on get fns (#838) 2024-04-16 14:55:50 -05:00