Commit Graph

723 Commits

Author SHA1 Message Date
Jeff Washington (jwash) 0732a6c45f
add 'oldest_root' stat in hash calc (#26896) 2022-08-03 10:02:08 -05:00
Jeff Washington (jwash) 6706be0f56
log oldest dirty slot on clean (#26897) 2022-08-03 10:01:29 -05:00
Jeff Washington (jwash) 224550d65f
allow caching on ancient append vecs for accounts hash calc (#26885) 2022-08-02 14:40:29 -05:00
Jeff Washington (jwash) acc1112e9c
calculate cache hash path for store_detailed_debug_info_on_failure (#26874) 2022-08-02 11:19:00 -05:00
Jeff Washington (jwash) e5c5055869
Revert "allow caching on ancient append vecs for accounts hash calc" (#26882)
Revert "allow caching on ancient append vecs for accounts hash calc (#26770)"

This reverts commit 30b408ed0c.
2022-08-02 10:23:11 -05:00
Jeff Washington (jwash) 050c66f197
enable accounts hash caching if store_detailed_debug_info_on_failure (#26864) 2022-08-01 14:27:20 -05:00
Jeff Washington (jwash) c7462b7a52
ledger tool verify can store debug info on hash calc (#26837) 2022-07-29 15:54:56 -05:00
Jeff Washington (jwash) 817f65bb50
add full_snapshot to hash config (#26811) 2022-07-28 09:46:34 -05:00
Jeff Washington (jwash) 1a5b830294
CalcAccountsHashConfig default for tests (#26810) 2022-07-27 13:21:38 -05:00
Jeff Washington (jwash) 1421833c5e
log when shrink still contains extra stores (#26794) 2022-07-27 09:14:16 -05:00
Jeff Washington (jwash) 30b408ed0c
allow caching on ancient append vecs for accounts hash calc (#26770) 2022-07-27 09:13:32 -05:00
HaoranYi 6ec89ee861
log error for cache_hash_data save failure (#26676)
log error for cach_hash_data save
2022-07-20 16:19:35 -05:00
Jeff Washington (jwash) 2d689ac53a
get rid of rwlock per bin in accounts index (#26675) 2022-07-19 17:48:40 -05:00
Jeff Washington (jwash) 263911e7fd
save off what we find when calculating hash (#26663) 2022-07-19 09:55:52 -05:00
Jeff Washington (jwash) 738c4a79b7
metric for 'ancient_account_cleans' (#26664) 2022-07-19 09:16:23 -05:00
Jeff Washington (jwash) c4d2c95540
move hash stat to correct level for ahv to work (#26649) 2022-07-18 12:44:33 -05:00
Jeff Washington (jwash) 1c08f83c7e
eliminate flatten().collect() of reclaims in clean (#26647) 2022-07-18 12:37:17 -05:00
Jeff Washington (jwash) be8813e588
shrink stops using HandleReclaims (#26594) 2022-07-15 17:54:48 -05:00
Jeff Washington (jwash) 83c614da3c
change type to be more clear (#26646) 2022-07-15 17:17:17 -05:00
Jeff Washington (jwash) 47716a5e01
async hash verify on load (#26208)
* verify accounts hash in bg on startup

* fix some tests and loading from genesis

* add extra state for when background thread has completed
2022-07-15 14:29:56 -05:00
Jeff Washington (jwash) 4dea32e8e5
fix active_scans metric to not swap (#26602) 2022-07-13 20:11:23 -05:00
Jeff Washington (jwash) 2b7cc107d8
typo (#26618) 2022-07-13 17:55:02 -05:00
HaoranYi 1aa9215411
Refactor: add account_iter api for appendvec (#26591)
* add account iterator for appendvec

* semicolon
2022-07-12 23:56:05 +00:00
Jeff Washington (jwash) 6b0eb5a42b
handle rent paying accounts differently (#26487)
* handle rent paying accounts differently

* restore collection

* downgrade assert to metric and warn until we get more runtime
2022-07-12 17:51:33 -05:00
Jeff Washington (jwash) c62c55a16b
shrink stores by id (#26490) 2022-07-12 15:43:41 -05:00
Jeff Washington (jwash) 9d31216d23
add StoreReclaims::Ignore (#26489) 2022-07-12 15:05:47 -05:00
Jeff Washington (jwash) 1db136a8be
do not count 0 lamport accounts as rent-paying (#26514) 2022-07-11 22:16:46 -05:00
Jeff Washington (jwash) 36bd46bece
ancient append vec: reuse AccountsToStore (#26572) 2022-07-11 22:13:09 -05:00
behzad nouri ba785cf8ab
removes erroneous uses of std::mem::swap (#26536)
All instances should be replace by std::mem::{replace,take},
or just plain assignment.
2022-07-11 11:33:15 +00:00
HaoranYi 995756a550
Use RefCount type alias instead of u64 (#26472)
use RefCount type alias instead of u64
2022-07-08 13:51:16 -05:00
Jeff Washington (jwash) d9eee72edb
remove FoundStoredAccount.account_size (#26460) 2022-07-07 18:37:14 -05:00
Jeff Washington (jwash) 38d53ef8a2
add metrics for # scans active and scan distance (#26395) 2022-07-07 16:20:32 -05:00
Jeff Washington (jwash) ee0a54ce80
improve clean acct idx calls (#26320) 2022-07-07 15:50:12 -05:00
Jeff Washington (jwash) b582e4ce0f
introduce UpsertReclaim (#26462) 2022-07-07 15:40:17 -05:00
Jeff Washington (jwash) c9f1d01e79
refactor, add GetUniqueAccountsResult (#26457) 2022-07-07 13:49:33 -05:00
Jeff Washington (jwash) 72256ac54d
use iter for append vecs (#26459) 2022-07-07 13:13:47 -05:00
Jeff Washington (jwash) b3a47de1ce
improve startup stats for rent payers (#26432) 2022-07-07 08:39:57 -05:00
Jeff Washington (jwash) 16219e300e
generating index sets uncleaned_roots correctly (#26431)
* generating index sets uncleaned_roots correctly

* fix test failures

* rename

* update comments
2022-07-06 16:12:47 -05:00
behzad nouri d7201a8d1a
names fields in RentResullt::CollectRent enum variant (#26449)
Avoiding ambiguous raw tuple:
    CollectRent((Epoch, u64))

Using named fields instead:
    CollectRent {
        new_rent_epoch: Epoch,
        rent_due: u64,
    },
2022-07-06 20:01:16 +00:00
Xiang Zhu 44f499cff3
Refactor account index scan function parameters (#26428)
* Refactor account index scan function parameters

F: FnMut(bool, &SlotList<T>, &'a Pubkey, RefCount) -> AccountsIndexScanResult,
Refactor to take  (&'a Pubkey, Option<(&SlotList<T>, RefCount))

* Update comments based on review feedback

* cargo fmt --all fixes
2022-07-06 09:49:11 -07:00
Jeff Washington (jwash) 17a99d98dd
Revert "avoid adding to 'uncleaned_roots' when generating index and c… (#26441)
Revert "avoid adding to 'uncleaned_roots' when generating index and caller passes accounts-db-skip-shrink (#25936)"

This reverts commit e24cc537a4.
2022-07-06 11:32:45 -05:00
Jeff Washington (jwash) ab164fc975
simplify code (#26443) 2022-07-06 11:22:26 -05:00
Jeff Washington (jwash) 5afe4d938d
combine lookup in calc_delete_dependencies (#26429) 2022-07-06 08:50:14 -05:00
Jeff Washington (jwash) 71c7b4f9cb
use append vec iter in construct_candidate_clean_keys (#26426) 2022-07-06 08:49:35 -05:00
Jeff Washington (jwash) 75149fd624
move mark_old_slots_as_dirty to be called from ahv (#26411) 2022-07-05 23:11:32 -05:00
Jeff Washington (jwash) 2aafef38f9
is_shrinking_productive calls avoid arc clone (#26422) 2022-07-05 17:53:29 -05:00
HaoranYi 728986ddf4
Spell correction and add remove slot in AccountDB purge assert message (#26358)
nit spell. add remove slots in accoutdb purge assert message
2022-07-01 17:39:35 +00:00
Jeff Washington (jwash) 557bf6e656
allow initial hash calc to occur in bg (#26271)
* allow initial hash calc to occur in bg

* validator_initialized -> startup_verification_complete

* add infos for leader and vote

* rework snapshot for startup verification

* change to assert
2022-06-29 16:48:33 -05:00
Jeff Washington (jwash) 608191352f
track # roots added/removed since last clean (#26319) 2022-06-29 16:47:57 -05:00
Jeff Washington (jwash) e241906db2
remove outdated, confusing comment (#26315) 2022-06-29 10:42:34 -05:00
Jeff Washington (jwash) ca83f143aa
load_accounts_index_for_shrink uses scan (#26260) 2022-06-29 09:02:27 -05:00
Jeff Washington (jwash) 89e5640c63
AccountsIndex::Scan takes iter (#26268) 2022-06-28 12:47:17 -05:00
Jeff Washington (jwash) bfc390f9ed
refactor acct idx scan func for reuse (#26252) 2022-06-28 09:01:04 -05:00
Jeff Washington (jwash) 4b591d7b59
AccountsDb::load_accounts_index_for_shrink takes slice (#26267) 2022-06-28 08:59:57 -05:00
Jeff Washington (jwash) e164879f20
load_accounts_index_for_shrink uses &item instead of (&item.0, &item.1) (#26266) 2022-06-27 22:40:45 -05:00
Jeff Washington (jwash) d6a28cb7a8
load_accounts_index_for_shrink unrefed_pubkeys is optional (#26265) 2022-06-27 17:23:46 -05:00
Jeff Washington (jwash) 9db44ce352
cleanup some AccountSharedData::from(Account (#26120) 2022-06-23 08:32:48 -05:00
apfitzge f4189c0305
ledger-tool minimized snapshots (#25334)
* working on local snapshot

* Parallelization for slot storage minimization

* Additional clean-up and fixes

* make --minimize an option of create-snapshot

* remove now unnecessary function

* Parallelize parts of minimized account set generation

* clippy fixes

* Add rent collection accounts and voting node_pubkeys

* Simplify programdata_accounts generation

* Loop over storages to get slot set

* Parallelize minimized slot set generation

* Parallelize adding owners and programdata_accounts

* Remove some now unncessary checks on the blockstore

* Add a warning for minimized snapshots across epoch boundary

* Simplify ledger-tool minimize

* Clarify names of bank's minimization helper functions

* Remove unnecesary funciton, fix line spacing

* Use DashSets instead of HashSets for minimized account and slot sets

* Filter storages uses all threads instead of thread_pool

* Add some additional comments on functions for minimization

* Moved more into bank and parallelized

* Update programs/bpf/Cargo.lock for dashmap in ledger

* Clippy fix

* ledger-tool: convert minimize_bank_for_snapshot Measure into measure!

* bank.rs: convert minimize_bank_for_snapshot Measure into measure!

* accounts_db.rs: convert minimize_accounts_db Measure into measure!

* accounts_db.rs: add comment about use of minimize_accounts_db

* ledger-tool: CLI argument clarification

* minimization functions: make infos unique

* bank.rs: Add test_get_rent_collection_accounts_between_slots

* bank.rs: Add test_minimization_add_vote_accounts

* bank.rs: Add test_minimization_add_stake_accounts

* bank.rs: Add test_minimization_add_owner_accounts

* bank.rs: Add test_minimization_add_programdata_accounts

* accounts_db.rs: Add test_minimize_accounts_db

* bank.rs: Add negative case and comments in test_get_rent_collection_accounts_between_slots

* bank.rs: Negative test in test_minimization_add_programdata_accounts

* use new static runtime and sdk ids

* bank comments to doc comments

* Only need to insert the maximum slot a key is found in

* rename remove_pubkeys to purge_pubkeys

* add comment on builtins::get_pubkeys

* prevent excessive logging of removed dead slots

* don't need to remove slot from shrink slot candidates

* blockstore.rs: get_accounts_used_in_range shouldn't return Result

* blockstore.rs: get_accounts_used_in_range: parallelize slot loop

* report filtering progress on time instead of count

* parallelize loop over snapshot storages

* WIP: move some bank minimization functionality into a new class

* WIP: move some accounts_db minimization functionality into SnapshotMinimizer

* WIP: Use new SnapshotMinimizer

* SnapshotMinimizer: fix use statements

* remove bank and accounts_db minimization code, where possible

* measure! doesn't take a closure

* fix use statement in blockstore

* log_dead_slots does not need pub(crate)

* get_unique_accounts_from_storages does not need pub(crate)

* different way to get stake accounts/nodes

* fix tests

* move rent collection account functionality to snapshot minimizer

* move accounts_db minimize behavior to snapshot minimizer

* clean up

* Use bank reference instead of Arc. Additional comments

* Add a comment to blockstore function

* Additional clarifying comments

* Moved all non-transaction account accumulation into the SnapshotMinimizer.

* transaction_account_set does not need to be mutable now

* Add comment about load_to_collect_rent_eagerly

* Update log_dead_slots comment

* remove duplicate measure/print of get_minimized_slot_set
2022-06-22 13:17:43 -04:00
Jeff Washington (jwash) d3829b1a76
store_cached takes StorableAccounts to eliminate collect (#26109) 2022-06-22 08:45:03 -05:00
Jeff Washington (jwash) b02c412d5b
generate_index straight to disk and batched (#25947)
* generate_index straight to disk and batched

* renames and comments

* handle in-mem case correctly

* use mutex
2022-06-15 18:14:39 -05:00
Jeff Washington (jwash) e24cc537a4
avoid adding to 'uncleaned_roots' when generating index and caller passes accounts-db-skip-shrink (#25936) 2022-06-14 10:10:44 -05:00
ivandzen 2251aa2809
Report transaction signature on update account (#25726)
There's no direct linkage between account-update events and transaction-events

Issue is linked with geyser plugin interface
Currently, there's no straightforward way to determine what notify_transaction call is related to particular update_account call. So, there's no simple way to determine what transaction caused what changes in accounts. This issue is especially valuable in case when several transactions modifies single account in the same slot.

Add txn_signature field to ReplicaTransactionInfo structure

This additional field will contain the signature of transaction which caused this account update event. This modification is not bringing significant overhead because all necessary information for such linkage already available inside Accounts::collect_accounts_to_store method
2022-06-10 10:24:35 -07:00
Jeff Washington (jwash) 8b0d5b2485
parallelize rent collection (#25767) 2022-06-03 18:00:02 -05:00
Jeff Washington (jwash) a9a94166ef
load_account_into_read_cache (#25760) 2022-06-03 13:34:17 -05:00
sakridge e6e2058758
Increase accounts stack size to 8mb (#25592) 2022-06-03 19:46:01 +02:00
carllin eeec3fb9c9
Remove unwrap (#25538) 2022-06-02 12:18:46 -05:00
Jeff Washington (jwash) d1a6cebaa2
u64 -> duration (#25708) 2022-06-02 07:56:47 -05:00
Jeff Washington (jwash) 149a54b786
remove par_iter on update index below threshold (#25699) 2022-06-01 17:41:56 -05:00
Jeff Washington (jwash) dd0bf4ff4b
add filler accounts gradually with cache flush (#25678) 2022-06-01 15:47:21 -05:00
Jeff Washington (jwash) 905fef29cb
add filter to AppendVecScan (#25664) 2022-06-01 11:55:18 -05:00
Jeff Washington (jwash) 94b0ce5d43
remove_old_historical_roots uses accounts_hash_complete_one_epoch_old (#25673) 2022-06-01 10:45:26 -05:00
Jeff Washington (jwash) c91fbd289f
set active state to shrink_ancient when doing intersting work (#25677) 2022-06-01 10:36:58 -05:00
Jeff Washington (jwash) 8bb76fcd87
add ancient metrics (#25656)
* add ancient metrics

* review feedback
2022-06-01 10:36:23 -05:00
Jeff Washington (jwash) 07958fd8ed
use self.init_accum (#25663) 2022-06-01 08:38:37 -05:00
Jeff Washington (jwash) 5994b4f53f
use object for hash scanning (#25418) 2022-05-31 11:51:56 -05:00
Jeff Washington (jwash) bac05dc55a
calc slot info on max slot once per hash calc (#25422) 2022-05-23 10:11:00 -05:00
Jeff Washington (jwash) 41f30a2383
stop logging misleading bank hash mismatch (#25427) 2022-05-23 08:43:25 -05:00
Michael Vines b05c7d91ed Fix derive_partial_eq_without_eq clippy lint 2022-05-22 22:22:21 -07:00
Jeff Washington (jwash) f584b249dd
special case for common case of 1 storage on append vec scan (#25414) 2022-05-20 17:36:30 -05:00
Jeff Washington (jwash) cfcc18f7f4
apply AccountsInAppendVec to more places (#25408) 2022-05-20 14:25:46 -05:00
Jeff Washington (jwash) 467431de89
add ledger-tool accounts-db-skip-initial-hash-calc (#25354) 2022-05-20 10:27:00 -05:00
Jeff Washington (jwash) 950245c29b
simulate filler account hash calc time at delta hash (#25387) 2022-05-20 10:20:44 -05:00
Jeff Washington (jwash) 1b4e4ef548
impl squash to ancient append vecs (#24538)
* squash to ancient append vecs

* pr feedback

* clippy
2022-05-20 10:20:17 -05:00
Jeff Washington (jwash) de03a3d3aa
filler accounts get a non-zero rent_epoch (#25386)
* fill accounts get a non-zero rent_epoch

* use snapshot_slot

* get rid of perf simulation code
2022-05-20 09:03:55 -05:00
Jeff Washington (jwash) 3fe7655378
use iterator instead of vec to iterate accounts (#25370)
* user iterator instead of vec to iterate accounts

* rename
2022-05-20 08:50:24 -05:00
Jeff Washington (jwash) 9ccdc8a050
populate ancient append vec metrics (#25204)
* populate ancient append vec metrics

* count ancient storages while enumerating storages anyway.
2022-05-13 18:20:15 -05:00
Jeff Washington (jwash) cb037e49ea
is_candidate_for_shrink respects ancient append vecs (#25205) 2022-05-13 16:26:46 -05:00
apfitzge 5eeb07fead
coalesce purge stats and reclaim result (#25183) 2022-05-13 12:34:11 -05:00
Jeff Washington (jwash) 6f1cc5a2a7
assert we haven't squashed to ancient after hash calc (#25168) 2022-05-12 18:26:01 -05:00
Jeff Washington (jwash) 896729f25e
keep track of oldest slot used by last hash calculation (#25152) 2022-05-12 11:18:08 -05:00
Jeff Washington (jwash) abb4993a3b
SortedStorages::max_slot_inclusive() (#25153) 2022-05-12 10:24:03 -05:00
Jeff Washington (jwash) c56d978cb1
minor cleanup (#25144) 2022-05-11 14:58:43 -05:00
Jeff Washington (jwash) a8930ee14b
add --accounts-db-ancient-append-vecs (#25125) 2022-05-11 08:47:07 -05:00
Jeff Washington (jwash) c4506269a6
add ancient_append_vecs (#25119) 2022-05-10 16:41:04 -05:00
Jeff Washington (jwash) dc6e28099b
mark_dirty_dead_stores (#25114) 2022-05-10 15:08:29 -05:00
Jeff Washington (jwash) e9e267947e
simplify storage_finder -> storage (#25109) 2022-05-10 14:18:51 -05:00
Jeff Washington (jwash) e9a9604ca7
avoid zeroing out root stats with empty data (#25111) 2022-05-10 13:56:30 -05:00
Jeff Washington (jwash) 2d7ce2a6c2
Revert "uses designated thread-pools for account db parallel ops (#24954)" (#25053)
This reverts commit e8bdc27080.
2022-05-06 15:20:50 -05:00
Jeff Washington (jwash) 7401bc0fa5
remove_old_historical_roots uses correct epoch schedule (#25020)
* remove_old_historical_roots uses correct epoch schedule

* Update runtime/src/accounts_db.rs

Co-authored-by: apfitzge <apfitzge@users.noreply.github.com>

Co-authored-by: apfitzge <apfitzge@users.noreply.github.com>
2022-05-06 12:38:09 -05:00
Jeff Washington (jwash) 0cc97689f9
add stats for stakes_cache_check_and_store_us (#24349)
* add stats for stakes_cache_check_and_store_us

* remove redundant metrics
2022-05-05 17:22:48 -05:00