Commit Graph

760 Commits

Author SHA1 Message Date
Jeff Washington (jwash) 263911e7fd
save off what we find when calculating hash (#26663) 2022-07-19 09:55:52 -05:00
Jeff Washington (jwash) 738c4a79b7
metric for 'ancient_account_cleans' (#26664) 2022-07-19 09:16:23 -05:00
Jeff Washington (jwash) c4d2c95540
move hash stat to correct level for ahv to work (#26649) 2022-07-18 12:44:33 -05:00
Jeff Washington (jwash) 1c08f83c7e
eliminate flatten().collect() of reclaims in clean (#26647) 2022-07-18 12:37:17 -05:00
Jeff Washington (jwash) be8813e588
shrink stops using HandleReclaims (#26594) 2022-07-15 17:54:48 -05:00
Jeff Washington (jwash) 83c614da3c
change type to be more clear (#26646) 2022-07-15 17:17:17 -05:00
Jeff Washington (jwash) 47716a5e01
async hash verify on load (#26208)
* verify accounts hash in bg on startup

* fix some tests and loading from genesis

* add extra state for when background thread has completed
2022-07-15 14:29:56 -05:00
Jeff Washington (jwash) 4dea32e8e5
fix active_scans metric to not swap (#26602) 2022-07-13 20:11:23 -05:00
Jeff Washington (jwash) 2b7cc107d8
typo (#26618) 2022-07-13 17:55:02 -05:00
HaoranYi 1aa9215411
Refactor: add account_iter api for appendvec (#26591)
* add account iterator for appendvec

* semicolon
2022-07-12 23:56:05 +00:00
Jeff Washington (jwash) 6b0eb5a42b
handle rent paying accounts differently (#26487)
* handle rent paying accounts differently

* restore collection

* downgrade assert to metric and warn until we get more runtime
2022-07-12 17:51:33 -05:00
Jeff Washington (jwash) c62c55a16b
shrink stores by id (#26490) 2022-07-12 15:43:41 -05:00
Jeff Washington (jwash) 9d31216d23
add StoreReclaims::Ignore (#26489) 2022-07-12 15:05:47 -05:00
Jeff Washington (jwash) 1db136a8be
do not count 0 lamport accounts as rent-paying (#26514) 2022-07-11 22:16:46 -05:00
Jeff Washington (jwash) 36bd46bece
ancient append vec: reuse AccountsToStore (#26572) 2022-07-11 22:13:09 -05:00
behzad nouri ba785cf8ab
removes erroneous uses of std::mem::swap (#26536)
All instances should be replace by std::mem::{replace,take},
or just plain assignment.
2022-07-11 11:33:15 +00:00
HaoranYi 995756a550
Use RefCount type alias instead of u64 (#26472)
use RefCount type alias instead of u64
2022-07-08 13:51:16 -05:00
Jeff Washington (jwash) d9eee72edb
remove FoundStoredAccount.account_size (#26460) 2022-07-07 18:37:14 -05:00
Jeff Washington (jwash) 38d53ef8a2
add metrics for # scans active and scan distance (#26395) 2022-07-07 16:20:32 -05:00
Jeff Washington (jwash) ee0a54ce80
improve clean acct idx calls (#26320) 2022-07-07 15:50:12 -05:00
Jeff Washington (jwash) b582e4ce0f
introduce UpsertReclaim (#26462) 2022-07-07 15:40:17 -05:00
Jeff Washington (jwash) c9f1d01e79
refactor, add GetUniqueAccountsResult (#26457) 2022-07-07 13:49:33 -05:00
Jeff Washington (jwash) 72256ac54d
use iter for append vecs (#26459) 2022-07-07 13:13:47 -05:00
Jeff Washington (jwash) b3a47de1ce
improve startup stats for rent payers (#26432) 2022-07-07 08:39:57 -05:00
Jeff Washington (jwash) 16219e300e
generating index sets uncleaned_roots correctly (#26431)
* generating index sets uncleaned_roots correctly

* fix test failures

* rename

* update comments
2022-07-06 16:12:47 -05:00
behzad nouri d7201a8d1a
names fields in RentResullt::CollectRent enum variant (#26449)
Avoiding ambiguous raw tuple:
    CollectRent((Epoch, u64))

Using named fields instead:
    CollectRent {
        new_rent_epoch: Epoch,
        rent_due: u64,
    },
2022-07-06 20:01:16 +00:00
Xiang Zhu 44f499cff3
Refactor account index scan function parameters (#26428)
* Refactor account index scan function parameters

F: FnMut(bool, &SlotList<T>, &'a Pubkey, RefCount) -> AccountsIndexScanResult,
Refactor to take  (&'a Pubkey, Option<(&SlotList<T>, RefCount))

* Update comments based on review feedback

* cargo fmt --all fixes
2022-07-06 09:49:11 -07:00
Jeff Washington (jwash) 17a99d98dd
Revert "avoid adding to 'uncleaned_roots' when generating index and c… (#26441)
Revert "avoid adding to 'uncleaned_roots' when generating index and caller passes accounts-db-skip-shrink (#25936)"

This reverts commit e24cc537a4.
2022-07-06 11:32:45 -05:00
Jeff Washington (jwash) ab164fc975
simplify code (#26443) 2022-07-06 11:22:26 -05:00
Jeff Washington (jwash) 5afe4d938d
combine lookup in calc_delete_dependencies (#26429) 2022-07-06 08:50:14 -05:00
Jeff Washington (jwash) 71c7b4f9cb
use append vec iter in construct_candidate_clean_keys (#26426) 2022-07-06 08:49:35 -05:00
Jeff Washington (jwash) 75149fd624
move mark_old_slots_as_dirty to be called from ahv (#26411) 2022-07-05 23:11:32 -05:00
Jeff Washington (jwash) 2aafef38f9
is_shrinking_productive calls avoid arc clone (#26422) 2022-07-05 17:53:29 -05:00
HaoranYi 728986ddf4
Spell correction and add remove slot in AccountDB purge assert message (#26358)
nit spell. add remove slots in accoutdb purge assert message
2022-07-01 17:39:35 +00:00
Jeff Washington (jwash) 557bf6e656
allow initial hash calc to occur in bg (#26271)
* allow initial hash calc to occur in bg

* validator_initialized -> startup_verification_complete

* add infos for leader and vote

* rework snapshot for startup verification

* change to assert
2022-06-29 16:48:33 -05:00
Jeff Washington (jwash) 608191352f
track # roots added/removed since last clean (#26319) 2022-06-29 16:47:57 -05:00
Jeff Washington (jwash) e241906db2
remove outdated, confusing comment (#26315) 2022-06-29 10:42:34 -05:00
Jeff Washington (jwash) ca83f143aa
load_accounts_index_for_shrink uses scan (#26260) 2022-06-29 09:02:27 -05:00
Jeff Washington (jwash) 89e5640c63
AccountsIndex::Scan takes iter (#26268) 2022-06-28 12:47:17 -05:00
Jeff Washington (jwash) bfc390f9ed
refactor acct idx scan func for reuse (#26252) 2022-06-28 09:01:04 -05:00
Jeff Washington (jwash) 4b591d7b59
AccountsDb::load_accounts_index_for_shrink takes slice (#26267) 2022-06-28 08:59:57 -05:00
Jeff Washington (jwash) e164879f20
load_accounts_index_for_shrink uses &item instead of (&item.0, &item.1) (#26266) 2022-06-27 22:40:45 -05:00
Jeff Washington (jwash) d6a28cb7a8
load_accounts_index_for_shrink unrefed_pubkeys is optional (#26265) 2022-06-27 17:23:46 -05:00
Jeff Washington (jwash) 9db44ce352
cleanup some AccountSharedData::from(Account (#26120) 2022-06-23 08:32:48 -05:00
apfitzge f4189c0305
ledger-tool minimized snapshots (#25334)
* working on local snapshot

* Parallelization for slot storage minimization

* Additional clean-up and fixes

* make --minimize an option of create-snapshot

* remove now unnecessary function

* Parallelize parts of minimized account set generation

* clippy fixes

* Add rent collection accounts and voting node_pubkeys

* Simplify programdata_accounts generation

* Loop over storages to get slot set

* Parallelize minimized slot set generation

* Parallelize adding owners and programdata_accounts

* Remove some now unncessary checks on the blockstore

* Add a warning for minimized snapshots across epoch boundary

* Simplify ledger-tool minimize

* Clarify names of bank's minimization helper functions

* Remove unnecesary funciton, fix line spacing

* Use DashSets instead of HashSets for minimized account and slot sets

* Filter storages uses all threads instead of thread_pool

* Add some additional comments on functions for minimization

* Moved more into bank and parallelized

* Update programs/bpf/Cargo.lock for dashmap in ledger

* Clippy fix

* ledger-tool: convert minimize_bank_for_snapshot Measure into measure!

* bank.rs: convert minimize_bank_for_snapshot Measure into measure!

* accounts_db.rs: convert minimize_accounts_db Measure into measure!

* accounts_db.rs: add comment about use of minimize_accounts_db

* ledger-tool: CLI argument clarification

* minimization functions: make infos unique

* bank.rs: Add test_get_rent_collection_accounts_between_slots

* bank.rs: Add test_minimization_add_vote_accounts

* bank.rs: Add test_minimization_add_stake_accounts

* bank.rs: Add test_minimization_add_owner_accounts

* bank.rs: Add test_minimization_add_programdata_accounts

* accounts_db.rs: Add test_minimize_accounts_db

* bank.rs: Add negative case and comments in test_get_rent_collection_accounts_between_slots

* bank.rs: Negative test in test_minimization_add_programdata_accounts

* use new static runtime and sdk ids

* bank comments to doc comments

* Only need to insert the maximum slot a key is found in

* rename remove_pubkeys to purge_pubkeys

* add comment on builtins::get_pubkeys

* prevent excessive logging of removed dead slots

* don't need to remove slot from shrink slot candidates

* blockstore.rs: get_accounts_used_in_range shouldn't return Result

* blockstore.rs: get_accounts_used_in_range: parallelize slot loop

* report filtering progress on time instead of count

* parallelize loop over snapshot storages

* WIP: move some bank minimization functionality into a new class

* WIP: move some accounts_db minimization functionality into SnapshotMinimizer

* WIP: Use new SnapshotMinimizer

* SnapshotMinimizer: fix use statements

* remove bank and accounts_db minimization code, where possible

* measure! doesn't take a closure

* fix use statement in blockstore

* log_dead_slots does not need pub(crate)

* get_unique_accounts_from_storages does not need pub(crate)

* different way to get stake accounts/nodes

* fix tests

* move rent collection account functionality to snapshot minimizer

* move accounts_db minimize behavior to snapshot minimizer

* clean up

* Use bank reference instead of Arc. Additional comments

* Add a comment to blockstore function

* Additional clarifying comments

* Moved all non-transaction account accumulation into the SnapshotMinimizer.

* transaction_account_set does not need to be mutable now

* Add comment about load_to_collect_rent_eagerly

* Update log_dead_slots comment

* remove duplicate measure/print of get_minimized_slot_set
2022-06-22 13:17:43 -04:00
Jeff Washington (jwash) d3829b1a76
store_cached takes StorableAccounts to eliminate collect (#26109) 2022-06-22 08:45:03 -05:00
Jeff Washington (jwash) b02c412d5b
generate_index straight to disk and batched (#25947)
* generate_index straight to disk and batched

* renames and comments

* handle in-mem case correctly

* use mutex
2022-06-15 18:14:39 -05:00
Jeff Washington (jwash) e24cc537a4
avoid adding to 'uncleaned_roots' when generating index and caller passes accounts-db-skip-shrink (#25936) 2022-06-14 10:10:44 -05:00
ivandzen 2251aa2809
Report transaction signature on update account (#25726)
There's no direct linkage between account-update events and transaction-events

Issue is linked with geyser plugin interface
Currently, there's no straightforward way to determine what notify_transaction call is related to particular update_account call. So, there's no simple way to determine what transaction caused what changes in accounts. This issue is especially valuable in case when several transactions modifies single account in the same slot.

Add txn_signature field to ReplicaTransactionInfo structure

This additional field will contain the signature of transaction which caused this account update event. This modification is not bringing significant overhead because all necessary information for such linkage already available inside Accounts::collect_accounts_to_store method
2022-06-10 10:24:35 -07:00
Jeff Washington (jwash) 8b0d5b2485
parallelize rent collection (#25767) 2022-06-03 18:00:02 -05:00
Jeff Washington (jwash) a9a94166ef
load_account_into_read_cache (#25760) 2022-06-03 13:34:17 -05:00
sakridge e6e2058758
Increase accounts stack size to 8mb (#25592) 2022-06-03 19:46:01 +02:00
carllin eeec3fb9c9
Remove unwrap (#25538) 2022-06-02 12:18:46 -05:00
Jeff Washington (jwash) d1a6cebaa2
u64 -> duration (#25708) 2022-06-02 07:56:47 -05:00
Jeff Washington (jwash) 149a54b786
remove par_iter on update index below threshold (#25699) 2022-06-01 17:41:56 -05:00
Jeff Washington (jwash) dd0bf4ff4b
add filler accounts gradually with cache flush (#25678) 2022-06-01 15:47:21 -05:00
Jeff Washington (jwash) 905fef29cb
add filter to AppendVecScan (#25664) 2022-06-01 11:55:18 -05:00
Jeff Washington (jwash) 94b0ce5d43
remove_old_historical_roots uses accounts_hash_complete_one_epoch_old (#25673) 2022-06-01 10:45:26 -05:00
Jeff Washington (jwash) c91fbd289f
set active state to shrink_ancient when doing intersting work (#25677) 2022-06-01 10:36:58 -05:00
Jeff Washington (jwash) 8bb76fcd87
add ancient metrics (#25656)
* add ancient metrics

* review feedback
2022-06-01 10:36:23 -05:00
Jeff Washington (jwash) 07958fd8ed
use self.init_accum (#25663) 2022-06-01 08:38:37 -05:00
Jeff Washington (jwash) 5994b4f53f
use object for hash scanning (#25418) 2022-05-31 11:51:56 -05:00
Jeff Washington (jwash) bac05dc55a
calc slot info on max slot once per hash calc (#25422) 2022-05-23 10:11:00 -05:00
Jeff Washington (jwash) 41f30a2383
stop logging misleading bank hash mismatch (#25427) 2022-05-23 08:43:25 -05:00
Michael Vines b05c7d91ed Fix derive_partial_eq_without_eq clippy lint 2022-05-22 22:22:21 -07:00
Jeff Washington (jwash) f584b249dd
special case for common case of 1 storage on append vec scan (#25414) 2022-05-20 17:36:30 -05:00
Jeff Washington (jwash) cfcc18f7f4
apply AccountsInAppendVec to more places (#25408) 2022-05-20 14:25:46 -05:00
Jeff Washington (jwash) 467431de89
add ledger-tool accounts-db-skip-initial-hash-calc (#25354) 2022-05-20 10:27:00 -05:00
Jeff Washington (jwash) 950245c29b
simulate filler account hash calc time at delta hash (#25387) 2022-05-20 10:20:44 -05:00
Jeff Washington (jwash) 1b4e4ef548
impl squash to ancient append vecs (#24538)
* squash to ancient append vecs

* pr feedback

* clippy
2022-05-20 10:20:17 -05:00
Jeff Washington (jwash) de03a3d3aa
filler accounts get a non-zero rent_epoch (#25386)
* fill accounts get a non-zero rent_epoch

* use snapshot_slot

* get rid of perf simulation code
2022-05-20 09:03:55 -05:00
Jeff Washington (jwash) 3fe7655378
use iterator instead of vec to iterate accounts (#25370)
* user iterator instead of vec to iterate accounts

* rename
2022-05-20 08:50:24 -05:00
Jeff Washington (jwash) 9ccdc8a050
populate ancient append vec metrics (#25204)
* populate ancient append vec metrics

* count ancient storages while enumerating storages anyway.
2022-05-13 18:20:15 -05:00
Jeff Washington (jwash) cb037e49ea
is_candidate_for_shrink respects ancient append vecs (#25205) 2022-05-13 16:26:46 -05:00
apfitzge 5eeb07fead
coalesce purge stats and reclaim result (#25183) 2022-05-13 12:34:11 -05:00
Jeff Washington (jwash) 6f1cc5a2a7
assert we haven't squashed to ancient after hash calc (#25168) 2022-05-12 18:26:01 -05:00
Jeff Washington (jwash) 896729f25e
keep track of oldest slot used by last hash calculation (#25152) 2022-05-12 11:18:08 -05:00
Jeff Washington (jwash) abb4993a3b
SortedStorages::max_slot_inclusive() (#25153) 2022-05-12 10:24:03 -05:00
Jeff Washington (jwash) c56d978cb1
minor cleanup (#25144) 2022-05-11 14:58:43 -05:00
Jeff Washington (jwash) a8930ee14b
add --accounts-db-ancient-append-vecs (#25125) 2022-05-11 08:47:07 -05:00
Jeff Washington (jwash) c4506269a6
add ancient_append_vecs (#25119) 2022-05-10 16:41:04 -05:00
Jeff Washington (jwash) dc6e28099b
mark_dirty_dead_stores (#25114) 2022-05-10 15:08:29 -05:00
Jeff Washington (jwash) e9e267947e
simplify storage_finder -> storage (#25109) 2022-05-10 14:18:51 -05:00
Jeff Washington (jwash) e9a9604ca7
avoid zeroing out root stats with empty data (#25111) 2022-05-10 13:56:30 -05:00
Jeff Washington (jwash) 2d7ce2a6c2
Revert "uses designated thread-pools for account db parallel ops (#24954)" (#25053)
This reverts commit e8bdc27080.
2022-05-06 15:20:50 -05:00
Jeff Washington (jwash) 7401bc0fa5
remove_old_historical_roots uses correct epoch schedule (#25020)
* remove_old_historical_roots uses correct epoch schedule

* Update runtime/src/accounts_db.rs

Co-authored-by: apfitzge <apfitzge@users.noreply.github.com>

Co-authored-by: apfitzge <apfitzge@users.noreply.github.com>
2022-05-06 12:38:09 -05:00
Jeff Washington (jwash) 0cc97689f9
add stats for stakes_cache_check_and_store_us (#24349)
* add stats for stakes_cache_check_and_store_us

* remove redundant metrics
2022-05-05 17:22:48 -05:00
behzad nouri e8bdc27080
uses designated thread-pools for account db parallel ops (#24954)
Instead of global rayon thread-pool, the commit uses
AccountsDb::{thread_pool,thread_pool_clean} for parallel ops in
AccountsDB::{update_index,clean_accounts_older_than_root}.
2022-05-04 22:47:12 +00:00
Jeff Washington (jwash) e8bf812a8a
allow ledger-tool halt at slot to calc hash using write cache (#24974) 2022-05-04 16:27:49 -05:00
Jeff Washington (jwash) 40986daddf
log write cache contents from purge_slots_from_cache_and_store (#24948) 2022-05-04 14:28:47 -05:00
Jeff Washington (jwash) 755100dd7c
use correct epoch schedule (#24904) 2022-05-03 09:21:47 -05:00
apfitzge 112a0b475a
Revert "Refactor to use EpochSchedule from within RentCollector struct" (#24893)
* Revert "Ran cargo fmt"

This reverts commit 9052e41b32.

* Revert "Fix build error introduced by my editor setup, part 2"

This reverts commit 4dfeab3b38.

* Revert "Fix build error introduced by my editor setup"

This reverts commit 87fb78dc56.

* Revert "Remove redundant epoch_schedule from AccountsPackage"

This reverts commit c2f7f2fff8.

* Revert "Fix a test"

This reverts commit 36c0bdaa78.

* Revert "Fixes to initial code"

This reverts commit ed7813e698.

* Revert "Removing redundant EpochSchedule param from fns"

This reverts commit 5472d2e605.
2022-05-02 13:46:17 -05:00
Jeff Washington (jwash) 099258e4eb
some cleanup (#24568) 2022-04-25 13:17:30 -05:00
carllin 8a062273de
Move error counters to be reported by leader only at end of slot (#24581)
* Add error counters to leader metrics only

* Add dependencies
2022-04-23 18:10:47 -05:00
Jeff Washington (jwash) ce7d2964c9
metrics on ancient slots (#24596) 2022-04-22 15:01:51 -05:00
Jeff Washington (jwash) 25e9199397
cleanup FlushStats (#24556) 2022-04-21 14:05:01 -05:00
Jeff Washington (jwash) 05438c3cd6
implement remove_old_historical_roots (#24492) 2022-04-20 10:43:32 -05:00
Jeff Washington (jwash) 0d797e2fff
contains_multiple_slots (#24500) 2022-04-20 09:53:36 -05:00
Jeff Washington (jwash) cfe2177e16
validator option: accounts-db-skip-rewrites (#24504) 2022-04-20 09:41:00 -05:00
Jeff Washington (jwash) 255a6a729d
add historical_roots_len stat (#24497) 2022-04-19 16:53:24 -05:00
Jeff Washington (jwash) d23c04bb68
max_slot_in_storages_exclusive -> INclusive (#24450) 2022-04-19 13:59:41 -05:00
Jeff Washington (jwash) c4923d29b4
bank_hash_at uses rewrites (#24439) 2022-04-19 11:29:29 -05:00
HaoranYi 4fdf966393
demote shink counter to debug (#24458) 2022-04-19 10:02:23 -05:00
Jeff Washington (jwash) dc98510d6d
accounts hash calls maybe_rehash_skipped_rewrite (#24316) 2022-04-19 08:29:09 -05:00
Jeff Washington (jwash) 0e0f306996
mechanism for disabling hash calc caching (#24437) 2022-04-18 22:44:30 -05:00
Jason Davis 9052e41b32 Ran cargo fmt 2022-04-18 11:57:40 -05:00
Jason Davis 87fb78dc56 Fix build error introduced by my editor setup 2022-04-18 11:57:40 -05:00
Jason Davis 5472d2e605 Removing redundant EpochSchedule param from fns 2022-04-18 11:57:40 -05:00
Jeff Washington (jwash) b4fd9124bf
log secondary index contents on startup (#24348) 2022-04-15 13:30:03 -05:00
Jeff Washington (jwash) d20b4c9958
rename function param to avoid conflict (#24342) 2022-04-14 11:56:32 -05:00
HaoranYi e3ef0741be
simplify bank drop calls (#24142)
* simplify bank drop calls

* clippy: import

* Update ledger/src/blockstore_processor.rs

Co-authored-by: Brooks Prumo <brooks@prumo.org>

* Update runtime/src/accounts_background_service.rs

Co-authored-by: Brooks Prumo <brooks@prumo.org>

* Update runtime/src/bank.rs

Co-authored-by: Brooks Prumo <brooks@prumo.org>

* cleanup

* format

* use msb of bank_id to indicates that we are dropping

* clippy

* restore bank id

* clippy

* revert is-serialized_with_abs flag

* assert

* clippy

* whitespace

* fix bank drop callback check

* more fix

* remove msb dropping implementation

* fix

Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-14 08:43:54 -05:00
Jeff Washington (jwash) 255ef66a27
move feature activation log to correct fn for hash (#24326) 2022-04-13 17:43:33 -05:00
HaoranYi 929753a11f
typo (#24291) 2022-04-12 16:46:59 -05:00
Brooks Prumo 4fd184c131
Use Release/Acquire instead of SeqCst for is_bank_drop_callback_enabled (#24134) 2022-04-11 19:19:17 -05:00
Jeff Washington (jwash) 6fbe2b936c
fix comment (#24254) 2022-04-11 18:53:45 -05:00
Giorgio Gambino 60b2155bd3
Add accounts-filler-size command line option (#23896) 2022-04-11 13:10:09 -05:00
Jeff Washington (jwash) 210f6a6fab
move hash calculation out of acct bg svc (#23689)
* move hash calculation out of acct bg svc

* pr feedback
2022-04-08 10:42:03 -05:00
Jeff Washington (jwash) c27150b1a3
reserialize_bank_fields_with_hash (#23916)
* reserialize_bank_with_new_accounts_hash

* Update runtime/src/serde_snapshot.rs

Co-authored-by: Brooks Prumo <brooks@prumo.org>

* Update runtime/src/serde_snapshot/tests.rs

Co-authored-by: Brooks Prumo <brooks@prumo.org>

* Update runtime/src/serde_snapshot/tests.rs

Co-authored-by: Brooks Prumo <brooks@prumo.org>

* pr feedback

Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-07 14:05:57 -05:00
Jeff Washington (jwash) 4a11fa072f
hash_account_with_rent_epoch (#24104) 2022-04-05 08:10:31 -05:00
Jeff Washington (jwash) 132f08486a
remove basically duplicate function (#24107) 2022-04-04 12:55:05 -05:00
Jeff Washington (jwash) ec97d6d078
rename remove_old_roots (#24059) 2022-04-02 12:01:13 -05:00
Jeff Washington (jwash) 3ca4fffa78
root -> alive_root (#24062) 2022-04-02 12:00:52 -05:00
Jeff Washington (jwash) 31997f8251
hash calc scanning takes config (#24016) 2022-03-31 14:26:37 -05:00
Jeff Washington (jwash) 9c8dad33c7
add epoch_schedule and rent_collector to hash calc (#24012) 2022-03-31 10:51:18 -05:00
Jeff Washington (jwash) da001d54e5
calculate_accounts_hash_helper uses config (#24003) 2022-03-31 09:29:45 -05:00
Jeff Washington (jwash) 125f9634fd
add hash calc config.use_write_cache (#24005) 2022-03-30 17:19:34 -05:00
Jeff Washington (jwash) af9344fe22
SortedStorages refactoring (#23998) 2022-03-30 09:19:03 -05:00
Jeff Washington (jwash) 5636570d6d
add roots_original to roots tracker (#23849) 2022-03-30 08:52:45 -05:00
Jeff Washington (jwash) 5a613e9b6e
use CalcAccountsHashConfig in calculate_accounts_hash (#23987) 2022-03-29 22:05:47 -05:00
Jeff Washington (jwash) 37c36ce3fa
pass stats separately from CalcAccountsHashConfig (#23892) 2022-03-24 12:48:47 -05:00
Jeff Washington (jwash) 82328fd9d8
move max_clean_root deeper in flush cache (#23869) 2022-03-24 12:45:49 -05:00
Jeff Washington (jwash) a22a2384bf
fix ci test error (#23908) 2022-03-24 11:30:20 -05:00
Jeff Washington (jwash) 5b916961b5
HashCalc uses self.accounts_cache (#23890) 2022-03-24 10:34:28 -05:00
Jeff Washington (jwash) f2aea3b7c7
flush_slot_cache takes [Slot] (#23865) 2022-03-24 10:24:36 -05:00
Jeff Washington (jwash) 9d3b17c635
HashCalc uses self.accounts_index (#23888) 2022-03-24 10:06:32 -05:00
Jeff Washington (jwash) b22165ad69
hash calc uses self.filler_account_suffix (#23887) 2022-03-24 09:58:06 -05:00
Jeff Washington (jwash) 9022931689
calc hash uses self.num_hash_scan_passes (#23883) 2022-03-24 09:44:42 -05:00
Jeff Washington (jwash) e3eb002f66
Log storage size stats at hash calc (#23843) 2022-03-24 09:40:35 -05:00
Jeff Washington (jwash) f1a411c897
add epoch_schedule and rent_collector to hash calc (#23857) 2022-03-24 09:39:22 -05:00
Jeff Washington (jwash) db5d68f01f
HashCalc uses self.accounts_hash_cache_path (#23882) 2022-03-24 09:31:55 -05:00
HaoranYi 90009f330b
small refactor to shorten the lock on slot_under_contention hashset (#23891)
* small refactor to shorten the lock on slot_under_contention hashset

* adding comments

* comments
2022-03-24 08:20:56 -05:00
Jeff Washington (jwash) 5a892af2fe
disable 'check_hash' on accounts hash calc (#23873) 2022-03-23 21:03:31 -05:00
Jeff Washington (jwash) 3e22d4b286
calc hash uses self.thread_pool_clean (#23881) 2022-03-23 20:52:38 -05:00
Jeff Washington (jwash) dc3863ef14
flush_slot_cache_with_clean (#23868) 2022-03-23 14:09:56 -05:00
Jeff Washington (jwash) 260f899eda
write cache: hashmap to set (#23866) 2022-03-23 14:05:45 -05:00
Jeff Washington (jwash) 9e61fe7583
add AccountsHashConfig to manage parameters (#23850) 2022-03-23 13:44:23 -05:00
Jeff Washington (jwash) b1280b670a
calculate_accounts_hash_without_index takes &self (#23846)
* calculate_accounts_hash_without_index takes &self

* Update runtime/src/snapshot_package.rs

Co-authored-by: Brooks Prumo <brooks@prumo.org>

Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-03-23 11:57:32 -05:00
Jeff Washington (jwash) 7b89222fde
don't start extra threads for shrink/clean/hash (#23858) 2022-03-23 11:53:37 -05:00
Jeff Washington (jwash) 493a8e2348
remove random flushing of write cache (#23845) 2022-03-23 08:45:44 -05:00
Jeff Washington (jwash) dd69f3baf5
throttle index adding to allow disk flushing to keep up and reduce startup ram usage (#23773) 2022-03-20 19:56:20 -05:00