765c628546
use exit signal for acct idx bg threads ( #27483 )
2022-09-12 11:51:12 -07:00
bd6177d5f2
move fn only used by tests to tests ( #27699 )
2022-09-10 13:57:52 -07:00
1811d684b5
clean race condition with extra unref ( #27682 )
...
* clean race condition with extra unref
* clarify comment
* fix test
* cleanup test
* clippy
2022-09-10 13:57:17 -07:00
1d9d5bef04
remove outdated comment ( #27703 )
2022-09-10 13:56:08 -07:00
6a322de845
Make Accounts Background Services aware of Epoch Accounts Hash ( #27626 )
2022-09-07 20:41:40 +00:00
22007a3c96
allow accounts hash calc to specify enable_rehashing ( #27615 )
2022-09-07 10:16:52 -07:00
a31d4a597d
serialize epoch_accounts_hash ( #27516 )
2022-09-07 10:07:00 -07:00
1cf9077d06
Include the Epoch Accounts Hash in the Bank hash ( #27541 )
2022-09-07 10:10:49 -04:00
acd846b90e
when storing detailed debug info on accounts hash, we're ok with > 1 storage ( #27617 )
2022-09-07 06:38:57 -07:00
72a2c02ee7
add --accounts-db-verify-refcounts for debugging ( #27504 )
2022-09-06 09:39:39 -07:00
ed132e75a9
rename get_account_maps_read_lock to get_bin since we don't use a loc… ( #27543 )
...
rename get_account_maps_read_lock to get_bin since we don't ues a lock anymore
2022-09-06 07:22:55 -07:00
f90e70ff5a
fix some comments ( #27547 )
...
remove stray comment
2022-09-06 07:21:19 -07:00
a6d737d773
Add EpochAccountsHash to AccountsDb ( #27538 )
2022-09-01 12:11:07 -04:00
0df0b94a72
randomly shrink ancient append vecs ( #27523 )
2022-09-01 08:32:43 -07:00
28c3552756
remove counter stat for clean which is already captured in clean_accounts_stats
2022-09-01 10:11:08 -05:00
43e3545c26
add stats of dirty_slots for cleaning during hash calc
2022-09-01 10:11:08 -05:00
407717eb2a
add timing stat for account hash calc with cache
2022-09-01 10:11:08 -05:00
874085aadc
improve performance of unrefing ancient pubkeys ( #27485 )
2022-09-01 07:34:52 -07:00
b5cca0bfa3
add tests to AccountsDb::unref_accounts ( #27474 )
2022-08-30 18:51:20 -05:00
2e4a52d010
remove storages that only contain empty ( #27452 )
2022-08-30 14:04:17 -05:00
1154bc8636
get rid of acquisition of read lock while unrefing accounts in scan ( #27388 )
2022-08-30 14:04:02 -05:00
9bde3261fd
cleanup ( #27453 )
...
* cleanup
* Update runtime/src/accounts_db.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-08-30 10:58:35 -05:00
e35cd7a34a
comment loop in function ( #27454 )
2022-08-30 10:58:02 -05:00
3d692ba046
refactor clean_dead_slots_from_accounts_index ( #27451 )
2022-08-30 10:06:52 -05:00
73ee94d674
remove old bank hashes with ancient shrink ( #27450 )
2022-08-30 09:31:20 -05:00
a4b8ab2f59
clean_dead_slots_from_accounts_index unrefs correctly ( #27461 )
2022-08-30 08:40:21 -05:00
49411aaae0
Feature: Interleaved Snapshot Untar and Indexing - Stage 1 ( #26590 )
...
* Add snapshot_storage_lengths_from_fields
* Add SnapshotUnpacker
* Add SnapshotStorageRebuilder
* Rebuild snapshot storage during unpack
* move snapshot unpacker back into snapshot_utils
* use SerializedAppendVecId in storage rebuilder
* create AccountStorageMap type alias
* Wrap storage and next_append_vec_id in a type
* fixed typo
* move use StorageAndNextAppendVecId
* calculate num_rebuilder_threads more concisely
2022-08-29 13:17:27 -05:00
cdb699d812
ancient append vecs and cleaning dead slots ( #27353 )
...
cleanup old code
2022-08-26 10:13:41 -07:00
841ed68088
clean_accounts add _inclusive ( #27282 )
2022-08-24 08:56:53 -05:00
66e2df8ddc
failed_store_id should be searched per pubkey ( #27342 )
2022-08-23 19:12:50 -05:00
1c4f7bcd37
clean optimization for store count ( #27277 )
2022-08-23 14:56:44 -05:00
326f43d644
don't mark ancient append vecs as dirty wrt clean ( #27312 )
2022-08-23 09:18:48 -05:00
e4a7d01e10
Rust v1.63 ( #27303 )
...
* Upgrade to Rust v1.63.0
* Add nightly_clippy_allows
* Resolve some new clippy nightly lints
* Increase QUIC packets completion timeout
* Update quinn-udp crate
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-22 18:01:03 -07:00
748c6ea609
in ancient shrink, unref accounts that were already ancient shrunk ( #27294 )
2022-08-22 18:37:46 -05:00
0ac114fc5d
add _inclusive ( #27308 )
2022-08-22 18:03:32 -05:00
0e45e17aca
log reason why oldest store_id isn't being cleaned ( #27298 )
2022-08-22 17:11:38 -05:00
9703855a80
add _inclusive ( #27300 )
2022-08-22 14:55:58 -05:00
ae02d955b4
add _inclusive ( #27301 )
2022-08-22 14:33:17 -05:00
97036b1c75
refactor/document calc_delete_dependencies ( #27272 )
2022-08-22 13:28:18 -05:00
5272a2852d
account_iter instead of accounts() ( #27252 )
2022-08-22 11:35:43 -05:00
fc1a4dd11a
run hash calc with index on failure ( #27279 )
2022-08-22 10:58:04 -05:00
ab826b71a9
rename unused parameter for clarity ( #27271 )
2022-08-22 10:27:24 -05:00
9d5029da15
slots_per_epoch shouldn't be optional ( #27256 )
2022-08-22 10:15:48 -05:00
8f98333395
remove inaccurate log ( #27255 )
2022-08-20 10:15:10 -05:00
58ebae5b9c
cleanup comment on filter_zero_lamport_clean_for_incremental_snapshots ( #27273 )
2022-08-20 10:14:49 -05:00
3f4731b37f
Standardize thread names
...
Tenets:
1. Limit thread names to 15 characters
2. Prefix all Solana-controlled threads with "sol"
3. Use Camel case. It's more character dense than Snake or Kebab case
2022-08-20 07:49:39 -07:00
3c786bab65
Parallel insertion of dirty store keys during clean ( #27058 )
...
parallelize dirty store key insertion
2022-08-20 06:52:32 -07:00
1b441f7a2d
make test code more clear ( #27260 )
2022-08-19 14:39:20 -05:00
42e227778b
eliminate unnecessary ZERO_RAW_LAMPORTS_SENTINEL ( #27218 )
2022-08-19 14:18:00 -05:00
e132583a24
add _inclusive for clarity ( #27239 )
2022-08-19 14:17:32 -05:00
7573000d87
Revert "Rust v1.63.0 ( #27148 )" ( #27245 )
...
This reverts commit a2e7bdf50a
2022-08-19 09:19:44 +01:00
f115e4d0ed
reorder slot # in debug hash data path ( #27217 )
2022-08-18 11:30:11 -05:00
7720b48aa6
add an assert for a debug feature to avoid wasted time ( #27210 )
2022-08-18 10:35:22 -05:00
a2e7bdf50a
Rust v1.63.0 ( #27148 )
...
* Upgrade to Rust v1.63.0
* Add nightly_clippy_allows
* Resolve some new clippy nightly lints
* Increase QUIC packets completion timeout
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-17 15:48:33 -07:00
2fd9a4f373
Add clean_accounts_for_tests() ( #27200 )
2022-08-17 18:45:59 -04:00
8c791f871e
store hash calc failures in a separate folder by slot ( #27146 )
...
* store hash calc failures in a separate folder by slot
* Update runtime/src/accounts_db.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-08-16 09:35:56 -05:00
1a90cffabe
Add stats for readonly cache evicts ( #26938 )
...
* add stats for readonly cache evicts
* bump up account cache to 400M
* aggregate num_evicts in the loop
2022-08-15 16:23:37 -05:00
0ca8239ef7
don't log when there is no work to do for combining ancient slots ( #26925 )
2022-08-13 14:38:41 -06:00
0732a6c45f
add 'oldest_root' stat in hash calc ( #26896 )
2022-08-03 10:02:08 -05:00
6706be0f56
log oldest dirty slot on clean ( #26897 )
2022-08-03 10:01:29 -05:00
224550d65f
allow caching on ancient append vecs for accounts hash calc ( #26885 )
2022-08-02 14:40:29 -05:00
acc1112e9c
calculate cache hash path for store_detailed_debug_info_on_failure ( #26874 )
2022-08-02 11:19:00 -05:00
e5c5055869
Revert "allow caching on ancient append vecs for accounts hash calc" ( #26882 )
...
Revert "allow caching on ancient append vecs for accounts hash calc (#26770 )"
This reverts commit 30b408ed0c
2022-08-02 10:23:11 -05:00
050c66f197
enable accounts hash caching if store_detailed_debug_info_on_failure ( #26864 )
2022-08-01 14:27:20 -05:00
c7462b7a52
ledger tool verify can store debug info on hash calc ( #26837 )
2022-07-29 15:54:56 -05:00
817f65bb50
add full_snapshot to hash config ( #26811 )
2022-07-28 09:46:34 -05:00
1a5b830294
CalcAccountsHashConfig default for tests ( #26810 )
2022-07-27 13:21:38 -05:00
1421833c5e
log when shrink still contains extra stores ( #26794 )
2022-07-27 09:14:16 -05:00
30b408ed0c
allow caching on ancient append vecs for accounts hash calc ( #26770 )
2022-07-27 09:13:32 -05:00
6ec89ee861
log error for cache_hash_data save failure ( #26676 )
...
log error for cach_hash_data save
2022-07-20 16:19:35 -05:00
2d689ac53a
get rid of rwlock per bin in accounts index ( #26675 )
2022-07-19 17:48:40 -05:00
263911e7fd
save off what we find when calculating hash ( #26663 )
2022-07-19 09:55:52 -05:00
738c4a79b7
metric for 'ancient_account_cleans' ( #26664 )
2022-07-19 09:16:23 -05:00
c4d2c95540
move hash stat to correct level for ahv to work ( #26649 )
2022-07-18 12:44:33 -05:00
1c08f83c7e
eliminate flatten().collect() of reclaims in clean ( #26647 )
2022-07-18 12:37:17 -05:00
be8813e588
shrink stops using HandleReclaims ( #26594 )
2022-07-15 17:54:48 -05:00
83c614da3c
change type to be more clear ( #26646 )
2022-07-15 17:17:17 -05:00
47716a5e01
async hash verify on load ( #26208 )
...
* verify accounts hash in bg on startup
* fix some tests and loading from genesis
* add extra state for when background thread has completed
2022-07-15 14:29:56 -05:00
4dea32e8e5
fix active_scans metric to not swap ( #26602 )
2022-07-13 20:11:23 -05:00
2b7cc107d8
typo ( #26618 )
2022-07-13 17:55:02 -05:00
1aa9215411
Refactor: add account_iter api for appendvec ( #26591 )
...
* add account iterator for appendvec
* semicolon
2022-07-12 23:56:05 +00:00
6b0eb5a42b
handle rent paying accounts differently ( #26487 )
...
* handle rent paying accounts differently
* restore collection
* downgrade assert to metric and warn until we get more runtime
2022-07-12 17:51:33 -05:00
c62c55a16b
shrink stores by id ( #26490 )
2022-07-12 15:43:41 -05:00
9d31216d23
add StoreReclaims::Ignore ( #26489 )
2022-07-12 15:05:47 -05:00
1db136a8be
do not count 0 lamport accounts as rent-paying ( #26514 )
2022-07-11 22:16:46 -05:00
36bd46bece
ancient append vec: reuse AccountsToStore ( #26572 )
2022-07-11 22:13:09 -05:00
ba785cf8ab
removes erroneous uses of std::mem::swap ( #26536 )
...
All instances should be replace by std::mem::{replace,take},
or just plain assignment.
2022-07-11 11:33:15 +00:00
995756a550
Use RefCount type alias instead of u64 ( #26472 )
...
use RefCount type alias instead of u64
2022-07-08 13:51:16 -05:00
d9eee72edb
remove FoundStoredAccount.account_size ( #26460 )
2022-07-07 18:37:14 -05:00
38d53ef8a2
add metrics for # scans active and scan distance ( #26395 )
2022-07-07 16:20:32 -05:00
ee0a54ce80
improve clean acct idx calls ( #26320 )
2022-07-07 15:50:12 -05:00
b582e4ce0f
introduce UpsertReclaim ( #26462 )
2022-07-07 15:40:17 -05:00
c9f1d01e79
refactor, add GetUniqueAccountsResult ( #26457 )
2022-07-07 13:49:33 -05:00
72256ac54d
use iter for append vecs ( #26459 )
2022-07-07 13:13:47 -05:00
b3a47de1ce
improve startup stats for rent payers ( #26432 )
2022-07-07 08:39:57 -05:00
16219e300e
generating index sets uncleaned_roots correctly ( #26431 )
...
* generating index sets uncleaned_roots correctly
* fix test failures
* rename
* update comments
2022-07-06 16:12:47 -05:00
d7201a8d1a
names fields in RentResullt::CollectRent enum variant ( #26449 )
...
Avoiding ambiguous raw tuple:
CollectRent((Epoch, u64))
Using named fields instead:
CollectRent {
new_rent_epoch: Epoch,
rent_due: u64,
},
2022-07-06 20:01:16 +00:00
44f499cff3
Refactor account index scan function parameters ( #26428 )
...
* Refactor account index scan function parameters
F: FnMut(bool, &SlotList<T>, &'a Pubkey, RefCount) -> AccountsIndexScanResult,
Refactor to take (&'a Pubkey, Option<(&SlotList<T>, RefCount))
* Update comments based on review feedback
* cargo fmt --all fixes
2022-07-06 09:49:11 -07:00
17a99d98dd
Revert "avoid adding to 'uncleaned_roots' when generating index and c… ( #26441 )
...
Revert "avoid adding to 'uncleaned_roots' when generating index and caller passes accounts-db-skip-shrink (#25936 )"
This reverts commit e24cc537a4
2022-07-06 11:32:45 -05:00
ab164fc975
simplify code ( #26443 )
2022-07-06 11:22:26 -05:00
5afe4d938d
combine lookup in calc_delete_dependencies ( #26429 )
2022-07-06 08:50:14 -05:00
71c7b4f9cb
use append vec iter in construct_candidate_clean_keys ( #26426 )
2022-07-06 08:49:35 -05:00
75149fd624
move mark_old_slots_as_dirty to be called from ahv ( #26411 )
2022-07-05 23:11:32 -05:00
2aafef38f9
is_shrinking_productive calls avoid arc clone ( #26422 )
2022-07-05 17:53:29 -05:00
728986ddf4
Spell correction and add remove slot in AccountDB purge assert message ( #26358 )
...
nit spell. add remove slots in accoutdb purge assert message
2022-07-01 17:39:35 +00:00
557bf6e656
allow initial hash calc to occur in bg ( #26271 )
...
* allow initial hash calc to occur in bg
* validator_initialized -> startup_verification_complete
* add infos for leader and vote
* rework snapshot for startup verification
* change to assert
2022-06-29 16:48:33 -05:00
608191352f
track # roots added/removed since last clean ( #26319 )
2022-06-29 16:47:57 -05:00
e241906db2
remove outdated, confusing comment ( #26315 )
2022-06-29 10:42:34 -05:00
ca83f143aa
load_accounts_index_for_shrink uses scan ( #26260 )
2022-06-29 09:02:27 -05:00
89e5640c63
AccountsIndex::Scan takes iter ( #26268 )
2022-06-28 12:47:17 -05:00
bfc390f9ed
refactor acct idx scan func for reuse ( #26252 )
2022-06-28 09:01:04 -05:00
4b591d7b59
AccountsDb::load_accounts_index_for_shrink takes slice ( #26267 )
2022-06-28 08:59:57 -05:00
e164879f20
load_accounts_index_for_shrink uses &item instead of (&item.0, &item.1) ( #26266 )
2022-06-27 22:40:45 -05:00
d6a28cb7a8
load_accounts_index_for_shrink unrefed_pubkeys is optional ( #26265 )
2022-06-27 17:23:46 -05:00
9db44ce352
cleanup some AccountSharedData::from(Account ( #26120 )
2022-06-23 08:32:48 -05:00
f4189c0305
ledger-tool minimized snapshots ( #25334 )
...
* working on local snapshot
* Parallelization for slot storage minimization
* Additional clean-up and fixes
* make --minimize an option of create-snapshot
* remove now unnecessary function
* Parallelize parts of minimized account set generation
* clippy fixes
* Add rent collection accounts and voting node_pubkeys
* Simplify programdata_accounts generation
* Loop over storages to get slot set
* Parallelize minimized slot set generation
* Parallelize adding owners and programdata_accounts
* Remove some now unncessary checks on the blockstore
* Add a warning for minimized snapshots across epoch boundary
* Simplify ledger-tool minimize
* Clarify names of bank's minimization helper functions
* Remove unnecesary funciton, fix line spacing
* Use DashSets instead of HashSets for minimized account and slot sets
* Filter storages uses all threads instead of thread_pool
* Add some additional comments on functions for minimization
* Moved more into bank and parallelized
* Update programs/bpf/Cargo.lock for dashmap in ledger
* Clippy fix
* ledger-tool: convert minimize_bank_for_snapshot Measure into measure!
* bank.rs: convert minimize_bank_for_snapshot Measure into measure!
* accounts_db.rs: convert minimize_accounts_db Measure into measure!
* accounts_db.rs: add comment about use of minimize_accounts_db
* ledger-tool: CLI argument clarification
* minimization functions: make infos unique
* bank.rs: Add test_get_rent_collection_accounts_between_slots
* bank.rs: Add test_minimization_add_vote_accounts
* bank.rs: Add test_minimization_add_stake_accounts
* bank.rs: Add test_minimization_add_owner_accounts
* bank.rs: Add test_minimization_add_programdata_accounts
* accounts_db.rs: Add test_minimize_accounts_db
* bank.rs: Add negative case and comments in test_get_rent_collection_accounts_between_slots
* bank.rs: Negative test in test_minimization_add_programdata_accounts
* use new static runtime and sdk ids
* bank comments to doc comments
* Only need to insert the maximum slot a key is found in
* rename remove_pubkeys to purge_pubkeys
* add comment on builtins::get_pubkeys
* prevent excessive logging of removed dead slots
* don't need to remove slot from shrink slot candidates
* blockstore.rs: get_accounts_used_in_range shouldn't return Result
* blockstore.rs: get_accounts_used_in_range: parallelize slot loop
* report filtering progress on time instead of count
* parallelize loop over snapshot storages
* WIP: move some bank minimization functionality into a new class
* WIP: move some accounts_db minimization functionality into SnapshotMinimizer
* WIP: Use new SnapshotMinimizer
* SnapshotMinimizer: fix use statements
* remove bank and accounts_db minimization code, where possible
* measure! doesn't take a closure
* fix use statement in blockstore
* log_dead_slots does not need pub(crate)
* get_unique_accounts_from_storages does not need pub(crate)
* different way to get stake accounts/nodes
* fix tests
* move rent collection account functionality to snapshot minimizer
* move accounts_db minimize behavior to snapshot minimizer
* clean up
* Use bank reference instead of Arc. Additional comments
* Add a comment to blockstore function
* Additional clarifying comments
* Moved all non-transaction account accumulation into the SnapshotMinimizer.
* transaction_account_set does not need to be mutable now
* Add comment about load_to_collect_rent_eagerly
* Update log_dead_slots comment
* remove duplicate measure/print of get_minimized_slot_set
2022-06-22 13:17:43 -04:00
d3829b1a76
store_cached takes StorableAccounts to eliminate collect ( #26109 )
2022-06-22 08:45:03 -05:00
b02c412d5b
generate_index straight to disk and batched ( #25947 )
...
* generate_index straight to disk and batched
* renames and comments
* handle in-mem case correctly
* use mutex
2022-06-15 18:14:39 -05:00
e24cc537a4
avoid adding to 'uncleaned_roots' when generating index and caller passes accounts-db-skip-shrink ( #25936 )
2022-06-14 10:10:44 -05:00
2251aa2809
Report transaction signature on update account ( #25726 )
...
There's no direct linkage between account-update events and transaction-events
Issue is linked with geyser plugin interface
Currently, there's no straightforward way to determine what notify_transaction call is related to particular update_account call. So, there's no simple way to determine what transaction caused what changes in accounts. This issue is especially valuable in case when several transactions modifies single account in the same slot.
Add txn_signature field to ReplicaTransactionInfo structure
This additional field will contain the signature of transaction which caused this account update event. This modification is not bringing significant overhead because all necessary information for such linkage already available inside Accounts::collect_accounts_to_store method
2022-06-10 10:24:35 -07:00
8b0d5b2485
parallelize rent collection ( #25767 )
2022-06-03 18:00:02 -05:00
a9a94166ef
load_account_into_read_cache ( #25760 )
2022-06-03 13:34:17 -05:00
e6e2058758
Increase accounts stack size to 8mb ( #25592 )
2022-06-03 19:46:01 +02:00
eeec3fb9c9
Remove unwrap ( #25538 )
2022-06-02 12:18:46 -05:00
d1a6cebaa2
u64 -> duration ( #25708 )
2022-06-02 07:56:47 -05:00
149a54b786
remove par_iter on update index below threshold ( #25699 )
2022-06-01 17:41:56 -05:00
dd0bf4ff4b
add filler accounts gradually with cache flush ( #25678 )
2022-06-01 15:47:21 -05:00
905fef29cb
add filter to AppendVecScan ( #25664 )
2022-06-01 11:55:18 -05:00
94b0ce5d43
remove_old_historical_roots uses accounts_hash_complete_one_epoch_old ( #25673 )
2022-06-01 10:45:26 -05:00
c91fbd289f
set active state to shrink_ancient when doing intersting work ( #25677 )
2022-06-01 10:36:58 -05:00
8bb76fcd87
add ancient metrics ( #25656 )
...
* add ancient metrics
* review feedback
2022-06-01 10:36:23 -05:00
07958fd8ed
use self.init_accum ( #25663 )
2022-06-01 08:38:37 -05:00
5994b4f53f
use object for hash scanning ( #25418 )
2022-05-31 11:51:56 -05:00
bac05dc55a
calc slot info on max slot once per hash calc ( #25422 )
2022-05-23 10:11:00 -05:00
41f30a2383
stop logging misleading bank hash mismatch ( #25427 )
2022-05-23 08:43:25 -05:00
b05c7d91ed
Fix derive_partial_eq_without_eq clippy lint
2022-05-22 22:22:21 -07:00
f584b249dd
special case for common case of 1 storage on append vec scan ( #25414 )
2022-05-20 17:36:30 -05:00
cfcc18f7f4
apply AccountsInAppendVec to more places ( #25408 )
2022-05-20 14:25:46 -05:00
467431de89
add ledger-tool accounts-db-skip-initial-hash-calc ( #25354 )
2022-05-20 10:27:00 -05:00
950245c29b
simulate filler account hash calc time at delta hash ( #25387 )
2022-05-20 10:20:44 -05:00
1b4e4ef548
impl squash to ancient append vecs ( #24538 )
...
* squash to ancient append vecs
* pr feedback
* clippy
2022-05-20 10:20:17 -05:00
de03a3d3aa
filler accounts get a non-zero rent_epoch ( #25386 )
...
* fill accounts get a non-zero rent_epoch
* use snapshot_slot
* get rid of perf simulation code
2022-05-20 09:03:55 -05:00
3fe7655378
use iterator instead of vec to iterate accounts ( #25370 )
...
* user iterator instead of vec to iterate accounts
* rename
2022-05-20 08:50:24 -05:00
9ccdc8a050
populate ancient append vec metrics ( #25204 )
...
* populate ancient append vec metrics
* count ancient storages while enumerating storages anyway.
2022-05-13 18:20:15 -05:00
cb037e49ea
is_candidate_for_shrink respects ancient append vecs ( #25205 )
2022-05-13 16:26:46 -05:00
5eeb07fead
coalesce purge stats and reclaim result ( #25183 )
2022-05-13 12:34:11 -05:00
6f1cc5a2a7
assert we haven't squashed to ancient after hash calc ( #25168 )
2022-05-12 18:26:01 -05:00
896729f25e
keep track of oldest slot used by last hash calculation ( #25152 )
2022-05-12 11:18:08 -05:00
abb4993a3b
SortedStorages::max_slot_inclusive() ( #25153 )
2022-05-12 10:24:03 -05:00
c56d978cb1
minor cleanup ( #25144 )
2022-05-11 14:58:43 -05:00
a8930ee14b
add --accounts-db-ancient-append-vecs ( #25125 )
2022-05-11 08:47:07 -05:00
c4506269a6
add ancient_append_vecs ( #25119 )
2022-05-10 16:41:04 -05:00
dc6e28099b
mark_dirty_dead_stores ( #25114 )
2022-05-10 15:08:29 -05:00
e9e267947e
simplify storage_finder -> storage ( #25109 )
2022-05-10 14:18:51 -05:00
e9a9604ca7
avoid zeroing out root stats with empty data ( #25111 )
2022-05-10 13:56:30 -05:00
2d7ce2a6c2
Revert "uses designated thread-pools for account db parallel ops ( #24954 )" ( #25053 )
...
This reverts commit e8bdc27080
2022-05-06 15:20:50 -05:00
7401bc0fa5
remove_old_historical_roots uses correct epoch schedule ( #25020 )
...
* remove_old_historical_roots uses correct epoch schedule
* Update runtime/src/accounts_db.rs
Co-authored-by: apfitzge <apfitzge@users.noreply.github.com>
Co-authored-by: apfitzge <apfitzge@users.noreply.github.com>
2022-05-06 12:38:09 -05:00
0cc97689f9
add stats for stakes_cache_check_and_store_us ( #24349 )
...
* add stats for stakes_cache_check_and_store_us
* remove redundant metrics
2022-05-05 17:22:48 -05:00
e8bdc27080
uses designated thread-pools for account db parallel ops ( #24954 )
...
Instead of global rayon thread-pool, the commit uses
AccountsDb::{thread_pool,thread_pool_clean} for parallel ops in
AccountsDB::{update_index,clean_accounts_older_than_root}.
2022-05-04 22:47:12 +00:00
e8bf812a8a
allow ledger-tool halt at slot to calc hash using write cache ( #24974 )
2022-05-04 16:27:49 -05:00
40986daddf
log write cache contents from purge_slots_from_cache_and_store ( #24948 )
2022-05-04 14:28:47 -05:00
755100dd7c
use correct epoch schedule ( #24904 )
2022-05-03 09:21:47 -05:00
112a0b475a
Revert "Refactor to use EpochSchedule from within RentCollector struct" ( #24893 )
...
* Revert "Ran cargo fmt"
This reverts commit 9052e41b324dfeab3b3887fb78dc56c2f7f2fff836c0bdaa78ed7813e6985472d2e605
2022-05-02 13:46:17 -05:00
099258e4eb
some cleanup ( #24568 )
2022-04-25 13:17:30 -05:00
8a062273de
Move error counters to be reported by leader only at end of slot ( #24581 )
...
* Add error counters to leader metrics only
* Add dependencies
2022-04-23 18:10:47 -05:00
ce7d2964c9
metrics on ancient slots ( #24596 )
2022-04-22 15:01:51 -05:00
25e9199397
cleanup FlushStats ( #24556 )
2022-04-21 14:05:01 -05:00
05438c3cd6
implement remove_old_historical_roots ( #24492 )
2022-04-20 10:43:32 -05:00
0d797e2fff
contains_multiple_slots ( #24500 )
2022-04-20 09:53:36 -05:00
cfe2177e16
validator option: accounts-db-skip-rewrites ( #24504 )
2022-04-20 09:41:00 -05:00
255a6a729d
add historical_roots_len stat ( #24497 )
2022-04-19 16:53:24 -05:00
d23c04bb68
max_slot_in_storages_exclusive -> INclusive ( #24450 )
2022-04-19 13:59:41 -05:00
c4923d29b4
bank_hash_at uses rewrites ( #24439 )
2022-04-19 11:29:29 -05:00
4fdf966393
demote shink counter to debug ( #24458 )
2022-04-19 10:02:23 -05:00
dc98510d6d
accounts hash calls maybe_rehash_skipped_rewrite ( #24316 )
2022-04-19 08:29:09 -05:00
0e0f306996
mechanism for disabling hash calc caching ( #24437 )
2022-04-18 22:44:30 -05:00
9052e41b32
Ran cargo fmt
2022-04-18 11:57:40 -05:00
87fb78dc56
Fix build error introduced by my editor setup
2022-04-18 11:57:40 -05:00
5472d2e605
Removing redundant EpochSchedule param from fns
2022-04-18 11:57:40 -05:00
b4fd9124bf
log secondary index contents on startup ( #24348 )
2022-04-15 13:30:03 -05:00
d20b4c9958
rename function param to avoid conflict ( #24342 )
2022-04-14 11:56:32 -05:00
e3ef0741be
simplify bank drop calls ( #24142 )
...
* simplify bank drop calls
* clippy: import
* Update ledger/src/blockstore_processor.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/accounts_background_service.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/bank.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* cleanup
* format
* use msb of bank_id to indicates that we are dropping
* clippy
* restore bank id
* clippy
* revert is-serialized_with_abs flag
* assert
* clippy
* whitespace
* fix bank drop callback check
* more fix
* remove msb dropping implementation
* fix
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-14 08:43:54 -05:00
255ef66a27
move feature activation log to correct fn for hash ( #24326 )
2022-04-13 17:43:33 -05:00
929753a11f
typo ( #24291 )
2022-04-12 16:46:59 -05:00
4fd184c131
Use Release/Acquire instead of SeqCst for is_bank_drop_callback_enabled ( #24134 )
2022-04-11 19:19:17 -05:00
6fbe2b936c
fix comment ( #24254 )
2022-04-11 18:53:45 -05:00
60b2155bd3
Add accounts-filler-size command line option ( #23896 )
2022-04-11 13:10:09 -05:00
210f6a6fab
move hash calculation out of acct bg svc ( #23689 )
...
* move hash calculation out of acct bg svc
* pr feedback
2022-04-08 10:42:03 -05:00
c27150b1a3
reserialize_bank_fields_with_hash ( #23916 )
...
* reserialize_bank_with_new_accounts_hash
* Update runtime/src/serde_snapshot.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* pr feedback
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-07 14:05:57 -05:00
4a11fa072f
hash_account_with_rent_epoch ( #24104 )
2022-04-05 08:10:31 -05:00
132f08486a
remove basically duplicate function ( #24107 )
2022-04-04 12:55:05 -05:00
ec97d6d078
rename remove_old_roots ( #24059 )
2022-04-02 12:01:13 -05:00
3ca4fffa78
root -> alive_root ( #24062 )
2022-04-02 12:00:52 -05:00
31997f8251
hash calc scanning takes config ( #24016 )
2022-03-31 14:26:37 -05:00
9c8dad33c7
add epoch_schedule and rent_collector to hash calc ( #24012 )
2022-03-31 10:51:18 -05:00
da001d54e5
calculate_accounts_hash_helper uses config ( #24003 )
2022-03-31 09:29:45 -05:00
125f9634fd
add hash calc config.use_write_cache ( #24005 )
2022-03-30 17:19:34 -05:00
af9344fe22
SortedStorages refactoring ( #23998 )
2022-03-30 09:19:03 -05:00
5636570d6d
add roots_original to roots tracker ( #23849 )
2022-03-30 08:52:45 -05:00
5a613e9b6e
use CalcAccountsHashConfig in calculate_accounts_hash ( #23987 )
2022-03-29 22:05:47 -05:00
Jeff Washington (jwash)
Brooks Prumo
Jeff Washington (jwash)
Haoran Yi
apfitzge
Brennan Watt
Michael Vines
behzad nouri
Xiang Zhu
ivandzen
sakridge
carllin
Jason Davis
Giorgio Gambino