c89bd0d6a4
make ancient shrink consistent with shrink ( #28457 )
2022-10-18 15:05:32 -07:00
825b83c95c
simple minor refactoring ( #28456 )
2022-10-18 15:05:06 -07:00
3835ac166e
refactor load_accounts_index_for_shrink for easier sharing ( #28455 )
2022-10-18 14:55:10 -07:00
d52e28ac22
combine sort into get_unique_accounts_from_storages ( #28450 )
2022-10-18 11:52:48 -07:00
b583ec79e1
add tests for should_move_to_ancient_append_vec ( #28438 )
...
add tests for should_moev_to_ancient_append_vec
2022-10-18 11:41:35 -07:00
fd2e671861
remove slot# from account hash ( #28405 )
...
* remove slot# from account hash
* add feature
* fix tests
* constants to help clarify 'irrelevant' changes
* move to enum for enforcing irrelevancy
* ignore unsupported tests
2022-10-18 08:03:37 -07:00
682999a423
accounts hash calculation uses files instead of memory ( #28065 )
2022-10-18 07:51:38 -07:00
b669b827fb
refactor and test get_sorted_potential_ancient_slots ( #28436 )
2022-10-18 07:45:43 -07:00
28a89a1d99
remove expected rent collection and rehashing completely ( #28422 )
2022-10-17 07:24:42 -07:00
0807105c5a
refactoring of accounts hash to prepare for using files always ( #28379 )
2022-10-14 12:29:43 -07:00
72d6927fa1
factor out hashing of storages for caching ( #28261 )
2022-10-13 08:53:54 -07:00
53a1a14868
eliminate worthless work in store_accounts_to ( #28365 )
2022-10-12 13:30:20 -07:00
db9e32d71d
Revert "log adding root every 10s ( #28280 )" ( #28334 )
...
This reverts commit 16853acf35
2022-10-11 12:49:49 -05:00
7a120b8b62
accounts_db::load returns None for zero lamport accounts ( #28311 )
2022-10-11 07:43:03 -07:00
16853acf35
log adding root every 10s ( #28280 )
2022-10-07 09:19:34 -07:00
a1fe8dd444
add info for load race condition asserts ( #28277 )
2022-10-07 08:55:01 -07:00
a400178744
hash calc chunking cleanup ( #28274 )
2022-10-06 23:23:26 -07:00
76c30e7bd0
move accounts_db::load_without_fixed_root to test mod ( #28271 )
2022-10-06 17:11:08 -07:00
66cd290d04
add chunk tracking in SplitAncientStorages ( #28202 )
2022-10-06 11:19:53 -07:00
bb9a19ac86
minor refactoring of accounts hash cache ( #28260 )
2022-10-06 10:50:16 -07:00
435d4aded9
run initial hash calc in background, using background threads ( #28239 )
2022-10-05 19:42:06 -07:00
456d4ae93a
accounts hash calc, add get_ancient_slots ( #28240 )
2022-10-05 15:32:39 -07:00
ad0742c7cf
exit early deep in bank freeze, avoiding some unnecessary work ( #28238 )
...
exit early on bank freeze
2022-10-05 14:49:33 -07:00
a8c6a9e5fc
Bank::freeze() waits for EAH calculation to complete ( #28170 )
2022-10-05 17:44:35 -04:00
3f5eec32cd
refactor SplitAncientStorages::new for testing ( #28235 )
2022-10-05 12:56:29 -07:00
f464c69764
refactor into SplitAncientStorages::new ( #28228 )
2022-10-05 08:46:53 -07:00
3339570a70
accounts hash scan refactoring ( #28220 )
2022-10-04 10:19:50 -07:00
fd11e261dd
use ref for snapshot_storages.iter_range ( #28205 )
2022-10-04 09:11:44 -07:00
39b37e2c6f
remove LoadZeroLamports enum ( #28204 )
...
remove feature return_none_for_zero_lamport_accounts
2022-10-04 08:06:56 -07:00
e2c0999568
rename fields in SplitAncientStorages ( #28200 )
2022-10-03 17:18:08 -07:00
df76cb5ad9
refactor into split_storages_ancient ( #28194 )
2022-10-03 14:25:33 -07:00
adc4e633a2
accounts hash calc refactoring ( #28164 )
2022-10-01 17:09:50 -07:00
929a311155
refactor accts hash tests ( #28165 )
2022-10-01 17:09:29 -07:00
cfc124c825
acct idx can no longer use write cache ( #28150 )
2022-09-30 10:55:27 -07:00
80c0173452
width of hash calc bins was off by 1 ( #28124 )
2022-09-29 11:45:53 -07:00
89b0abcf4d
get_one_epoch_old_slot_for_hash_calc_scan to handle many old append vecs ( #28000 )
2022-09-23 11:17:02 -07:00
9a57c64f21
patches clippy errors from new rust nightly release ( #27996 )
2022-09-22 22:23:03 +00:00
1ee595ca9c
remove AccountsDb::initial_blockstore_processing_complete ( #27974 )
2022-09-22 13:52:04 -04:00
6e24edb771
log metric in case of race condition ( #27951 )
2022-09-20 16:23:25 -07:00
d74480a125
when shrinking, don't send entire old append vec to clean ( #27914 )
2022-09-20 06:32:07 -07:00
f2d6a7ecea
bank.initial_blockstore_processing_complete to avoid concurrent hash calculations ( #27776 )
...
* bank.initial_blockstore_processing_complete to avoid concurrent hash calculations
* Update runtime/src/bank.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/bank.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Rename TestValidator::set_startup_verification_complete()
* Initialize with `AtomicBool::new(false)` instead of `default()`
* snapshot tests: move where `initial_blockstore_processing_completed()` is called
* fixup bank_forks.rs calling `is_initial_blockstore_processing_complete()`
* only call initial_blockstore_processing_completed() in blockstore_processor
Co-authored-by: Brooks Prumo <brooks@prumo.org>
Co-authored-by: Brooks Prumo <brooks@solana.com>
2022-09-19 13:00:21 -07:00
9f097301df
remove dead code ( #27858 )
2022-09-19 07:36:40 -07:00
9cec2ef577
runtime: fix typo situtation -> situation ( #27893 )
2022-09-17 21:57:13 -06:00
1d701aa4f9
ancient append vec stats: ancient_append_vecs_shrunk ( #27824 )
2022-09-16 12:58:48 -07:00
1a4b8b5b79
recycle_store_count is moved to try_recycle_store ( #27797 )
2022-09-15 21:27:09 +02:00
138d5ed5b0
don't return zero lamport accounts from 'load' with feature ( #27793 )
...
* don't return zero lamport accounts from 'load'
* add feature
* rename
2022-09-15 08:46:32 -07:00
6a7fbf7aba
move create_store counting to create_store ( #27781 )
2022-09-14 22:02:29 +02:00
4f232250e5
add `dirty_ancient_stores` to clean metrics ( #27774 )
2022-09-14 06:52:51 -07:00
17007fdda8
remove ancient append vec stores from shrink_candidate_slots ( #27763 )
2022-09-14 06:50:34 -07:00
0bcd9134bc
Rework 'remove_dead_slots_metadata: slots' info log message
2022-09-12 13:36:05 -07:00
d242c0fc97
pass 'enable_rehashing' through all accounts hash functions ( #27701 )
2022-09-12 12:18:09 -07:00
ac74fc16b6
cleanup test constants ( #27704 )
2022-09-12 12:00:52 -07:00
765c628546
use exit signal for acct idx bg threads ( #27483 )
2022-09-12 11:51:12 -07:00
bd6177d5f2
move fn only used by tests to tests ( #27699 )
2022-09-10 13:57:52 -07:00
1811d684b5
clean race condition with extra unref ( #27682 )
...
* clean race condition with extra unref
* clarify comment
* fix test
* cleanup test
* clippy
2022-09-10 13:57:17 -07:00
1d9d5bef04
remove outdated comment ( #27703 )
2022-09-10 13:56:08 -07:00
6a322de845
Make Accounts Background Services aware of Epoch Accounts Hash ( #27626 )
2022-09-07 20:41:40 +00:00
22007a3c96
allow accounts hash calc to specify enable_rehashing ( #27615 )
2022-09-07 10:16:52 -07:00
a31d4a597d
serialize epoch_accounts_hash ( #27516 )
2022-09-07 10:07:00 -07:00
1cf9077d06
Include the Epoch Accounts Hash in the Bank hash ( #27541 )
2022-09-07 10:10:49 -04:00
acd846b90e
when storing detailed debug info on accounts hash, we're ok with > 1 storage ( #27617 )
2022-09-07 06:38:57 -07:00
72a2c02ee7
add --accounts-db-verify-refcounts for debugging ( #27504 )
2022-09-06 09:39:39 -07:00
ed132e75a9
rename get_account_maps_read_lock to get_bin since we don't use a loc… ( #27543 )
...
rename get_account_maps_read_lock to get_bin since we don't ues a lock anymore
2022-09-06 07:22:55 -07:00
f90e70ff5a
fix some comments ( #27547 )
...
remove stray comment
2022-09-06 07:21:19 -07:00
a6d737d773
Add EpochAccountsHash to AccountsDb ( #27538 )
2022-09-01 12:11:07 -04:00
0df0b94a72
randomly shrink ancient append vecs ( #27523 )
2022-09-01 08:32:43 -07:00
28c3552756
remove counter stat for clean which is already captured in clean_accounts_stats
2022-09-01 10:11:08 -05:00
43e3545c26
add stats of dirty_slots for cleaning during hash calc
2022-09-01 10:11:08 -05:00
407717eb2a
add timing stat for account hash calc with cache
2022-09-01 10:11:08 -05:00
874085aadc
improve performance of unrefing ancient pubkeys ( #27485 )
2022-09-01 07:34:52 -07:00
b5cca0bfa3
add tests to AccountsDb::unref_accounts ( #27474 )
2022-08-30 18:51:20 -05:00
2e4a52d010
remove storages that only contain empty ( #27452 )
2022-08-30 14:04:17 -05:00
1154bc8636
get rid of acquisition of read lock while unrefing accounts in scan ( #27388 )
2022-08-30 14:04:02 -05:00
9bde3261fd
cleanup ( #27453 )
...
* cleanup
* Update runtime/src/accounts_db.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-08-30 10:58:35 -05:00
e35cd7a34a
comment loop in function ( #27454 )
2022-08-30 10:58:02 -05:00
3d692ba046
refactor clean_dead_slots_from_accounts_index ( #27451 )
2022-08-30 10:06:52 -05:00
73ee94d674
remove old bank hashes with ancient shrink ( #27450 )
2022-08-30 09:31:20 -05:00
a4b8ab2f59
clean_dead_slots_from_accounts_index unrefs correctly ( #27461 )
2022-08-30 08:40:21 -05:00
49411aaae0
Feature: Interleaved Snapshot Untar and Indexing - Stage 1 ( #26590 )
...
* Add snapshot_storage_lengths_from_fields
* Add SnapshotUnpacker
* Add SnapshotStorageRebuilder
* Rebuild snapshot storage during unpack
* move snapshot unpacker back into snapshot_utils
* use SerializedAppendVecId in storage rebuilder
* create AccountStorageMap type alias
* Wrap storage and next_append_vec_id in a type
* fixed typo
* move use StorageAndNextAppendVecId
* calculate num_rebuilder_threads more concisely
2022-08-29 13:17:27 -05:00
cdb699d812
ancient append vecs and cleaning dead slots ( #27353 )
...
cleanup old code
2022-08-26 10:13:41 -07:00
841ed68088
clean_accounts add _inclusive ( #27282 )
2022-08-24 08:56:53 -05:00
66e2df8ddc
failed_store_id should be searched per pubkey ( #27342 )
2022-08-23 19:12:50 -05:00
1c4f7bcd37
clean optimization for store count ( #27277 )
2022-08-23 14:56:44 -05:00
326f43d644
don't mark ancient append vecs as dirty wrt clean ( #27312 )
2022-08-23 09:18:48 -05:00
e4a7d01e10
Rust v1.63 ( #27303 )
...
* Upgrade to Rust v1.63.0
* Add nightly_clippy_allows
* Resolve some new clippy nightly lints
* Increase QUIC packets completion timeout
* Update quinn-udp crate
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-22 18:01:03 -07:00
748c6ea609
in ancient shrink, unref accounts that were already ancient shrunk ( #27294 )
2022-08-22 18:37:46 -05:00
0ac114fc5d
add _inclusive ( #27308 )
2022-08-22 18:03:32 -05:00
0e45e17aca
log reason why oldest store_id isn't being cleaned ( #27298 )
2022-08-22 17:11:38 -05:00
9703855a80
add _inclusive ( #27300 )
2022-08-22 14:55:58 -05:00
ae02d955b4
add _inclusive ( #27301 )
2022-08-22 14:33:17 -05:00
97036b1c75
refactor/document calc_delete_dependencies ( #27272 )
2022-08-22 13:28:18 -05:00
5272a2852d
account_iter instead of accounts() ( #27252 )
2022-08-22 11:35:43 -05:00
fc1a4dd11a
run hash calc with index on failure ( #27279 )
2022-08-22 10:58:04 -05:00
ab826b71a9
rename unused parameter for clarity ( #27271 )
2022-08-22 10:27:24 -05:00
9d5029da15
slots_per_epoch shouldn't be optional ( #27256 )
2022-08-22 10:15:48 -05:00
8f98333395
remove inaccurate log ( #27255 )
2022-08-20 10:15:10 -05:00
58ebae5b9c
cleanup comment on filter_zero_lamport_clean_for_incremental_snapshots ( #27273 )
2022-08-20 10:14:49 -05:00
3f4731b37f
Standardize thread names
...
Tenets:
1. Limit thread names to 15 characters
2. Prefix all Solana-controlled threads with "sol"
3. Use Camel case. It's more character dense than Snake or Kebab case
2022-08-20 07:49:39 -07:00
3c786bab65
Parallel insertion of dirty store keys during clean ( #27058 )
...
parallelize dirty store key insertion
2022-08-20 06:52:32 -07:00
1b441f7a2d
make test code more clear ( #27260 )
2022-08-19 14:39:20 -05:00
Jeff Washington (jwash)
steviez
Brooks Prumo
behzad nouri
Ian Macalinao
kirill lykov
Michael Vines
Jeff Washington (jwash)
Haoran Yi
apfitzge
Brennan Watt