6fbe2b936c
fix comment ( #24254 )
2022-04-11 18:53:45 -05:00
60b2155bd3
Add accounts-filler-size command line option ( #23896 )
2022-04-11 13:10:09 -05:00
210f6a6fab
move hash calculation out of acct bg svc ( #23689 )
...
* move hash calculation out of acct bg svc
* pr feedback
2022-04-08 10:42:03 -05:00
c27150b1a3
reserialize_bank_fields_with_hash ( #23916 )
...
* reserialize_bank_with_new_accounts_hash
* Update runtime/src/serde_snapshot.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* pr feedback
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-07 14:05:57 -05:00
4a11fa072f
hash_account_with_rent_epoch ( #24104 )
2022-04-05 08:10:31 -05:00
132f08486a
remove basically duplicate function ( #24107 )
2022-04-04 12:55:05 -05:00
ec97d6d078
rename remove_old_roots ( #24059 )
2022-04-02 12:01:13 -05:00
3ca4fffa78
root -> alive_root ( #24062 )
2022-04-02 12:00:52 -05:00
31997f8251
hash calc scanning takes config ( #24016 )
2022-03-31 14:26:37 -05:00
9c8dad33c7
add epoch_schedule and rent_collector to hash calc ( #24012 )
2022-03-31 10:51:18 -05:00
da001d54e5
calculate_accounts_hash_helper uses config ( #24003 )
2022-03-31 09:29:45 -05:00
125f9634fd
add hash calc config.use_write_cache ( #24005 )
2022-03-30 17:19:34 -05:00
af9344fe22
SortedStorages refactoring ( #23998 )
2022-03-30 09:19:03 -05:00
5636570d6d
add roots_original to roots tracker ( #23849 )
2022-03-30 08:52:45 -05:00
5a613e9b6e
use CalcAccountsHashConfig in calculate_accounts_hash ( #23987 )
2022-03-29 22:05:47 -05:00
37c36ce3fa
pass stats separately from CalcAccountsHashConfig ( #23892 )
2022-03-24 12:48:47 -05:00
82328fd9d8
move max_clean_root deeper in flush cache ( #23869 )
2022-03-24 12:45:49 -05:00
a22a2384bf
fix ci test error ( #23908 )
2022-03-24 11:30:20 -05:00
5b916961b5
HashCalc uses self.accounts_cache ( #23890 )
2022-03-24 10:34:28 -05:00
f2aea3b7c7
flush_slot_cache takes [Slot] ( #23865 )
2022-03-24 10:24:36 -05:00
9d3b17c635
HashCalc uses self.accounts_index ( #23888 )
2022-03-24 10:06:32 -05:00
b22165ad69
hash calc uses self.filler_account_suffix ( #23887 )
2022-03-24 09:58:06 -05:00
9022931689
calc hash uses self.num_hash_scan_passes ( #23883 )
2022-03-24 09:44:42 -05:00
e3eb002f66
Log storage size stats at hash calc ( #23843 )
2022-03-24 09:40:35 -05:00
f1a411c897
add epoch_schedule and rent_collector to hash calc ( #23857 )
2022-03-24 09:39:22 -05:00
db5d68f01f
HashCalc uses self.accounts_hash_cache_path ( #23882 )
2022-03-24 09:31:55 -05:00
90009f330b
small refactor to shorten the lock on slot_under_contention hashset ( #23891 )
...
* small refactor to shorten the lock on slot_under_contention hashset
* adding comments
* comments
2022-03-24 08:20:56 -05:00
5a892af2fe
disable 'check_hash' on accounts hash calc ( #23873 )
2022-03-23 21:03:31 -05:00
3e22d4b286
calc hash uses self.thread_pool_clean ( #23881 )
2022-03-23 20:52:38 -05:00
dc3863ef14
flush_slot_cache_with_clean ( #23868 )
2022-03-23 14:09:56 -05:00
260f899eda
write cache: hashmap to set ( #23866 )
2022-03-23 14:05:45 -05:00
9e61fe7583
add AccountsHashConfig to manage parameters ( #23850 )
2022-03-23 13:44:23 -05:00
b1280b670a
calculate_accounts_hash_without_index takes &self ( #23846 )
...
* calculate_accounts_hash_without_index takes &self
* Update runtime/src/snapshot_package.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-03-23 11:57:32 -05:00
7b89222fde
don't start extra threads for shrink/clean/hash ( #23858 )
2022-03-23 11:53:37 -05:00
493a8e2348
remove random flushing of write cache ( #23845 )
2022-03-23 08:45:44 -05:00
dd69f3baf5
throttle index adding to allow disk flushing to keep up and reduce startup ram usage ( #23773 )
2022-03-20 19:56:20 -05:00
a419374fa4
factor out function ( #23742 )
2022-03-18 14:10:52 -05:00
b5a99b9b09
avoid data copies in writing to cache ( #23674 )
2022-03-15 16:42:26 -05:00
102dd68a03
Rename AccountsDb plugins to Geyser plugins ( #23604 )
2022-03-14 19:18:46 -06:00
a3242b6b86
remove unnecessary namespaces ( #23646 )
2022-03-14 12:28:55 -05:00
e8a8f4e9e2
fix filler acct slots in epoch ( #23536 )
2022-03-14 09:57:44 -05:00
463cd564cf
fix typos ( #23495 )
...
* fix typos
2022-03-05 20:46:46 -06:00
ddfd4f86f3
exit early from acct bg svc if no shrink candidates ( #23459 )
2022-03-03 08:47:39 -06:00
a99fd09c16
allow index update to change storage slot # ( #23311 )
2022-03-03 08:40:48 -06:00
d909b7c80b
log hash and lamport result of calculate_accounts_hash_without_index ( #23425 )
2022-03-02 11:10:40 -06:00
7b7fdb42d9
minor test refactoring ( #23423 )
2022-03-02 08:31:14 -06:00
0ad4757159
plumbing for 'other_slot' in 'update_index' ( #23330 )
2022-02-25 12:58:08 -06:00
7ee549e5ae
fix params for store_accounts_frozen ( #23312 )
2022-02-23 17:40:51 -06:00
cafc18c3f9
update_index uses ReadableAccount to reduce params ( #23305 )
2022-02-23 17:01:23 -06:00
a245efe83d
trait for acct data with slot per item ( #23285 )
2022-02-23 15:22:29 -06:00
a102453bae
accounts_index: Add SPL Token account indexing for token-2022 accounts ( #23043 )
2022-02-16 16:23:25 -08:00
a160fc30f2
make max_roots_inclusive clear ( #22942 )
2022-02-07 13:26:53 -06:00
350207b05f
impl ReadableAccount for LoadedAccount ( #22793 )
2022-01-27 16:59:30 -06:00
18d69c8d2b
SortedStoragesIter to remove random access requirement ( #22713 )
2022-01-27 10:25:13 -06:00
8b90084ebc
fix comment typos ( #22723 )
2022-01-25 11:10:42 -06:00
6d5bbca630
Pacify clippy
2022-01-21 19:12:57 -08:00
4abf7a7f88
add ActiveStats to track when long running, expensive bg processes are running ( #22608 )
2022-01-21 10:51:23 -06:00
7d7228c356
cleanup assert ( #22495 )
2022-01-14 11:53:24 -06:00
cddab635ff
Cleanup activated rent_for_sysvars feature ( #22454 )
2022-01-14 20:34:09 +08:00
d3b95b9226
Cleanup AccountStorage struct ( #22463 )
...
* Revert "chore: bump dashmap from 4.0.2 to 5.0.0 (#21824 )"
This reverts commit 8aa3d690b5
2022-01-13 17:51:53 -06:00
637e366b18
Prevent rent-paying account creation ( #22292 )
...
* Fixup typo
* Add new feature
* Add new TransactionError
* Add framework for checking account state before and after transaction processing
* Fail transactions that leave new rent-paying accounts
* Only check rent-state of writable tx accounts
* Review comments: combine process_result success behavior; log and metrics before feature activation
* Fix tests that assume rent-exempt accounts are okay
* Remove test no longer relevant
* Remove native/sysvar special case
* Move metrics submission to report legacy->legacy rent paying transitions as well
2022-01-11 11:32:25 -07:00
a49ef49f87
optimizes ReadOnlyAccountsCache LRU eviction implementation ( #22403 )
...
ReadOnlyAccountsCache is using a background thread, table scan and sort
to implement LRU eviction policy:
https://github.com/solana-labs/solana/blob/eaa52bc93/runtime/src/read_only_accounts_cache.rs#L66-L73
https://github.com/solana-labs/solana/blob/eaa52bc93/runtime/src/read_only_accounts_cache.rs#L186-L191
https://github.com/solana-labs/solana/blob/eaa52bc93/runtime/src/read_only_accounts_cache.rs#L222
DashMap internally locks each shard when accessed; so a table scan in
the background thread can create a lot of lock contention.
This commit adds an index-list queue containing cached keys in the order
that they are accessed. Each hash-map entry also includes its index into
this queue.
When an item is first entered into the cache, it is added to the end of
the queue. Also each time an entry is looked up from the cache it is
moved to the end of queue. As a result, items in the queue are always
sorted in the order that they have last been accessed. When doing LRU
eviction, cache entries are evicted from the front of the queue.
Using index-list, all queue operations above are O(1) with low overhead
and so above achieves an efficient implementation of LRU cache eviction
policy.
2022-01-11 17:25:28 +00:00
d90d5ee9b6
Refactor Rent::due() with RentDue enum ( #22346 )
2022-01-08 09:03:46 -06:00
e0c091a9f4
factor out code in do_shrink_slot_stores ( #22306 )
2022-01-06 10:49:24 -06:00
100293c4b5
refactor get_store_for_shrink ( #22307 )
2022-01-06 10:49:01 -06:00
ab13e39518
Only sum accounts data len from non-zero lamport accounts ( #22309 )
2022-01-05 23:23:37 +00:00
fb62407232
AcctIdx: appendvecid: u32 ( #21842 )
2022-01-03 10:35:17 -06:00
729698e815
AcctIdx: items() uses held ranges ( #21954 )
2021-12-17 09:59:29 -06:00
5d40da5688
log "next_id" to track append vec ids ( #21971 )
2021-12-16 22:41:34 -06:00
e11a1911ad
load_accounts_index_for_shrink ignores cached entries ( #21951 )
2021-12-16 16:37:08 -06:00
347323cbb2
use bg thread pool for shrink ( #21950 )
2021-12-16 15:54:38 -06:00
46e5350d8c
AcctInfo: store offset in AccountInfo as u32 ( #21895 )
2021-12-15 15:41:11 -06:00
e5be96d8bf
AcctIdx: consolidate next_id calls ( #21929 )
2021-12-15 15:39:54 -06:00
41dd31e5f4
AcctIdx: move cached store id to bit ( #21892 )
...
* AcctIdx: move cached store id to bit
* add comments/rename
2021-12-15 11:49:24 -06:00
02fa135815
AcctIdx: create test fn get_test() to isolate changes to AcctIdx::get() ( #21909 )
2021-12-15 09:09:56 -06:00
f4308bdb64
AcctIdx: StoredSize is u32 ( #21894 )
2021-12-14 13:50:19 -06:00
ec583bd12d
AcctIdex: use StorageLocation ( #21853 )
2021-12-14 10:27:17 -06:00
6fc6673ead
AcctInfo: store_id private and add accessor ( #21839 )
2021-12-13 21:35:30 -06:00
98e5ea9dce
AcctIdx: simplify AccountIndexGetResult ( #21857 )
2021-12-13 21:16:17 -06:00
50f26ea9c0
AcctInfo: create AcctInfo with cache explicitly ( #21847 )
2021-12-13 17:35:07 -06:00
6155ef6377
AcctInfo: make offset private, add accessor ( #21838 )
2021-12-13 14:43:26 -06:00
c9e0bde407
AcctIdx: AccountInfo::stored_size private ( #21821 )
2021-12-13 10:10:06 -06:00
025a5a3b9c
AccountInfo: construct with new() ( #21788 )
2021-12-12 14:36:05 -06:00
825f8bcea4
use AppendVecId instead of usize ( #21792 )
2021-12-11 21:38:13 -06:00
c5b6ea74ef
replace some .lamports checks with trait ( #21783 )
2021-12-11 21:37:33 -06:00
4c0373b72a
introduce AtomicAppendVecId ( #21793 )
2021-12-11 21:34:35 -06:00
ef7f49a21d
move AccountInfo to its own file ( #21784 )
2021-12-11 11:47:05 -06:00
ec7e17787e
Compute accounts data len during generate_index() ( #21757 )
2021-12-10 13:27:59 -06:00
1528f85112
Add return types to generate_index() ( #21735 )
2021-12-09 17:15:13 -06:00
a2df1eb502
AcctIdx: disk generate index and filler accounts use more threads ( #21566 )
2021-12-09 11:54:14 -06:00
b8837c04ec
Reformat imports to a consistent style for imports
...
rustfmt.toml configuration:
imports_granularity = "One"
group_imports = "One"
2021-12-03 09:19:13 -08:00
f0b32b75ab
AcctIdx: use ZeroLamport trait ( #21552 )
2021-12-02 12:10:11 -06:00
308d7d40d0
aggressively flush cache based on cache size ( #21332 )
...
* aggressively flush cache based on cache size
* find_older_frozen_slots -> cached_frozen_slots
* remove 'WRITE_CACHE_LIMIT_BYTES_DEFAULT'
* tweaks to stats
* fix tests
2021-12-01 13:10:48 -06:00
7ec88226ee
AcctIdx: stats are usize ( #21493 )
...
* AcctIdx: stats are usize
* rename test function
2021-11-30 11:35:21 -06:00
ba9dfa0d22
Remove frozen account support
2021-11-29 08:38:11 -08:00
c4d68063c7
Add timing for accounts add_root ( #21379 )
2021-11-22 18:29:45 +01:00
02bc4e3fc1
spl-token: New program feature flag ( #21354 )
...
* spl-token: Add feature flag for new release
* Remove all spl token version declarations
2021-11-21 14:27:03 +01:00
79d21d6805
add --accounts-index-scan-results-limit-mb to allow scans to abort ( #21327 )
...
* ScanConfig -> &ScanConfig
* add --accounts-index-scan-results-limit-mb to allow scans to abort
* feedback
2021-11-19 09:00:19 -06:00
c24e30f689
Constrain memory ordering on AccountsDb::next_id ( #21311 )
2021-11-18 20:34:37 -06:00
f7152c889c
Constrain memory ordering on AppendVec::id ( #21310 )
2021-11-18 20:34:20 -06:00
Jeff Washington (jwash)
Giorgio Gambino
HaoranYi
Tyera Eulberg
Michael Vines
Justin Starry
Tyera Eulberg
behzad nouri
Brooks Prumo
sakridge
Jon Cinque