a3242b6b86
remove unnecessary namespaces ( #23646 )
2022-03-14 12:28:55 -05:00
e8a8f4e9e2
fix filler acct slots in epoch ( #23536 )
2022-03-14 09:57:44 -05:00
463cd564cf
fix typos ( #23495 )
...
* fix typos
2022-03-05 20:46:46 -06:00
ddfd4f86f3
exit early from acct bg svc if no shrink candidates ( #23459 )
2022-03-03 08:47:39 -06:00
a99fd09c16
allow index update to change storage slot # ( #23311 )
2022-03-03 08:40:48 -06:00
d909b7c80b
log hash and lamport result of calculate_accounts_hash_without_index ( #23425 )
2022-03-02 11:10:40 -06:00
7b7fdb42d9
minor test refactoring ( #23423 )
2022-03-02 08:31:14 -06:00
0ad4757159
plumbing for 'other_slot' in 'update_index' ( #23330 )
2022-02-25 12:58:08 -06:00
7ee549e5ae
fix params for store_accounts_frozen ( #23312 )
2022-02-23 17:40:51 -06:00
cafc18c3f9
update_index uses ReadableAccount to reduce params ( #23305 )
2022-02-23 17:01:23 -06:00
a245efe83d
trait for acct data with slot per item ( #23285 )
2022-02-23 15:22:29 -06:00
a102453bae
accounts_index: Add SPL Token account indexing for token-2022 accounts ( #23043 )
2022-02-16 16:23:25 -08:00
a160fc30f2
make max_roots_inclusive clear ( #22942 )
2022-02-07 13:26:53 -06:00
350207b05f
impl ReadableAccount for LoadedAccount ( #22793 )
2022-01-27 16:59:30 -06:00
18d69c8d2b
SortedStoragesIter to remove random access requirement ( #22713 )
2022-01-27 10:25:13 -06:00
8b90084ebc
fix comment typos ( #22723 )
2022-01-25 11:10:42 -06:00
6d5bbca630
Pacify clippy
2022-01-21 19:12:57 -08:00
4abf7a7f88
add ActiveStats to track when long running, expensive bg processes are running ( #22608 )
2022-01-21 10:51:23 -06:00
7d7228c356
cleanup assert ( #22495 )
2022-01-14 11:53:24 -06:00
cddab635ff
Cleanup activated rent_for_sysvars feature ( #22454 )
2022-01-14 20:34:09 +08:00
d3b95b9226
Cleanup AccountStorage struct ( #22463 )
...
* Revert "chore: bump dashmap from 4.0.2 to 5.0.0 (#21824 )"
This reverts commit 8aa3d690b5
2022-01-13 17:51:53 -06:00
637e366b18
Prevent rent-paying account creation ( #22292 )
...
* Fixup typo
* Add new feature
* Add new TransactionError
* Add framework for checking account state before and after transaction processing
* Fail transactions that leave new rent-paying accounts
* Only check rent-state of writable tx accounts
* Review comments: combine process_result success behavior; log and metrics before feature activation
* Fix tests that assume rent-exempt accounts are okay
* Remove test no longer relevant
* Remove native/sysvar special case
* Move metrics submission to report legacy->legacy rent paying transitions as well
2022-01-11 11:32:25 -07:00
a49ef49f87
optimizes ReadOnlyAccountsCache LRU eviction implementation ( #22403 )
...
ReadOnlyAccountsCache is using a background thread, table scan and sort
to implement LRU eviction policy:
https://github.com/solana-labs/solana/blob/eaa52bc93/runtime/src/read_only_accounts_cache.rs#L66-L73
https://github.com/solana-labs/solana/blob/eaa52bc93/runtime/src/read_only_accounts_cache.rs#L186-L191
https://github.com/solana-labs/solana/blob/eaa52bc93/runtime/src/read_only_accounts_cache.rs#L222
DashMap internally locks each shard when accessed; so a table scan in
the background thread can create a lot of lock contention.
This commit adds an index-list queue containing cached keys in the order
that they are accessed. Each hash-map entry also includes its index into
this queue.
When an item is first entered into the cache, it is added to the end of
the queue. Also each time an entry is looked up from the cache it is
moved to the end of queue. As a result, items in the queue are always
sorted in the order that they have last been accessed. When doing LRU
eviction, cache entries are evicted from the front of the queue.
Using index-list, all queue operations above are O(1) with low overhead
and so above achieves an efficient implementation of LRU cache eviction
policy.
2022-01-11 17:25:28 +00:00
d90d5ee9b6
Refactor Rent::due() with RentDue enum ( #22346 )
2022-01-08 09:03:46 -06:00
e0c091a9f4
factor out code in do_shrink_slot_stores ( #22306 )
2022-01-06 10:49:24 -06:00
100293c4b5
refactor get_store_for_shrink ( #22307 )
2022-01-06 10:49:01 -06:00
ab13e39518
Only sum accounts data len from non-zero lamport accounts ( #22309 )
2022-01-05 23:23:37 +00:00
fb62407232
AcctIdx: appendvecid: u32 ( #21842 )
2022-01-03 10:35:17 -06:00
729698e815
AcctIdx: items() uses held ranges ( #21954 )
2021-12-17 09:59:29 -06:00
5d40da5688
log "next_id" to track append vec ids ( #21971 )
2021-12-16 22:41:34 -06:00
e11a1911ad
load_accounts_index_for_shrink ignores cached entries ( #21951 )
2021-12-16 16:37:08 -06:00
347323cbb2
use bg thread pool for shrink ( #21950 )
2021-12-16 15:54:38 -06:00
46e5350d8c
AcctInfo: store offset in AccountInfo as u32 ( #21895 )
2021-12-15 15:41:11 -06:00
e5be96d8bf
AcctIdx: consolidate next_id calls ( #21929 )
2021-12-15 15:39:54 -06:00
41dd31e5f4
AcctIdx: move cached store id to bit ( #21892 )
...
* AcctIdx: move cached store id to bit
* add comments/rename
2021-12-15 11:49:24 -06:00
02fa135815
AcctIdx: create test fn get_test() to isolate changes to AcctIdx::get() ( #21909 )
2021-12-15 09:09:56 -06:00
f4308bdb64
AcctIdx: StoredSize is u32 ( #21894 )
2021-12-14 13:50:19 -06:00
ec583bd12d
AcctIdex: use StorageLocation ( #21853 )
2021-12-14 10:27:17 -06:00
6fc6673ead
AcctInfo: store_id private and add accessor ( #21839 )
2021-12-13 21:35:30 -06:00
98e5ea9dce
AcctIdx: simplify AccountIndexGetResult ( #21857 )
2021-12-13 21:16:17 -06:00
50f26ea9c0
AcctInfo: create AcctInfo with cache explicitly ( #21847 )
2021-12-13 17:35:07 -06:00
6155ef6377
AcctInfo: make offset private, add accessor ( #21838 )
2021-12-13 14:43:26 -06:00
c9e0bde407
AcctIdx: AccountInfo::stored_size private ( #21821 )
2021-12-13 10:10:06 -06:00
025a5a3b9c
AccountInfo: construct with new() ( #21788 )
2021-12-12 14:36:05 -06:00
825f8bcea4
use AppendVecId instead of usize ( #21792 )
2021-12-11 21:38:13 -06:00
c5b6ea74ef
replace some .lamports checks with trait ( #21783 )
2021-12-11 21:37:33 -06:00
4c0373b72a
introduce AtomicAppendVecId ( #21793 )
2021-12-11 21:34:35 -06:00
ef7f49a21d
move AccountInfo to its own file ( #21784 )
2021-12-11 11:47:05 -06:00
ec7e17787e
Compute accounts data len during generate_index() ( #21757 )
2021-12-10 13:27:59 -06:00
1528f85112
Add return types to generate_index() ( #21735 )
2021-12-09 17:15:13 -06:00
a2df1eb502
AcctIdx: disk generate index and filler accounts use more threads ( #21566 )
2021-12-09 11:54:14 -06:00
b8837c04ec
Reformat imports to a consistent style for imports
...
rustfmt.toml configuration:
imports_granularity = "One"
group_imports = "One"
2021-12-03 09:19:13 -08:00
f0b32b75ab
AcctIdx: use ZeroLamport trait ( #21552 )
2021-12-02 12:10:11 -06:00
308d7d40d0
aggressively flush cache based on cache size ( #21332 )
...
* aggressively flush cache based on cache size
* find_older_frozen_slots -> cached_frozen_slots
* remove 'WRITE_CACHE_LIMIT_BYTES_DEFAULT'
* tweaks to stats
* fix tests
2021-12-01 13:10:48 -06:00
7ec88226ee
AcctIdx: stats are usize ( #21493 )
...
* AcctIdx: stats are usize
* rename test function
2021-11-30 11:35:21 -06:00
ba9dfa0d22
Remove frozen account support
2021-11-29 08:38:11 -08:00
c4d68063c7
Add timing for accounts add_root ( #21379 )
2021-11-22 18:29:45 +01:00
02bc4e3fc1
spl-token: New program feature flag ( #21354 )
...
* spl-token: Add feature flag for new release
* Remove all spl token version declarations
2021-11-21 14:27:03 +01:00
79d21d6805
add --accounts-index-scan-results-limit-mb to allow scans to abort ( #21327 )
...
* ScanConfig -> &ScanConfig
* add --accounts-index-scan-results-limit-mb to allow scans to abort
* feedback
2021-11-19 09:00:19 -06:00
c24e30f689
Constrain memory ordering on AccountsDb::next_id ( #21311 )
2021-11-18 20:34:37 -06:00
f7152c889c
Constrain memory ordering on AppendVec::id ( #21310 )
2021-11-18 20:34:20 -06:00
5788973153
Constrain memory ordering on AccountStorageEntry::alive_bytes ( #21309 )
2021-11-18 20:34:08 -06:00
9cb74e8421
remove store_cached stall ( #21326 )
2021-11-17 18:30:40 -06:00
0f69a14247
Add ability to abort scan ( #21314 )
2021-11-17 13:10:29 -06:00
c3e5927d16
Constrain memory ordering on AccountsDb::write_version ( #21312 )
2021-11-17 07:11:16 -06:00
39340ed25b
throttle store_cached when cache size is too large ( #21188 )
...
* throttle store_cached when cache size is too large
* reduce max delay
* 100ms max
* 10ms max delay
2021-11-13 14:00:37 -06:00
171243a370
add rent exempt count to get_total_accounts_stats ( #21153 )
2021-11-04 11:17:19 -05:00
76e533be46
LoadedAccountAccessor::Cached does not store Pubkey ( #21030 )
2021-11-04 10:28:04 -05:00
b1e3a82c11
refactor rent collector and logs ( #21155 )
2021-11-04 06:28:41 -05:00
c59c56c0f8
log rent exempt accounts ( #21137 )
2021-11-03 14:44:52 -05:00
c0952831be
correctly clean filler accounts ( #21139 )
2021-11-03 11:27:15 -05:00
7a41b2c5d1
optimize get_accounts_delta_hash ( #21027 )
2021-10-28 10:40:32 -05:00
e9ab214237
filler accts: only add filler accts to slots in the current epoch ( #21024 )
2021-10-28 10:26:30 -05:00
9d330fc638
FillerAccts: use variable cycle partitions ( #20963 )
2021-10-27 11:18:27 -05:00
bbe3ce3db5
Accountsdb plugin write ordering ( #20948 )
...
Use the write_version in the Accounts's meta data so that account write with lower write_version would not overwrite the higher ones.
2021-10-25 14:07:56 -07:00
9e85499d4c
fix hash calc # storages, # slots ( #20832 )
2021-10-25 19:37:50 +00:00
43ea579f63
add cli for --accounts-hash-num-passes ( #20827 )
2021-10-25 09:45:46 -05:00
f14365f4b4
Accountsdb plugin postgres -- bulk insertion at startup ( #20763 )
...
Use bulk insertion to Postgres at startup to reduce time taken for initial snapshot restore for postgres plugin. Avoid duplicate writes of accounts at startup. Doing account plugin notification and indexing in parallel.
Improved error handling for postgres plugin to show the real db issues for debug purpose
Added more metrics for postgres plugin.
Refactored plugin centric code out to a sub module from accounts_db and added unit tests
2021-10-24 12:43:33 -07:00
47a58a38c2
clean_accounts calls AcctIdx: get_many ( #20715 )
...
* AcctIdx: get_many
* keep read lock
* AcctIdx: get_many optionally adds to cache
* rename
2021-10-19 15:54:06 -05:00
2b76ea51b4
Reduce visibility of `Hash` struct contents
2021-10-19 09:30:47 -07:00
46bf7d4a4a
AcctIdx: support 2^24 bins ( #20739 )
2021-10-18 14:05:16 -05:00
6ec159a1ae
parallelize update_index ( #20601 )
2021-10-18 09:07:01 -05:00
0d934d311e
sort shrink acct indexes ( #20708 )
2021-10-18 09:06:32 -05:00
70b2e5fef2
hash calc uses all threads during startup ( #20735 )
2021-10-15 17:41:23 -05:00
254ad45878
sort pubkeys for clean ( #20706 )
2021-10-15 14:31:06 -05:00
6d7da6dbee
clean:clone uncleaned_roots for loop ( #20714 )
2021-10-15 13:18:00 -05:00
1e064c32e2
pull append_vec_id() out of loop ( #20713 )
2021-10-15 10:41:23 -05:00
00b9056004
clean metric: total_us ( #20707 )
2021-10-15 10:41:05 -05:00
7eead4e25f
clean stats on acct idx lookup ( #20632 )
2021-10-14 08:37:42 -05:00
c43b6e0248
log metric for clean_accounts.delta_insert_us ( #20634 )
2021-10-14 08:37:14 -05:00
aec9d8bf2f
add clean stats ( #20631 )
2021-10-13 08:22:11 -05:00
a8e000a2a6
add filler accounts to bloat validator and predict failure ( #20491 )
...
* add filler accounts to bloat validator and predict failure
* assert no accounts match filler
* cleanup magic numbers
* panic if can't load from snapshot with filler accounts specified
* some renames
* renames
* into_par_iter
* clean filler accts, too
2021-10-11 12:46:27 -05:00
df39b37cb8
improve clean stats ( #20469 )
2021-10-06 20:04:26 -05:00
7027d56064
Resolve nightly-2021-10-05 clippy complaints
2021-10-06 10:37:58 -07:00
895b46e0c3
simplify AccountsDb::new_with_config ( #20473 )
2021-10-06 11:41:23 -05:00
64cf354651
Add clean metrics ( #20410 )
2021-10-04 18:21:45 -05:00
2d78f8ad2a
parallelize unref_from_storage from clean_dead_slots_from_accounts_index ( #20411 )
2021-10-04 18:20:10 -05:00
b6a6d6986d
lift store.slot() from loop ( #20409 )
2021-10-04 16:07:01 -05:00
fe97cb2ddf
AccountsDb plugin framework ( #20047 )
...
Summary of Changes
Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows
Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.
The code comprises 4 major parts:
accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
2021-09-30 14:26:17 -07:00
9d91a6f1b7
shrink slots at startup uses more threads ( #20251 )
2021-09-27 14:14:55 -05:00
Jeff Washington (jwash)
HaoranYi
Michael Vines
Justin Starry
Tyera Eulberg
behzad nouri
Brooks Prumo
sakridge
Jon Cinque
Jeff Washington (jwash)
Lijun Wang