a102453bae
accounts_index: Add SPL Token account indexing for token-2022 accounts ( #23043 )
2022-02-16 16:23:25 -08:00
a160fc30f2
make max_roots_inclusive clear ( #22942 )
2022-02-07 13:26:53 -06:00
350207b05f
impl ReadableAccount for LoadedAccount ( #22793 )
2022-01-27 16:59:30 -06:00
18d69c8d2b
SortedStoragesIter to remove random access requirement ( #22713 )
2022-01-27 10:25:13 -06:00
8b90084ebc
fix comment typos ( #22723 )
2022-01-25 11:10:42 -06:00
6d5bbca630
Pacify clippy
2022-01-21 19:12:57 -08:00
4abf7a7f88
add ActiveStats to track when long running, expensive bg processes are running ( #22608 )
2022-01-21 10:51:23 -06:00
7d7228c356
cleanup assert ( #22495 )
2022-01-14 11:53:24 -06:00
cddab635ff
Cleanup activated rent_for_sysvars feature ( #22454 )
2022-01-14 20:34:09 +08:00
d3b95b9226
Cleanup AccountStorage struct ( #22463 )
...
* Revert "chore: bump dashmap from 4.0.2 to 5.0.0 (#21824 )"
This reverts commit 8aa3d690b5
2022-01-13 17:51:53 -06:00
637e366b18
Prevent rent-paying account creation ( #22292 )
...
* Fixup typo
* Add new feature
* Add new TransactionError
* Add framework for checking account state before and after transaction processing
* Fail transactions that leave new rent-paying accounts
* Only check rent-state of writable tx accounts
* Review comments: combine process_result success behavior; log and metrics before feature activation
* Fix tests that assume rent-exempt accounts are okay
* Remove test no longer relevant
* Remove native/sysvar special case
* Move metrics submission to report legacy->legacy rent paying transitions as well
2022-01-11 11:32:25 -07:00
a49ef49f87
optimizes ReadOnlyAccountsCache LRU eviction implementation ( #22403 )
...
ReadOnlyAccountsCache is using a background thread, table scan and sort
to implement LRU eviction policy:
https://github.com/solana-labs/solana/blob/eaa52bc93/runtime/src/read_only_accounts_cache.rs#L66-L73
https://github.com/solana-labs/solana/blob/eaa52bc93/runtime/src/read_only_accounts_cache.rs#L186-L191
https://github.com/solana-labs/solana/blob/eaa52bc93/runtime/src/read_only_accounts_cache.rs#L222
DashMap internally locks each shard when accessed; so a table scan in
the background thread can create a lot of lock contention.
This commit adds an index-list queue containing cached keys in the order
that they are accessed. Each hash-map entry also includes its index into
this queue.
When an item is first entered into the cache, it is added to the end of
the queue. Also each time an entry is looked up from the cache it is
moved to the end of queue. As a result, items in the queue are always
sorted in the order that they have last been accessed. When doing LRU
eviction, cache entries are evicted from the front of the queue.
Using index-list, all queue operations above are O(1) with low overhead
and so above achieves an efficient implementation of LRU cache eviction
policy.
2022-01-11 17:25:28 +00:00
d90d5ee9b6
Refactor Rent::due() with RentDue enum ( #22346 )
2022-01-08 09:03:46 -06:00
e0c091a9f4
factor out code in do_shrink_slot_stores ( #22306 )
2022-01-06 10:49:24 -06:00
100293c4b5
refactor get_store_for_shrink ( #22307 )
2022-01-06 10:49:01 -06:00
ab13e39518
Only sum accounts data len from non-zero lamport accounts ( #22309 )
2022-01-05 23:23:37 +00:00
fb62407232
AcctIdx: appendvecid: u32 ( #21842 )
2022-01-03 10:35:17 -06:00
729698e815
AcctIdx: items() uses held ranges ( #21954 )
2021-12-17 09:59:29 -06:00
5d40da5688
log "next_id" to track append vec ids ( #21971 )
2021-12-16 22:41:34 -06:00
e11a1911ad
load_accounts_index_for_shrink ignores cached entries ( #21951 )
2021-12-16 16:37:08 -06:00
347323cbb2
use bg thread pool for shrink ( #21950 )
2021-12-16 15:54:38 -06:00
46e5350d8c
AcctInfo: store offset in AccountInfo as u32 ( #21895 )
2021-12-15 15:41:11 -06:00
e5be96d8bf
AcctIdx: consolidate next_id calls ( #21929 )
2021-12-15 15:39:54 -06:00
41dd31e5f4
AcctIdx: move cached store id to bit ( #21892 )
...
* AcctIdx: move cached store id to bit
* add comments/rename
2021-12-15 11:49:24 -06:00
02fa135815
AcctIdx: create test fn get_test() to isolate changes to AcctIdx::get() ( #21909 )
2021-12-15 09:09:56 -06:00
f4308bdb64
AcctIdx: StoredSize is u32 ( #21894 )
2021-12-14 13:50:19 -06:00
ec583bd12d
AcctIdex: use StorageLocation ( #21853 )
2021-12-14 10:27:17 -06:00
6fc6673ead
AcctInfo: store_id private and add accessor ( #21839 )
2021-12-13 21:35:30 -06:00
98e5ea9dce
AcctIdx: simplify AccountIndexGetResult ( #21857 )
2021-12-13 21:16:17 -06:00
50f26ea9c0
AcctInfo: create AcctInfo with cache explicitly ( #21847 )
2021-12-13 17:35:07 -06:00
6155ef6377
AcctInfo: make offset private, add accessor ( #21838 )
2021-12-13 14:43:26 -06:00
c9e0bde407
AcctIdx: AccountInfo::stored_size private ( #21821 )
2021-12-13 10:10:06 -06:00
025a5a3b9c
AccountInfo: construct with new() ( #21788 )
2021-12-12 14:36:05 -06:00
825f8bcea4
use AppendVecId instead of usize ( #21792 )
2021-12-11 21:38:13 -06:00
c5b6ea74ef
replace some .lamports checks with trait ( #21783 )
2021-12-11 21:37:33 -06:00
4c0373b72a
introduce AtomicAppendVecId ( #21793 )
2021-12-11 21:34:35 -06:00
ef7f49a21d
move AccountInfo to its own file ( #21784 )
2021-12-11 11:47:05 -06:00
ec7e17787e
Compute accounts data len during generate_index() ( #21757 )
2021-12-10 13:27:59 -06:00
1528f85112
Add return types to generate_index() ( #21735 )
2021-12-09 17:15:13 -06:00
a2df1eb502
AcctIdx: disk generate index and filler accounts use more threads ( #21566 )
2021-12-09 11:54:14 -06:00
b8837c04ec
Reformat imports to a consistent style for imports
...
rustfmt.toml configuration:
imports_granularity = "One"
group_imports = "One"
2021-12-03 09:19:13 -08:00
f0b32b75ab
AcctIdx: use ZeroLamport trait ( #21552 )
2021-12-02 12:10:11 -06:00
308d7d40d0
aggressively flush cache based on cache size ( #21332 )
...
* aggressively flush cache based on cache size
* find_older_frozen_slots -> cached_frozen_slots
* remove 'WRITE_CACHE_LIMIT_BYTES_DEFAULT'
* tweaks to stats
* fix tests
2021-12-01 13:10:48 -06:00
7ec88226ee
AcctIdx: stats are usize ( #21493 )
...
* AcctIdx: stats are usize
* rename test function
2021-11-30 11:35:21 -06:00
ba9dfa0d22
Remove frozen account support
2021-11-29 08:38:11 -08:00
c4d68063c7
Add timing for accounts add_root ( #21379 )
2021-11-22 18:29:45 +01:00
02bc4e3fc1
spl-token: New program feature flag ( #21354 )
...
* spl-token: Add feature flag for new release
* Remove all spl token version declarations
2021-11-21 14:27:03 +01:00
79d21d6805
add --accounts-index-scan-results-limit-mb to allow scans to abort ( #21327 )
...
* ScanConfig -> &ScanConfig
* add --accounts-index-scan-results-limit-mb to allow scans to abort
* feedback
2021-11-19 09:00:19 -06:00
c24e30f689
Constrain memory ordering on AccountsDb::next_id ( #21311 )
2021-11-18 20:34:37 -06:00
f7152c889c
Constrain memory ordering on AppendVec::id ( #21310 )
2021-11-18 20:34:20 -06:00
5788973153
Constrain memory ordering on AccountStorageEntry::alive_bytes ( #21309 )
2021-11-18 20:34:08 -06:00
9cb74e8421
remove store_cached stall ( #21326 )
2021-11-17 18:30:40 -06:00
0f69a14247
Add ability to abort scan ( #21314 )
2021-11-17 13:10:29 -06:00
c3e5927d16
Constrain memory ordering on AccountsDb::write_version ( #21312 )
2021-11-17 07:11:16 -06:00
39340ed25b
throttle store_cached when cache size is too large ( #21188 )
...
* throttle store_cached when cache size is too large
* reduce max delay
* 100ms max
* 10ms max delay
2021-11-13 14:00:37 -06:00
171243a370
add rent exempt count to get_total_accounts_stats ( #21153 )
2021-11-04 11:17:19 -05:00
76e533be46
LoadedAccountAccessor::Cached does not store Pubkey ( #21030 )
2021-11-04 10:28:04 -05:00
b1e3a82c11
refactor rent collector and logs ( #21155 )
2021-11-04 06:28:41 -05:00
c59c56c0f8
log rent exempt accounts ( #21137 )
2021-11-03 14:44:52 -05:00
c0952831be
correctly clean filler accounts ( #21139 )
2021-11-03 11:27:15 -05:00
7a41b2c5d1
optimize get_accounts_delta_hash ( #21027 )
2021-10-28 10:40:32 -05:00
e9ab214237
filler accts: only add filler accts to slots in the current epoch ( #21024 )
2021-10-28 10:26:30 -05:00
9d330fc638
FillerAccts: use variable cycle partitions ( #20963 )
2021-10-27 11:18:27 -05:00
bbe3ce3db5
Accountsdb plugin write ordering ( #20948 )
...
Use the write_version in the Accounts's meta data so that account write with lower write_version would not overwrite the higher ones.
2021-10-25 14:07:56 -07:00
9e85499d4c
fix hash calc # storages, # slots ( #20832 )
2021-10-25 19:37:50 +00:00
43ea579f63
add cli for --accounts-hash-num-passes ( #20827 )
2021-10-25 09:45:46 -05:00
f14365f4b4
Accountsdb plugin postgres -- bulk insertion at startup ( #20763 )
...
Use bulk insertion to Postgres at startup to reduce time taken for initial snapshot restore for postgres plugin. Avoid duplicate writes of accounts at startup. Doing account plugin notification and indexing in parallel.
Improved error handling for postgres plugin to show the real db issues for debug purpose
Added more metrics for postgres plugin.
Refactored plugin centric code out to a sub module from accounts_db and added unit tests
2021-10-24 12:43:33 -07:00
47a58a38c2
clean_accounts calls AcctIdx: get_many ( #20715 )
...
* AcctIdx: get_many
* keep read lock
* AcctIdx: get_many optionally adds to cache
* rename
2021-10-19 15:54:06 -05:00
2b76ea51b4
Reduce visibility of `Hash` struct contents
2021-10-19 09:30:47 -07:00
46bf7d4a4a
AcctIdx: support 2^24 bins ( #20739 )
2021-10-18 14:05:16 -05:00
6ec159a1ae
parallelize update_index ( #20601 )
2021-10-18 09:07:01 -05:00
0d934d311e
sort shrink acct indexes ( #20708 )
2021-10-18 09:06:32 -05:00
70b2e5fef2
hash calc uses all threads during startup ( #20735 )
2021-10-15 17:41:23 -05:00
254ad45878
sort pubkeys for clean ( #20706 )
2021-10-15 14:31:06 -05:00
6d7da6dbee
clean:clone uncleaned_roots for loop ( #20714 )
2021-10-15 13:18:00 -05:00
1e064c32e2
pull append_vec_id() out of loop ( #20713 )
2021-10-15 10:41:23 -05:00
00b9056004
clean metric: total_us ( #20707 )
2021-10-15 10:41:05 -05:00
7eead4e25f
clean stats on acct idx lookup ( #20632 )
2021-10-14 08:37:42 -05:00
c43b6e0248
log metric for clean_accounts.delta_insert_us ( #20634 )
2021-10-14 08:37:14 -05:00
aec9d8bf2f
add clean stats ( #20631 )
2021-10-13 08:22:11 -05:00
a8e000a2a6
add filler accounts to bloat validator and predict failure ( #20491 )
...
* add filler accounts to bloat validator and predict failure
* assert no accounts match filler
* cleanup magic numbers
* panic if can't load from snapshot with filler accounts specified
* some renames
* renames
* into_par_iter
* clean filler accts, too
2021-10-11 12:46:27 -05:00
df39b37cb8
improve clean stats ( #20469 )
2021-10-06 20:04:26 -05:00
7027d56064
Resolve nightly-2021-10-05 clippy complaints
2021-10-06 10:37:58 -07:00
895b46e0c3
simplify AccountsDb::new_with_config ( #20473 )
2021-10-06 11:41:23 -05:00
64cf354651
Add clean metrics ( #20410 )
2021-10-04 18:21:45 -05:00
2d78f8ad2a
parallelize unref_from_storage from clean_dead_slots_from_accounts_index ( #20411 )
2021-10-04 18:20:10 -05:00
b6a6d6986d
lift store.slot() from loop ( #20409 )
2021-10-04 16:07:01 -05:00
fe97cb2ddf
AccountsDb plugin framework ( #20047 )
...
Summary of Changes
Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows
Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.
The code comprises 4 major parts:
accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
2021-09-30 14:26:17 -07:00
9d91a6f1b7
shrink slots at startup uses more threads ( #20251 )
2021-09-27 14:14:55 -05:00
1347b507da
Handle duplicate AppendVec IDs ( #20096 )
...
When reconstructing the AccountsDb, if the storages came from full and
incremental snapshots generated on different nodes, it's possible that
the AppendVec IDs could overlap/have duplicates, which would cause the
reconstruction to fail.
This commit handles this issue by unconditionally remapping the
AppendVec ID for every AppendVec.
Fixes #17088
2021-09-23 17:34:32 -05:00
742155c214
call set_startup and add metrics on generate_index ( #20006 )
2021-09-18 22:08:58 -05:00
fc647eed19
introduce trait IndexValue for accounts index ( #19859 )
2021-09-14 17:51:07 -05:00
2a795614d3
accounts index keys returns by value ( #19858 )
2021-09-14 09:38:59 -05:00
b57e86abf2
cache account hash info ( #19426 )
...
* cache account hash info
* ledger_path -> accounts_hash_cache_path
2021-09-13 20:39:26 -05:00
ef749a2506
Flush accounts cache before calling clean_accounts() ( #19839 )
2021-09-13 14:56:13 -05:00
b992c02708
in_mem_accounts_index filters by range ( #19779 )
2021-09-12 21:57:15 -05:00
91c3b18e1e
InMemAccountsIndex iter returns vec<owned> ( #19778 )
2021-09-11 20:33:04 +00:00
2de2c6ddd7
log accounts loaded for shrink stats ( #19747 )
2021-09-10 12:51:58 -05:00
97418bede3
introduce BinnedHashData type to improve some code readability ( #19722 )
2021-09-08 19:25:52 -05:00
001ed66b4d
minor constant cleanup ( #19721 )
2021-09-08 18:26:10 -05:00
38bbb77989
Return error if Transaction contains writable executable or ProgramData accounts ( #19629 )
...
* Return error if Transaction locks an executable as writable
* Return error if a ProgramData account is writable but the upgradable loader isn't present
* Remove unreachable clause
* Fixup bpf tests
* Review comments
* Add new TransactionError
* Disallow writes to any upgradeable-loader account when loader not present; remove is_upgradeable_loader_present exception for all other executables
2021-09-08 15:21:52 -06:00
456bf15012
AccountsIndexConfig -> AccountsDbConfig ( #19687 )
2021-09-08 04:30:38 +00:00
982454a455
refactor do_shrink_slot_stores ( #19560 )
2021-09-07 15:10:49 -05:00
b3fae0a01a
move constants in functions to global space ( #19622 )
2021-09-03 19:46:42 -05:00
3936c10aa0
rename var ( #19588 )
2021-09-03 00:55:35 +00:00
7eb793a55e
add debugging to test ( #19563 )
...
* add debugging to test
* update cargo
2021-09-02 02:52:16 +00:00
ed2c071fe1
remove unused is_startup ( #19561 )
2021-09-01 18:39:00 -05:00
a5f74d86c8
make test pass with 1 bin to remove noisy failure during testing ( #19554 )
2021-09-01 15:58:56 -05:00
dd9481c403
parallelize index_read in shrink ( #19506 )
2021-09-01 15:40:59 -05:00
82a6bbe068
add Debug trait to range for accounts index ( #19531 )
2021-09-01 08:13:56 -05:00
12dc8749cf
add Copy trait to AccountInfo for fast copies to mmapped file ( #19524 )
2021-09-01 08:13:08 -05:00
e1939688c2
parallel shrink ( #19507 )
2021-08-31 11:57:34 -05:00
0e94b8fd4e
remove redundant cancelled_shrink ( #19522 )
2021-08-31 11:51:48 -05:00
9d3afba045
Debug shrink stats ( #19505 )
...
* add some shrink metrics
* renames
2021-08-31 09:04:56 -05:00
09458cc802
Shrink mmap leakage ( #19373 )
...
* Putting stores satisfying future shrink into the candidate list.
* Fixed unit tests for select_shrink_candidates
* Added metrics on stores readded back for next round
* Ehance the metrics on accountsdb to report the total bytes and alive bytes and alive ratio
* Enhance select_shrink_candidates metrics
2021-08-31 00:55:16 -07:00
02b050e0f5
replace AccountsIndex btree with hashmap of 8k bins ( #19212 )
2021-08-26 18:12:43 -05:00
98bc694606
hash calculation adds really old slots to dirty_stores ( #19434 )
2021-08-26 14:32:43 -05:00
535de3b302
add test for test_clean_nonrooted ( #19409 )
2021-08-26 13:56:09 -05:00
234461f779
Add base_slot to get_snapshot_storages() ( #19348 )
2021-08-20 16:23:43 -05:00
6327e006df
CalculateHashIntermediate::new_without_slot -> new ( #19274 )
2021-08-18 09:07:34 -05:00
7c70f2158b
accounts_index_bins to AccountsIndexConfig ( #19257 )
...
* accounts_index_bins to AccountsIndexConfig
* rename param bins -> config
* rename BINS_FOR* to ACCOUNTS_INDEX_CONFIG_FOR*
2021-08-17 14:50:01 -05:00
26e963f436
remove par iter for 32k wide code path ( #18764 )
2021-08-13 10:07:47 -05:00
5fb6b341c4
Handle cleaning of zero-lamport accounts w.r.t. Incremental Snapshots ( #18870 )
...
* Handle cleaning zero-lamport accounts
Handle cleaning zero-lamport accounts in slots higher than the last full
snapshot slot. This is part of the Incremental Snapshot work.
Fixes #18825
2021-08-12 15:56:08 -05:00
651343688d
Use option for account index bucket parameter ( #19150 )
2021-08-10 11:32:25 -05:00
c18bd08021
calculate store info while generating index ( #19107 )
...
* calculate store info while generating index
* update store info during index generation pass
2021-08-10 07:39:59 -05:00
47e0d9aa95
plumb more accounts_index bins ( #19123 )
2021-08-10 05:45:46 -05:00
f506da5e19
introduce StorageSizeAndCount ( #19137 )
2021-08-09 22:20:03 -05:00
c56bccdf32
clean up types on account index generation ( #19105 )
2021-08-09 17:29:36 -05:00
1951fe4e80
accounts default stragglers ( #19101 )
2021-08-09 09:00:30 -05:00
9616ae0f2c
signal to upsert whether reclaims are important ( #18900 )
...
* signal to upsert whether reclaims are important
* reclaims_must_be_empty -> previous_slot_entry_was_cached
* UPSERT_RECLAIMS_MUST_BE_EMPTY_FALSE -> UPSERT_PREVIOUS_SLOT_ENTRY_WAS_CACHED_FALSE
2021-08-09 08:58:59 -05:00
7923c26939
AccountsDb::new_with_config_for_tests ( #19103 )
2021-08-06 17:18:16 -05:00
c12289fd1b
remove AccountsDb::new() from public api ( #19098 )
2021-08-06 13:07:50 -05:00
592013eaf4
Clean within shrink_all_slots ( #19042 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-08-05 23:26:38 -07:00
0028442e14
rework defaults and construction of accountsdb, accounts, bank ( #19083 )
...
* rework defaults and construction of accountsdb, accounts, bank
* merge issues
2021-08-05 17:27:13 -05:00
24207a09ac
remove AccountsIndex::default ( #19082 )
...
* accounts_db calls AccountsDb::new(bins)
* remove AccountsIndex::default
2021-08-05 11:38:53 -05:00
5cf28689e6
accounts_db calls AccountsDb::new(bins) ( #19068 )
2021-08-05 11:15:26 -05:00
6a995f5dfd
rename to AccountsDb::new_single_for_tests ( #19039 )
2021-08-04 11:47:11 -05:00
31a620c42b
move towards accounts index being dynamic ( #19032 )
2021-08-04 09:18:05 -05:00
84e78316b1
Write helper for multithread update ( #18808 )
2021-07-29 03:16:36 +02:00
5cabb5bb11
Fix race in remove_unrooted_race and flush_slot_cache ( #18785 )
2021-07-22 13:49:08 -07:00
da9fdd785c
measure and speed up 2nd half of generate_index ( #18503 )
...
* add metrics and speedup storage calcs in generate_index
* add tests, refactor
* rename map variable
* rename another index variable
2021-07-15 10:26:00 -05:00
d092fa1f03
add ledger-tool verify verify-accounts-index option ( #18375 )
...
* add ledger-tool verify verify-accounts-index option
* comment, merge, respond to feedback, cleanup
2021-07-13 11:06:18 -05:00
eca0ceb04c
eliminate unnecessary copies in accounts index generation ( #18466 )
2021-07-07 15:36:05 -05:00
49c4e54b28
accounts index insert uses pubkey by value ( #18460 )
2021-07-07 15:35:35 -05:00
d951916d40
Make test_hash_stored_account pass in release mode
...
The hash differs between debug and release mode, which makes the test
fail when running under "cargo test --release". Use cfg! to include a
different expected hash in release mode.
2021-07-01 18:27:24 -07:00
531f1bce78
report on min/max bin stats for accounts index ( #18338 )
2021-07-01 00:46:59 +00:00
47cafb70da
test ( #18159 )
2021-06-28 18:27:26 -07:00
afd64d27c9
bin accounts index ( #18111 )
2021-06-28 15:03:57 -05:00
e06376664b
replace account index bulk insert with iterator ( #18198 )
2021-06-25 14:31:55 -05:00
f2a2581259
metric for accounts index insertion time ( #18202 )
2021-06-25 08:36:55 -05:00
Michael Vines
Jeff Washington (jwash)
Justin Starry
Tyera Eulberg
behzad nouri
Brooks Prumo
sakridge
Jon Cinque
Jeff Washington (jwash)
Lijun Wang
carllin
Ruud van Asseldonk