Tyera Eulberg
38bbb77989
Return error if Transaction contains writable executable or ProgramData accounts ( #19629 )
...
* Return error if Transaction locks an executable as writable
* Return error if a ProgramData account is writable but the upgradable loader isn't present
* Remove unreachable clause
* Fixup bpf tests
* Review comments
* Add new TransactionError
* Disallow writes to any upgradeable-loader account when loader not present; remove is_upgradeable_loader_present exception for all other executables
2021-09-08 15:21:52 -06:00
Jeff Washington (jwash)
456bf15012
AccountsIndexConfig -> AccountsDbConfig ( #19687 )
2021-09-08 04:30:38 +00:00
Jeff Washington (jwash)
982454a455
refactor do_shrink_slot_stores ( #19560 )
2021-09-07 15:10:49 -05:00
Jeff Washington (jwash)
b3fae0a01a
move constants in functions to global space ( #19622 )
2021-09-03 19:46:42 -05:00
Jeff Washington (jwash)
3936c10aa0
rename var ( #19588 )
2021-09-03 00:55:35 +00:00
Jeff Washington (jwash)
7eb793a55e
add debugging to test ( #19563 )
...
* add debugging to test
* update cargo
2021-09-02 02:52:16 +00:00
Jeff Washington (jwash)
ed2c071fe1
remove unused is_startup ( #19561 )
2021-09-01 18:39:00 -05:00
Jeff Washington (jwash)
a5f74d86c8
make test pass with 1 bin to remove noisy failure during testing ( #19554 )
2021-09-01 15:58:56 -05:00
Jeff Washington (jwash)
dd9481c403
parallelize index_read in shrink ( #19506 )
2021-09-01 15:40:59 -05:00
Jeff Washington (jwash)
82a6bbe068
add Debug trait to range for accounts index ( #19531 )
2021-09-01 08:13:56 -05:00
Jeff Washington (jwash)
12dc8749cf
add Copy trait to AccountInfo for fast copies to mmapped file ( #19524 )
2021-09-01 08:13:08 -05:00
Jeff Washington (jwash)
e1939688c2
parallel shrink ( #19507 )
2021-08-31 11:57:34 -05:00
Jeff Washington (jwash)
0e94b8fd4e
remove redundant cancelled_shrink ( #19522 )
2021-08-31 11:51:48 -05:00
Jeff Washington (jwash)
9d3afba045
Debug shrink stats ( #19505 )
...
* add some shrink metrics
* renames
2021-08-31 09:04:56 -05:00
Lijun Wang
09458cc802
Shrink mmap leakage ( #19373 )
...
* Putting stores satisfying future shrink into the candidate list.
* Fixed unit tests for select_shrink_candidates
* Added metrics on stores readded back for next round
* Ehance the metrics on accountsdb to report the total bytes and alive bytes and alive ratio
* Enhance select_shrink_candidates metrics
2021-08-31 00:55:16 -07:00
Jeff Washington (jwash)
02b050e0f5
replace AccountsIndex btree with hashmap of 8k bins ( #19212 )
2021-08-26 18:12:43 -05:00
Jeff Washington (jwash)
98bc694606
hash calculation adds really old slots to dirty_stores ( #19434 )
2021-08-26 14:32:43 -05:00
Jeff Washington (jwash)
535de3b302
add test for test_clean_nonrooted ( #19409 )
2021-08-26 13:56:09 -05:00
Brooks Prumo
234461f779
Add base_slot to get_snapshot_storages() ( #19348 )
2021-08-20 16:23:43 -05:00
Jeff Washington (jwash)
6327e006df
CalculateHashIntermediate::new_without_slot -> new ( #19274 )
2021-08-18 09:07:34 -05:00
Jeff Washington (jwash)
7c70f2158b
accounts_index_bins to AccountsIndexConfig ( #19257 )
...
* accounts_index_bins to AccountsIndexConfig
* rename param bins -> config
* rename BINS_FOR* to ACCOUNTS_INDEX_CONFIG_FOR*
2021-08-17 14:50:01 -05:00
Jeff Washington (jwash)
26e963f436
remove par iter for 32k wide code path ( #18764 )
2021-08-13 10:07:47 -05:00
Brooks Prumo
5fb6b341c4
Handle cleaning of zero-lamport accounts w.r.t. Incremental Snapshots ( #18870 )
...
* Handle cleaning zero-lamport accounts
Handle cleaning zero-lamport accounts in slots higher than the last full
snapshot slot. This is part of the Incremental Snapshot work.
Fixes #18825
2021-08-12 15:56:08 -05:00
Jeff Washington (jwash)
651343688d
Use option for account index bucket parameter ( #19150 )
2021-08-10 11:32:25 -05:00
Jeff Washington (jwash)
c18bd08021
calculate store info while generating index ( #19107 )
...
* calculate store info while generating index
* update store info during index generation pass
2021-08-10 07:39:59 -05:00
Jeff Washington (jwash)
47e0d9aa95
plumb more accounts_index bins ( #19123 )
2021-08-10 05:45:46 -05:00
Jeff Washington (jwash)
f506da5e19
introduce StorageSizeAndCount ( #19137 )
2021-08-09 22:20:03 -05:00
Jeff Washington (jwash)
c56bccdf32
clean up types on account index generation ( #19105 )
2021-08-09 17:29:36 -05:00
Jeff Washington (jwash)
1951fe4e80
accounts default stragglers ( #19101 )
2021-08-09 09:00:30 -05:00
Jeff Washington (jwash)
9616ae0f2c
signal to upsert whether reclaims are important ( #18900 )
...
* signal to upsert whether reclaims are important
* reclaims_must_be_empty -> previous_slot_entry_was_cached
* UPSERT_RECLAIMS_MUST_BE_EMPTY_FALSE -> UPSERT_PREVIOUS_SLOT_ENTRY_WAS_CACHED_FALSE
2021-08-09 08:58:59 -05:00
Jeff Washington (jwash)
7923c26939
AccountsDb::new_with_config_for_tests ( #19103 )
2021-08-06 17:18:16 -05:00
Jeff Washington (jwash)
c12289fd1b
remove AccountsDb::new() from public api ( #19098 )
2021-08-06 13:07:50 -05:00
sakridge
592013eaf4
Clean within shrink_all_slots ( #19042 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-08-05 23:26:38 -07:00
Jeff Washington (jwash)
0028442e14
rework defaults and construction of accountsdb, accounts, bank ( #19083 )
...
* rework defaults and construction of accountsdb, accounts, bank
* merge issues
2021-08-05 17:27:13 -05:00
Jeff Washington (jwash)
24207a09ac
remove AccountsIndex::default ( #19082 )
...
* accounts_db calls AccountsDb::new(bins)
* remove AccountsIndex::default
2021-08-05 11:38:53 -05:00
Jeff Washington (jwash)
5cf28689e6
accounts_db calls AccountsDb::new(bins) ( #19068 )
2021-08-05 11:15:26 -05:00
Jeff Washington (jwash)
6a995f5dfd
rename to AccountsDb::new_single_for_tests ( #19039 )
2021-08-04 11:47:11 -05:00
Jeff Washington (jwash)
31a620c42b
move towards accounts index being dynamic ( #19032 )
2021-08-04 09:18:05 -05:00
sakridge
84e78316b1
Write helper for multithread update ( #18808 )
2021-07-29 03:16:36 +02:00
carllin
5cabb5bb11
Fix race in remove_unrooted_race and flush_slot_cache ( #18785 )
2021-07-22 13:49:08 -07:00
Jeff Washington (jwash)
da9fdd785c
measure and speed up 2nd half of generate_index ( #18503 )
...
* add metrics and speedup storage calcs in generate_index
* add tests, refactor
* rename map variable
* rename another index variable
2021-07-15 10:26:00 -05:00
Jeff Washington (jwash)
d092fa1f03
add ledger-tool verify verify-accounts-index option ( #18375 )
...
* add ledger-tool verify verify-accounts-index option
* comment, merge, respond to feedback, cleanup
2021-07-13 11:06:18 -05:00
Jeff Washington (jwash)
eca0ceb04c
eliminate unnecessary copies in accounts index generation ( #18466 )
2021-07-07 15:36:05 -05:00
Jeff Washington (jwash)
49c4e54b28
accounts index insert uses pubkey by value ( #18460 )
2021-07-07 15:35:35 -05:00
Ruud van Asseldonk
d951916d40
Make test_hash_stored_account pass in release mode
...
The hash differs between debug and release mode, which makes the test
fail when running under "cargo test --release". Use cfg! to include a
different expected hash in release mode.
2021-07-01 18:27:24 -07:00
Jeff Washington (jwash)
531f1bce78
report on min/max bin stats for accounts index ( #18338 )
2021-07-01 00:46:59 +00:00
carllin
47cafb70da
test ( #18159 )
2021-06-28 18:27:26 -07:00
Jeff Washington (jwash)
afd64d27c9
bin accounts index ( #18111 )
2021-06-28 15:03:57 -05:00
Jeff Washington (jwash)
e06376664b
replace account index bulk insert with iterator ( #18198 )
2021-06-25 14:31:55 -05:00
Jeff Washington (jwash)
f2a2581259
metric for accounts index insertion time ( #18202 )
2021-06-25 08:36:55 -05:00
Jeff Washington (jwash)
4b314be5bd
rework dirty_pubkeys from insert_new_if_missing_into_primary_index ( #18200 )
2021-06-24 14:52:11 -05:00
sakridge
3b1738c000
Keep track of dirty stores on remove accounts to clean ( #17601 )
...
* Keep track of dirty stores on remove accounts to clean
and not zero_lamport key set
* Only dirty when count==0?
* Add another clean
2021-06-23 10:28:35 +02:00
Jeff Washington (jwash)
d3ee73e151
capture sort time in hash calculation ( #18118 )
2021-06-22 16:02:21 -05:00
Jeff Washington (jwash)
2087f5da94
refactor generate_index_for_slot ( #17984 )
2021-06-21 15:32:50 -05:00
Jeff Washington (jwash)
bf97627021
eliminate flatten and sort in hash calculation ( #17802 )
...
* eliminate flatten and sort in hash calculation
* reduce critical section time
* remove now no-longer necessary test code
* conflict with reset bins to 0 pr
2021-06-21 15:32:03 -05:00
Alexander Meißner
789f33e8db
chore: cargo fmt
2021-06-18 10:42:46 -07:00
Alexander Meißner
6514096a67
chore: cargo +nightly clippy --fix -Z unstable-options
2021-06-18 10:42:46 -07:00
Jeff Washington (jwash)
1feaaf009d
generate_index timings ( #17983 )
2021-06-17 13:35:25 -05:00
Jeff Washington (jwash)
20e714b3d0
remove unused code ( #17981 )
2021-06-17 13:30:46 -05:00
Jeff Washington (jwash)
71796f4951
break out generate index reporting ( #17980 )
2021-06-16 12:09:17 -05:00
Jeff Washington (jwash)
7ca04d6a86
refactor generate_index process_storage_slot ( #17982 )
2021-06-16 11:02:34 -05:00
Jeff Washington (jwash)
eee5414c64
avoid unnecessary empty arrays when binning ( #17944 )
2021-06-16 09:24:59 -05:00
Jeff Washington (jwash)
adc683956f
do not use index for verify_bank_hash_and_lamports ( #17812 )
2021-06-16 09:10:38 -05:00
Jeff Washington (jwash)
55ee3b5f2f
set hash bins to 65k ( #17912 )
2021-06-16 09:09:55 -05:00
Jeff Washington (jwash)
f558b9b6bf
verify bank hash on startup with ledger tool option ( #17939 )
2021-06-15 11:52:12 -05:00
Jeff Washington (jwash)
4d8ffdcc11
remove unused parameters from CalculateHashIntermediate ( #17949 )
2021-06-15 09:00:43 -05:00
carllin
ccc013e134
Handle removing slots during account scans ( #17471 )
2021-06-14 21:04:01 -07:00
Jeff Washington (jwash)
7fde9b6ff0
name arguments to help with confusion ( #17942 )
2021-06-14 16:21:48 -05:00
Jeff Washington (jwash)
62e1b20e56
refactor so hash verify can be done by more callers ( #17941 )
2021-06-14 16:20:03 -05:00
Jeff Washington (jwash)
d4cc975fe9
calculate_capitalization uses hash calculation ( #17443 )
...
* calculate_capitalization uses hash calculation
* feedback
* remove debugging code, clean up slot math
2021-06-14 08:53:07 -05:00
Lijun Wang
269d995832
Make account shrink configurable #17544 ( #17778 )
...
1. Added both options for measuring space usage using total accounts usage and for individual store shrink ratio using an enum. Validator CLI options: --accounts-shrink-optimize-total-space and --accounts-shrink-ratio
2. Added code for selecting candidates based on total usage in a separate function select_candidates_by_total_usage
3. Added unit tests for the new functions added
4. The default implementations is kept at 0.8 shrink ratio with --accounts-shrink-optimize-total-space set to true
Fixes #17544
2021-06-09 21:21:32 -07:00
Jeff Washington (jwash)
576e3d95f7
refactor: create type ( #17818 )
2021-06-08 14:49:08 -06:00
Jeff Washington (jwash)
2a9b127029
Pass iterator for SortedStorages::new_with_slots ( #17811 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-06-08 13:59:50 -06:00
Jeff Washington (jwash)
b5bb91b50f
rework hash calculation to not keep slot and write version ( #17685 )
...
* rework hash calculation to not keep slot and write version
* refactor functions and add tests
* always use multiple slot code path
2021-06-07 09:01:16 -05:00
Jeff Washington (jwash)
5197454fea
increase bin count for hash scan ( #17562 )
2021-06-04 10:53:13 -05:00
Jeff Washington (jwash)
9388aaca15
use slots returned from get_snapshot_storages to sort ( #17638 )
...
* use slots returned from get_snapshot_storages to sort
* add tests
2021-06-02 18:24:55 -05:00
Jeff Washington (jwash)
4bd32d891f
remove slot lookup ( #17691 )
2021-06-02 17:56:19 -05:00
Jeff Washington (jwash)
83ceedc091
simplify test construction ( #17686 )
2021-06-02 17:32:31 -05:00
Jeff Washington (jwash)
db8811eacd
support bin divisions up to 65536 ( #17563 )
...
* support bin divisions up to 65536
* add tests
2021-06-02 17:31:15 -05:00
Jeff Washington (jwash)
d802eb303c
reduce copy ( #17672 )
2021-06-02 14:03:02 -05:00
carllin
bbcdf073ba
Support out of band dumping of unrooted slots in AccountsDb ( #17269 )
...
* Accounts dumping logic
* Add test for interaction between cache flush and remove_unrooted_slot()
* Update comments
* Rename
* renaming
* Add more comments
* Renaming
* Fixup test and bad check
2021-06-02 09:51:10 +00:00
sakridge
14c52ab018
Skip shrink when it doesn't save anything ( #17405 )
2021-06-02 09:51:46 +02:00
Jeff Washington (jwash)
738cc9549f
parallel get_snapshot_storages ( #17589 )
2021-06-01 19:33:12 -05:00
Jeff Washington (jwash)
c8fdf0762c
add Ancestors to get_snapshot_storages to prepare for the future ( #17591 )
2021-06-01 15:14:35 -05:00
Jeff Washington (jwash)
ef5169ff24
sort storages by slot before scan ( #17411 )
...
* sort storages by slot before scan
* fix return value
2021-06-01 15:07:46 -05:00
Jeff Washington (jwash)
72bb271a94
add metric for collecting storages ( #17527 )
2021-06-01 13:17:49 -05:00
Jeff Washington (jwash)
886898eabf
rework test for flexibility ( #17592 )
...
* rework test for flexibility
* respond to pr feedback
2021-05-31 22:49:17 -05:00
Jeff Washington (jwash)
654918ab27
refactor SnapshotStorage helpers to prepare for later changes ( #17560 )
2021-05-28 13:54:53 -05:00
Jeff Washington (jwash)
55c22d3b76
add check_hash to non-index hash calculation ( #17558 )
2021-05-28 10:24:40 -05:00
sakridge
49402b7d82
Only add keys with multiple index entries to the clean set ( #17473 )
2021-05-27 16:31:24 +02:00
Jeff Washington (jwash)
3973bf08f1
move local out of call ( #17519 )
2021-05-27 08:53:11 -05:00
sakridge
3f9e3c7375
Remove copied shrink code ( #17385 )
2021-05-26 19:27:18 +02:00
Jeff Washington (jwash)
6b9d8d41a3
add --limit_load_slot_count_from_snapshot to ledger-tool ( #17417 )
2021-05-26 10:36:12 -05:00
Jeff Washington (jwash)
40bb6577fc
initialize purge stats ( #17345 )
2021-05-25 09:24:38 -05:00
Jeff Washington (jwash)
d4fc9e7ed4
ad metric for collect ( #17418 )
2021-05-24 18:14:52 -05:00
carllin
d8bc56fa51
Refactor purge_slots_from_cache_and_store() and handle_reclaims() ( #17319 )
2021-05-24 13:51:17 -07:00
Jeff Washington (jwash)
3f3324231d
Hold read lock during startup shrink ( #17309 )
...
* hold read lock during account scan of shrink
* rename and improve rusty
2021-05-21 13:59:32 -05:00
Jeff Washington (jwash)
2f55547d37
remove flatten in storage scan ( #17354 )
...
* remove flatten in storage scan
* fix tests
2021-05-21 09:26:46 -05:00
Jeff Washington (jwash)
45552d271a
take mutex once in process_dead_slots ( #17357 )
2021-05-20 19:50:58 -05:00
Jeff Washington (jwash)
0486df02ba
don't limit to thread pool when cleaning on startup ( #17317 )
2021-05-20 14:36:35 -05:00