54ef065cc8
Use slot 1 for tests' sample store and accounts
...
Anticipating the removal of the old account hashing way, the Account DB
tests use slot 0 for the sample storages and accounts data. Slot 0 is a
special case, and so to make future changes not break any tests, update
just the tests here now to use any other slot.
The golden hash values were obtained by running `cargo test` and copying
in the "actual" values from the test output into the `raw_expected`
vector of the test code.
Validated by running `cargo test` successfully.
2021-04-12 17:09:07 -05:00
6930a77a0f
prepare replace Ancestors HashMap for performance ( #16476 )
2021-04-12 10:51:57 -05:00
e69f37a464
Remove old function: account_balance_for_capitalization ( #16383 )
...
This function currently returns one of its' parameters and is thus useless.
2021-04-07 23:57:40 -05:00
3996b699dc
read only account cache for executable accounts - improve replay ( #16150 )
...
* read only account cache
* tests
* clippy
* cleanup
* new file, add tests
* remove copy/paste code from test
* remove dead code
* all loads use cache
* remove stale comments
* add metrics logging for read only cache size
* report read only cache hits and misses
* consistency
* formatting
* rename, add comment
* u64
* better interaction with existing cache
* lru list saved between cleans
2021-04-01 07:16:34 -05:00
f374b35944
Hash stored accounts in bg ( #16157 )
...
* lazy calculate account hash
* push to bg thread
* remove deadlock
* logs
* format
* some cleanup on aisle 9
* format, fix up some metrics
* fix test, remove legacy function only there for tests
* cleanup
* remove unused store_hasher
* Switch to crossbeam
* clippy
* format
* use iter()
* rework from feedback
* hash_slot -> slot
* hash(cluster_type)
Co-authored-by: Carl Lin <carl@solana.com>
2021-03-31 15:39:34 -05:00
6d5c6c17c5
Simplify account.rent_epoch handling for sysvar rent ( #16049 )
...
* Add some code for special local testing
* Add comment to store_account_and_update_capitalization
* Simplify account.rent_epoch handling for sysvar rent
* Introduce *_for_test functions
* Add deprecation messages to existing api
2021-03-25 15:23:20 +09:00
26af2d4c26
Up store limit for test_account_grow ( #16099 )
...
There can be more than 5 stores
2021-03-24 15:24:17 -07:00
4beb39f7a1
multi-pass bin scanning ( #15377 )
...
* multi-pass bin scanning
* pr feedback
* format
* fix typo
* adjust metrics for code changes
* merge errors
2021-03-18 10:32:07 -05:00
4bbeb9c033
Remove old feature: simple_capitalization ( #15763 )
...
* Remove old feature: simple_capitalization
* Fix another failing test in core
* Finish up test cleanup
* Further clean up a bit
2021-03-12 11:12:40 +09:00
952c3bcbb7
AccountSharedData construction ( #15790 )
2021-03-11 18:09:04 -06:00
3419a5446e
AccountSharedData.set_data ( #15781 )
...
* account.set_data and resize_data
* remove data_resize
2021-03-11 16:40:45 -06:00
1061d021c9
Unpack snapshot AppendVecs directly into account paths
2021-03-11 08:25:24 -08:00
65dd177601
AppendVec cleanup: remove dead code, rename non-idiomatic "new" function
2021-03-10 14:50:59 -08:00
1135ffd595
mut data refs as slice ( #15782 )
2021-03-10 15:28:03 -06:00
52e54e1100
account.data -> data() ( #15778 )
2021-03-09 22:31:33 +00:00
8a3135d17b
Account->AccountSharedData ( #15691 )
2021-03-09 15:06:07 -06:00
c5371fdc85
More debug on shrink and flush ( #15518 )
2021-03-05 14:10:34 -08:00
edd159e7d7
Fix typos ( #15721 )
2021-03-05 07:16:53 +00:00
a2c47db920
extract expected cluster type ( #15615 )
...
* extract expected cluster type
* fix clippy warning
* more clippy
2021-03-02 13:55:34 -06:00
da47ab7e50
fix indentation format problem ( #15618 )
...
* fix indentation format problem
* fix indentation format problem
2021-03-02 10:46:00 -06:00
2463cc1e6a
Fix typos ( #15610 )
2021-03-02 06:36:49 -08:00
97eaf3c334
Fix finalize_dead_slot_removal() of cached slots decrementing refcount ( #15534 )
2021-02-26 17:49:37 -08:00
21b43009f6
Introduce ttl eviction for RecycleStore ( #15513 )
2021-02-25 17:27:27 +09:00
5df36aec7d
Pacify clippy
2021-02-19 20:08:41 -08:00
7fee0bc69c
move hashing functionality to accounts_hash ( #15353 )
2021-02-17 00:29:50 +00:00
8367740ff9
compute merkle root on chunks of fanout^3 ( #15344 )
...
* compute merkle root on chunks of fanout^3
* improve test_accountsdb_compute_merkle_root_large
2021-02-16 17:03:35 -06:00
d83aeebcca
change 2d cumulative calculation ( #15346 )
2021-02-16 14:24:45 -06:00
708b8380b7
assert 2 items in stores don't have same slot and version with different values ( #15250 )
...
* assert 2 items in stores don't have same slot and version with different values
* add message to assert
2021-02-16 18:12:59 +00:00
53959b4bbc
divide account scan into bins for more efficient downstream processing ( #15303 )
2021-02-15 21:27:39 -06:00
78a176fc00
rename CumulativeOffset in preparation of 2d support ( #15338 )
2021-02-15 20:57:05 -06:00
30f18319f2
Track RecycleStore basic stats with needed refactor ( #15291 )
...
* Track RecycleStore basic stats with needed refactor
* Fix another wrong metrics def
2021-02-15 16:09:17 +09:00
b8448f4189
merkle root code no longer adds lamports ( #15298 )
2021-02-12 17:30:14 -06:00
8944efddf7
combine flatten and hash ( #15269 )
2021-02-12 11:35:54 -06:00
eced3776b0
add metrics to hash calculation and consolidate ( #15165 )
...
* add metrics to hash calculation
* add metrics to hash calculation
* create HashStats struct to consolidate hash timing
* formatting
2021-02-11 20:06:04 -06:00
eb54e544c2
add div_ceil ( #15267 )
2021-02-11 20:05:41 -06:00
59e55f0371
remove hash accumulation with lamports ( #15275 )
2021-02-11 20:05:22 -06:00
e59a24d9f9
pass expected capitalization to hash calculation to improve assert msg ( #15191 )
...
* cleanup if
* pass expected capitalization to hash calculation to improve assert message
* fix bank function
* one more level
* calculate_accounts_hash_helper
* add slot to error message
* success
2021-02-10 14:38:00 -06:00
600cea274d
change store account scan to not use dashmap ( #15104 )
...
* change store account scan to not use dashmap
* add test_accountsdb_de_dup_accounts_from_stores
* add tests
* add test_accountsdb_flatten_hash_intermediate
* add tests
* add sort test
* add test
* clippy
* first_slice -> is_first_slice
* comment
* use partial_cmp
2021-02-05 21:16:30 -06:00
fbf9dc47e9
Threadpool2 ( #15151 )
...
* rework thread pool for hash calculation
* rename
2021-02-05 18:48:16 -06:00
fabecdc86c
use thread pool for non-index hash calculations ( #15149 )
2021-02-05 19:48:55 +00:00
f85be6259b
sentinel value for zero lamport accounts in hash scanning ( #15097 )
...
* sentinel value for zero lamport accounts in hash scanning
* fix test
2021-02-05 11:23:48 -06:00
7d9f5ad525
remove unused arg from function ( #15096 )
2021-02-04 20:46:30 -06:00
e4d0d4bfae
Fix pubkey refcount for shrink + clean ( #14987 )
2021-02-04 12:44:19 -08:00
f49a70e626
borrow storages ( #15088 )
2021-02-04 11:29:03 -06:00
e5225b7e68
Add ref count from storage ( #15078 )
2021-02-04 09:11:05 -08:00
e7f53182e2
fix merge conflict ( #15085 )
2021-02-04 09:20:50 -06:00
600ff0d915
calculate hash from store instead of index ( #15034 )
...
* calculate hash from store instead of index
* restore update hash in abs
2021-02-04 09:00:33 -06:00
2970b59853
Don't load all accounts into memory for capitalization check ( #14957 )
...
* Don't load all accounts into memory for capitalization check
2021-02-03 15:00:42 -08:00
2b9a0311d3
Don't reset accounts if the remove_account comes from a clean ( #15022 )
...
Store may be in-use with a snapshot creation, so don't disturb
it's state.
2021-02-03 13:06:28 -08:00
37aac5a12d
Cleanup v1 shrink path ( #15009 )
...
move legacy functions to another impl block
2021-02-02 13:29:46 -08:00
22800054e6
streamline calculate_accounts_hash ( #14980 )
2021-02-02 10:58:19 -06:00
63c44bd690
Use helper for count() in accountsDB ( #14953 )
2021-01-31 09:32:02 -08:00
65315fa4c2
Use already-generated key set to populate dirty keys for clean ( #14905 )
...
Don't need to scan the stores again when we already found the key
set of updates per slot. Just insert it earlier.
2021-01-29 11:44:43 -08:00
a53b8558cd
Only mmap file from snapshot once ( #14815 )
2021-01-28 08:15:33 -08:00
72f10f5f29
Aggregate purge and shrink metrics ( #14763 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-27 01:39:47 -08:00
daddcd361a
Snapshots missing slots from accounts cache clean optimization ( #14852 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-26 20:08:55 -08:00
1b85114a9c
remove legacy merkle root ( #14772 )
...
* remove legacy merkle root
f78197a
* clippy
* compile error
* borrow error
* derministic results
* clippy
* borrow
2021-01-25 15:26:08 -06:00
0d32a0e0f4
Partial clean ( #14800 )
...
* Revert "Revert "Partial accounts clean (#14652 )" (#14777 )"
This reverts commit ad2e10e17b
2021-01-24 09:50:19 -08:00
cbffab7850
Upgrade to Rust v1.49.0
2021-01-23 19:16:36 -08:00
424bb797a6
Speed up generate_index ( #14792 )
2021-01-23 08:05:05 -08:00
c77461e428
Remove unnecesary flushes in previous roots ( #14596 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-23 04:02:44 -08:00
ad2e10e17b
Revert "Partial accounts clean ( #14652 )" ( #14777 )
...
This reverts commit ca7914aa3d
2021-01-22 18:05:49 -08:00
18bd0c9a5b
speed up merkle calculation ( #14710 )
2021-01-22 09:21:50 -06:00
ca7914aa3d
Partial accounts clean ( #14652 )
...
Clean less keys by tracking the two cases:
* Touched keys per slot in uncleaned_keys derived from
accounts delta hash operation.
* Set of keys with any zero-lamport updates.
2021-01-21 11:01:53 -08:00
2745b79b74
Parallel cache scan ( #14544 )
...
* Parallel cache scan
* PR comments
* PR comments
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-20 08:50:17 +00:00
5f14f45282
More generic accounts purge functions ( #14595 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-17 20:31:03 -08:00
6dfad0652f
Cache account stores, flush from AccountsBackgroundService ( #13140 )
2021-01-11 17:00:23 -08:00
53d65009a0
Lower recycle store count ( #14429 )
...
Too many stores can cause swap usage which
is detrimental to account store times.
2021-01-05 12:21:17 -08:00
b6dcdb90e8
Remove dead code
2021-01-02 17:26:33 -08:00
5affd8aa72
Add secondary indexes ( #14212 )
2020-12-31 18:06:03 -08:00
1c61d005b5
Re-enable shrink test ( #14331 )
2020-12-30 08:25:45 -08:00
c693ffaa08
Fix subtraction overflow in metrics ( #14290 )
2020-12-27 16:26:22 -08:00
c276513b84
Ignore long/hanging test ( #14261 )
2020-12-22 16:13:03 -07:00
baa9602411
Add shrink paths ( #14238 )
2020-12-21 21:33:37 -08:00
7143aaa89b
Clippy
2020-12-14 08:03:29 -08:00
de9ac43ebf
Simple cap by including sysvars and native programs ( #13884 )
...
* Simpler cap by including sysvars and native programs
* Fix tests
* Add comment
* revert some unrelated code
* Update test_bank_update_sysvar_account for cap.
* Test cap. for add_native_program using new helper
* Improve the cap adjustment with new tests
* Fix typo...
* Adjust test for improved code coverage
* Rename simpler_capitalization => simple_capitalization
* More rename and bonus commenting
2020-12-14 21:03:24 +09:00
5788e75188
Shrink slots by sparseness of written data size ( #14072 )
2020-12-14 02:58:38 +09:00
164b7895b3
Tiny add_native_program bug fixes with cleanups ( #14042 )
...
* Tiny add_native_program bug fixes with cleanups
* Fix typo
2020-12-11 11:03:31 +09:00
68109a46e0
Lower priority for hashing threads. ( #14043 )
2020-12-10 12:26:47 -08:00
5dceddd21d
Remove associated functions on AccountsDb, replace with methods ( #13953 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-12-03 17:53:42 -08:00
955b99cf69
Fix roots_and_ref_count ( #13827 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-26 13:09:20 -08:00
791fb17437
Prevent scans on unrooted slots from seeing partial clean ( #13628 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-20 13:01:04 -08:00
6276360468
Prevent scans from seeing root updates/clean ( #13464 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-16 17:23:11 -08:00
2c2432fddc
Refactor function ( #13294 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-10 17:20:35 -08:00
43053dcc90
Re-use accounts stores ( #12885 )
...
* Re-use accounts_db stores
Creating files and dropping mmap areas can be expensive
* Add test for storage finder
Can encounter an infinite loop when the store is too small, but
smaller than the normal store size.
* Fix storage finding
* Check for strong_count == 1
* try_recycle helper
2020-11-04 09:17:05 -08:00
c458d4b213
move Account to solana-sdk ( #13198 )
2020-10-28 22:01:07 -07:00
d5e439037b
Make bad_bank_hash test faster ( #13206 )
2020-10-28 21:44:42 -07:00
7d2729f6bd
shrink debug ( #13089 )
2020-10-23 08:05:19 -07:00
959880db60
Remove unused pubkey::Pubkey imports
2020-10-21 19:08:13 -07:00
7bc073defe
Run `codemod --extensions rs Pubkey::new_rand solana_sdk::pubkey::new_rand`
2020-10-21 19:08:13 -07:00
e6b821c392
Finer grained AccountsIndex locking ( #12787 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-21 17:05:27 -07:00
c8fc0a6ba1
Don't hold dashmap write lock in store create ( #13007 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-21 12:54:09 -07:00
6858950f76
Remove frozen ABI modules from solana-sdk
2020-10-20 16:11:30 -07:00
456eae6ccb
Calculate accounts hash async in accounts background service ( #12852 )
2020-10-19 09:48:29 -07:00
4f6ebbdbe0
Add timings for store accounts ( #12882 )
2020-10-15 12:14:39 -07:00
f8d338c9cb
Switch accounts storage lock to DashMap ( #12126 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-13 18:29:50 -07:00
16d45b8480
Fix zero-lamport accounts preventing slot cleanup ( #12606 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-09 12:40:08 -07:00
adf6f748c0
Disable rent in flaky test ( #12658 )
...
* Disable rent in test
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-03 12:44:03 -07:00
06f84c65f1
Fix rooted accounts cleanup, simplify locking ( #12194 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-28 16:04:46 -07:00
00b36e6db2
Fix ref-count for multiple stores to the same pubkey in a slot, fixes zero lamport purge detection ( #12462 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-25 18:54:48 -07:00
Brooks Prumo
Jeff Washington (jwash)
steviez
Ryo Onodera
sakridge
Michael Vines
Greg Fitzgerald
carllin
carllin
Tyera Eulberg
Jack May