e69f37a464
Remove old function: account_balance_for_capitalization ( #16383 )
...
This function currently returns one of its' parameters and is thus useless.
2021-04-07 23:57:40 -05:00
3996b699dc
read only account cache for executable accounts - improve replay ( #16150 )
...
* read only account cache
* tests
* clippy
* cleanup
* new file, add tests
* remove copy/paste code from test
* remove dead code
* all loads use cache
* remove stale comments
* add metrics logging for read only cache size
* report read only cache hits and misses
* consistency
* formatting
* rename, add comment
* u64
* better interaction with existing cache
* lru list saved between cleans
2021-04-01 07:16:34 -05:00
f374b35944
Hash stored accounts in bg ( #16157 )
...
* lazy calculate account hash
* push to bg thread
* remove deadlock
* logs
* format
* some cleanup on aisle 9
* format, fix up some metrics
* fix test, remove legacy function only there for tests
* cleanup
* remove unused store_hasher
* Switch to crossbeam
* clippy
* format
* use iter()
* rework from feedback
* hash_slot -> slot
* hash(cluster_type)
Co-authored-by: Carl Lin <carl@solana.com>
2021-03-31 15:39:34 -05:00
6d5c6c17c5
Simplify account.rent_epoch handling for sysvar rent ( #16049 )
...
* Add some code for special local testing
* Add comment to store_account_and_update_capitalization
* Simplify account.rent_epoch handling for sysvar rent
* Introduce *_for_test functions
* Add deprecation messages to existing api
2021-03-25 15:23:20 +09:00
26af2d4c26
Up store limit for test_account_grow ( #16099 )
...
There can be more than 5 stores
2021-03-24 15:24:17 -07:00
4beb39f7a1
multi-pass bin scanning ( #15377 )
...
* multi-pass bin scanning
* pr feedback
* format
* fix typo
* adjust metrics for code changes
* merge errors
2021-03-18 10:32:07 -05:00
4bbeb9c033
Remove old feature: simple_capitalization ( #15763 )
...
* Remove old feature: simple_capitalization
* Fix another failing test in core
* Finish up test cleanup
* Further clean up a bit
2021-03-12 11:12:40 +09:00
952c3bcbb7
AccountSharedData construction ( #15790 )
2021-03-11 18:09:04 -06:00
3419a5446e
AccountSharedData.set_data ( #15781 )
...
* account.set_data and resize_data
* remove data_resize
2021-03-11 16:40:45 -06:00
1061d021c9
Unpack snapshot AppendVecs directly into account paths
2021-03-11 08:25:24 -08:00
65dd177601
AppendVec cleanup: remove dead code, rename non-idiomatic "new" function
2021-03-10 14:50:59 -08:00
1135ffd595
mut data refs as slice ( #15782 )
2021-03-10 15:28:03 -06:00
52e54e1100
account.data -> data() ( #15778 )
2021-03-09 22:31:33 +00:00
8a3135d17b
Account->AccountSharedData ( #15691 )
2021-03-09 15:06:07 -06:00
c5371fdc85
More debug on shrink and flush ( #15518 )
2021-03-05 14:10:34 -08:00
edd159e7d7
Fix typos ( #15721 )
2021-03-05 07:16:53 +00:00
a2c47db920
extract expected cluster type ( #15615 )
...
* extract expected cluster type
* fix clippy warning
* more clippy
2021-03-02 13:55:34 -06:00
da47ab7e50
fix indentation format problem ( #15618 )
...
* fix indentation format problem
* fix indentation format problem
2021-03-02 10:46:00 -06:00
2463cc1e6a
Fix typos ( #15610 )
2021-03-02 06:36:49 -08:00
97eaf3c334
Fix finalize_dead_slot_removal() of cached slots decrementing refcount ( #15534 )
2021-02-26 17:49:37 -08:00
21b43009f6
Introduce ttl eviction for RecycleStore ( #15513 )
2021-02-25 17:27:27 +09:00
5df36aec7d
Pacify clippy
2021-02-19 20:08:41 -08:00
7fee0bc69c
move hashing functionality to accounts_hash ( #15353 )
2021-02-17 00:29:50 +00:00
8367740ff9
compute merkle root on chunks of fanout^3 ( #15344 )
...
* compute merkle root on chunks of fanout^3
* improve test_accountsdb_compute_merkle_root_large
2021-02-16 17:03:35 -06:00
d83aeebcca
change 2d cumulative calculation ( #15346 )
2021-02-16 14:24:45 -06:00
708b8380b7
assert 2 items in stores don't have same slot and version with different values ( #15250 )
...
* assert 2 items in stores don't have same slot and version with different values
* add message to assert
2021-02-16 18:12:59 +00:00
53959b4bbc
divide account scan into bins for more efficient downstream processing ( #15303 )
2021-02-15 21:27:39 -06:00
78a176fc00
rename CumulativeOffset in preparation of 2d support ( #15338 )
2021-02-15 20:57:05 -06:00
30f18319f2
Track RecycleStore basic stats with needed refactor ( #15291 )
...
* Track RecycleStore basic stats with needed refactor
* Fix another wrong metrics def
2021-02-15 16:09:17 +09:00
b8448f4189
merkle root code no longer adds lamports ( #15298 )
2021-02-12 17:30:14 -06:00
8944efddf7
combine flatten and hash ( #15269 )
2021-02-12 11:35:54 -06:00
eced3776b0
add metrics to hash calculation and consolidate ( #15165 )
...
* add metrics to hash calculation
* add metrics to hash calculation
* create HashStats struct to consolidate hash timing
* formatting
2021-02-11 20:06:04 -06:00
eb54e544c2
add div_ceil ( #15267 )
2021-02-11 20:05:41 -06:00
59e55f0371
remove hash accumulation with lamports ( #15275 )
2021-02-11 20:05:22 -06:00
e59a24d9f9
pass expected capitalization to hash calculation to improve assert msg ( #15191 )
...
* cleanup if
* pass expected capitalization to hash calculation to improve assert message
* fix bank function
* one more level
* calculate_accounts_hash_helper
* add slot to error message
* success
2021-02-10 14:38:00 -06:00
600cea274d
change store account scan to not use dashmap ( #15104 )
...
* change store account scan to not use dashmap
* add test_accountsdb_de_dup_accounts_from_stores
* add tests
* add test_accountsdb_flatten_hash_intermediate
* add tests
* add sort test
* add test
* clippy
* first_slice -> is_first_slice
* comment
* use partial_cmp
2021-02-05 21:16:30 -06:00
fbf9dc47e9
Threadpool2 ( #15151 )
...
* rework thread pool for hash calculation
* rename
2021-02-05 18:48:16 -06:00
fabecdc86c
use thread pool for non-index hash calculations ( #15149 )
2021-02-05 19:48:55 +00:00
f85be6259b
sentinel value for zero lamport accounts in hash scanning ( #15097 )
...
* sentinel value for zero lamport accounts in hash scanning
* fix test
2021-02-05 11:23:48 -06:00
7d9f5ad525
remove unused arg from function ( #15096 )
2021-02-04 20:46:30 -06:00
e4d0d4bfae
Fix pubkey refcount for shrink + clean ( #14987 )
2021-02-04 12:44:19 -08:00
f49a70e626
borrow storages ( #15088 )
2021-02-04 11:29:03 -06:00
e5225b7e68
Add ref count from storage ( #15078 )
2021-02-04 09:11:05 -08:00
e7f53182e2
fix merge conflict ( #15085 )
2021-02-04 09:20:50 -06:00
600ff0d915
calculate hash from store instead of index ( #15034 )
...
* calculate hash from store instead of index
* restore update hash in abs
2021-02-04 09:00:33 -06:00
2970b59853
Don't load all accounts into memory for capitalization check ( #14957 )
...
* Don't load all accounts into memory for capitalization check
2021-02-03 15:00:42 -08:00
2b9a0311d3
Don't reset accounts if the remove_account comes from a clean ( #15022 )
...
Store may be in-use with a snapshot creation, so don't disturb
it's state.
2021-02-03 13:06:28 -08:00
37aac5a12d
Cleanup v1 shrink path ( #15009 )
...
move legacy functions to another impl block
2021-02-02 13:29:46 -08:00
22800054e6
streamline calculate_accounts_hash ( #14980 )
2021-02-02 10:58:19 -06:00
63c44bd690
Use helper for count() in accountsDB ( #14953 )
2021-01-31 09:32:02 -08:00
65315fa4c2
Use already-generated key set to populate dirty keys for clean ( #14905 )
...
Don't need to scan the stores again when we already found the key
set of updates per slot. Just insert it earlier.
2021-01-29 11:44:43 -08:00
a53b8558cd
Only mmap file from snapshot once ( #14815 )
2021-01-28 08:15:33 -08:00
72f10f5f29
Aggregate purge and shrink metrics ( #14763 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-27 01:39:47 -08:00
daddcd361a
Snapshots missing slots from accounts cache clean optimization ( #14852 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-26 20:08:55 -08:00
1b85114a9c
remove legacy merkle root ( #14772 )
...
* remove legacy merkle root
f78197a
* clippy
* compile error
* borrow error
* derministic results
* clippy
* borrow
2021-01-25 15:26:08 -06:00
0d32a0e0f4
Partial clean ( #14800 )
...
* Revert "Revert "Partial accounts clean (#14652 )" (#14777 )"
This reverts commit ad2e10e17b
2021-01-24 09:50:19 -08:00
cbffab7850
Upgrade to Rust v1.49.0
2021-01-23 19:16:36 -08:00
424bb797a6
Speed up generate_index ( #14792 )
2021-01-23 08:05:05 -08:00
c77461e428
Remove unnecesary flushes in previous roots ( #14596 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-23 04:02:44 -08:00
ad2e10e17b
Revert "Partial accounts clean ( #14652 )" ( #14777 )
...
This reverts commit ca7914aa3d
2021-01-22 18:05:49 -08:00
18bd0c9a5b
speed up merkle calculation ( #14710 )
2021-01-22 09:21:50 -06:00
ca7914aa3d
Partial accounts clean ( #14652 )
...
Clean less keys by tracking the two cases:
* Touched keys per slot in uncleaned_keys derived from
accounts delta hash operation.
* Set of keys with any zero-lamport updates.
2021-01-21 11:01:53 -08:00
2745b79b74
Parallel cache scan ( #14544 )
...
* Parallel cache scan
* PR comments
* PR comments
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-20 08:50:17 +00:00
5f14f45282
More generic accounts purge functions ( #14595 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-17 20:31:03 -08:00
6dfad0652f
Cache account stores, flush from AccountsBackgroundService ( #13140 )
2021-01-11 17:00:23 -08:00
53d65009a0
Lower recycle store count ( #14429 )
...
Too many stores can cause swap usage which
is detrimental to account store times.
2021-01-05 12:21:17 -08:00
b6dcdb90e8
Remove dead code
2021-01-02 17:26:33 -08:00
5affd8aa72
Add secondary indexes ( #14212 )
2020-12-31 18:06:03 -08:00
1c61d005b5
Re-enable shrink test ( #14331 )
2020-12-30 08:25:45 -08:00
c693ffaa08
Fix subtraction overflow in metrics ( #14290 )
2020-12-27 16:26:22 -08:00
c276513b84
Ignore long/hanging test ( #14261 )
2020-12-22 16:13:03 -07:00
baa9602411
Add shrink paths ( #14238 )
2020-12-21 21:33:37 -08:00
7143aaa89b
Clippy
2020-12-14 08:03:29 -08:00
de9ac43ebf
Simple cap by including sysvars and native programs ( #13884 )
...
* Simpler cap by including sysvars and native programs
* Fix tests
* Add comment
* revert some unrelated code
* Update test_bank_update_sysvar_account for cap.
* Test cap. for add_native_program using new helper
* Improve the cap adjustment with new tests
* Fix typo...
* Adjust test for improved code coverage
* Rename simpler_capitalization => simple_capitalization
* More rename and bonus commenting
2020-12-14 21:03:24 +09:00
5788e75188
Shrink slots by sparseness of written data size ( #14072 )
2020-12-14 02:58:38 +09:00
164b7895b3
Tiny add_native_program bug fixes with cleanups ( #14042 )
...
* Tiny add_native_program bug fixes with cleanups
* Fix typo
2020-12-11 11:03:31 +09:00
68109a46e0
Lower priority for hashing threads. ( #14043 )
2020-12-10 12:26:47 -08:00
5dceddd21d
Remove associated functions on AccountsDb, replace with methods ( #13953 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-12-03 17:53:42 -08:00
955b99cf69
Fix roots_and_ref_count ( #13827 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-26 13:09:20 -08:00
791fb17437
Prevent scans on unrooted slots from seeing partial clean ( #13628 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-20 13:01:04 -08:00
6276360468
Prevent scans from seeing root updates/clean ( #13464 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-16 17:23:11 -08:00
2c2432fddc
Refactor function ( #13294 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-10 17:20:35 -08:00
43053dcc90
Re-use accounts stores ( #12885 )
...
* Re-use accounts_db stores
Creating files and dropping mmap areas can be expensive
* Add test for storage finder
Can encounter an infinite loop when the store is too small, but
smaller than the normal store size.
* Fix storage finding
* Check for strong_count == 1
* try_recycle helper
2020-11-04 09:17:05 -08:00
c458d4b213
move Account to solana-sdk ( #13198 )
2020-10-28 22:01:07 -07:00
d5e439037b
Make bad_bank_hash test faster ( #13206 )
2020-10-28 21:44:42 -07:00
7d2729f6bd
shrink debug ( #13089 )
2020-10-23 08:05:19 -07:00
959880db60
Remove unused pubkey::Pubkey imports
2020-10-21 19:08:13 -07:00
7bc073defe
Run `codemod --extensions rs Pubkey::new_rand solana_sdk::pubkey::new_rand`
2020-10-21 19:08:13 -07:00
e6b821c392
Finer grained AccountsIndex locking ( #12787 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-21 17:05:27 -07:00
c8fc0a6ba1
Don't hold dashmap write lock in store create ( #13007 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-21 12:54:09 -07:00
6858950f76
Remove frozen ABI modules from solana-sdk
2020-10-20 16:11:30 -07:00
456eae6ccb
Calculate accounts hash async in accounts background service ( #12852 )
2020-10-19 09:48:29 -07:00
4f6ebbdbe0
Add timings for store accounts ( #12882 )
2020-10-15 12:14:39 -07:00
f8d338c9cb
Switch accounts storage lock to DashMap ( #12126 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-13 18:29:50 -07:00
16d45b8480
Fix zero-lamport accounts preventing slot cleanup ( #12606 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-09 12:40:08 -07:00
adf6f748c0
Disable rent in flaky test ( #12658 )
...
* Disable rent in test
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-03 12:44:03 -07:00
06f84c65f1
Fix rooted accounts cleanup, simplify locking ( #12194 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-28 16:04:46 -07:00
00b36e6db2
Fix ref-count for multiple stores to the same pubkey in a slot, fixes zero lamport purge detection ( #12462 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-25 18:54:48 -07:00
55be8d4016
Move dropping AppendVecs outside lock ( #12408 )
...
* Move drop outside lock
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-23 14:17:49 -07:00
3930cb865a
Add keccak-secp256k1 instruction ( #11839 )
...
* Implement keccak-secp256k1 instruction
Verifies eth addreses with ecrecover function
* Move secp256k1 test
2020-09-15 18:23:21 -07:00
steviez
Jeff Washington (jwash)
Ryo Onodera
sakridge
Michael Vines
Greg Fitzgerald
carllin
carllin
Tyera Eulberg
Jack May