Michael Vines
65dd177601
AppendVec cleanup: remove dead code, rename non-idiomatic "new" function
2021-03-10 14:50:59 -08:00
Jeff Washington (jwash)
1135ffd595
mut data refs as slice ( #15782 )
2021-03-10 15:28:03 -06:00
Jeff Washington (jwash)
52e54e1100
account.data -> data() ( #15778 )
2021-03-09 22:31:33 +00:00
Jeff Washington (jwash)
8a3135d17b
Account->AccountSharedData ( #15691 )
2021-03-09 15:06:07 -06:00
sakridge
c5371fdc85
More debug on shrink and flush ( #15518 )
2021-03-05 14:10:34 -08:00
Greg Fitzgerald
edd159e7d7
Fix typos ( #15721 )
2021-03-05 07:16:53 +00:00
Jeff Washington (jwash)
a2c47db920
extract expected cluster type ( #15615 )
...
* extract expected cluster type
* fix clippy warning
* more clippy
2021-03-02 13:55:34 -06:00
Jeff Washington (jwash)
da47ab7e50
fix indentation format problem ( #15618 )
...
* fix indentation format problem
* fix indentation format problem
2021-03-02 10:46:00 -06:00
Greg Fitzgerald
2463cc1e6a
Fix typos ( #15610 )
2021-03-02 06:36:49 -08:00
carllin
97eaf3c334
Fix finalize_dead_slot_removal() of cached slots decrementing refcount ( #15534 )
2021-02-26 17:49:37 -08:00
Ryo Onodera
21b43009f6
Introduce ttl eviction for RecycleStore ( #15513 )
2021-02-25 17:27:27 +09:00
Michael Vines
5df36aec7d
Pacify clippy
2021-02-19 20:08:41 -08:00
Jeff Washington (jwash)
7fee0bc69c
move hashing functionality to accounts_hash ( #15353 )
2021-02-17 00:29:50 +00:00
Jeff Washington (jwash)
8367740ff9
compute merkle root on chunks of fanout^3 ( #15344 )
...
* compute merkle root on chunks of fanout^3
* improve test_accountsdb_compute_merkle_root_large
2021-02-16 17:03:35 -06:00
Jeff Washington (jwash)
d83aeebcca
change 2d cumulative calculation ( #15346 )
2021-02-16 14:24:45 -06:00
Jeff Washington (jwash)
708b8380b7
assert 2 items in stores don't have same slot and version with different values ( #15250 )
...
* assert 2 items in stores don't have same slot and version with different values
* add message to assert
2021-02-16 18:12:59 +00:00
Jeff Washington (jwash)
53959b4bbc
divide account scan into bins for more efficient downstream processing ( #15303 )
2021-02-15 21:27:39 -06:00
Jeff Washington (jwash)
78a176fc00
rename CumulativeOffset in preparation of 2d support ( #15338 )
2021-02-15 20:57:05 -06:00
Ryo Onodera
30f18319f2
Track RecycleStore basic stats with needed refactor ( #15291 )
...
* Track RecycleStore basic stats with needed refactor
* Fix another wrong metrics def
2021-02-15 16:09:17 +09:00
Jeff Washington (jwash)
b8448f4189
merkle root code no longer adds lamports ( #15298 )
2021-02-12 17:30:14 -06:00
Jeff Washington (jwash)
8944efddf7
combine flatten and hash ( #15269 )
2021-02-12 11:35:54 -06:00
Jeff Washington (jwash)
eced3776b0
add metrics to hash calculation and consolidate ( #15165 )
...
* add metrics to hash calculation
* add metrics to hash calculation
* create HashStats struct to consolidate hash timing
* formatting
2021-02-11 20:06:04 -06:00
Jeff Washington (jwash)
eb54e544c2
add div_ceil ( #15267 )
2021-02-11 20:05:41 -06:00
Jeff Washington (jwash)
59e55f0371
remove hash accumulation with lamports ( #15275 )
2021-02-11 20:05:22 -06:00
Jeff Washington (jwash)
e59a24d9f9
pass expected capitalization to hash calculation to improve assert msg ( #15191 )
...
* cleanup if
* pass expected capitalization to hash calculation to improve assert message
* fix bank function
* one more level
* calculate_accounts_hash_helper
* add slot to error message
* success
2021-02-10 14:38:00 -06:00
Jeff Washington (jwash)
600cea274d
change store account scan to not use dashmap ( #15104 )
...
* change store account scan to not use dashmap
* add test_accountsdb_de_dup_accounts_from_stores
* add tests
* add test_accountsdb_flatten_hash_intermediate
* add tests
* add sort test
* add test
* clippy
* first_slice -> is_first_slice
* comment
* use partial_cmp
2021-02-05 21:16:30 -06:00
Jeff Washington (jwash)
fbf9dc47e9
Threadpool2 ( #15151 )
...
* rework thread pool for hash calculation
* rename
2021-02-05 18:48:16 -06:00
Jeff Washington (jwash)
fabecdc86c
use thread pool for non-index hash calculations ( #15149 )
2021-02-05 19:48:55 +00:00
Jeff Washington (jwash)
f85be6259b
sentinel value for zero lamport accounts in hash scanning ( #15097 )
...
* sentinel value for zero lamport accounts in hash scanning
* fix test
2021-02-05 11:23:48 -06:00
Jeff Washington (jwash)
7d9f5ad525
remove unused arg from function ( #15096 )
2021-02-04 20:46:30 -06:00
carllin
e4d0d4bfae
Fix pubkey refcount for shrink + clean ( #14987 )
2021-02-04 12:44:19 -08:00
Jeff Washington (jwash)
f49a70e626
borrow storages ( #15088 )
2021-02-04 11:29:03 -06:00
sakridge
e5225b7e68
Add ref count from storage ( #15078 )
2021-02-04 09:11:05 -08:00
Jeff Washington (jwash)
e7f53182e2
fix merge conflict ( #15085 )
2021-02-04 09:20:50 -06:00
Jeff Washington (jwash)
600ff0d915
calculate hash from store instead of index ( #15034 )
...
* calculate hash from store instead of index
* restore update hash in abs
2021-02-04 09:00:33 -06:00
carllin
2970b59853
Don't load all accounts into memory for capitalization check ( #14957 )
...
* Don't load all accounts into memory for capitalization check
2021-02-03 15:00:42 -08:00
sakridge
2b9a0311d3
Don't reset accounts if the remove_account comes from a clean ( #15022 )
...
Store may be in-use with a snapshot creation, so don't disturb
it's state.
2021-02-03 13:06:28 -08:00
sakridge
37aac5a12d
Cleanup v1 shrink path ( #15009 )
...
move legacy functions to another impl block
2021-02-02 13:29:46 -08:00
Jeff Washington (jwash)
22800054e6
streamline calculate_accounts_hash ( #14980 )
2021-02-02 10:58:19 -06:00
sakridge
63c44bd690
Use helper for count() in accountsDB ( #14953 )
2021-01-31 09:32:02 -08:00
sakridge
65315fa4c2
Use already-generated key set to populate dirty keys for clean ( #14905 )
...
Don't need to scan the stores again when we already found the key
set of updates per slot. Just insert it earlier.
2021-01-29 11:44:43 -08:00
sakridge
a53b8558cd
Only mmap file from snapshot once ( #14815 )
2021-01-28 08:15:33 -08:00
carllin
72f10f5f29
Aggregate purge and shrink metrics ( #14763 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-27 01:39:47 -08:00
carllin
daddcd361a
Snapshots missing slots from accounts cache clean optimization ( #14852 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-26 20:08:55 -08:00
Jeff Washington (jwash)
1b85114a9c
remove legacy merkle root ( #14772 )
...
* remove legacy merkle root
f78197a
* clippy
* compile error
* borrow error
* derministic results
* clippy
* borrow
2021-01-25 15:26:08 -06:00
sakridge
0d32a0e0f4
Partial clean ( #14800 )
...
* Revert "Revert "Partial accounts clean (#14652 )" (#14777 )"
This reverts commit ad2e10e17b
.
* Remove squashed uncleaned keys
2021-01-24 09:50:19 -08:00
Michael Vines
cbffab7850
Upgrade to Rust v1.49.0
2021-01-23 19:16:36 -08:00
sakridge
424bb797a6
Speed up generate_index ( #14792 )
2021-01-23 08:05:05 -08:00
carllin
c77461e428
Remove unnecesary flushes in previous roots ( #14596 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-23 04:02:44 -08:00
sakridge
ad2e10e17b
Revert "Partial accounts clean ( #14652 )" ( #14777 )
...
This reverts commit ca7914aa3d
.
2021-01-22 18:05:49 -08:00
Jeff Washington (jwash)
18bd0c9a5b
speed up merkle calculation ( #14710 )
2021-01-22 09:21:50 -06:00
sakridge
ca7914aa3d
Partial accounts clean ( #14652 )
...
Clean less keys by tracking the two cases:
* Touched keys per slot in uncleaned_keys derived from
accounts delta hash operation.
* Set of keys with any zero-lamport updates.
2021-01-21 11:01:53 -08:00
carllin
2745b79b74
Parallel cache scan ( #14544 )
...
* Parallel cache scan
* PR comments
* PR comments
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-20 08:50:17 +00:00
carllin
5f14f45282
More generic accounts purge functions ( #14595 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-17 20:31:03 -08:00
carllin
6dfad0652f
Cache account stores, flush from AccountsBackgroundService ( #13140 )
2021-01-11 17:00:23 -08:00
sakridge
53d65009a0
Lower recycle store count ( #14429 )
...
Too many stores can cause swap usage which
is detrimental to account store times.
2021-01-05 12:21:17 -08:00
Michael Vines
b6dcdb90e8
Remove dead code
2021-01-02 17:26:33 -08:00
carllin
5affd8aa72
Add secondary indexes ( #14212 )
2020-12-31 18:06:03 -08:00
sakridge
1c61d005b5
Re-enable shrink test ( #14331 )
2020-12-30 08:25:45 -08:00
sakridge
c693ffaa08
Fix subtraction overflow in metrics ( #14290 )
2020-12-27 16:26:22 -08:00
Tyera Eulberg
c276513b84
Ignore long/hanging test ( #14261 )
2020-12-22 16:13:03 -07:00
sakridge
baa9602411
Add shrink paths ( #14238 )
2020-12-21 21:33:37 -08:00
Michael Vines
7143aaa89b
Clippy
2020-12-14 08:03:29 -08:00
Ryo Onodera
de9ac43ebf
Simple cap by including sysvars and native programs ( #13884 )
...
* Simpler cap by including sysvars and native programs
* Fix tests
* Add comment
* revert some unrelated code
* Update test_bank_update_sysvar_account for cap.
* Test cap. for add_native_program using new helper
* Improve the cap adjustment with new tests
* Fix typo...
* Adjust test for improved code coverage
* Rename simpler_capitalization => simple_capitalization
* More rename and bonus commenting
2020-12-14 21:03:24 +09:00
Ryo Onodera
5788e75188
Shrink slots by sparseness of written data size ( #14072 )
2020-12-14 02:58:38 +09:00
Ryo Onodera
164b7895b3
Tiny add_native_program bug fixes with cleanups ( #14042 )
...
* Tiny add_native_program bug fixes with cleanups
* Fix typo
2020-12-11 11:03:31 +09:00
sakridge
68109a46e0
Lower priority for hashing threads. ( #14043 )
2020-12-10 12:26:47 -08:00
carllin
5dceddd21d
Remove associated functions on AccountsDb, replace with methods ( #13953 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-12-03 17:53:42 -08:00
carllin
955b99cf69
Fix roots_and_ref_count ( #13827 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-26 13:09:20 -08:00
carllin
791fb17437
Prevent scans on unrooted slots from seeing partial clean ( #13628 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-20 13:01:04 -08:00
carllin
6276360468
Prevent scans from seeing root updates/clean ( #13464 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-16 17:23:11 -08:00
carllin
2c2432fddc
Refactor function ( #13294 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-10 17:20:35 -08:00
sakridge
43053dcc90
Re-use accounts stores ( #12885 )
...
* Re-use accounts_db stores
Creating files and dropping mmap areas can be expensive
* Add test for storage finder
Can encounter an infinite loop when the store is too small, but
smaller than the normal store size.
* Fix storage finding
* Check for strong_count == 1
* try_recycle helper
2020-11-04 09:17:05 -08:00
Jack May
c458d4b213
move Account to solana-sdk ( #13198 )
2020-10-28 22:01:07 -07:00
sakridge
d5e439037b
Make bad_bank_hash test faster ( #13206 )
2020-10-28 21:44:42 -07:00
sakridge
7d2729f6bd
shrink debug ( #13089 )
2020-10-23 08:05:19 -07:00
Michael Vines
959880db60
Remove unused pubkey::Pubkey imports
2020-10-21 19:08:13 -07:00
Michael Vines
7bc073defe
Run `codemod --extensions rs Pubkey::new_rand solana_sdk::pubkey::new_rand`
2020-10-21 19:08:13 -07:00
carllin
e6b821c392
Finer grained AccountsIndex locking ( #12787 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-21 17:05:27 -07:00
carllin
c8fc0a6ba1
Don't hold dashmap write lock in store create ( #13007 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-21 12:54:09 -07:00
Michael Vines
6858950f76
Remove frozen ABI modules from solana-sdk
2020-10-20 16:11:30 -07:00
sakridge
456eae6ccb
Calculate accounts hash async in accounts background service ( #12852 )
2020-10-19 09:48:29 -07:00
sakridge
4f6ebbdbe0
Add timings for store accounts ( #12882 )
2020-10-15 12:14:39 -07:00
carllin
f8d338c9cb
Switch accounts storage lock to DashMap ( #12126 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-13 18:29:50 -07:00
carllin
16d45b8480
Fix zero-lamport accounts preventing slot cleanup ( #12606 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-09 12:40:08 -07:00
carllin
adf6f748c0
Disable rent in flaky test ( #12658 )
...
* Disable rent in test
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-03 12:44:03 -07:00
carllin
06f84c65f1
Fix rooted accounts cleanup, simplify locking ( #12194 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-28 16:04:46 -07:00
carllin
00b36e6db2
Fix ref-count for multiple stores to the same pubkey in a slot, fixes zero lamport purge detection ( #12462 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-25 18:54:48 -07:00
carllin
55be8d4016
Move dropping AppendVecs outside lock ( #12408 )
...
* Move drop outside lock
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-23 14:17:49 -07:00
sakridge
3930cb865a
Add keccak-secp256k1 instruction ( #11839 )
...
* Implement keccak-secp256k1 instruction
Verifies eth addreses with ecrecover function
* Move secp256k1 test
2020-09-15 18:23:21 -07:00
Ryo Onodera
de4a613610
Check bank capitalization ( #11927 )
...
* Check bank capitalization
* Simplify and unify capitalization calculation
* Improve and add tests
* Avoid overflow and inhibit automatic restart
* Fix test
* Tweak checked sum for cap. and add tests
* Fix broken build after merge conflicts..
* Rename to ClusterType
* Rename confusing method
* Clarify comment
* Verify cap. in rent and inflation tests
Co-authored-by: Stephen Akridge <sakridge@gmail.com>
2020-09-12 01:48:06 +09:00
Ryo Onodera
c274e26eb8
Prevent unbound memory growth by blockstore_processor ( #12110 )
...
* Prevent unbound memory growth by blockstore_processor
* Promote log to info! considering infrequency
* Exclude the time of freeing from interval...
* Skip not-shrinkable slots even if forced
* Add comment
2020-09-09 12:05:10 +09:00
Ryo Onodera
53b8ea4464
Rename to ClusterType and restore devnet compat. ( #12068 )
...
* Rename to ClusterType and restore devnet compat.
* De-duplicate parse code and add comments
* Adjust default Devnet genesis & reduce it in tests
2020-09-08 23:55:09 +09:00
carllin
a13efc52b3
Add unlock epochs for blake3 ( #12054 )
...
Co-authored-by: Carl <carl@solana.com>
2020-09-04 15:21:15 -07:00
carllin
af08221aec
Switch account hashing to blake3 ( #11969 )
...
* Switch account hashing to blake3
Co-authored-by: Carl <carl@solana.com>
2020-09-02 00:37:36 -07:00
sakridge
770d3d383c
Accounts hash calculation metrics ( #11433 )
2020-08-06 21:39:10 -07:00
Michael Vines
eefcf484cb
clippy
2020-08-03 18:35:15 +00:00
Ryo Onodera
3e4f49f9c9
Fix race condition between shrinking and cleaning ( #11235 )
...
* Fix race condition between shrinking and cleaning
* Minor formatting
* fix ci
* Update comments
* More update comment
* Adjust fn naming
2020-07-29 06:06:14 +09:00
Michael Vines
e5d8c4383f
Add Bank support for "upgrade epochs" where all non-vote transactions will be rejected
2020-07-17 08:48:42 -07:00
sakridge
8bf3a0aa05
accounts_clean: Convert stack dependency calculation with iterative ( #11067 )
...
* accounts_clean: Convert stack dependency calculation with iterative
* optimize clean with by creating a reverse-lookup hashset of the affected
keys
* Add dependency bench
reduce bench
* Huge clean
2020-07-15 06:49:22 -07:00