Jeff Washington (jwash)
1135ffd595
mut data refs as slice ( #15782 )
2021-03-10 15:28:03 -06:00
Jeff Washington (jwash)
52e54e1100
account.data -> data() ( #15778 )
2021-03-09 22:31:33 +00:00
Jeff Washington (jwash)
8a3135d17b
Account->AccountSharedData ( #15691 )
2021-03-09 15:06:07 -06:00
sakridge
c5371fdc85
More debug on shrink and flush ( #15518 )
2021-03-05 14:10:34 -08:00
Greg Fitzgerald
edd159e7d7
Fix typos ( #15721 )
2021-03-05 07:16:53 +00:00
Jeff Washington (jwash)
a2c47db920
extract expected cluster type ( #15615 )
...
* extract expected cluster type
* fix clippy warning
* more clippy
2021-03-02 13:55:34 -06:00
Jeff Washington (jwash)
da47ab7e50
fix indentation format problem ( #15618 )
...
* fix indentation format problem
* fix indentation format problem
2021-03-02 10:46:00 -06:00
Greg Fitzgerald
2463cc1e6a
Fix typos ( #15610 )
2021-03-02 06:36:49 -08:00
carllin
97eaf3c334
Fix finalize_dead_slot_removal() of cached slots decrementing refcount ( #15534 )
2021-02-26 17:49:37 -08:00
Ryo Onodera
21b43009f6
Introduce ttl eviction for RecycleStore ( #15513 )
2021-02-25 17:27:27 +09:00
Michael Vines
5df36aec7d
Pacify clippy
2021-02-19 20:08:41 -08:00
Jeff Washington (jwash)
7fee0bc69c
move hashing functionality to accounts_hash ( #15353 )
2021-02-17 00:29:50 +00:00
Jeff Washington (jwash)
8367740ff9
compute merkle root on chunks of fanout^3 ( #15344 )
...
* compute merkle root on chunks of fanout^3
* improve test_accountsdb_compute_merkle_root_large
2021-02-16 17:03:35 -06:00
Jeff Washington (jwash)
d83aeebcca
change 2d cumulative calculation ( #15346 )
2021-02-16 14:24:45 -06:00
Jeff Washington (jwash)
708b8380b7
assert 2 items in stores don't have same slot and version with different values ( #15250 )
...
* assert 2 items in stores don't have same slot and version with different values
* add message to assert
2021-02-16 18:12:59 +00:00
Jeff Washington (jwash)
53959b4bbc
divide account scan into bins for more efficient downstream processing ( #15303 )
2021-02-15 21:27:39 -06:00
Jeff Washington (jwash)
78a176fc00
rename CumulativeOffset in preparation of 2d support ( #15338 )
2021-02-15 20:57:05 -06:00
Ryo Onodera
30f18319f2
Track RecycleStore basic stats with needed refactor ( #15291 )
...
* Track RecycleStore basic stats with needed refactor
* Fix another wrong metrics def
2021-02-15 16:09:17 +09:00
Jeff Washington (jwash)
b8448f4189
merkle root code no longer adds lamports ( #15298 )
2021-02-12 17:30:14 -06:00
Jeff Washington (jwash)
8944efddf7
combine flatten and hash ( #15269 )
2021-02-12 11:35:54 -06:00
Jeff Washington (jwash)
eced3776b0
add metrics to hash calculation and consolidate ( #15165 )
...
* add metrics to hash calculation
* add metrics to hash calculation
* create HashStats struct to consolidate hash timing
* formatting
2021-02-11 20:06:04 -06:00
Jeff Washington (jwash)
eb54e544c2
add div_ceil ( #15267 )
2021-02-11 20:05:41 -06:00
Jeff Washington (jwash)
59e55f0371
remove hash accumulation with lamports ( #15275 )
2021-02-11 20:05:22 -06:00
Jeff Washington (jwash)
e59a24d9f9
pass expected capitalization to hash calculation to improve assert msg ( #15191 )
...
* cleanup if
* pass expected capitalization to hash calculation to improve assert message
* fix bank function
* one more level
* calculate_accounts_hash_helper
* add slot to error message
* success
2021-02-10 14:38:00 -06:00
Jeff Washington (jwash)
600cea274d
change store account scan to not use dashmap ( #15104 )
...
* change store account scan to not use dashmap
* add test_accountsdb_de_dup_accounts_from_stores
* add tests
* add test_accountsdb_flatten_hash_intermediate
* add tests
* add sort test
* add test
* clippy
* first_slice -> is_first_slice
* comment
* use partial_cmp
2021-02-05 21:16:30 -06:00
Jeff Washington (jwash)
fbf9dc47e9
Threadpool2 ( #15151 )
...
* rework thread pool for hash calculation
* rename
2021-02-05 18:48:16 -06:00
Jeff Washington (jwash)
fabecdc86c
use thread pool for non-index hash calculations ( #15149 )
2021-02-05 19:48:55 +00:00
Jeff Washington (jwash)
f85be6259b
sentinel value for zero lamport accounts in hash scanning ( #15097 )
...
* sentinel value for zero lamport accounts in hash scanning
* fix test
2021-02-05 11:23:48 -06:00
Jeff Washington (jwash)
7d9f5ad525
remove unused arg from function ( #15096 )
2021-02-04 20:46:30 -06:00
carllin
e4d0d4bfae
Fix pubkey refcount for shrink + clean ( #14987 )
2021-02-04 12:44:19 -08:00
Jeff Washington (jwash)
f49a70e626
borrow storages ( #15088 )
2021-02-04 11:29:03 -06:00
sakridge
e5225b7e68
Add ref count from storage ( #15078 )
2021-02-04 09:11:05 -08:00
Jeff Washington (jwash)
e7f53182e2
fix merge conflict ( #15085 )
2021-02-04 09:20:50 -06:00
Jeff Washington (jwash)
600ff0d915
calculate hash from store instead of index ( #15034 )
...
* calculate hash from store instead of index
* restore update hash in abs
2021-02-04 09:00:33 -06:00
carllin
2970b59853
Don't load all accounts into memory for capitalization check ( #14957 )
...
* Don't load all accounts into memory for capitalization check
2021-02-03 15:00:42 -08:00
sakridge
2b9a0311d3
Don't reset accounts if the remove_account comes from a clean ( #15022 )
...
Store may be in-use with a snapshot creation, so don't disturb
it's state.
2021-02-03 13:06:28 -08:00
sakridge
37aac5a12d
Cleanup v1 shrink path ( #15009 )
...
move legacy functions to another impl block
2021-02-02 13:29:46 -08:00
Jeff Washington (jwash)
22800054e6
streamline calculate_accounts_hash ( #14980 )
2021-02-02 10:58:19 -06:00
sakridge
63c44bd690
Use helper for count() in accountsDB ( #14953 )
2021-01-31 09:32:02 -08:00
sakridge
65315fa4c2
Use already-generated key set to populate dirty keys for clean ( #14905 )
...
Don't need to scan the stores again when we already found the key
set of updates per slot. Just insert it earlier.
2021-01-29 11:44:43 -08:00
sakridge
a53b8558cd
Only mmap file from snapshot once ( #14815 )
2021-01-28 08:15:33 -08:00
carllin
72f10f5f29
Aggregate purge and shrink metrics ( #14763 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-27 01:39:47 -08:00
carllin
daddcd361a
Snapshots missing slots from accounts cache clean optimization ( #14852 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-26 20:08:55 -08:00
Jeff Washington (jwash)
1b85114a9c
remove legacy merkle root ( #14772 )
...
* remove legacy merkle root
f78197a
* clippy
* compile error
* borrow error
* derministic results
* clippy
* borrow
2021-01-25 15:26:08 -06:00
sakridge
0d32a0e0f4
Partial clean ( #14800 )
...
* Revert "Revert "Partial accounts clean (#14652 )" (#14777 )"
This reverts commit ad2e10e17b
.
* Remove squashed uncleaned keys
2021-01-24 09:50:19 -08:00
Michael Vines
cbffab7850
Upgrade to Rust v1.49.0
2021-01-23 19:16:36 -08:00
sakridge
424bb797a6
Speed up generate_index ( #14792 )
2021-01-23 08:05:05 -08:00
carllin
c77461e428
Remove unnecesary flushes in previous roots ( #14596 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-23 04:02:44 -08:00
sakridge
ad2e10e17b
Revert "Partial accounts clean ( #14652 )" ( #14777 )
...
This reverts commit ca7914aa3d
.
2021-01-22 18:05:49 -08:00
Jeff Washington (jwash)
18bd0c9a5b
speed up merkle calculation ( #14710 )
2021-01-22 09:21:50 -06:00
sakridge
ca7914aa3d
Partial accounts clean ( #14652 )
...
Clean less keys by tracking the two cases:
* Touched keys per slot in uncleaned_keys derived from
accounts delta hash operation.
* Set of keys with any zero-lamport updates.
2021-01-21 11:01:53 -08:00
carllin
2745b79b74
Parallel cache scan ( #14544 )
...
* Parallel cache scan
* PR comments
* PR comments
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-20 08:50:17 +00:00
carllin
5f14f45282
More generic accounts purge functions ( #14595 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-17 20:31:03 -08:00
carllin
6dfad0652f
Cache account stores, flush from AccountsBackgroundService ( #13140 )
2021-01-11 17:00:23 -08:00
sakridge
53d65009a0
Lower recycle store count ( #14429 )
...
Too many stores can cause swap usage which
is detrimental to account store times.
2021-01-05 12:21:17 -08:00
Michael Vines
b6dcdb90e8
Remove dead code
2021-01-02 17:26:33 -08:00
carllin
5affd8aa72
Add secondary indexes ( #14212 )
2020-12-31 18:06:03 -08:00
sakridge
1c61d005b5
Re-enable shrink test ( #14331 )
2020-12-30 08:25:45 -08:00
sakridge
c693ffaa08
Fix subtraction overflow in metrics ( #14290 )
2020-12-27 16:26:22 -08:00
Tyera Eulberg
c276513b84
Ignore long/hanging test ( #14261 )
2020-12-22 16:13:03 -07:00
sakridge
baa9602411
Add shrink paths ( #14238 )
2020-12-21 21:33:37 -08:00
Michael Vines
7143aaa89b
Clippy
2020-12-14 08:03:29 -08:00
Ryo Onodera
de9ac43ebf
Simple cap by including sysvars and native programs ( #13884 )
...
* Simpler cap by including sysvars and native programs
* Fix tests
* Add comment
* revert some unrelated code
* Update test_bank_update_sysvar_account for cap.
* Test cap. for add_native_program using new helper
* Improve the cap adjustment with new tests
* Fix typo...
* Adjust test for improved code coverage
* Rename simpler_capitalization => simple_capitalization
* More rename and bonus commenting
2020-12-14 21:03:24 +09:00
Ryo Onodera
5788e75188
Shrink slots by sparseness of written data size ( #14072 )
2020-12-14 02:58:38 +09:00
Ryo Onodera
164b7895b3
Tiny add_native_program bug fixes with cleanups ( #14042 )
...
* Tiny add_native_program bug fixes with cleanups
* Fix typo
2020-12-11 11:03:31 +09:00
sakridge
68109a46e0
Lower priority for hashing threads. ( #14043 )
2020-12-10 12:26:47 -08:00
carllin
5dceddd21d
Remove associated functions on AccountsDb, replace with methods ( #13953 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-12-03 17:53:42 -08:00
carllin
955b99cf69
Fix roots_and_ref_count ( #13827 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-26 13:09:20 -08:00
carllin
791fb17437
Prevent scans on unrooted slots from seeing partial clean ( #13628 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-20 13:01:04 -08:00
carllin
6276360468
Prevent scans from seeing root updates/clean ( #13464 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-16 17:23:11 -08:00
carllin
2c2432fddc
Refactor function ( #13294 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-10 17:20:35 -08:00
sakridge
43053dcc90
Re-use accounts stores ( #12885 )
...
* Re-use accounts_db stores
Creating files and dropping mmap areas can be expensive
* Add test for storage finder
Can encounter an infinite loop when the store is too small, but
smaller than the normal store size.
* Fix storage finding
* Check for strong_count == 1
* try_recycle helper
2020-11-04 09:17:05 -08:00
Jack May
c458d4b213
move Account to solana-sdk ( #13198 )
2020-10-28 22:01:07 -07:00
sakridge
d5e439037b
Make bad_bank_hash test faster ( #13206 )
2020-10-28 21:44:42 -07:00
sakridge
7d2729f6bd
shrink debug ( #13089 )
2020-10-23 08:05:19 -07:00
Michael Vines
959880db60
Remove unused pubkey::Pubkey imports
2020-10-21 19:08:13 -07:00
Michael Vines
7bc073defe
Run `codemod --extensions rs Pubkey::new_rand solana_sdk::pubkey::new_rand`
2020-10-21 19:08:13 -07:00
carllin
e6b821c392
Finer grained AccountsIndex locking ( #12787 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-21 17:05:27 -07:00
carllin
c8fc0a6ba1
Don't hold dashmap write lock in store create ( #13007 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-21 12:54:09 -07:00
Michael Vines
6858950f76
Remove frozen ABI modules from solana-sdk
2020-10-20 16:11:30 -07:00
sakridge
456eae6ccb
Calculate accounts hash async in accounts background service ( #12852 )
2020-10-19 09:48:29 -07:00
sakridge
4f6ebbdbe0
Add timings for store accounts ( #12882 )
2020-10-15 12:14:39 -07:00
carllin
f8d338c9cb
Switch accounts storage lock to DashMap ( #12126 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-13 18:29:50 -07:00
carllin
16d45b8480
Fix zero-lamport accounts preventing slot cleanup ( #12606 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-09 12:40:08 -07:00
carllin
adf6f748c0
Disable rent in flaky test ( #12658 )
...
* Disable rent in test
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-03 12:44:03 -07:00
carllin
06f84c65f1
Fix rooted accounts cleanup, simplify locking ( #12194 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-28 16:04:46 -07:00
carllin
00b36e6db2
Fix ref-count for multiple stores to the same pubkey in a slot, fixes zero lamport purge detection ( #12462 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-25 18:54:48 -07:00
carllin
55be8d4016
Move dropping AppendVecs outside lock ( #12408 )
...
* Move drop outside lock
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-23 14:17:49 -07:00
sakridge
3930cb865a
Add keccak-secp256k1 instruction ( #11839 )
...
* Implement keccak-secp256k1 instruction
Verifies eth addreses with ecrecover function
* Move secp256k1 test
2020-09-15 18:23:21 -07:00
Ryo Onodera
de4a613610
Check bank capitalization ( #11927 )
...
* Check bank capitalization
* Simplify and unify capitalization calculation
* Improve and add tests
* Avoid overflow and inhibit automatic restart
* Fix test
* Tweak checked sum for cap. and add tests
* Fix broken build after merge conflicts..
* Rename to ClusterType
* Rename confusing method
* Clarify comment
* Verify cap. in rent and inflation tests
Co-authored-by: Stephen Akridge <sakridge@gmail.com>
2020-09-12 01:48:06 +09:00
Ryo Onodera
c274e26eb8
Prevent unbound memory growth by blockstore_processor ( #12110 )
...
* Prevent unbound memory growth by blockstore_processor
* Promote log to info! considering infrequency
* Exclude the time of freeing from interval...
* Skip not-shrinkable slots even if forced
* Add comment
2020-09-09 12:05:10 +09:00
Ryo Onodera
53b8ea4464
Rename to ClusterType and restore devnet compat. ( #12068 )
...
* Rename to ClusterType and restore devnet compat.
* De-duplicate parse code and add comments
* Adjust default Devnet genesis & reduce it in tests
2020-09-08 23:55:09 +09:00
carllin
a13efc52b3
Add unlock epochs for blake3 ( #12054 )
...
Co-authored-by: Carl <carl@solana.com>
2020-09-04 15:21:15 -07:00
carllin
af08221aec
Switch account hashing to blake3 ( #11969 )
...
* Switch account hashing to blake3
Co-authored-by: Carl <carl@solana.com>
2020-09-02 00:37:36 -07:00
sakridge
770d3d383c
Accounts hash calculation metrics ( #11433 )
2020-08-06 21:39:10 -07:00
Michael Vines
eefcf484cb
clippy
2020-08-03 18:35:15 +00:00
Ryo Onodera
3e4f49f9c9
Fix race condition between shrinking and cleaning ( #11235 )
...
* Fix race condition between shrinking and cleaning
* Minor formatting
* fix ci
* Update comments
* More update comment
* Adjust fn naming
2020-07-29 06:06:14 +09:00
Michael Vines
e5d8c4383f
Add Bank support for "upgrade epochs" where all non-vote transactions will be rejected
2020-07-17 08:48:42 -07:00
sakridge
8bf3a0aa05
accounts_clean: Convert stack dependency calculation with iterative ( #11067 )
...
* accounts_clean: Convert stack dependency calculation with iterative
* optimize clean with by creating a reverse-lookup hashset of the affected
keys
* Add dependency bench
reduce bench
* Huge clean
2020-07-15 06:49:22 -07:00
Kristofer Peterson
ed5a2f2a90
Fold bank serialisation into serde snapshot ( #10581 )
...
* Move bank (de)serialisation logic from bank and snapshot_utils to serde_snapshot.
Add sanity assertions between genesis config and bank fields on deserialisation.
Atomically update atomic bool in quote_for_specialization_detection().
Use same genesis config when restoring snapshots in test cases.
* Tidy up namings and duplicate structs to version
* Apply struct renames to tests
* Update abi hashes
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-07-13 23:00:59 +09:00
sakridge
edf2b1ee85
Check for deleting key, make sure list is empty again ( #11007 )
2020-07-11 20:38:34 -07:00
sakridge
631f051c68
Add jitter to cleanup to prevent all nodes cleaning at the same time ( #10936 )
2020-07-10 11:54:45 -07:00
sakridge
6429042b6e
Add --print-account-stats/--snapshot-archive-path arguments to ledger-tool ( #10945 )
...
Allows for seeing how the accounts are being stored
and specifying a different snapshot source directory.
2020-07-08 09:32:11 -07:00
Ryo Onodera
39b3ac6a8d
Introduce automatic ABI maintenance mechanism (2/2; rollout) ( #8012 )
...
* Introduce automatic ABI maintenance mechanism (2/2; rollout)
* Fix stable clippy
* Change to symlink
* Freeze abi of Tower
* fmt...
* Improve dev-experience!
* Update BankSlotDelta
$ diff -u /tmp/abi8/*7dg6BreYxTuxiVz6aLvk3p2Z7GQk2cJqfGvC9h4FAoSj* /tmp/abi8/*9chBcbXVJ4fK7uGgydQzam5aHipaAKFw6V4LDFpjbE4w*
--- /tmp/abi8/bank__BankSlotDelta_frozen_abi__test_abi_digest_7dg6BreYxTuxiVz6aLvk3p2Z7GQk2cJqfGvC9h4FAoSj 2020-06-18 18:01:22.831228087 +0900
+++ /tmp/abi8/bank__BankSlotDelta_frozen_abi__test_abi_digest_9chBcbXVJ4fK7uGgydQzam5aHipaAKFw6V4LDFpjbE4w 2020-07-03 15:59:58.430695244 +0900
@@ -140,7 +140,7 @@
field u8
primitive u8
field solana_sdk::instruction::InstructionError
- enum InstructionError (variants = 34)
+ enum InstructionError (variants = 35)
variant(0) GenericError (unit)
variant(1) InvalidArgument (unit)
variant(2) InvalidInstructionData (unit)
@@ -176,6 +176,7 @@
variant(31) CallDepth (unit)
variant(32) MissingAccount (unit)
variant(33) ReentrancyNotAllowed (unit)
+ variant(34) MaxSeedLengthExceeded (unit)
variant(9) CallChainTooDeep (unit)
variant(10) MissingSignatureForFee (unit)
variant(11) InvalidAccountIndex (unit)
* Fix some merge conflicts...
2020-07-06 20:22:23 +09:00
sakridge
832d47317e
Move clean accounts to background service ( #10898 )
2020-07-02 22:25:17 -07:00
Greg Fitzgerald
0550b893b0
Fix typos ( #10675 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2020-06-17 20:54:52 -07:00
carllin
8bd62d78eb
Fix race in remove_unrooted_slot ( #10607 )
...
* Fix race
* clippy fixes
* Rename and add comment
Co-authored-by: Carl <carl@solana.com>
2020-06-15 19:41:26 -07:00
Ryo Onodera
dfe72d5242
Optimize stale slot shrinking for previously cleaned roots ( #10099 )
...
* Prioritize shrinking of previously cleaned roots
* measure time of stale slot shrinking
* Disable shrink for test
* shrink: budgeting, store count, force for snapshot
* Polish implementation and fix tests
* Fix ci..
* Clean up a bit
* Further polish implementation and fix/add tests
* Rebase fixes
* Remove unneeded Default for AccountStorageEntry
* Address review comments
* More cleanup
* More cleanup
2020-06-12 14:51:43 +09:00
Boqin Qin
0a638a8f17
runtime: fix possible deadlock in accounts_db ( #10469 )
2020-06-10 19:37:15 +09:00
Kristofer Peterson
e23340d89e
Clippy cleanup for all targets and nighly rust (also support 1.44.0) ( #10445 )
...
* address warnings from 'rustup run beta cargo clippy --workspace'
minor refactoring in:
- cli/src/cli.rs
- cli/src/offline/blockhash_query.rs
- logger/src/lib.rs
- runtime/src/accounts_db.rs
expect some performance improvement AccountsDB::clean_accounts()
* address warnings from 'rustup run beta cargo clippy --workspace --tests'
* address warnings from 'rustup run nightly cargo clippy --workspace --all-targets'
* rustfmt
* fix warning stragglers
* properly fix clippy warnings test_vote_subscribe()
replace ref-to-arc with ref parameters where arc not cloned
* Remove lock around JsonRpcRequestProcessor (#10417 )
automerge
* make ancestors parameter optional to avoid forcing construction of empty hash maps
Co-authored-by: Greg Fitzgerald <greg@solana.com>
2020-06-09 09:38:14 +09:00
Michael Vines
e3b444834f
Add more logging while unpacking snapshots ( #10266 )
2020-05-27 10:41:05 -07:00
Kristofer Peterson
b7a32f01c0
Multi-version snapshot support ( #9980 )
...
* Multi-version snapshot support
* rustfmt
* Remove CLI options and runtime support for selection output snapshot version.
Address some clippy complaints.
* Muzzle clippy type complexity warning.
Despite clippy's suggestion, it is not currently possible to create type aliases
for traits and so everything within the 'Box<...>' cannot be type aliased.
This then leaves creating full blown traits, and either implementing
said traits by closure (somehow) or moving the closures into new structs
implementing said traits which seems a bit of a palaver.
Alternatively it is possible to define and use the type alias 'type ResultBox<T> = Result<Box<T>>'
which does seems rather pointless and not a great reduction in complexity but is enough to keep clippy happy.
In the end I simply went with squelching the clippy warning.
* Remove now unused Serialize/Deserialize trait implementations for AccountStorageEntry and AppendVec
* refactor versioned de/serialisers
* rename serde_utils to serde_snapshot
* move call to accounts_db.generate_index() back down to context_accountsdb_from_stream()
* update version 1.1.1 to 1.2.0
remove nested use of serialize_bytes
* cleanups
* Add back measurement of account storage entry serialization.
Remove construction of Vec and HashMap temporaries during serialization.
* consolidate serialisation test cases into serde_snapshot.
clean up leakage of implementation details in serde_snapshot.
* move short term / legacy snapshot code into child module
* add serialize_iter_as_tuple
* preliminary integration of following commit
commit 6d58b73c47294bfb93465d5a83cd2175660b6e6d
Author: Ryo Onodera <ryoqun@gmail.com>
Date: Wed May 20 14:02:02 2020 +0900
Confine snapshot 1.1 relic to versioned codepath
* refactored serde_snapshot, rustfmt
legacy accounts_db format now "owns" both leading u64s, legacy bank_rc format has none
* reduce type complexity (clippy)
2020-05-23 02:54:24 +09:00
Kristofer Peterson
58ef02f02b
9951 clippy errors in the test suite ( #10030 )
...
automerge
2020-05-15 09:35:43 -07:00
Ryo Onodera
1eb40c3fe0
Introduce eager rent collection ( #9527 )
...
* Switch AccountsIndex.account_maps from HashMap to BTreeMap
* Introduce eager rent collection
* Start to add tests
* Avoid too short eager rent collection cycles
* Add more tests
* Add more tests...
* Refacotr!!!!!!
* Refactoring follow up
* More tiny cleanups
* Don't rewrite 0-lamport accounts to be deterministic
* Refactor a bit
* Do hard fork, restore tests, and perf. mitigation
* Fix build...
* Refactor and add switch over for testnet (TdS)
* Use to_be_bytes
* cleanup
* More tiny cleanup
* Rebase cleanup
* Set Bank::genesis_hash when resuming from snapshot
* Reorder fns and clean ups
* Better naming and commenting
* Yet more naming clarifications
* Make prefix width strictly uniform for 2-base partition_count
* Fix typo...
* Revert cluster-dependent gate
* kick ci?
* kick ci?
* kick ci?
2020-05-13 16:22:14 +09:00
Justin Starry
5cc252d471
Remove hash field from account ( #9915 )
2020-05-12 23:39:46 +08:00
Ryo Onodera
2ac50177a6
Include account.owner into account hash ( #9917 )
...
automerge
2020-05-07 13:01:11 -07:00
carllin
3442f36f8a
Repair alternate versions of dead slots ( #9805 )
...
Co-authored-by: Carl <carl@solana.com>
2020-05-05 14:07:21 -07:00
Michael Vines
230df0ec0c
Upgrade to Rust 1.43.0 ( #9754 )
2020-04-29 18:02:05 -07:00
Ryo Onodera
9918539229
Introduce type alias Ancestors ( #9699 )
...
* Introduce type alias AncestorList
* Rename AncestorList => Ancestors
2020-04-27 11:07:03 +09:00
sakridge
217828a849
Reduce accounts_db prints ( #9469 )
2020-04-15 09:05:54 -07:00
sakridge
99655206c8
Calculate account refs fix ( #9447 )
2020-04-11 12:52:10 -07:00
Ryo Onodera
b28ec430e4
Introduce background stale AppendVec shrink mechanism ( #9219 )
...
* Introduce background AppendVec shrink mechanism
* Support ledger tool
* Clean up
* save
* save
* Fix CI
* More clean up
* Add tests
* Clean up yet more
* Use account.hash...
* Fix typo....
* Add comment
* Rename accounts_cleanup_service
2020-04-06 17:30:23 +09:00
sakridge
b1771b92ec
Calculate ref counts earlier to prevent bad clean ( #9147 )
2020-03-29 14:42:34 -07:00
Ryo Onodera
2623c71ed3
Use type aliases/resulting var names consistently ( #9060 )
2020-03-26 13:08:56 +09:00
Justin Starry
ef3af104ae
Return appropriate error for invalid program account ( #9047 )
...
automerge
2020-03-25 13:23:05 -07:00
sakridge
45348b2c83
Remove accounts unwrap ( #9062 )
...
automerge
2020-03-25 10:21:30 -07:00
sakridge
c530fbd22b
Remove thread-priority crate which is not cross-platform ( #9023 )
2020-03-23 12:18:52 -07:00
sakridge
4b397d15b3
Accounts cleanup service and perf improvements ( #8799 )
...
* Use atomic for ref count instead of taking rwlock
* Accounts cleanup service
* Review comments
2020-03-23 08:50:23 -07:00
Michael Vines
88ba8439fc
Add frozen account support ( #8989 )
...
automerge
2020-03-22 11:10:04 -07:00
sakridge
4dd0367136
Rwlock storage opt ( #9006 )
...
* Remove unecessary account paths rwlock
* Remove path rwlock in accounts_db and optimize storage critical section
2020-03-22 10:04:03 -07:00
sakridge
3ea556bc24
Drop storage lock ( #8997 )
2020-03-21 13:37:52 -07:00
Ryo Onodera
4bbf09f582
Enable conservative out-of-bound snapshot cleaning ( #8811 )
...
* Enable conservative out-of-bound snapshot cleaning
* Add tests
2020-03-13 14:44:00 +09:00
Ryo Onodera
952cd38b7b
Avoid early clean and bad snapshot by ref-counting ( #8724 )
...
* Avoid early clean and bad snapshot by ref-counting
* Add measure
* Clean ups
* clean ups
2020-03-13 14:14:37 +09:00
Michael Vines
f146c92e88
Always and fully normalize stored 0-lamport accts. ( #8657 )
2020-03-05 09:14:40 -07:00
sakridge
b084c1d437
Remove accounts hack and correctly restore accounts store counts ( #8569 )
...
* Remove accounts hack and correctly restore append-vec counts
* Add test
2020-03-03 20:48:55 -08:00
sakridge
4f05f08f5d
Use fs::rename which is much faster than move_items ( #8579 )
2020-03-03 10:03:17 -08:00
Ryo Onodera
d86103383a
Do periodic inbound cleaning for rooted slots ( #8436 )
...
* Do periodic inbound compaction for rooted slots
* Add comment
* nits
* Consider not_compacted_roots in cleanup_dead_slot
* Renames in AccountsIndex
* Rename to reflect expansion of removed accounts
* Fix a comment
* rename
* Parallelize clean over AccountsIndex
* Some niceties
* Reduce locks and real chunked parallelism
* Measure each step for sampling opportunities
* Just noticed par iter is maybe lazy
* Replace storage scan with optimized index scan
* Various clean-ups
* Clear uncleared_roots even if no updates
2020-03-03 14:57:25 +09:00
Ryo Onodera
5d9130a3c4
Hack to skip cleanup_dead_slots upon snapshot load
2020-03-02 10:24:12 -07:00
Ryo Onodera
0b48c8eb35
Promote dangerous cond. from just warning to panic ( #8439 )
2020-02-26 05:09:57 +09:00
sakridge
947a339714
Add snapshot hash of full accounts state ( #8295 )
...
* Add snapshot hash of full accounts state
* Use normal hashing for the accounts delta state
* Add merkle
2020-02-22 13:46:40 -08:00
Ryo Onodera
d238371b0c
Correct missing entry handling to avoid bad warns ( #8339 )
...
* Correct missing entry handling to avoid bad warns
* Pass storage entries to AccountStorageSerialize
* Fix CI.....
* Add tests and reorder condition for cheapest first
* Remove unneeded reference
2020-02-21 15:27:55 +09:00
Tyera Eulberg
ab361a8073
Rename KeypairUtil to Signer ( #8360 )
...
automerge
2020-02-20 13:28:55 -08:00
Ryo Onodera
027ec71aa9
Remove AccountInfo's (De)Serialize ( #8313 )
...
automerge
2020-02-17 03:07:36 -08:00
Ryo Onodera
58727463e1
Remove needless last_root for better reclaims ( #8148 )
...
* Restore last_root to fix unintended storage delete
* Remove last_root thing altogether
* Remove unneeded test...
2020-02-13 08:19:53 +09:00
sakridge
a8028fbb93
Fix accounts_db store counts in purging accounts logic ( #8218 )
...
* Show insufficient purge_zero_lamport_account logic
* Add another pass to detect non-deleted values and increment the count
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-02-12 08:51:03 -08:00
Michael Vines
3ac0192d40
Better surface bank hash verification failures
2020-02-05 11:39:47 -07:00
sakridge
78f6ddc5b7
Fix spelling of verification in accounts_db ( #8117 )
...
automerge
2020-02-04 11:45:47 -08:00
Jack May
83718a3b3e
Cleanup runtime use syntax ( #8002 )
2020-01-28 17:03:20 -08:00
Ryo Onodera
2c7447b73e
Secure sysvars under hash by freezing all strictly ( #7892 )
...
* Secure sysvars under hash by freezing all strictly
* Fix hash's non-idempotnet and add new test
* Clean up
* More cleanups
2020-01-24 16:10:32 +09:00
Michael Vines
8b8033c72b
Set BankRc slot correctly when restoring a bank snapshot
2020-01-23 13:37:13 -07:00