Commit Graph

144 Commits

Author SHA1 Message Date
Michael Vines 6858950f76 Remove frozen ABI modules from solana-sdk 2020-10-20 16:11:30 -07:00
sakridge 456eae6ccb
Calculate accounts hash async in accounts background service (#12852) 2020-10-19 09:48:29 -07:00
sakridge 4f6ebbdbe0
Add timings for store accounts (#12882) 2020-10-15 12:14:39 -07:00
carllin f8d338c9cb
Switch accounts storage lock to DashMap (#12126)
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-13 18:29:50 -07:00
carllin 16d45b8480
Fix zero-lamport accounts preventing slot cleanup (#12606)
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-09 12:40:08 -07:00
carllin adf6f748c0
Disable rent in flaky test (#12658)
* Disable rent in test

Co-authored-by: Carl Lin <carl@solana.com>
2020-10-03 12:44:03 -07:00
carllin 06f84c65f1
Fix rooted accounts cleanup, simplify locking (#12194)
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-28 16:04:46 -07:00
carllin 00b36e6db2
Fix ref-count for multiple stores to the same pubkey in a slot, fixes zero lamport purge detection (#12462)
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-25 18:54:48 -07:00
carllin 55be8d4016
Move dropping AppendVecs outside lock (#12408)
* Move drop outside lock

Co-authored-by: Carl Lin <carl@solana.com>
2020-09-23 14:17:49 -07:00
sakridge 3930cb865a
Add keccak-secp256k1 instruction (#11839)
* Implement keccak-secp256k1 instruction

Verifies eth addreses with ecrecover function

* Move secp256k1 test
2020-09-15 18:23:21 -07:00
Ryo Onodera de4a613610
Check bank capitalization (#11927)
* Check bank capitalization

* Simplify and unify capitalization calculation

* Improve and add tests

* Avoid overflow and inhibit automatic restart

* Fix test

* Tweak checked sum for cap. and add tests

* Fix broken build after merge conflicts..

* Rename to ClusterType

* Rename confusing method

* Clarify comment

* Verify cap. in rent and inflation tests

Co-authored-by: Stephen Akridge <sakridge@gmail.com>
2020-09-12 01:48:06 +09:00
Ryo Onodera c274e26eb8
Prevent unbound memory growth by blockstore_processor (#12110)
* Prevent unbound memory growth by blockstore_processor

* Promote log to info! considering infrequency

* Exclude the time of freeing from interval...

* Skip not-shrinkable slots even if forced

* Add comment
2020-09-09 12:05:10 +09:00
Ryo Onodera 53b8ea4464
Rename to ClusterType and restore devnet compat. (#12068)
* Rename to ClusterType and restore devnet compat.

* De-duplicate parse code and add comments

* Adjust default Devnet genesis & reduce it in tests
2020-09-08 23:55:09 +09:00
carllin a13efc52b3
Add unlock epochs for blake3 (#12054)
Co-authored-by: Carl <carl@solana.com>
2020-09-04 15:21:15 -07:00
carllin af08221aec
Switch account hashing to blake3 (#11969)
* Switch account hashing to blake3

Co-authored-by: Carl <carl@solana.com>
2020-09-02 00:37:36 -07:00
sakridge 770d3d383c
Accounts hash calculation metrics (#11433) 2020-08-06 21:39:10 -07:00
Michael Vines eefcf484cb clippy 2020-08-03 18:35:15 +00:00
Ryo Onodera 3e4f49f9c9
Fix race condition between shrinking and cleaning (#11235)
* Fix race condition between shrinking and cleaning

* Minor formatting

* fix ci

* Update comments

* More update comment

* Adjust fn naming
2020-07-29 06:06:14 +09:00
Michael Vines e5d8c4383f Add Bank support for "upgrade epochs" where all non-vote transactions will be rejected 2020-07-17 08:48:42 -07:00
sakridge 8bf3a0aa05
accounts_clean: Convert stack dependency calculation with iterative (#11067)
* accounts_clean: Convert stack dependency calculation with iterative

* optimize clean with by creating a reverse-lookup hashset of the affected
keys

* Add dependency bench

reduce bench

* Huge clean
2020-07-15 06:49:22 -07:00
Kristofer Peterson ed5a2f2a90
Fold bank serialisation into serde snapshot (#10581)
* Move bank (de)serialisation logic from bank and snapshot_utils to serde_snapshot.
Add sanity assertions between genesis config and bank fields on deserialisation.
Atomically update atomic bool in quote_for_specialization_detection().
Use same genesis config when restoring snapshots in test cases.

* Tidy up namings and duplicate structs to version

* Apply struct renames to tests

* Update abi hashes

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-07-13 23:00:59 +09:00
sakridge edf2b1ee85
Check for deleting key, make sure list is empty again (#11007) 2020-07-11 20:38:34 -07:00
sakridge 631f051c68
Add jitter to cleanup to prevent all nodes cleaning at the same time (#10936) 2020-07-10 11:54:45 -07:00
sakridge 6429042b6e
Add --print-account-stats/--snapshot-archive-path arguments to ledger-tool (#10945)
Allows for seeing how the accounts are being stored
and specifying a different snapshot source directory.
2020-07-08 09:32:11 -07:00
Ryo Onodera 39b3ac6a8d
Introduce automatic ABI maintenance mechanism (2/2; rollout) (#8012)
* Introduce automatic ABI maintenance mechanism (2/2; rollout)

* Fix stable clippy

* Change to symlink

* Freeze abi of Tower

* fmt...

* Improve dev-experience!

* Update BankSlotDelta

$ diff -u /tmp/abi8/*7dg6BreYxTuxiVz6aLvk3p2Z7GQk2cJqfGvC9h4FAoSj* /tmp/abi8/*9chBcbXVJ4fK7uGgydQzam5aHipaAKFw6V4LDFpjbE4w*
--- /tmp/abi8/bank__BankSlotDelta_frozen_abi__test_abi_digest_7dg6BreYxTuxiVz6aLvk3p2Z7GQk2cJqfGvC9h4FAoSj      2020-06-18 18:01:22.831228087 +0900
+++ /tmp/abi8/bank__BankSlotDelta_frozen_abi__test_abi_digest_9chBcbXVJ4fK7uGgydQzam5aHipaAKFw6V4LDFpjbE4w      2020-07-03 15:59:58.430695244 +0900
@@ -140,7 +140,7 @@
                                                         field u8
                                                             primitive u8
                                                         field solana_sdk::instruction::InstructionError
-                                                            enum InstructionError (variants = 34)
+                                                            enum InstructionError (variants = 35)
                                                                 variant(0) GenericError (unit)
                                                                 variant(1) InvalidArgument (unit)
                                                                 variant(2) InvalidInstructionData (unit)
@@ -176,6 +176,7 @@
                                                                 variant(31) CallDepth (unit)
                                                                 variant(32) MissingAccount (unit)
                                                                 variant(33) ReentrancyNotAllowed (unit)
+                                                                variant(34) MaxSeedLengthExceeded (unit)
                                                     variant(9) CallChainTooDeep (unit)
                                                     variant(10) MissingSignatureForFee (unit)
                                                     variant(11) InvalidAccountIndex (unit)

* Fix some merge conflicts...
2020-07-06 20:22:23 +09:00
sakridge 832d47317e
Move clean accounts to background service (#10898) 2020-07-02 22:25:17 -07:00
Greg Fitzgerald 0550b893b0
Fix typos (#10675)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2020-06-17 20:54:52 -07:00
carllin 8bd62d78eb
Fix race in remove_unrooted_slot (#10607)
* Fix race

* clippy fixes

* Rename and add comment

Co-authored-by: Carl <carl@solana.com>
2020-06-15 19:41:26 -07:00
Ryo Onodera dfe72d5242
Optimize stale slot shrinking for previously cleaned roots (#10099)
* Prioritize shrinking of previously cleaned roots

* measure time of stale slot shrinking

* Disable shrink for test

* shrink: budgeting, store count, force for snapshot

* Polish implementation and fix tests

* Fix ci..

* Clean up a bit

* Further polish implementation and fix/add tests

* Rebase fixes

* Remove unneeded Default for AccountStorageEntry

* Address review comments

* More cleanup

* More cleanup
2020-06-12 14:51:43 +09:00
Boqin Qin 0a638a8f17
runtime: fix possible deadlock in accounts_db (#10469) 2020-06-10 19:37:15 +09:00
Kristofer Peterson e23340d89e
Clippy cleanup for all targets and nighly rust (also support 1.44.0) (#10445)
* address warnings from 'rustup run beta cargo clippy --workspace'

minor refactoring in:
- cli/src/cli.rs
- cli/src/offline/blockhash_query.rs
- logger/src/lib.rs
- runtime/src/accounts_db.rs

expect some performance improvement AccountsDB::clean_accounts()

* address warnings from 'rustup run beta cargo clippy --workspace --tests'

* address warnings from 'rustup run nightly cargo clippy --workspace --all-targets'

* rustfmt

* fix warning stragglers

* properly fix clippy warnings test_vote_subscribe()
replace ref-to-arc with ref parameters where arc not cloned

* Remove lock around JsonRpcRequestProcessor (#10417)

automerge

* make ancestors parameter optional to avoid forcing construction of empty hash maps

Co-authored-by: Greg Fitzgerald <greg@solana.com>
2020-06-09 09:38:14 +09:00
Michael Vines e3b444834f
Add more logging while unpacking snapshots (#10266) 2020-05-27 10:41:05 -07:00
Kristofer Peterson b7a32f01c0
Multi-version snapshot support (#9980)
* Multi-version snapshot support

* rustfmt

* Remove CLI options and runtime support for selection output snapshot version.
Address some clippy complaints.

* Muzzle clippy type complexity warning.

Despite clippy's suggestion, it is not currently possible to create type aliases
for traits and so everything within the 'Box<...>' cannot be type aliased.

This then leaves creating full blown traits, and either implementing
said traits by closure (somehow) or moving the closures into new structs
implementing said traits which seems a bit of a palaver.

Alternatively it is possible to define and use the type alias 'type ResultBox<T> = Result<Box<T>>'
which does seems rather pointless and not a great reduction in complexity but is enough to keep clippy happy.

In the end I simply went with squelching the clippy warning.

* Remove now unused Serialize/Deserialize trait implementations for AccountStorageEntry and AppendVec

* refactor versioned de/serialisers

* rename serde_utils to serde_snapshot

* move call to accounts_db.generate_index() back down to context_accountsdb_from_stream()

* update version 1.1.1 to 1.2.0
remove nested use of serialize_bytes

* cleanups

* Add back measurement of account storage entry serialization.
Remove construction of Vec and HashMap temporaries during serialization.

* consolidate serialisation test cases into serde_snapshot.
clean up leakage of implementation details in serde_snapshot.

* move short term / legacy snapshot code into child module

* add serialize_iter_as_tuple

* preliminary integration of following commit

commit 6d58b73c47294bfb93465d5a83cd2175660b6e6d
Author: Ryo Onodera <ryoqun@gmail.com>
Date:   Wed May 20 14:02:02 2020 +0900

    Confine snapshot 1.1 relic to versioned codepath

* refactored serde_snapshot, rustfmt
legacy accounts_db format now "owns" both leading u64s, legacy bank_rc format has none

* reduce type complexity (clippy)
2020-05-23 02:54:24 +09:00
Kristofer Peterson 58ef02f02b
9951 clippy errors in the test suite (#10030)
automerge
2020-05-15 09:35:43 -07:00
Ryo Onodera 1eb40c3fe0
Introduce eager rent collection (#9527)
* Switch AccountsIndex.account_maps from HashMap to BTreeMap

* Introduce eager rent collection

* Start to add tests

* Avoid too short eager rent collection cycles

* Add more tests

* Add more tests...

* Refacotr!!!!!!

* Refactoring follow up

* More tiny cleanups

* Don't rewrite 0-lamport accounts to be deterministic

* Refactor a bit

* Do hard fork, restore tests, and perf. mitigation

* Fix build...

* Refactor and add switch over for testnet (TdS)

* Use to_be_bytes

* cleanup

* More tiny cleanup

* Rebase cleanup

* Set Bank::genesis_hash when resuming from snapshot

* Reorder fns and clean ups

* Better naming and commenting

* Yet more naming clarifications

* Make prefix width strictly uniform for 2-base partition_count

* Fix typo...

* Revert cluster-dependent gate

* kick ci?

* kick ci?

* kick ci?
2020-05-13 16:22:14 +09:00
Justin Starry 5cc252d471
Remove hash field from account (#9915) 2020-05-12 23:39:46 +08:00
Ryo Onodera 2ac50177a6
Include account.owner into account hash (#9917)
automerge
2020-05-07 13:01:11 -07:00
carllin 3442f36f8a
Repair alternate versions of dead slots (#9805)
Co-authored-by: Carl <carl@solana.com>
2020-05-05 14:07:21 -07:00
Michael Vines 230df0ec0c
Upgrade to Rust 1.43.0 (#9754) 2020-04-29 18:02:05 -07:00
Ryo Onodera 9918539229
Introduce type alias Ancestors (#9699)
* Introduce type alias AncestorList

* Rename AncestorList => Ancestors
2020-04-27 11:07:03 +09:00
sakridge 217828a849
Reduce accounts_db prints (#9469) 2020-04-15 09:05:54 -07:00
sakridge 99655206c8
Calculate account refs fix (#9447) 2020-04-11 12:52:10 -07:00
Ryo Onodera b28ec430e4
Introduce background stale AppendVec shrink mechanism (#9219)
* Introduce background AppendVec shrink mechanism

* Support ledger tool

* Clean up

* save

* save

* Fix CI

* More clean up

* Add tests

* Clean up yet more

* Use account.hash...

* Fix typo....

* Add comment

* Rename accounts_cleanup_service
2020-04-06 17:30:23 +09:00
sakridge b1771b92ec
Calculate ref counts earlier to prevent bad clean (#9147) 2020-03-29 14:42:34 -07:00
Ryo Onodera 2623c71ed3
Use type aliases/resulting var names consistently (#9060) 2020-03-26 13:08:56 +09:00
Justin Starry ef3af104ae
Return appropriate error for invalid program account (#9047)
automerge
2020-03-25 13:23:05 -07:00
sakridge 45348b2c83
Remove accounts unwrap (#9062)
automerge
2020-03-25 10:21:30 -07:00
sakridge c530fbd22b
Remove thread-priority crate which is not cross-platform (#9023) 2020-03-23 12:18:52 -07:00
sakridge 4b397d15b3
Accounts cleanup service and perf improvements (#8799)
* Use atomic for ref count instead of taking rwlock

* Accounts cleanup service

* Review comments
2020-03-23 08:50:23 -07:00
Michael Vines 88ba8439fc
Add frozen account support (#8989)
automerge
2020-03-22 11:10:04 -07:00