Commit Graph

200 Commits

Author SHA1 Message Date
carllin 6276360468
Prevent scans from seeing root updates/clean (#13464)
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-16 17:23:11 -08:00
joshua-maros a8a77614fe
Fix potential undefined behavior (#13555)
* Switch to ouroboros 0.5.1

* Update other lock files
2020-11-13 18:12:41 +09:00
Michael Vines 959880db60 Remove unused pubkey::Pubkey imports 2020-10-21 19:08:13 -07:00
carllin e6b821c392
Finer grained AccountsIndex locking (#12787)
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-21 17:05:27 -07:00
carllin 64c4861e04
Fix error in max root calculation (#12661)
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-03 15:18:58 -07:00
carllin 06f84c65f1
Fix rooted accounts cleanup, simplify locking (#12194)
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-28 16:04:46 -07:00
carllin 00b36e6db2
Fix ref-count for multiple stores to the same pubkey in a slot, fixes zero lamport purge detection (#12462)
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-25 18:54:48 -07:00
sakridge 832d47317e
Move clean accounts to background service (#10898) 2020-07-02 22:25:17 -07:00
Ryo Onodera 50f7ed80c8
Dont skip eager rent collect across gapped epochs (#10206)
* Dont skip eager rent collect across gapped epochs

* Adjust style and comment

* Adjust ascii chart and comment a bit

* Moar assert

* Relax the partition_count assert for completeness

* Tweak comment...

* tweak a bit

* Add gating logic

* Address reviews

* small formatting

* Clarify the code by replacing auto_generated...

* small formatting

* small formatting

* small formatting

* small formatting

* Narrow down conditional compilation scope
2020-06-25 23:11:33 +09:00
Ryo Onodera dfe72d5242
Optimize stale slot shrinking for previously cleaned roots (#10099)
* Prioritize shrinking of previously cleaned roots

* measure time of stale slot shrinking

* Disable shrink for test

* shrink: budgeting, store count, force for snapshot

* Polish implementation and fix tests

* Fix ci..

* Clean up a bit

* Further polish implementation and fix/add tests

* Rebase fixes

* Remove unneeded Default for AccountStorageEntry

* Address review comments

* More cleanup

* More cleanup
2020-06-12 14:51:43 +09:00
Kristofer Peterson e23340d89e
Clippy cleanup for all targets and nighly rust (also support 1.44.0) (#10445)
* address warnings from 'rustup run beta cargo clippy --workspace'

minor refactoring in:
- cli/src/cli.rs
- cli/src/offline/blockhash_query.rs
- logger/src/lib.rs
- runtime/src/accounts_db.rs

expect some performance improvement AccountsDB::clean_accounts()

* address warnings from 'rustup run beta cargo clippy --workspace --tests'

* address warnings from 'rustup run nightly cargo clippy --workspace --all-targets'

* rustfmt

* fix warning stragglers

* properly fix clippy warnings test_vote_subscribe()
replace ref-to-arc with ref parameters where arc not cloned

* Remove lock around JsonRpcRequestProcessor (#10417)

automerge

* make ancestors parameter optional to avoid forcing construction of empty hash maps

Co-authored-by: Greg Fitzgerald <greg@solana.com>
2020-06-09 09:38:14 +09:00
Ryo Onodera 1eb40c3fe0
Introduce eager rent collection (#9527)
* Switch AccountsIndex.account_maps from HashMap to BTreeMap

* Introduce eager rent collection

* Start to add tests

* Avoid too short eager rent collection cycles

* Add more tests

* Add more tests...

* Refacotr!!!!!!

* Refactoring follow up

* More tiny cleanups

* Don't rewrite 0-lamport accounts to be deterministic

* Refactor a bit

* Do hard fork, restore tests, and perf. mitigation

* Fix build...

* Refactor and add switch over for testnet (TdS)

* Use to_be_bytes

* cleanup

* More tiny cleanup

* Rebase cleanup

* Set Bank::genesis_hash when resuming from snapshot

* Reorder fns and clean ups

* Better naming and commenting

* Yet more naming clarifications

* Make prefix width strictly uniform for 2-base partition_count

* Fix typo...

* Revert cluster-dependent gate

* kick ci?

* kick ci?

* kick ci?
2020-05-13 16:22:14 +09:00
carllin 3442f36f8a
Repair alternate versions of dead slots (#9805)
Co-authored-by: Carl <carl@solana.com>
2020-05-05 14:07:21 -07:00
Ryo Onodera 9918539229
Introduce type alias Ancestors (#9699)
* Introduce type alias AncestorList

* Rename AncestorList => Ancestors
2020-04-27 11:07:03 +09:00
Ryo Onodera 2623c71ed3
Use type aliases/resulting var names consistently (#9060) 2020-03-26 13:08:56 +09:00
sakridge 4b397d15b3
Accounts cleanup service and perf improvements (#8799)
* Use atomic for ref count instead of taking rwlock

* Accounts cleanup service

* Review comments
2020-03-23 08:50:23 -07:00
Ryo Onodera 952cd38b7b
Avoid early clean and bad snapshot by ref-counting (#8724)
* Avoid early clean and bad snapshot by ref-counting

* Add measure

* Clean ups

* clean ups
2020-03-13 14:14:37 +09:00
Ryo Onodera d86103383a
Do periodic inbound cleaning for rooted slots (#8436)
* Do periodic inbound compaction for rooted slots

* Add comment

* nits

* Consider not_compacted_roots in cleanup_dead_slot

* Renames in AccountsIndex

* Rename to reflect expansion of removed accounts

* Fix a comment

* rename

* Parallelize clean over AccountsIndex

* Some niceties

* Reduce locks and real chunked parallelism

* Measure each step for sampling opportunities

* Just noticed par iter is maybe lazy

* Replace storage scan with optimized index scan

* Various clean-ups

* Clear uncleared_roots even if no updates
2020-03-03 14:57:25 +09:00
Tyera Eulberg ab361a8073
Rename KeypairUtil to Signer (#8360)
automerge
2020-02-20 13:28:55 -08:00
Ryo Onodera 58727463e1
Remove needless last_root for better reclaims (#8148)
* Restore last_root to fix unintended storage delete

* Remove last_root thing altogether

* Remove unneeded test...
2020-02-13 08:19:53 +09:00
François Garillot f016c9a669
Maintenance : simplify a few patterns, remove unneeded dependencies (#8137)
* Simplify a few pattern matches

* Removing unneeded dependencies, upgrading internal version #s

 net-shaper: Removing log, semver, serde_derive
 bench-tps: Removing serde, serde_derive
 banking-bench: Removing solana
 ledger-tool: Removing bincode, serde, serde_derive
 librapay: Removing solana, language_e2e_tests
 log-analyzer: Removing log, semver, serde_derive
 exchange: Removing solana
 core: Removing crc, memmap, symlink, untrusted
 perf: Removing serde_derive
 genesis: Removing hex, serde_derive
 sdk-c: Removing sha2
 sys-tuner: Removing semver
 bench-exchange: Removing bincode, bs58, env_logger, serde, serde_derive, untrusted, ws
 btc_spv_bin: Removing serde_json
 btc_spv: Removing chrono
 bpf_loader: Removing serde
 ledger: Removing dlopen, dlopen_derive, serde_derive
 move_loader: Removing byteorder, libc, language_e2e_tests
 ownable: Removing serde, serde_derive
 client: Removing rand
 archiver-utils: Removing rand_chacha
 validator: Removing serde_json, tempfile
 param_passing_dep: Removing solana
 failure: Removing log
 vest: Removing log
 vote-signer: Removing bs58, serde
 local-cluster: Removing symlink
 keygen: Removing rpassword
 install: Removing bs58, log
 upload-perf: Removing log
 runtime: Removing serde_json
 stake: Removing rand

* Add modified Cargo.lock

* fixup! Simplify a few pattern matches

* fixup! Simplify a few pattern matches
2020-02-06 10:02:38 -07:00
Jack May 83718a3b3e
Cleanup runtime use syntax (#8002) 2020-01-28 17:03:20 -08:00
Jack May 7c92bf15e2
Only return accounts that have changed since the bank's parent (#7520)
* Only return accounts that have changed since the bank's parent

* add test
2019-12-16 23:24:18 -08:00
sakridge 0aa4dc904e
Accounts index updates, remove pubkey when dead account (#7408) 2019-12-11 11:11:31 -08:00
sakridge 887bff572a
More conservative purge_zero_lamport_accounts purge logic (#7157) 2019-12-02 09:51:05 -08:00
Rob Walker b4119c454a
credit_only credits forwarding (#6509)
* credit_only_credits_forwarding

* whack transfer_now()

* fixup

* bench should retry the airdrop TX

* fixup

* try to make bench-exchange a bit more robust, informative
2019-10-23 22:01:22 -07:00
sakridge f1172617cc
Purge accounts with lamports=0 on rooted forks (#6315) 2019-10-23 12:46:48 -07:00
Greg Fitzgerald fb39bd45d7
Revert "Rename solana-runtime to sealevel (#6239)" (#6247)
This reverts commit 2e921437cd.
2019-10-04 19:33:29 -06:00
Greg Fitzgerald 2e921437cd Rename solana-runtime to sealevel (#6239)
automerge
2019-10-04 15:02:44 -07:00
Rob Walker 52f6da5cee
upgrade rust to 1.37 (#5611) 2019-08-23 08:55:51 -07:00
Greg Fitzgerald a43922ccbf
Boot hashbrown (#5505)
As of Rust 1.36.0, hashbrown now implements the HashMap in std (which
implements HashSet).

https://blog.rust-lang.org/2019/07/04/Rust-1.36.0.html#a-new-hashmapk,-v%3E-implementation
2019-08-12 16:46:49 -06:00
Sagar Dhawan 9dcf3347f5 Refactor status cache and remove complex serialize/deserialize (#5335)
automerge
2019-08-06 18:47:30 -07:00
carllin 1dbb5c8647
Deserialize snapshots (#5417)
* Deserialize snapshots
2019-08-05 22:53:19 -07:00
sakridge 595017499e
accounts_index: RwLock per-account (#5198)
* accounts_index: RwLock per-account

Lots of lock contention on the accounts_index lock,
only take write-lock on accounts_index if we need to insert/remove an
account.
For updates, take a read-lock and then write-lock on the individual
account.

* Remove unneeded enumerate and add comments.
2019-07-20 17:58:39 -07:00
Rob Walker 2abbc89dcd
add accounts_index_scan_accounts (#5020) 2019-07-10 22:06:32 -07:00
Sathish 9e7f618cff Set proper count value for account stores (#4797)
* set count values for store accounts

* Use AppendVecId type
2019-06-25 07:21:45 -07:00
Sathish Ambley 44967abd1c update storage len 2019-06-17 22:48:27 -07:00
sakridge 9e3758983d
Find max root and purge roots below it. (#4645)
* Test for forking accounts

* Find max root and purge roots below it.
2019-06-13 17:35:16 -07:00
sakridge 966b6999d1
Accounts index opt (#4621)
* Add accounts_index bench

* Don't take the accounts index lock unless needed

* Accounts_index remove insert return vec and add capacity stats

* Use hashbrown hashmap for accounts_index
2019-06-10 18:15:39 -07:00
Sagar Dhawan 9e14cde461 Revert "Fix roots never being purged (#4134)" (#4628)
automerge
2019-06-10 14:08:09 -07:00
Sathish 182096dc1a
Create bank snapshots (#4244)
* Revert "Revert "Create bank snapshots (#3671)" (#4243)"

This reverts commit 81fa69d347.

* keep saved and unsaved copies of status cache

* fix format check

* bench for status cache serialize

* misc cleanup

* remove appendvec storage on purge

* fix accounts restore

* cleanup

* Pass snapshot path as args

* Fix clippy
2019-05-30 21:31:35 -07:00
Rob Walker 81fa69d347
Revert "Create bank snapshots (#3671)" (#4243)
This reverts commit abf2b300da.
2019-05-09 19:27:27 -07:00
Sathish abf2b300da Create bank snapshots (#3671)
* Be able to create bank snapshots

* fix clippy

* load snapshot on start

* regenerate account index from the storage

* Remove rc feature dependency

* cleanup

* save snapshot for slot 0
2019-05-09 19:27:06 -07:00
Tyera Eulberg 71f9b44687
Add Confirmations parameter to RPC Subscriptions (#4154)
* Add optional depth parameter to pubsub, and store in subscriptions

* Pass bank_forks into rpc_subscription; add method to check depth before notify and impl for account subscriptions

* Impl check-depth for signature subscriptions

* Impl check-depth for program subscriptions

* Plumb fork id through accounts

* Use fork id and root to prevent repeated account notifications; also s/Depth/Confirmations

* Write tests in terms of bank_forks

* Fixup accounts tests

* Add pubsub-confirmations tests

* Update pubsub documentation
2019-05-06 08:31:50 -06:00
carllin 5bb75a5894
Fix roots never being purged (#4134) 2019-05-03 15:17:37 -07:00
anatoly yakovenko acba1d6f9e
Roots are added out of order to the accounts index (#4051)
* fix root race

* assert root order

* fixup! assert root order

* last root test

* update

* fix tests
2019-04-28 10:27:37 -07:00
Rob Walker 5406d82d89
fix race in get_exclusive_storage() and load() (#3941)
* Revert "Revert "account storage is not in sync with the index after gc (#3914)" (#3936)"

This reverts commit 4f47fc00bc.

* fix get_exclusive_storage

* clippy
2019-04-23 09:56:36 -07:00
Pankaj Garg 4f47fc00bc
Revert "account storage is not in sync with the index after gc (#3914)" (#3936)
This reverts commit 101d6b92ee.
2019-04-22 17:14:41 -07:00
anatoly yakovenko 101d6b92ee
account storage is not in sync with the index after gc (#3914)
* account storage is not in sync with the index after gc

* builds

* clippy fmt

* test

* purge dead forks on store

* rm println

* also fixed count_stores

* comments
2019-04-20 20:50:50 -05:00
anatoly yakovenko 68fc303b9b
Rework Accounts for fast squash, hashing state and checkpoint recovery. (#3613)
* accounts rewrite

* ignore grow tests

* skip duplicate roots

* allow for a root race

* logger

* accounts_index tests

* tests

* tests
2019-04-15 17:15:50 -07:00