Commit Graph

42 Commits

Author SHA1 Message Date
Jeff Washington (jwash) b0b4e1f0c0
remove IncludeSlotInHash after feature activation on mnb (#33816)
* remove IncludeSlotInHash after feature activation on mnb

* fix compile errors

* compile errors

* fix tests

* fix test results
2023-10-23 15:12:02 -07:00
Pankaj Garg f4287d70bb
Move accounts-db code to its own crate (#32766) 2023-08-09 13:03:36 -07:00
Alexander Meißner 520c647918
Refactor - Merges `BuiltinPrograms` into `LoadedPrograms` (#31654)
* Merges BuiltinPrograms into LoadedPrograms.

* Prevents built-ins from being pruned.
2023-05-18 22:18:28 +02:00
Alexander Meißner 9e72668a77
Refactor - Simplify built-in program feature transition (#31565)
* Removes builtins::get_builtins(), instead derive them from Bank::builtin_programs.

* Removes BuiltinAction.
Turns BuiltinFeatureTransition from an enum into a struct.

* Makes Bank::remove_builtin() place tombstones.
2023-05-10 16:53:36 +02:00
Brennan a540690778
Move partition logic out of bank (#31343)
* Move partition logic out of bank
2023-04-26 12:37:02 -07:00
Brooks d3273ba118
Removes unused `ancestors` param from `get_snapshot_storages` (#30773) 2023-03-18 17:17:41 -04:00
Yueh-Hsuan Chiang 33e179905f
getter functions for StoredAccountMeta (#30447)
This PR makes all the StoredAccountMeta fields pub(crate)
and provides getter functions to access its member fields.

This PR is a preparation step for abstracting out StoredAccountMeta.
2023-02-27 10:03:22 -08:00
Jeff Washington (jwash) 12411e5215
pass shrink_can_be_active through shrink fns (#30244) 2023-02-10 10:06:40 -08:00
Jeff Washington (jwash) 927da903a8
StoreTo::Storage holds storage (#30156) 2023-02-07 11:54:26 -08:00
Brooks bda0c606a1
Renames fn to calculate_accounts_delta_hash() (#29876) 2023-01-24 18:55:56 -05:00
Jeff Washington (jwash) aef8692c8f
remove SnapshotStorageOne (#29747) 2023-01-18 07:10:56 -08:00
Jeff Washington (jwash) 14055fc3e2
remove type SnapshotStorage and uses (#29661) 2023-01-11 17:57:44 -08:00
Jeff Washington (jwash) 544b9745c2
snapshot storage path uses 1 append vec per slot (#29627) 2023-01-11 12:05:15 -08:00
Jeff Washington (jwash) 0f16120b03
remove FoundStoredAccount (#29591) 2023-01-10 13:57:34 -08:00
Jeff Washington (jwash) 6441ed121e
get_unique_accounts_from_storages drops plural (#29600) 2023-01-10 07:49:38 -08:00
Jeff Washington (jwash) dda34b208c
get_unique_accounts_from_storages takes 1 append vec (#29588) 2023-01-09 14:53:50 -08:00
Jeff Washington (jwash) 8e19909c7d
stop returning remaining # stores from mark_dirty_dead_stores (#29545) 2023-01-05 19:21:01 -08:00
Jeff Washington (jwash) 150ac80cfa
filter_storages ignores reclaims (#29512) 2023-01-04 12:38:30 -08:00
Jeff Washington (jwash) 9ee4555f53
remove should_retain from mark_dirty_dead_stores (#29358) 2023-01-03 09:53:43 -08:00
Jeff Washington (jwash) 0f7ef085a0
introduce ShrinkInProgress (#29329)
* introduce ShrinkInProgress

* remove redundant check

* add comments
2022-12-21 09:03:29 -08:00
Jeff Washington (jwash) 098b86b883
test_minimize_accounts_db uses write cache correctly (#29333) 2022-12-20 09:50:56 -08:00
Jeff Washington (jwash) f9db75768c
Vec<Arc<AccountStorageEntry>> becomes SnapshotStorage (#29253) 2022-12-14 12:18:43 -08:00
Jeff Washington (jwash) b77bef4ee3
write_version -> write_version_obsolete (#29176) 2022-12-14 07:43:40 -08:00
Brooks Prumo c65a8ce6c3
AccountsDb::get_snapshot_storages() takes a Range for slots (#29054) 2022-12-05 11:21:21 -05:00
Jeff Washington (jwash) 5d88a9b32b
improve perf of storing accounts for shrink/ancient (#28880)
* improve perf of storing accounts for shrink/ancient

* rename

* phantom data

* update comment

* make impl Borrow<Hash> consistent

* remove unused static
2022-11-22 15:36:57 -08:00
Jeff Washington (jwash) 552248906e
report ancient stats correctly in drop_or_recycle_stores (#28747) 2022-11-10 07:27:38 -08:00
Jeff Washington (jwash) 0b51b15af2
remove duplicate pubkey during shrink (#28689) 2022-11-01 07:45:52 -07:00
Jeff Washington (jwash) 51e1a7f2af
simplify mark_dirty_dead_stores (#28565) 2022-10-25 11:34:58 -07:00
Jeff Washington (jwash) 993583d9b3
simplify get_store_for_shrink (#28524) 2022-10-21 14:49:34 -07:00
Jeff Washington (jwash) d52e28ac22
combine sort into get_unique_accounts_from_storages (#28450) 2022-10-18 11:52:48 -07:00
Jeff Washington (jwash) fd2e671861
remove slot# from account hash (#28405)
* remove slot# from account hash

* add feature

* fix tests

* constants to help clarify 'irrelevant' changes

* move to enum for enforcing irrelevancy

* ignore unsupported tests
2022-10-18 08:03:37 -07:00
Jeff Washington (jwash) d74480a125
when shrinking, don't send entire old append vec to clean (#27914) 2022-09-20 06:32:07 -07:00
Jeff Washington (jwash) 1811d684b5
clean race condition with extra unref (#27682)
* clean race condition with extra unref

* clarify comment

* fix test

* cleanup test

* clippy
2022-09-10 13:57:17 -07:00
Brennan Watt e4a7d01e10
Rust v1.63 (#27303)
* Upgrade to Rust v1.63.0

* Add nightly_clippy_allows

* Resolve some new clippy nightly lints

* Increase QUIC packets completion timeout

* Update quinn-udp crate

Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-22 18:01:03 -07:00
Brennan Watt 7573000d87
Revert "Rust v1.63.0 (#27148)" (#27245)
This reverts commit a2e7bdf50a.
2022-08-19 09:19:44 +01:00
Brennan Watt a2e7bdf50a
Rust v1.63.0 (#27148)
* Upgrade to Rust v1.63.0

* Add nightly_clippy_allows

* Resolve some new clippy nightly lints

* Increase QUIC packets completion timeout

Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-17 15:48:33 -07:00
HaoranYi 1aa9215411
Refactor: add account_iter api for appendvec (#26591)
* add account iterator for appendvec

* semicolon
2022-07-12 23:56:05 +00:00
Jeff Washington (jwash) 9d31216d23
add StoreReclaims::Ignore (#26489) 2022-07-12 15:05:47 -05:00
Jeff Washington (jwash) d9eee72edb
remove FoundStoredAccount.account_size (#26460) 2022-07-07 18:37:14 -05:00
Jeff Washington (jwash) c9f1d01e79
refactor, add GetUniqueAccountsResult (#26457) 2022-07-07 13:49:33 -05:00
apfitzge 4a729effff
Bugfix/test get rent collection accounts remove random account (#26178)
* Use default GenisisConfig so we don't have a random account

* Formatting
2022-06-24 09:22:50 -04:00
apfitzge f4189c0305
ledger-tool minimized snapshots (#25334)
* working on local snapshot

* Parallelization for slot storage minimization

* Additional clean-up and fixes

* make --minimize an option of create-snapshot

* remove now unnecessary function

* Parallelize parts of minimized account set generation

* clippy fixes

* Add rent collection accounts and voting node_pubkeys

* Simplify programdata_accounts generation

* Loop over storages to get slot set

* Parallelize minimized slot set generation

* Parallelize adding owners and programdata_accounts

* Remove some now unncessary checks on the blockstore

* Add a warning for minimized snapshots across epoch boundary

* Simplify ledger-tool minimize

* Clarify names of bank's minimization helper functions

* Remove unnecesary funciton, fix line spacing

* Use DashSets instead of HashSets for minimized account and slot sets

* Filter storages uses all threads instead of thread_pool

* Add some additional comments on functions for minimization

* Moved more into bank and parallelized

* Update programs/bpf/Cargo.lock for dashmap in ledger

* Clippy fix

* ledger-tool: convert minimize_bank_for_snapshot Measure into measure!

* bank.rs: convert minimize_bank_for_snapshot Measure into measure!

* accounts_db.rs: convert minimize_accounts_db Measure into measure!

* accounts_db.rs: add comment about use of minimize_accounts_db

* ledger-tool: CLI argument clarification

* minimization functions: make infos unique

* bank.rs: Add test_get_rent_collection_accounts_between_slots

* bank.rs: Add test_minimization_add_vote_accounts

* bank.rs: Add test_minimization_add_stake_accounts

* bank.rs: Add test_minimization_add_owner_accounts

* bank.rs: Add test_minimization_add_programdata_accounts

* accounts_db.rs: Add test_minimize_accounts_db

* bank.rs: Add negative case and comments in test_get_rent_collection_accounts_between_slots

* bank.rs: Negative test in test_minimization_add_programdata_accounts

* use new static runtime and sdk ids

* bank comments to doc comments

* Only need to insert the maximum slot a key is found in

* rename remove_pubkeys to purge_pubkeys

* add comment on builtins::get_pubkeys

* prevent excessive logging of removed dead slots

* don't need to remove slot from shrink slot candidates

* blockstore.rs: get_accounts_used_in_range shouldn't return Result

* blockstore.rs: get_accounts_used_in_range: parallelize slot loop

* report filtering progress on time instead of count

* parallelize loop over snapshot storages

* WIP: move some bank minimization functionality into a new class

* WIP: move some accounts_db minimization functionality into SnapshotMinimizer

* WIP: Use new SnapshotMinimizer

* SnapshotMinimizer: fix use statements

* remove bank and accounts_db minimization code, where possible

* measure! doesn't take a closure

* fix use statement in blockstore

* log_dead_slots does not need pub(crate)

* get_unique_accounts_from_storages does not need pub(crate)

* different way to get stake accounts/nodes

* fix tests

* move rent collection account functionality to snapshot minimizer

* move accounts_db minimize behavior to snapshot minimizer

* clean up

* Use bank reference instead of Arc. Additional comments

* Add a comment to blockstore function

* Additional clarifying comments

* Moved all non-transaction account accumulation into the SnapshotMinimizer.

* transaction_account_set does not need to be mutable now

* Add comment about load_to_collect_rent_eagerly

* Update log_dead_slots comment

* remove duplicate measure/print of get_minimized_slot_set
2022-06-22 13:17:43 -04:00