Commit Graph

155 Commits

Author SHA1 Message Date
Brooks 45eaa4c1a6
Removes unnecessary Accounts constructors (#34471) 2023-12-14 22:50:15 -05:00
Brooks e79c09bdc4
Refactors Accounts constructors (#34466) 2023-12-14 16:36:47 -05:00
Brooks d1cfd26abb
Refactors AccountsDb constructors (#34456) 2023-12-14 12:34:15 -05:00
Brooks e02f25d5a2
Removes filler accounts (#34115) 2023-11-19 20:36:57 -05:00
Jon Cinque 0fe902ced7
Bump rand to 0.8, rand_chacha to 0.3, getrandom to 0.2 (#32871)
* sdk: Add concurrent support for rand 0.7 and 0.8

* Update rand, rand_chacha, and getrandom versions

* Run command to replace `gen_range`

Run `git grep -l gen_range | xargs sed -i'' -e 's/gen_range(\(\S*\), /gen_range(\1../'

* sdk: Fix users of older `gen_range`

* Replace `hash::new_rand` with `hash::new_with_thread_rng`

Run:
```
git grep -l hash::new_rand | xargs sed -i'' -e 's/hash::new_rand([^)]*/hash::new_with_thread_rng(/'
```

* perf: Use `Keypair::new()` instead of `generate`

* Use older rand version in zk-token-sdk

* program-runtime: Inline random key generation

* bloom: Fix clippy warnings in tests

* streamer: Scope rng usage correctly

* perf: Fix clippy warning

* accounts-db: Map to char to generate a random string

* Remove `from_secret_key_bytes`, it's just `keypair_from_seed`

* ledger: Generate keypairs by hand

* ed25519-tests: Use new rand

* runtime: Use new rand in all tests

* gossip: Clean up clippy and inline keypair generators

* core: Inline keypair generation for tests

* Push sbf lockfile change

* sdk: Sort dependencies correctly

* Remove `hash::new_with_thread_rng`, use `Hash::new_unique()`

* Use Keypair::new where chacha isn't used

* sdk: Fix build by marking rand 0.7 optional

* Hardcode secret key length, add static assertion

* Unify `getrandom` crate usage to fix linking errors

* bloom: Fix tests that require a random hash

* Remove some dependencies, try to unify others

* Remove unnecessary uses of rand and rand_core

* Update lockfiles

* Add back some dependencies to reduce rebuilds

* Increase max rebuilds from 14 to 15

* frozen-abi: Remove `getrandom`

* Bump rebuilds to 17

* Remove getrandom from zk-token-proof
2023-08-21 19:11:21 +02:00
Brooks e316db2ab4
dcou: set_accounts_hash() (#32829) 2023-08-15 07:18:59 -04:00
Pankaj Garg f4287d70bb
Move accounts-db code to its own crate (#32766) 2023-08-09 13:03:36 -07:00
Brooks 4894eb0333
Moves some accounts-db test-only code into a dev-context-only-utils feature (#32748) 2023-08-07 17:28:15 -04:00
Pankaj Garg 511cf28be8
Move `serde-snapshot dependent accounts-db tests (#32671)
Move serde-snapshot dependent accounts-db tests
2023-08-07 09:35:41 -07:00
Pankaj Garg faff34403c
Move bank dependent tests out of serde_snapshot/tests.rs (#32571) 2023-07-21 12:08:32 -07:00
Brooks 3b0e4c8452
Removes unnecessary default() (#32477) 2023-07-13 13:59:57 -04:00
Jeff Washington (jwash) 174ceba82b
rename to stake_rewards_by_partition for clarity (#32165) 2023-06-16 19:28:53 +00:00
Brooks 47ff3cecc9
Enables creating snapshots after booting from local state (#32137) 2023-06-15 22:54:32 -04:00
Jeff Washington (jwash) 6280c63e1a
epoch rewards are stored partitioned as Vec<Vec< (#32076)
* epoch rewards are stored partitioned as Vec<Vec<

* update comment
2023-06-12 13:06:22 -07:00
Jeff Washington (jwash) 37efc82a76
snapshot epoch reward fields tests (#32064) 2023-06-12 07:39:59 -07:00
Jeff Washington (jwash) 36ccc71220
move StakeReward to stake_rewards.rs (#32014) 2023-06-07 12:35:28 -07:00
steviez debe794987
Replace improper &Arc<...> with Arc<...> in Bank and Accounts (#31892)
The callstack updated in this PR passed an &Arc<...> down only to have
the bottom level clone the reference. Thus, we are giving shared
ownership so the reference is a bit redundant and arguably obscures the
intention to clone further down the callstack.
2023-05-31 12:36:44 -05:00
Jeff Washington (jwash) e396a10312
change StartBlockHeightAndRewards from parent to current (#31781) 2023-05-23 20:10:51 -07:00
Alexander Meißner 1e8e024ec5
Cleanup - disable_fee_calculator (#31766)
disable_fee_calculator
2023-05-23 18:20:38 +02:00
Yueh-Hsuan Chiang c79a6e74e2
Add new error type AccountsFileError (#31631)
#### Problem
AccountsFile- and AppendVec-related code uses io::Error for their main error type.  However, there're many errors under AccountsFile that don't belong to io::Error.

#### Summary of Changes
This PR introduces AccountsFileError and makes minimum changes to keep the PR small.
Subsequent changes related to this will be in separate PRs.
2023-05-22 14:09:09 -07:00
Jeff Washington (jwash) 0291454876
persist EpochRewardStatus::Active (#31697)
persist `EpochRewardStatus::Active`
2023-05-18 07:14:41 -07:00
Jeff Washington (jwash) 8d98589c85
remove unused historical_roots (#31539) 2023-05-08 10:50:27 -07:00
Brooks fdf7bdae1a
Fixes name of snapshot_utils::get_bank_snapshot_dir() (#31432) 2023-05-01 19:28:35 +00:00
Xiang Zhu 0a2e897f16
Clean up the outdated SnapshotPackage snapshot_links field (#31360)
* Remove snapshot_links

* Change the function name from snapshot_dir to bank_snapshot_dir

* Format fix

* Fix test_concurrent_snapshot_packaging

* Fix clippy error

* Fix nits

* Fix nits 2nd try

* Use get_bank_snapshots_dir

* Use slot_dir

* Revert "Use get_bank_snapshots_dir" because get_bank_snapshots_dir is private to crate

This reverts commit 1ed9b3b2c8e84689a918beee7159f63c56500a96.
2023-05-01 11:24:59 -07:00
Brennan 2164a50d00
Move BankIncrementalSnapshotPersistence (#31236)
* Move BankIncrementalSnapshotPersistence

* Update bank serialize ABI digest
2023-04-18 11:18:17 -07:00
steviez cc8e531a5d
Enforce a minimum of 1 on full and incremental snapshot retention (#30968) 2023-03-30 10:16:36 -05:00
Brooks e5edcb9c48
Sets IncrementalAccountsHash in serde_snapshot when reconstructing AccountsDb (#30801) 2023-03-21 10:51:04 -04:00
Brooks d3273ba118
Removes unused `ancestors` param from `get_snapshot_storages` (#30773) 2023-03-18 17:17:41 -04:00
Yueh-Hsuan Chiang fce949009d
Add new_from_file() API to AccountsFile (#30687)
#### Problem
Accounts db currently use AppendVec::new_from_file() directly
to create a new AcountsFile instance from an existing file.
However, this method should be abstracted out to AccountsFile
so that an existing file can be opened correctly using the right format.

#### Summary of Changes
Add new_from_file() API to AccountsFile which will open an existing
file based on its accounts file format.

Currently, it only supports AppendVec.
2023-03-15 20:38:20 -07:00
Brooks 263342a686
Adds accounts hashes from full & incremental snapshots at startup (#30656) 2023-03-13 14:47:25 -04:00
Brooks 5e5b7f00a2
Stores capitalization in account hashes map (#30635) 2023-03-13 10:50:45 -04:00
Brooks 64c7b401cd
BankIncrementalSnapshotPersistence uses strong accounts hash types (#30502) 2023-02-24 18:15:23 -05:00
Brooks ffe2d309b2
Adds serde_snapshot::types module (#30501) 2023-02-24 15:27:12 -05:00
Brooks ec288ded9b
Serde snapshot uses own types (#30391) 2023-02-20 12:25:15 -05:00
Brooks ad6a3abd8b
Uses set_accounts_{delta,}_hash_for_tests() in serde tests (#30390) 2023-02-17 16:00:25 -05:00
Yueh-Hsuan Chiang aeb6df3507
Abstract out AppendVec into AccountsFile enum (#29815)
Abstracts out AppendVec into AccountsFile enum.
This will allow different implementations of AccountsFile that support the AccountsDB.
2023-02-15 17:24:04 -08:00
Xiang Zhu 4909267c88
Add accounts hard-link files into the bank snapshot directory (#29496)
* Add accounts hard-link files into the bank snapshot directory

* Small adjustments and fixes.

* Address some of the review issues

* Fix compilation issues

* Change the latest slot snapshot storage from VecDeque to Option

* IoWithSourceAndFile and expanded comments on accounts

* last_slot_snapshot_storages in return value

* Update comments following the review input

* rename dir_accounts_hard_links to hard_link_path

* Add dir_full_state flag for add_bank_snapshot

* Let appendvec files hardlinking work with multiple accounts paths across multiple partitions

* Fixes for rebasing

* fix tests which generates account_path without adding run/

* rebasing fixes

* fix account path test failures

* fix test test_concurrent_snapshot_packaging

* review comments.  renamed the path setup function

* Addressed most of the review comments

* update with more review comments

* handle error from create_accounts_run_and_snapshot_dirs

* fix rebasing duplicate

* minor accounts_dir path cleanup

* minor cleanup, remove commented code

* misc review comments

* build error fix

* Fix test_incremental_snapshot_download_with_crossing_full_snapshot_interval_at_startup

* fix build error on MAX_BANK_SNAPSHOTS_TO_RETAIN

* rebase fix, update hardlink filename

* minor comment spelling fix

* rebasing fixes

* fix rebase issues; with_extension

* comments changes for review

* misc minor review issues

* bank.fill_bank_with_ticks_for_tests

* error handling on appendvec path

* fix use_jit

* minor comments refining

* Remove type AccountStorages

* get_account_path_from_appendvec_path return changed to Option

* removed appendvec_path.to_path_buf in create_accounts_run_and_snapshot_dirs

* add test_get_snapshot_accounts_hardlink_dir

* update last_snapshot_storages comment

* update last_snapshot_storages comment

* symlink map_err

* simplify test_get_snapshot_accounts_hardlink_dir with fake paths

* log last_snapshot_storages at the end of the loop
2023-02-15 09:52:07 -08:00
Brooks 06e801a8a5
Moves BankHashInfo into serde_snapshot (#30120) 2023-02-03 13:17:50 -05:00
Brooks 3249e4a123
Removes {get,set}_bank_hash_info_from_snapshot() (#30087) 2023-02-02 16:50:44 -05:00
Brooks d048a1903f
Splits up AccountsDb::bank_hashes (#30024) 2023-02-01 14:32:35 -05:00
Xiang Zhu 856598969c
Account path add run parent with old path cleanup (#29942)
* Add run parent directory for accounts files

* fix test test_concurrent_snapshot_packaging

* review comments.  renamed the path setup function

* Addressed most of the review comments

* remove explict type def for map result

* handle create_accounts_run_and_snapshot_dirs error with expect

* update with more review comments

* minor fixes from review comments

* simplify account_filename option assignment

* handle error from create_accounts_run_and_snapshot_dirs

* use then instead of then_some for lazy evaluation

* Clean up files in the old account_path before trasitioning to the new run path

* try_exist and accounts_dir removing extra

* sync rmdir, is_dir check

* handle the account_path not deletable case
2023-01-30 10:26:43 -08:00
Brooks d76c598691
Snapshot serde test uses calculate_accounts_delta_hash() (#29905) 2023-01-25 16:26:15 -05:00
Xiang Zhu 4ebcacb4a3
Revert "Add run parent directory for accounts files (#29794)" (#29899)
This PR is causing OOM on master.  Reverting it for now.

This reverts commit 74f89d1494.
2023-01-25 10:03:01 -08:00
Xiang Zhu 74f89d1494
Add run parent directory for accounts files (#29794)
* Add run parent directory for accounts files

* fix test test_concurrent_snapshot_packaging

* review comments.  renamed the path setup function

* Addressed most of the review comments

* remove explict type def for map result

* handle create_accounts_run_and_snapshot_dirs error with expect

* update with more review comments

* minor fixes from review comments

* simplify account_filename option assignment

* handle error from create_accounts_run_and_snapshot_dirs

* use then instead of then_some for lazy evaluation
2023-01-24 16:44:35 -08:00
Brooks db10073c36
Promotes accounts delta hash to its own type (#29768) 2023-01-20 14:23:56 -05:00
Jeff Washington (jwash) aef8692c8f
remove SnapshotStorageOne (#29747) 2023-01-18 07:10:56 -08:00
Jeff Washington (jwash) 461dafb887
AccountStorage only holds a single append vec (#29723)
* AccountStorageReference

* tests build

* pr feedback
2023-01-16 16:22:30 -08:00
Jeff Washington (jwash) 14055fc3e2
remove type SnapshotStorage and uses (#29661) 2023-01-11 17:57:44 -08:00
Jeff Washington (jwash) 544b9745c2
snapshot storage path uses 1 append vec per slot (#29627) 2023-01-11 12:05:15 -08:00
Jeff Washington (jwash) 4a64f6d421
test_utils::create_test_accounts pre-allocates an append vec first (#29336)
* test_utils::create_test_accounts pre-allocates an append vec first

* remove comment
2022-12-21 09:02:42 -08:00