Commit Graph

161 Commits

Author SHA1 Message Date
Jeff Washington (jwash) e396a10312
change StartBlockHeightAndRewards from parent to current (#31781) 2023-05-23 20:10:51 -07:00
Alexander Meißner 1e8e024ec5
Cleanup - disable_fee_calculator (#31766)
disable_fee_calculator
2023-05-23 18:20:38 +02:00
Yueh-Hsuan Chiang c79a6e74e2
Add new error type AccountsFileError (#31631)
#### Problem
AccountsFile- and AppendVec-related code uses io::Error for their main error type.  However, there're many errors under AccountsFile that don't belong to io::Error.

#### Summary of Changes
This PR introduces AccountsFileError and makes minimum changes to keep the PR small.
Subsequent changes related to this will be in separate PRs.
2023-05-22 14:09:09 -07:00
Jeff Washington (jwash) 3ac5bb2a60
deserialize epoch_reward_status into BankFieldsToDeserialize (#31698) 2023-05-18 07:14:59 -07:00
Jeff Washington (jwash) 0291454876
persist EpochRewardStatus::Active (#31697)
persist `EpochRewardStatus::Active`
2023-05-18 07:14:41 -07:00
Jeff Washington (jwash) 3f625e47df
working towards persistence for EpochRewardStatus (#31678) 2023-05-17 13:23:20 -07:00
Jeff Washington (jwash) 8ed4962229
refactor serialization tuple into an enum (#31679) 2023-05-17 10:53:28 -07:00
Jeff Washington (jwash) ce32cb43db
add bank.epoch_reward_status (#31680) 2023-05-17 10:01:14 -07:00
Jeff Washington (jwash) d841ff7fe1
support snapshot deserialization with EpochRewardStatus (#31672)
* support snapshot deserialization with EpochRewardStatus

* pub(crate)
2023-05-16 12:53:52 -07:00
Jeff Washington (jwash) 8d98589c85
remove unused historical_roots (#31539) 2023-05-08 10:50:27 -07:00
Brooks fdf7bdae1a
Fixes name of snapshot_utils::get_bank_snapshot_dir() (#31432) 2023-05-01 19:28:35 +00:00
Xiang Zhu 0a2e897f16
Clean up the outdated SnapshotPackage snapshot_links field (#31360)
* Remove snapshot_links

* Change the function name from snapshot_dir to bank_snapshot_dir

* Format fix

* Fix test_concurrent_snapshot_packaging

* Fix clippy error

* Fix nits

* Fix nits 2nd try

* Use get_bank_snapshots_dir

* Use slot_dir

* Revert "Use get_bank_snapshots_dir" because get_bank_snapshots_dir is private to crate

This reverts commit 1ed9b3b2c8e84689a918beee7159f63c56500a96.
2023-05-01 11:24:59 -07:00
Brennan 2164a50d00
Move BankIncrementalSnapshotPersistence (#31236)
* Move BankIncrementalSnapshotPersistence

* Update bank serialize ABI digest
2023-04-18 11:18:17 -07:00
steviez cc8e531a5d
Enforce a minimum of 1 on full and incremental snapshot retention (#30968) 2023-03-30 10:16:36 -05:00
Brooks e5edcb9c48
Sets IncrementalAccountsHash in serde_snapshot when reconstructing AccountsDb (#30801) 2023-03-21 10:51:04 -04:00
Brooks d3273ba118
Removes unused `ancestors` param from `get_snapshot_storages` (#30773) 2023-03-18 17:17:41 -04:00
Yueh-Hsuan Chiang fce949009d
Add new_from_file() API to AccountsFile (#30687)
#### Problem
Accounts db currently use AppendVec::new_from_file() directly
to create a new AcountsFile instance from an existing file.
However, this method should be abstracted out to AccountsFile
so that an existing file can be opened correctly using the right format.

#### Summary of Changes
Add new_from_file() API to AccountsFile which will open an existing
file based on its accounts file format.

Currently, it only supports AppendVec.
2023-03-15 20:38:20 -07:00
Brooks 263342a686
Adds accounts hashes from full & incremental snapshots at startup (#30656) 2023-03-13 14:47:25 -04:00
Brooks 5e5b7f00a2
Stores capitalization in account hashes map (#30635) 2023-03-13 10:50:45 -04:00
Brooks 64c7b401cd
BankIncrementalSnapshotPersistence uses strong accounts hash types (#30502) 2023-02-24 18:15:23 -05:00
Brooks ffe2d309b2
Adds serde_snapshot::types module (#30501) 2023-02-24 15:27:12 -05:00
Brooks ec288ded9b
Serde snapshot uses own types (#30391) 2023-02-20 12:25:15 -05:00
Brooks ad6a3abd8b
Uses set_accounts_{delta,}_hash_for_tests() in serde tests (#30390) 2023-02-17 16:00:25 -05:00
Yueh-Hsuan Chiang aeb6df3507
Abstract out AppendVec into AccountsFile enum (#29815)
Abstracts out AppendVec into AccountsFile enum.
This will allow different implementations of AccountsFile that support the AccountsDB.
2023-02-15 17:24:04 -08:00
Xiang Zhu 4909267c88
Add accounts hard-link files into the bank snapshot directory (#29496)
* Add accounts hard-link files into the bank snapshot directory

* Small adjustments and fixes.

* Address some of the review issues

* Fix compilation issues

* Change the latest slot snapshot storage from VecDeque to Option

* IoWithSourceAndFile and expanded comments on accounts

* last_slot_snapshot_storages in return value

* Update comments following the review input

* rename dir_accounts_hard_links to hard_link_path

* Add dir_full_state flag for add_bank_snapshot

* Let appendvec files hardlinking work with multiple accounts paths across multiple partitions

* Fixes for rebasing

* fix tests which generates account_path without adding run/

* rebasing fixes

* fix account path test failures

* fix test test_concurrent_snapshot_packaging

* review comments.  renamed the path setup function

* Addressed most of the review comments

* update with more review comments

* handle error from create_accounts_run_and_snapshot_dirs

* fix rebasing duplicate

* minor accounts_dir path cleanup

* minor cleanup, remove commented code

* misc review comments

* build error fix

* Fix test_incremental_snapshot_download_with_crossing_full_snapshot_interval_at_startup

* fix build error on MAX_BANK_SNAPSHOTS_TO_RETAIN

* rebase fix, update hardlink filename

* minor comment spelling fix

* rebasing fixes

* fix rebase issues; with_extension

* comments changes for review

* misc minor review issues

* bank.fill_bank_with_ticks_for_tests

* error handling on appendvec path

* fix use_jit

* minor comments refining

* Remove type AccountStorages

* get_account_path_from_appendvec_path return changed to Option

* removed appendvec_path.to_path_buf in create_accounts_run_and_snapshot_dirs

* add test_get_snapshot_accounts_hardlink_dir

* update last_snapshot_storages comment

* update last_snapshot_storages comment

* symlink map_err

* simplify test_get_snapshot_accounts_hardlink_dir with fake paths

* log last_snapshot_storages at the end of the loop
2023-02-15 09:52:07 -08:00
Brooks 06e801a8a5
Moves BankHashInfo into serde_snapshot (#30120) 2023-02-03 13:17:50 -05:00
Brooks 3249e4a123
Removes {get,set}_bank_hash_info_from_snapshot() (#30087) 2023-02-02 16:50:44 -05:00
Brooks d048a1903f
Splits up AccountsDb::bank_hashes (#30024) 2023-02-01 14:32:35 -05:00
Brooks 005dc96bc5
serde-snapshot uses methods to access AccountsDb::bank_hashes (#29989) 2023-01-30 21:32:04 -05:00
Xiang Zhu 856598969c
Account path add run parent with old path cleanup (#29942)
* Add run parent directory for accounts files

* fix test test_concurrent_snapshot_packaging

* review comments.  renamed the path setup function

* Addressed most of the review comments

* remove explict type def for map result

* handle create_accounts_run_and_snapshot_dirs error with expect

* update with more review comments

* minor fixes from review comments

* simplify account_filename option assignment

* handle error from create_accounts_run_and_snapshot_dirs

* use then instead of then_some for lazy evaluation

* Clean up files in the old account_path before trasitioning to the new run path

* try_exist and accounts_dir removing extra

* sync rmdir, is_dir check

* handle the account_path not deletable case
2023-01-30 10:26:43 -08:00
Brooks d76c598691
Snapshot serde test uses calculate_accounts_delta_hash() (#29905) 2023-01-25 16:26:15 -05:00
Xiang Zhu 4ebcacb4a3
Revert "Add run parent directory for accounts files (#29794)" (#29899)
This PR is causing OOM on master.  Reverting it for now.

This reverts commit 74f89d1494.
2023-01-25 10:03:01 -08:00
Xiang Zhu 74f89d1494
Add run parent directory for accounts files (#29794)
* Add run parent directory for accounts files

* fix test test_concurrent_snapshot_packaging

* review comments.  renamed the path setup function

* Addressed most of the review comments

* remove explict type def for map result

* handle create_accounts_run_and_snapshot_dirs error with expect

* update with more review comments

* minor fixes from review comments

* simplify account_filename option assignment

* handle error from create_accounts_run_and_snapshot_dirs

* use then instead of then_some for lazy evaluation
2023-01-24 16:44:35 -08:00
Brooks db10073c36
Promotes accounts delta hash to its own type (#29768) 2023-01-20 14:23:56 -05:00
Jeff Washington (jwash) aef8692c8f
remove SnapshotStorageOne (#29747) 2023-01-18 07:10:56 -08:00
Jeff Washington (jwash) 461dafb887
AccountStorage only holds a single append vec (#29723)
* AccountStorageReference

* tests build

* pr feedback
2023-01-16 16:22:30 -08:00
Jeff Washington (jwash) 14055fc3e2
remove type SnapshotStorage and uses (#29661) 2023-01-11 17:57:44 -08:00
Jeff Washington (jwash) 544b9745c2
snapshot storage path uses 1 append vec per slot (#29627) 2023-01-11 12:05:15 -08:00
Jeff Washington (jwash) 4a64f6d421
test_utils::create_test_accounts pre-allocates an append vec first (#29336)
* test_utils::create_test_accounts pre-allocates an append vec first

* remove comment
2022-12-21 09:02:42 -08:00
Jeff Washington (jwash) 49a136a374
move AccountStorage out of accounts_db.rs (#29261) 2022-12-14 15:08:39 -08:00
Jeff Washington (jwash) f355066456
remove caching_enabled from serde_snapshot (#29246)
* fix tests broken by add_root change

* remove caching_enabled from serde_snapshot
2022-12-13 16:32:43 -08:00
Jeff Washington (jwash) c8e7c706df
remove bank_from_streams(caching_enabled) (#29235) 2022-12-13 07:49:33 -08:00
Jeff Washington (jwash) db2cc53967
remove skip rewrites from bank (#29182) 2022-12-10 09:27:15 -08:00
Jeff Washington (jwash) 45ba5ef6fd
remove bank_from_snapshot_archives caching_enabled (#29171) 2022-12-09 10:45:21 -08:00
Jeff Washington (jwash) 99f0113450
remove Accounts::new_with_config_for_tests.caching_enabled (#29169) 2022-12-09 10:44:56 -08:00
Jeff Washington (jwash) ec5098a723
remove bank_test_config_caching_enabled (#29170) 2022-12-09 08:28:02 -08:00
Brooks Prumo c65a8ce6c3
AccountsDb::get_snapshot_storages() takes a Range for slots (#29054) 2022-12-05 11:21:21 -05:00
Jeff Washington (jwash) 631d93dd2d
tests use write cache (#29012) 2022-12-01 12:17:13 -08:00
Brooks Prumo 9327658007
Promotes accounts hash to a strong type (#28930) 2022-11-28 10:09:47 -05:00
Brooks Prumo 04016e3bcf
Don't wait for EAH unless feature is enabled (#28938) 2022-11-23 09:11:28 -05:00