Commit Graph

137 Commits

Author SHA1 Message Date
Brooks 1689586213
Uses a channel for AHV -> SPS (#30406) 2023-02-22 03:36:29 +00:00
Brooks 35328ca63d
Makes AccountsHash an enum (#30416) 2023-02-21 15:20:51 -05:00
Brooks bcc4bc80c9
Removes unnecessary derives from Accounts{Delta}Hash (#30392) 2023-02-20 16:00:53 -05:00
Xiang Zhu 4909267c88
Add accounts hard-link files into the bank snapshot directory (#29496)
* Add accounts hard-link files into the bank snapshot directory

* Small adjustments and fixes.

* Address some of the review issues

* Fix compilation issues

* Change the latest slot snapshot storage from VecDeque to Option

* IoWithSourceAndFile and expanded comments on accounts

* last_slot_snapshot_storages in return value

* Update comments following the review input

* rename dir_accounts_hard_links to hard_link_path

* Add dir_full_state flag for add_bank_snapshot

* Let appendvec files hardlinking work with multiple accounts paths across multiple partitions

* Fixes for rebasing

* fix tests which generates account_path without adding run/

* rebasing fixes

* fix account path test failures

* fix test test_concurrent_snapshot_packaging

* review comments.  renamed the path setup function

* Addressed most of the review comments

* update with more review comments

* handle error from create_accounts_run_and_snapshot_dirs

* fix rebasing duplicate

* minor accounts_dir path cleanup

* minor cleanup, remove commented code

* misc review comments

* build error fix

* Fix test_incremental_snapshot_download_with_crossing_full_snapshot_interval_at_startup

* fix build error on MAX_BANK_SNAPSHOTS_TO_RETAIN

* rebase fix, update hardlink filename

* minor comment spelling fix

* rebasing fixes

* fix rebase issues; with_extension

* comments changes for review

* misc minor review issues

* bank.fill_bank_with_ticks_for_tests

* error handling on appendvec path

* fix use_jit

* minor comments refining

* Remove type AccountStorages

* get_account_path_from_appendvec_path return changed to Option

* removed appendvec_path.to_path_buf in create_accounts_run_and_snapshot_dirs

* add test_get_snapshot_accounts_hardlink_dir

* update last_snapshot_storages comment

* update last_snapshot_storages comment

* symlink map_err

* simplify test_get_snapshot_accounts_hardlink_dir with fake paths

* log last_snapshot_storages at the end of the loop
2023-02-15 09:52:07 -08:00
behzad nouri ded457cd73
embeds the new gossip ContactInfo in ClusterInfo (#30022)
Working towards replacing the legacy gossip contact-info with the new
one, the commit updates the respective field in gossip cluster-info.
2023-02-10 20:07:45 +00:00
Xiang Zhu f107b8b607
Add slot deltas into the bank snapshot directory (#29409) 2023-02-01 16:51:32 -08:00
Brooks d048a1903f
Splits up AccountsDb::bank_hashes (#30024) 2023-02-01 14:32:35 -05:00
Xiang Zhu 856598969c
Account path add run parent with old path cleanup (#29942)
* Add run parent directory for accounts files

* fix test test_concurrent_snapshot_packaging

* review comments.  renamed the path setup function

* Addressed most of the review comments

* remove explict type def for map result

* handle create_accounts_run_and_snapshot_dirs error with expect

* update with more review comments

* minor fixes from review comments

* simplify account_filename option assignment

* handle error from create_accounts_run_and_snapshot_dirs

* use then instead of then_some for lazy evaluation

* Clean up files in the old account_path before trasitioning to the new run path

* try_exist and accounts_dir removing extra

* sync rmdir, is_dir check

* handle the account_path not deletable case
2023-01-30 10:26:43 -08:00
Xiang Zhu 4ebcacb4a3
Revert "Add run parent directory for accounts files (#29794)" (#29899)
This PR is causing OOM on master.  Reverting it for now.

This reverts commit 74f89d1494.
2023-01-25 10:03:01 -08:00
Xiang Zhu 74f89d1494
Add run parent directory for accounts files (#29794)
* Add run parent directory for accounts files

* fix test test_concurrent_snapshot_packaging

* review comments.  renamed the path setup function

* Addressed most of the review comments

* remove explict type def for map result

* handle create_accounts_run_and_snapshot_dirs error with expect

* update with more review comments

* minor fixes from review comments

* simplify account_filename option assignment

* handle error from create_accounts_run_and_snapshot_dirs

* use then instead of then_some for lazy evaluation
2023-01-24 16:44:35 -08:00
behzad nouri 1c7662a37f
asserts that cluster-info keypair is consistent with contact-info id (#29818) 2023-01-24 16:57:55 +00:00
Jeff Washington (jwash) 544b9745c2
snapshot storage path uses 1 append vec per slot (#29627) 2023-01-11 12:05:15 -08:00
behzad nouri 8c212f59ad
renames ContactInfo to LegacyContactInfo (#29566)
Working towards adding a new ContactInfo where new sockets can be
added in a backward compatible way.
2023-01-08 16:00:55 +00:00
Jeff Washington (jwash) b95835143e
remove AccountsBackgroundService::new(caching_enabled) (#29234) 2022-12-13 07:18:02 -08:00
Jeff Washington (jwash) bb0bfc4214
remove bank_from_latest_snapshot_archives(caching_enabled) (#29238) 2022-12-13 07:16:24 -08:00
Jeff Washington (jwash) 2c2324f4ea
remove caching_enabled from Bank::new_with_paths_for_tests (#29214) 2022-12-12 15:30:46 -08:00
apfitzge 249607dbfe
Use a different tempdir for unpacking snapshots (#29219) 2022-12-12 17:26:52 -06:00
Jeff Washington (jwash) 6a90abd056
remove handle_snapshot_requests.caching_enabled (#29174) 2022-12-09 10:51:44 -08:00
Jeff Washington (jwash) 45ba5ef6fd
remove bank_from_snapshot_archives caching_enabled (#29171) 2022-12-09 10:45:21 -08:00
steviez aeb6b53502
Remove unused Option<> around ValidatorConfig's SnapshotConfig (#29090)
Remove Option<> around ValidatorConfig's SnapshotConfig

The SnapshotConfig is required and is currently hard-coded to be a
Some().
2022-12-06 22:47:55 +00:00
behzad nouri 9524c9dbff patches errors from clippy::uninlined_format_args
https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
2022-12-06 19:32:15 +00:00
Brooks Prumo 9327658007
Promotes accounts hash to a strong type (#28930) 2022-11-28 10:09:47 -05:00
Jeff Washington (jwash) f22104d46b
use write cache by default in some tests (#28876) 2022-11-18 14:35:52 -08:00
Brooks Prumo 2bafb0cb12
Requires EAH state cannot be Invalid (#28817) 2022-11-17 11:01:01 -05:00
Brooks Prumo ae0bb44401
Fixes test_snapshots_with_background_services (#28848) 2022-11-17 00:49:28 -05:00
Brooks Prumo d798e751a0
Disables EAH with short epochs (#28803) 2022-11-15 13:26:19 -05:00
Brooks Prumo 0bfea02056
Snapshots wait for EAH calculations to complete (#28777) 2022-11-14 11:34:44 -06:00
Brooks Prumo d1ba42180d
clippy for rust 1.65.0 (#28765) 2022-11-09 19:39:38 +00:00
Brooks Prumo 9e1cdc7e60
Enables not taking a bank snapshot (#28756) 2022-11-09 12:43:33 -05:00
Brooks Prumo d4c2900590
Removes `snapshot_bank()` wrapper fn (#28753) 2022-11-07 15:09:31 +00:00
Brooks Prumo 59bf1809fe
Uses SnapshotHash type in snapshot archive fields (#28681) 2022-10-31 14:28:35 -04:00
Brooks Prumo 37507a2de6
Removes EAH parameter from serde_snapshot::reserialize_bank() (#28669) 2022-10-31 09:43:17 -04:00
Brooks Prumo 5a3d252899
Renames fn to Bank::update_accounts_hash_for_tests() (#28620) 2022-10-28 14:33:05 -04:00
Brooks Prumo 12f3e8c9cc
Ignores errors when joining background threads in snapshot tests (#28480) 2022-10-19 16:54:59 -04:00
Brooks Prumo dd7fee8f32
Re-enqueues unhandled ABS requests (#28362) 2022-10-13 16:25:39 -04:00
Brooks Prumo 9cbd00fdbc
Converts PendingAccountsPackage to a channel (#28352) 2022-10-13 12:47:36 -04:00
Brooks Prumo a8c6a9e5fc
Bank::freeze() waits for EAH calculation to complete (#28170) 2022-10-05 17:44:35 -04:00
Brooks Prumo 2f8f6c6a31
Send Epoch Accounts Hash requests from set_root() (#27764) 2022-09-30 14:59:41 -04:00
behzad nouri 9a57c64f21
patches clippy errors from new rust nightly release (#27996) 2022-09-22 22:23:03 +00:00
Brooks Prumo ff71df4695
Remove unnecessary call to `set_startup_verification_complete()` (#27986) 2022-09-22 16:54:17 -04:00
Brooks Prumo 1ee595ca9c
remove AccountsDb::initial_blockstore_processing_complete (#27974) 2022-09-22 13:52:04 -04:00
Jeff Washington (jwash) f2d6a7ecea
bank.initial_blockstore_processing_complete to avoid concurrent hash calculations (#27776)
* bank.initial_blockstore_processing_complete to avoid concurrent hash calculations

* Update runtime/src/bank.rs

Co-authored-by: Brooks Prumo <brooks@prumo.org>

* Update runtime/src/bank.rs

Co-authored-by: Brooks Prumo <brooks@prumo.org>

* Rename TestValidator::set_startup_verification_complete()

* Initialize with `AtomicBool::new(false)` instead of `default()`

* snapshot tests: move where `initial_blockstore_processing_completed()` is called

* fixup bank_forks.rs calling `is_initial_blockstore_processing_complete()`

* only call initial_blockstore_processing_completed() in blockstore_processor

Co-authored-by: Brooks Prumo <brooks@prumo.org>
Co-authored-by: Brooks Prumo <brooks@solana.com>
2022-09-19 13:00:21 -07:00
Jeff Washington (jwash) 765c628546
use exit signal for acct idx bg threads (#27483) 2022-09-12 11:51:12 -07:00
Jeff Washington (jwash) abd01553d5
tests: Keypair::new().pubkey() -> pubkey::new_rand (#27705) 2022-09-10 13:56:45 -07:00
Brooks Prumo 6a322de845
Make Accounts Background Services aware of Epoch Accounts Hash (#27626) 2022-09-07 20:41:40 +00:00
Jeff Washington (jwash) a31d4a597d
serialize epoch_accounts_hash (#27516) 2022-09-07 10:07:00 -07:00
Brooks Prumo 93a4f80a2c
Handling snapshot requests is now required (#27537) 2022-09-07 10:08:42 -04:00
Brooks Prumo 3c7cd62030
Move pruned_banks_receiver into PrunedBanksRequestHandler (#27445) 2022-08-29 13:30:06 -04:00
Jeff Washington (jwash) 225cddcffb
serialize incremental_snapshot_hash (#26839)
* serialize incremental_snapshot_hash

* pr feedback
2022-08-17 15:14:31 -05:00
Justin Starry 69598ed4c0
Refactor: Add `RuntimeConfig` field to Bank (#26946)
* Refactor: Simplify arguments for bank constructor methods

* Refactor: Add RuntimeConfig to Bank fields

* Arc wrap runtime_config

* Arc wrap all runtime config usages

* Remove Copy trait derivation from RuntimeConfig

* Remove some arc wrapping
2022-08-05 20:49:00 +01:00