Commit Graph

280 Commits

Author SHA1 Message Date
Brooks 644cdaaa48
Adds comments in BankSnapshotInfo::new_from_dir() when selecting Pre or Post (#32261) 2023-06-26 11:53:25 -04:00
Brooks 4fb6bbf92e
Refactors datapoints and logs in bank_from_xxx fns (#32255) 2023-06-26 11:33:58 -04:00
Andrew Fitzgerald 4a566ec938
create and canonicalize account_paths (#32037)
Co-authored-by: Brooks <brooks@prumo.org>
2023-06-09 10:04:25 -07:00
Brooks afbb0fa92f
Removes init-time cleanup in add_bank_snapshot() (#31997) 2023-06-09 08:30:35 -04:00
Andrew Fitzgerald 3ba05d9c3d
Fix move_and_async_delete_path (#32020) 2023-06-08 10:55:52 -07:00
steviez 4b2db0d546
Add explaining comment for Err case in get_bank_snapshots() (#31998) 2023-06-08 12:54:24 -05:00
Brooks ddbc35fcc3
Add logging to move_and_async_delete_path (#32017) 2023-06-07 16:31:16 -04:00
Brooks f08fb3e158
Downgrades log to debug if reading bank snapshot from dir fails (#31990) 2023-06-06 15:07:45 -04:00
steviez 858ea5a143
chore: Add name string for bank_serialize measure! (#31926)
The measurement is later logged, so include a name to more clearly
identify what the measurement was.
2023-06-02 06:54:04 -05:00
steviez debe794987
Replace improper &Arc<...> with Arc<...> in Bank and Accounts (#31892)
The callstack updated in this PR passed an &Arc<...> down only to have
the bottom level clone the reference. Thus, we are giving shared
ownership so the reference is a bit redundant and arguably obscures the
intention to clone further down the callstack.
2023-05-31 12:36:44 -05:00
Brooks 8e191bbc56
Downgrades log to warn if reading bank snapshot from dir fails (#31885) 2023-05-30 19:35:06 -04:00
Brooks 8a3f446db0
Removes MAX_BANK_SNAPSHOTS_TO_RETAIN constant (#31827) 2023-05-25 17:13:28 -04:00
Andrew Fitzgerald f52ded35f4
async delete contents but leave directory (#31737)
* async delete contents but leave directory

* Clarified comment
2023-05-23 15:33:09 -07:00
Brooks 61afb07dbd
Purges old bank snapshots at startup (#31656)
Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>
2023-05-23 13:18:05 -04:00
Yueh-Hsuan Chiang c79a6e74e2
Add new error type AccountsFileError (#31631)
#### Problem
AccountsFile- and AppendVec-related code uses io::Error for their main error type.  However, there're many errors under AccountsFile that don't belong to io::Error.

#### Summary of Changes
This PR introduces AccountsFileError and makes minimum changes to keep the PR small.
Subsequent changes related to this will be in separate PRs.
2023-05-22 14:09:09 -07:00
Alexander Meißner 520c647918
Refactor - Merges `BuiltinPrograms` into `LoadedPrograms` (#31654)
* Merges BuiltinPrograms into LoadedPrograms.

* Prevents built-ins from being pruned.
2023-05-18 22:18:28 +02:00
Brooks 8e5e66fdb4
Revert "Revert "SnapshotPackagerService purges old bank snapshots (#31511)" (#31524)" (#31558)
This reverts commit 775639c058.
2023-05-12 15:39:14 -04:00
Brooks 7dbcf5a0fd
Do not remove bank snapshots in new_from_dir() (#31556) 2023-05-12 13:19:07 -04:00
Jeff Washington (jwash) 122b05b9f5
pass include_slot_in_hash through hash calcs to allow rehashing if hash is not stored (#31579)
* pass include_slot_in_hash through hash calcs to allow rehashing

* tests use each include_slot_in_hash value

* move include_slot_in_hash

* typo

* reorder struct init

* spelling is hard
2023-05-11 13:23:29 -07:00
Brooks 3bb2e3b546
Purges incomplete snapshot dirs at startup (#31555) 2023-05-09 14:08:12 -04:00
Brooks 775639c058
Revert "SnapshotPackagerService purges old bank snapshots (#31511)" (#31524)
This reverts commit a6c39ded8e.
2023-05-06 09:18:03 -04:00
Brooks a6c39ded8e
SnapshotPackagerService purges old bank snapshots (#31511) 2023-05-05 17:22:48 -04:00
Brooks 39a092c3dc
Adds snapshot_utils::purge_bank_snapshot() (#31500)
Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>
2023-05-05 14:01:08 -04:00
Brooks c7d922f229
Uses SNAPSHOT_VERSION_FILENAME constant (#31508) 2023-05-05 14:00:08 -04:00
Brooks 1d861ad558
Removes solana_logger from snapshot_utils unit tests (#31510) 2023-05-05 12:52:36 -04:00
Brooks 10b8b32a31
Uses SNAPSHOT_ARCHIVE_DOWNLOAD_DIR constant (#31507) 2023-05-05 12:36:32 -04:00
Brooks ab6094a94f
Uses SNAPSHOT_ACCOUNTS_HARDLINKS constant (#31499) 2023-05-05 12:36:21 -04:00
Brooks 4196dbcbf4
Replaces snapshot_utils::remove_bank_snapshot() (#31498) 2023-05-05 07:43:28 -04:00
Brooks 0e0c91faac
Adds snapshot_utils::purge_bank_snapshot() (#31497) 2023-05-04 19:19:26 -04:00
Brooks fdf7bdae1a
Fixes name of snapshot_utils::get_bank_snapshot_dir() (#31432) 2023-05-01 19:28:35 +00:00
Xiang Zhu 0a2e897f16
Clean up the outdated SnapshotPackage snapshot_links field (#31360)
* Remove snapshot_links

* Change the function name from snapshot_dir to bank_snapshot_dir

* Format fix

* Fix test_concurrent_snapshot_packaging

* Fix clippy error

* Fix nits

* Fix nits 2nd try

* Use get_bank_snapshots_dir

* Use slot_dir

* Revert "Use get_bank_snapshots_dir" because get_bank_snapshots_dir is private to crate

This reverts commit 1ed9b3b2c8e84689a918beee7159f63c56500a96.
2023-05-01 11:24:59 -07:00
steviez 1a995ade3f
Combine multiple snapshot_utils metrics (#31406)
Within the contents of snapshot_utils::add_bank_snapshot(), metrics were
being reported in several datapoints and counters. This adds extra
overhead, and makes it harder to correlate fields that correspond to the
same snapshot.

So, combine the counters and multiple datapoints into a single
datapoint.
2023-04-30 16:36:08 -04:00
Xiang Zhu bb95d44328
Check incomplete file flag first (#31409)
* Check incomplete file flag first

* Use the simple is_file call, no need for metadata
2023-04-28 15:44:59 -07:00
Brooks d621b48026
Cleanup usage of AsRef<Path> in snapshot_utils (#31393) 2023-04-28 14:46:57 +00:00
Brooks 6393c7d98d
Removes erroneous Copy on AsRef<Path> params in snapshot_utils (#31382) 2023-04-27 22:45:59 -04:00
Xiang Zhu a52f44c342
Use async delete to save time and gracefully handle missing dir (#31380) 2023-04-27 18:13:34 -07:00
Xiang Zhu f3e94ca73c
AHV processes the snapshot dirs in place (#30978)
* AHV processes the snapshot dirs in place

Let account pacakge use the snapshot dir, so AHV computes the accounts hash and turns the pre snapshot dir into a post snapshot dir

* fix status cache path to maintain the archive layout for the in-place snapshot dir archiving

* fix test_package_snapshots

* Fix test_concurrent_snapshot_packaging

* Remove debug change.

* Fix snapshot_links path

* change to borrow for bank_snapshots_dir

* Reverted changes in create_and_verify_snapshot

* Fix param errors

* Fix rebase errors

* Remove NOTE 1

* Remove unwrap

* Remove the variables to make it apparent taht snapshot_links is the bank_snapshots_dir

* Use soft link instead of hard link for snapshot and status cache

* After switching to soft symlinking, the src path should be absolute
2023-04-26 11:48:48 -07:00
Brooks 18aa54a4b6
Removes bank_snapshots_dir param from bank_fields_from_snapshot_archives() (#31349) 2023-04-26 11:45:00 -04:00
Xiang Zhu 16f3dcd5d2
Update fn create_and_verify_snapshot (#31245)
* only 1 snapshot per archive in create_and_verify_snapshot

* Update create_and_verify_snapshot with the newer funtion calls

* Fix test_package_snapshots

* Remove account path access change

* Rename slot to num_snapshots

* Remove unncessary purge_old_bank_snapshots in test

* Update non-deterministic format comment

* Cleanup unnecessary hash calls

* Use get_accounts_hash

* Remove extra declaration

* Remove rehash

* Remove clean_accounts

* Revert "Cleanup unnecessary hash calls"

This reverts commit 06b1457462cf6d7acf62e0e5531633caf5d9fc58.

Removing unncessary hash calls should be done for create_and_verify_snapshot,
not bank_to_full_snapshot_archive

* Fix typo appenedvecs

* Remove bank_snapshots_dir after rebasing
2023-04-24 18:52:50 -07:00
Xiang Zhu a5275f8839
Remove bank_snapshots_dir param (#31249)
* Remove bank_snapshots_dir param

* Remove outdated comment

* Revert "Remove outdated comment"

This reverts commit e4441432bec57edb0dc22c4bacf4d48ce26ed818.

* Handle parent() error

* Fix format error
2023-04-19 09:37:46 -07:00
Brennan 2164a50d00
Move BankIncrementalSnapshotPersistence (#31236)
* Move BankIncrementalSnapshotPersistence

* Update bank serialize ABI digest
2023-04-18 11:18:17 -07:00
Xiang Zhu e74bc4e2e7
Add a filter_by_type param for purge_old_bank_snapshots (#31191)
* Add a type_select param for purge_old_bank_snapshots

* Use flags to make the function calls more readable

* Remove the extra purge calls

* replace select_type with filter_by_type

* Add test

* Use matches

* Fix CI test on reference

* use match and call do_purge once

* Let bank_snapshots_dir be TempDir

* remove account_paths in the test

* replace bank with _bank

* Remove create_snapshot_dirs_for_tests, will take the lastest from master

* Fix merge errors
2023-04-17 16:16:41 -07:00
Xiang Zhu 667fad6da7
Refactor the common test snapshot dirs creation code (#31223)
* Create fn create_snapshot_dirs_for_tests to refactor the common test snapshot dirs creation code

* Fix trailing space

* Fix missed set_accounts_delta_hash
2023-04-17 14:14:01 -07:00
Xiang Zhu f066ea9f71
Define bank_snapshots_dir as tmpdir (#31216)
* Let bank_snapshots_dir be TempDir

* Same change in test_bank_from_latest_snapshot_dir

* Use borrow to avoid releasing the tmp dir

* One more to change to reference
2023-04-17 08:07:24 -07:00
Xiang Zhu 9f53c42339
bank_from_latest_snapshot_dir (#31115)
* bank_from_latest_snapshot_dir

* Clear the account paths before constructing a bank from a snapshot dir

* Clear the account paths before constructing a bank from a snapshot dir (#31170)

* use the highest POST snapshot dir to construct a bank

* Use POST to fix test_bank_from_latest_snapshot_dir

* Revert "Clear the account paths before constructing a bank from a snapshot dir"

This reverts commit 272bb9fee14a09d7fde3cddb3d439f2cdadb5493.

* Remove the set_accounts_hash_for_tests call

* Revert "Revert "Clear the account paths before constructing a bank from a snapshot dir""

This reverts commit 2e345d8b98b8fb3b07f00ce050132c678acf88d4.
2023-04-13 12:44:56 -07:00
Xiang Zhu 78d061a3d3
Clear the account paths before constructing a bank from a snapshot dir (#31170) 2023-04-12 16:03:19 -07:00
Xiang Zhu 31784b2ecc
Add remove_incomplete_bank_snapshot_dir (#31131)
* Add remove_incomplete_bank_snapshot_dir

* Remove incomplete snapshot dir if incomplete

* Revert "Add remove_incomplete_bank_snapshot_dir"

This reverts commit aaafef74803ca4e2050b8cd83ef352c5a59509b4.

* Replace metadata's is_file with simple PathBuf's is_file

* In test verify the deletion of the incomplete snapshot dir

* Add comments to explain the snapshot dir cleanup
2023-04-11 17:56:26 -07:00
Xiang Zhu fac76715e9
bank_from_snapshot_dir returns BankFromDirTimings (#31051) 2023-04-04 17:10:56 -07:00
steviez cc8e531a5d
Enforce a minimum of 1 on full and incremental snapshot retention (#30968) 2023-03-30 10:16:36 -05:00
Jeff Washington (jwash) d542496d10
fix typo in info log (#30964)
fix typo
2023-03-29 13:23:06 -07:00