Brooks
0db14ad39c
Removes full_snapshot from CalcAccountsHashConfig ( #29722 )
2023-01-16 16:22:46 -05:00
Jeff Washington (jwash)
544b9745c2
snapshot storage path uses 1 append vec per slot ( #29627 )
2023-01-11 12:05:15 -08:00
behzad nouri
8c212f59ad
renames ContactInfo to LegacyContactInfo ( #29566 )
...
Working towards adding a new ContactInfo where new sockets can be
added in a backward compatible way.
2023-01-08 16:00:55 +00:00
Jeff Washington (jwash)
b95835143e
remove AccountsBackgroundService::new(caching_enabled) ( #29234 )
2022-12-13 07:18:02 -08:00
Jeff Washington (jwash)
bb0bfc4214
remove bank_from_latest_snapshot_archives(caching_enabled) ( #29238 )
2022-12-13 07:16:24 -08:00
Jeff Washington (jwash)
2c2324f4ea
remove caching_enabled from Bank::new_with_paths_for_tests ( #29214 )
2022-12-12 15:30:46 -08:00
Brooks Prumo
1b0aaf1607
Makes a new PathBuf instead of moving the test's TempDir ( #29220 )
2022-12-12 18:29:36 -05:00
apfitzge
249607dbfe
Use a different tempdir for unpacking snapshots ( #29219 )
2022-12-12 17:26:52 -06:00
Brooks Prumo
391f68da61
Uses Storages to calculate accounts hash in EAH warp tests ( #29192 )
2022-12-12 13:30:23 -05:00
Jeff Washington (jwash)
631a98a3b6
warp_from_parents works with write_cache enabled ( #29185 )
2022-12-09 14:28:18 -08:00
Jeff Washington (jwash)
6a90abd056
remove handle_snapshot_requests.caching_enabled ( #29174 )
2022-12-09 10:51:44 -08:00
Jeff Washington (jwash)
45ba5ef6fd
remove bank_from_snapshot_archives caching_enabled ( #29171 )
2022-12-09 10:45:21 -08:00
Jeff Washington (jwash)
ec5098a723
remove bank_test_config_caching_enabled ( #29170 )
2022-12-09 08:28:02 -08:00
steviez
aeb6b53502
Remove unused Option<> around ValidatorConfig's SnapshotConfig ( #29090 )
...
Remove Option<> around ValidatorConfig's SnapshotConfig
The SnapshotConfig is required and is currently hard-coded to be a
Some().
2022-12-06 22:47:55 +00:00
behzad nouri
9524c9dbff
patches errors from clippy::uninlined_format_args
...
https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
2022-12-06 19:32:15 +00:00
steviez
3c42c87098
Remove obsoleted return value from Blockstore insert shred method ( #28992 )
2022-12-01 11:17:46 -06:00
Brooks Prumo
9327658007
Promotes accounts hash to a strong type ( #28930 )
2022-11-28 10:09:47 -05:00
Brooks Prumo
638b26ea65
Renames EAH test fn ( #28939 )
2022-11-23 05:18:50 +00:00
Jeff Washington (jwash)
f22104d46b
use write cache by default in some tests ( #28876 )
2022-11-18 14:35:52 -08:00
Brooks Prumo
2bafb0cb12
Requires EAH state cannot be Invalid ( #28817 )
2022-11-17 11:01:01 -05:00
Brooks Prumo
ae0bb44401
Fixes test_snapshots_with_background_services ( #28848 )
2022-11-17 00:49:28 -05:00
Brooks Prumo
d798e751a0
Disables EAH with short epochs ( #28803 )
2022-11-15 13:26:19 -05:00
Brooks Prumo
d4cf18421d
Use 400 slots-per-epoch in EAH tests ( #28801 )
2022-11-14 17:49:20 -05:00
Brooks Prumo
0bfea02056
Snapshots wait for EAH calculations to complete ( #28777 )
2022-11-14 11:34:44 -06:00
Brooks Prumo
d1ba42180d
clippy for rust 1.65.0 ( #28765 )
2022-11-09 19:39:38 +00:00
Brooks Prumo
9e1cdc7e60
Enables not taking a bank snapshot ( #28756 )
2022-11-09 12:43:33 -05:00
Brooks Prumo
d4c2900590
Removes `snapshot_bank()` wrapper fn ( #28753 )
2022-11-07 15:09:31 +00:00
Brooks Prumo
59bf1809fe
Uses SnapshotHash type in snapshot archive fields ( #28681 )
2022-10-31 14:28:35 -04:00
Brooks Prumo
37507a2de6
Removes EAH parameter from serde_snapshot::reserialize_bank() ( #28669 )
2022-10-31 09:43:17 -04:00
Brooks Prumo
5a3d252899
Renames fn to Bank::update_accounts_hash_for_tests() ( #28620 )
2022-10-28 14:33:05 -04:00
steviez
2272fd807e
Remove Blockstore manual compaction code ( #28409 )
...
The manual Blockstore compaction that was being initiated from
LedgerCleanupService has been disabled for quite some time in favor of
several optimizations.
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2022-10-28 10:39:00 +02:00
Brooks Prumo
2354a0a343
Renames fn to calculate_accounts_hash_from_index() ( #28568 )
2022-10-24 19:20:08 -04:00
Brooks Prumo
12f3e8c9cc
Ignores errors when joining background threads in snapshot tests ( #28480 )
2022-10-19 16:54:59 -04:00
HaoranYi
d81d2bba59
comments out print in test ( #28475 )
2022-10-19 10:25:11 -05:00
Brooks Prumo
1cc9cf927c
Supports warping with Epoch Accounts Hash ( #28459 )
2022-10-19 10:37:14 -04:00
Jeff Washington (jwash)
28a89a1d99
remove expected rent collection and rehashing completely ( #28422 )
2022-10-17 07:24:42 -07:00
Brooks Prumo
31c2b29941
Sends both an EAH and a snapshot request from `set_root()` ( #28363 )
2022-10-14 11:00:04 -04:00
Brooks Prumo
dd7fee8f32
Re-enqueues unhandled ABS requests ( #28362 )
2022-10-13 16:25:39 -04:00
Brooks Prumo
9cbd00fdbc
Converts PendingAccountsPackage to a channel ( #28352 )
2022-10-13 12:47:36 -04:00
Brooks Prumo
5a08eed82d
Cleans up debugging code in EAH tests ( #28324 )
2022-10-10 16:07:55 +00:00
Brooks Prumo
27cd2c324e
Adds tests for EAH and snapshot interactions ( #28304 )
2022-10-10 10:16:13 -04:00
Brooks Prumo
981c9d07a4
Rearranges eah TestEnvironment fields to ensure drop order ( #28270 )
2022-10-06 16:17:32 -04:00
Brooks Prumo
2d936784dd
Ignore errors when joining background threads for EAH tests ( #28263 )
2022-10-06 18:43:56 +00:00
Brooks Prumo
a8c6a9e5fc
Bank::freeze() waits for EAH calculation to complete ( #28170 )
2022-10-05 17:44:35 -04:00
Yueh-Hsuan Chiang
6b17bee5a8
Remove the const default for RocksFifo ( #27965 )
...
#### Summary of Changes
Removes the constant default for ShredStorageType::RocksFifo
as the shred storage size is either user-specified or derived
from --limit-ledger-size in #27459 .
2022-10-01 15:10:54 -07:00
Brooks Prumo
8877ac2aa9
Fix call to calculate_accounts_hash() ( #28169 )
2022-09-30 15:29:18 -04:00
Brooks Prumo
2f8f6c6a31
Send Epoch Accounts Hash requests from set_root() ( #27764 )
2022-09-30 14:59:41 -04:00
behzad nouri
97c9af4c6b
plumbs through flag to generate merkle variant of shreds
2022-09-23 16:45:18 +00:00
behzad nouri
9a57c64f21
patches clippy errors from new rust nightly release ( #27996 )
2022-09-22 22:23:03 +00:00
Brooks Prumo
ff71df4695
Remove unnecessary call to `set_startup_verification_complete()` ( #27986 )
2022-09-22 16:54:17 -04:00
Brooks Prumo
1ee595ca9c
remove AccountsDb::initial_blockstore_processing_complete ( #27974 )
2022-09-22 13:52:04 -04:00
Jeff Washington (jwash)
f2d6a7ecea
bank.initial_blockstore_processing_complete to avoid concurrent hash calculations ( #27776 )
...
* bank.initial_blockstore_processing_complete to avoid concurrent hash calculations
* Update runtime/src/bank.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/bank.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Rename TestValidator::set_startup_verification_complete()
* Initialize with `AtomicBool::new(false)` instead of `default()`
* snapshot tests: move where `initial_blockstore_processing_completed()` is called
* fixup bank_forks.rs calling `is_initial_blockstore_processing_complete()`
* only call initial_blockstore_processing_completed() in blockstore_processor
Co-authored-by: Brooks Prumo <brooks@prumo.org>
Co-authored-by: Brooks Prumo <brooks@solana.com>
2022-09-19 13:00:21 -07:00
Jeff Washington (jwash)
765c628546
use exit signal for acct idx bg threads ( #27483 )
2022-09-12 11:51:12 -07:00
Jeff Washington (jwash)
abd01553d5
tests: Keypair::new().pubkey() -> pubkey::new_rand ( #27705 )
2022-09-10 13:56:45 -07:00
Brooks Prumo
6a322de845
Make Accounts Background Services aware of Epoch Accounts Hash ( #27626 )
2022-09-07 20:41:40 +00:00
Jeff Washington (jwash)
a31d4a597d
serialize epoch_accounts_hash ( #27516 )
2022-09-07 10:07:00 -07:00
Brooks Prumo
93a4f80a2c
Handling snapshot requests is now required ( #27537 )
2022-09-07 10:08:42 -04:00
Brooks Prumo
3c7cd62030
Move pruned_banks_receiver into PrunedBanksRequestHandler ( #27445 )
2022-08-29 13:30:06 -04:00
Jeff Washington (jwash)
225cddcffb
serialize incremental_snapshot_hash ( #26839 )
...
* serialize incremental_snapshot_hash
* pr feedback
2022-08-17 15:14:31 -05:00
Justin Starry
69598ed4c0
Refactor: Add `RuntimeConfig` field to Bank ( #26946 )
...
* Refactor: Simplify arguments for bank constructor methods
* Refactor: Add RuntimeConfig to Bank fields
* Arc wrap runtime_config
* Arc wrap all runtime config usages
* Remove Copy trait derivation from RuntimeConfig
* Remove some arc wrapping
2022-08-05 20:49:00 +01:00
Jeff Washington (jwash)
47716a5e01
async hash verify on load ( #26208 )
...
* verify accounts hash in bg on startup
* fix some tests and loading from genesis
* add extra state for when background thread has completed
2022-07-15 14:29:56 -05:00
Jeff Washington (jwash)
602da5e51f
add accounts db config to bank tests ( #26517 )
2022-07-10 19:42:06 -05:00
Brooks Prumo
9ec38a3191
Cleanup snapshot integration tests ( #26390 )
2022-07-05 09:23:23 -05:00
Brooks Prumo
877fedadac
Remove StatusCacheRc type and use StatusCache directly ( #26184 )
2022-06-24 08:38:56 -05:00
Brooks Prumo
23c50a2389
Add StatusCache::root_slot_deltas() and use it ( #26170 )
2022-06-23 15:19:06 -05:00
Yueh-Hsuan Chiang
5b67960c76
(Refactor) Move blocktore options related stuff to blockstore_options.rs ( #25509 )
...
#### Problem
blockstore_db.rs has a mutual dependency between blockstore_metrics.rs.
#### Summary of Changes
This PR removes the mutual dependency by moving the option-related stuff
out from blockstore_db.rs to its new home --- blockstore_options.rs.
By doing this, we address the mutual dependency and also make the code cleaner.
2022-05-26 16:59:26 -07:00
DimAn
2fa9bc3e70
Add options to store full and/or incremental snapshots in separate locations ( #24247 )
2022-05-10 16:37:41 -04:00
Justin Starry
4e58b3870c
Update all BankForks methods to return owned values ( #24801 )
2022-04-28 18:51:00 +00:00
behzad nouri
454ef38e43
caches StakeAccount instead of Delegation in Stakes
...
The commit makes values in stake_delegations map in Stakes struct
generic. Stakes<Delegation> is equivalent to the old code and is used
for backward compatibility in BankFieldsTo{Serialize,Deserialize}.
But banks cache Stakes<StakeAccount> which includes the entire stake
account and StakeState deserialized from account. Doing so, will remove
the need to load stake account from accounts-db when working with
stake-delegations.
2022-04-21 15:28:41 +00:00
Michael Vines
0e2e0c8b7d
Extract most storage-related services from the Tvu abstraction
2022-04-19 19:23:58 -07:00
Jeff Washington (jwash)
210f6a6fab
move hash calculation out of acct bg svc ( #23689 )
...
* move hash calculation out of acct bg svc
* pr feedback
2022-04-08 10:42:03 -05:00
Jeff Washington (jwash)
c27150b1a3
reserialize_bank_fields_with_hash ( #23916 )
...
* reserialize_bank_with_new_accounts_hash
* Update runtime/src/serde_snapshot.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* pr feedback
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-07 14:05:57 -05:00
Jeff Washington (jwash)
550ca7bf92
compare contents of serialized banks instead of exact file format ( #24141 )
...
* compare contents of serialized banks instead of exact file format
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* pr feedback
* get rid of clone
* pr feedback
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-06 21:55:44 -05:00
Jeff Washington (jwash)
fddd162645
reserialize bank in ahv by first writing to temp file in abs ( #23947 )
2022-04-06 21:39:26 -05:00
Brooks Prumo
c322842257
Replace channel with Mutex<Option> for AccountsPackage ( #24013 )
2022-04-06 05:47:19 -05:00
Jeff Washington (jwash)
c24de17278
remove index hash calculation as an option ( #23928 )
2022-03-25 15:32:53 -05:00
Jeff Washington (jwash)
db5d68f01f
HashCalc uses self.accounts_hash_cache_path ( #23882 )
2022-03-24 09:31:55 -05:00
Yueh-Hsuan Chiang
ae75b1a25f
(LedgerStore) Add compression type ( #23578 )
...
This PR adds `--rocksdb-ledger-compression` as a hidden argument to the validator
for specifying the compression algorithm for TransactionStatus. Available compression
algorithms include `lz4`, `snappy`, `zlib`. The default value is `none`.
Experimental results show that with lz4 compression, we can achieve ~37% size-reduction
on the TransactionStatus column family, or ~8% size-reduction of the ledger store size.
2022-03-22 02:27:09 -07:00
Yueh-Hsuan Chiang
f999eef452
(LedgerStore) Rename BlockstoreAdvancedOptions to LedgerColumnOptions ( #23764 )
...
This PR renames BlockstoreAdvancedOptions to LedgerColumnOptions, as we will
pass-down this struct to LedgerColumn to allow it to perform metric reporting.
2022-03-18 11:13:35 -07:00
Yueh-Hsuan Chiang
1e20bd8f9a
(LedgerStore) Include storage type as a tag in RocksDB metric reporting ( #23523 )
...
#### Summary of Changes
This PR further enables group by operation on storage type in blockstore_rocksdb_cfs metrics.
Such group-by allows us to further compare the performance metrics between rocks-level and
rocks-fifo.
To make things extensible, this PR introduces BlockstoreAdvancedOptions and move shred_storage_type.
All fields in BlockstoreAdvancedOptions will support group-by operation in blockstore_rocksdb_cfs.
Dependency: #23580
2022-03-11 15:17:34 -08:00
Brooks Prumo
9bbccbe27c
Use `AsRef<Path>` instead of `PathBuf` for parameters ( #23560 )
2022-03-09 16:08:33 -06:00
HaoranYi
a1c45d5acb
typo ( #23564 )
2022-03-09 15:11:21 -06:00
HaoranYi
181fffb916
rename status filename to be consistent ( #23501 )
2022-03-07 17:34:35 +00:00
Yueh-Hsuan Chiang
b8b7163b66
(Ledger Store) Report RocksDB Column Family Metrics ( #22503 )
...
This PR enables blockstore to periodically report RocksDB column family properties.
The reported properties are under blockstore_rocksdb_cfs, and the properties also
support group by operation on cf_name.
2022-03-05 16:13:03 -08:00
HaoranYi
4f0070a5c6
unittest for bind two consecutive ports ( #23008 )
...
* minor fix of comments in fork-selection tests
* fix doc link
* add unittest for bind_two_consecutive_in_range
2022-03-02 09:10:29 -06:00
Yueh-Hsuan Chiang
8c872e9ce0
(LedgerStore/FIFO) Refactor FIFO options and sanity check. ( #23131 )
2022-02-19 00:58:38 -08:00
Yueh-Hsuan Chiang
1add82aa9e
(Ledger Store Benchmark) Add flags for checking shred storage size. ( #22451 )
2022-02-18 19:35:28 -08:00
Yueh-Hsuan Chiang
824446710b
(Ledger Store Benchmark) Enable FIFO compmaction in the benchmark. ( #22162 )
2022-02-12 00:43:25 -08:00
Yueh-Hsuan Chiang
f8db314134
(Ledger Store Benchmark) Display storage size of all data shreds ( #22445 )
...
* (Ledger Store) APIs for obtaining physical size of all data and coding shreds
* (Ledger Store Benchmark) Display total data shred storage size.
2022-01-19 19:33:08 -08:00
Jeff Biseda
8b66625c95
convert std::sync::mpsc to crossbeam_channel ( #22264 )
2022-01-11 02:44:46 -08:00
Yueh-Hsuan Chiang
5771c36d3f
Rename open_with_access_type() to open_with_options() ( #22123 )
2022-01-07 12:11:43 -08:00
Yueh-Hsuan Chiang
47b74e28ec
Add CLEANUP_SERVICE flag to ledger cleanup benchmark ( #22108 )
2022-01-05 23:46:02 -08:00
Yueh-Hsuan Chiang
44d61465f1
(Ledger store benchmark - 3/N) Add comments about the benchmark and its arguments ( #22160 )
...
* Avoid shred generation in ledger_cleanup.rs
* Update comment for test_ledger_cleanup_compaction to include benchmark information.
2022-01-04 23:35:55 -10:00
Yueh-Hsuan Chiang
e201b41341
Avoid shred generation in ledger_cleanup.rs ( #22091 )
2022-01-04 15:29:43 -10:00
Yueh-Hsuan Chiang
557d35ec79
Remove duplicate code in ledger_cleanup_compaction_test ( #22204 )
2021-12-31 11:19:33 -05:00
Yueh-Hsuan Chiang
f479ab7af2
ledger_cleanup test improvement (1/N) -- make the test lockless and simplify the logic ( #22090 )
2021-12-30 20:18:47 -10:00
steviez
e83ca4bb28
Clean up test_ledger_cleanup_compaction prints ( #21875 )
...
- Use info!()/warn!() over println!()/eprintln!()
- Make status prints consistent
- Add default RUST_LOG filter to see test printouts
- Adjust reported data to show shreds and rates we care about
2021-12-16 11:24:29 -06:00
steviez
4a9d7318d1
Rework test parameters to be shreds instead of entries ( #21780 )
...
The number of shreds that result from a given number of entries is
variable and in our test case, somewhat unintuitive to think about when
trying to determine how much data we're pushing into the blockstore. So,
this change converts the unit of test parameters from entries to shreds.
This change also cleans up some variable naming for clarity and prints.
2021-12-13 23:34:43 -06:00
Yueh-Hsuan Chiang
65194c7ae8
Add NUM_WRITERS to ledger_cleanup to enable multiple writers. ( #21729 )
...
Summary:
* Add NUM_WRITERS to ledger_cleanup to enable multiple writers.
(Note that our insert_shreds() is still single threaded because
it has a lock that limits only one writer at a time.)
* Make pre-generated slots more performent by directly inserting
into the shared queue. Otherwise, the main-thread which
prepares the slots will be slower than the writers.
* Correct the shred insertion time -- before this diff it did not
wait for joining all writer threads.
2021-12-10 09:42:51 -08:00
Michael Vines
b8837c04ec
Reformat imports to a consistent style for imports
...
rustfmt.toml configuration:
imports_granularity = "One"
group_imports = "One"
2021-12-03 09:19:13 -08:00