1c7662a37f
asserts that cluster-info keypair is consistent with contact-info id ( #29818 )
2023-01-24 16:57:55 +00:00
0db14ad39c
Removes full_snapshot from CalcAccountsHashConfig ( #29722 )
2023-01-16 16:22:46 -05:00
544b9745c2
snapshot storage path uses 1 append vec per slot ( #29627 )
2023-01-11 12:05:15 -08:00
8c212f59ad
renames ContactInfo to LegacyContactInfo ( #29566 )
...
Working towards adding a new ContactInfo where new sockets can be
added in a backward compatible way.
2023-01-08 16:00:55 +00:00
b95835143e
remove AccountsBackgroundService::new(caching_enabled) ( #29234 )
2022-12-13 07:18:02 -08:00
bb0bfc4214
remove bank_from_latest_snapshot_archives(caching_enabled) ( #29238 )
2022-12-13 07:16:24 -08:00
2c2324f4ea
remove caching_enabled from Bank::new_with_paths_for_tests ( #29214 )
2022-12-12 15:30:46 -08:00
1b0aaf1607
Makes a new PathBuf instead of moving the test's TempDir ( #29220 )
2022-12-12 18:29:36 -05:00
249607dbfe
Use a different tempdir for unpacking snapshots ( #29219 )
2022-12-12 17:26:52 -06:00
391f68da61
Uses Storages to calculate accounts hash in EAH warp tests ( #29192 )
2022-12-12 13:30:23 -05:00
631a98a3b6
warp_from_parents works with write_cache enabled ( #29185 )
2022-12-09 14:28:18 -08:00
6a90abd056
remove handle_snapshot_requests.caching_enabled ( #29174 )
2022-12-09 10:51:44 -08:00
45ba5ef6fd
remove bank_from_snapshot_archives caching_enabled ( #29171 )
2022-12-09 10:45:21 -08:00
ec5098a723
remove bank_test_config_caching_enabled ( #29170 )
2022-12-09 08:28:02 -08:00
aeb6b53502
Remove unused Option<> around ValidatorConfig's SnapshotConfig ( #29090 )
...
Remove Option<> around ValidatorConfig's SnapshotConfig
The SnapshotConfig is required and is currently hard-coded to be a
Some().
2022-12-06 22:47:55 +00:00
9524c9dbff
patches errors from clippy::uninlined_format_args
...
https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
2022-12-06 19:32:15 +00:00
3c42c87098
Remove obsoleted return value from Blockstore insert shred method ( #28992 )
2022-12-01 11:17:46 -06:00
9327658007
Promotes accounts hash to a strong type ( #28930 )
2022-11-28 10:09:47 -05:00
638b26ea65
Renames EAH test fn ( #28939 )
2022-11-23 05:18:50 +00:00
f22104d46b
use write cache by default in some tests ( #28876 )
2022-11-18 14:35:52 -08:00
2bafb0cb12
Requires EAH state cannot be Invalid ( #28817 )
2022-11-17 11:01:01 -05:00
ae0bb44401
Fixes test_snapshots_with_background_services ( #28848 )
2022-11-17 00:49:28 -05:00
d798e751a0
Disables EAH with short epochs ( #28803 )
2022-11-15 13:26:19 -05:00
d4cf18421d
Use 400 slots-per-epoch in EAH tests ( #28801 )
2022-11-14 17:49:20 -05:00
0bfea02056
Snapshots wait for EAH calculations to complete ( #28777 )
2022-11-14 11:34:44 -06:00
d1ba42180d
clippy for rust 1.65.0 ( #28765 )
2022-11-09 19:39:38 +00:00
9e1cdc7e60
Enables not taking a bank snapshot ( #28756 )
2022-11-09 12:43:33 -05:00
d4c2900590
Removes `snapshot_bank()` wrapper fn ( #28753 )
2022-11-07 15:09:31 +00:00
59bf1809fe
Uses SnapshotHash type in snapshot archive fields ( #28681 )
2022-10-31 14:28:35 -04:00
37507a2de6
Removes EAH parameter from serde_snapshot::reserialize_bank() ( #28669 )
2022-10-31 09:43:17 -04:00
5a3d252899
Renames fn to Bank::update_accounts_hash_for_tests() ( #28620 )
2022-10-28 14:33:05 -04:00
2272fd807e
Remove Blockstore manual compaction code ( #28409 )
...
The manual Blockstore compaction that was being initiated from
LedgerCleanupService has been disabled for quite some time in favor of
several optimizations.
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2022-10-28 10:39:00 +02:00
2354a0a343
Renames fn to calculate_accounts_hash_from_index() ( #28568 )
2022-10-24 19:20:08 -04:00
12f3e8c9cc
Ignores errors when joining background threads in snapshot tests ( #28480 )
2022-10-19 16:54:59 -04:00
d81d2bba59
comments out print in test ( #28475 )
2022-10-19 10:25:11 -05:00
1cc9cf927c
Supports warping with Epoch Accounts Hash ( #28459 )
2022-10-19 10:37:14 -04:00
28a89a1d99
remove expected rent collection and rehashing completely ( #28422 )
2022-10-17 07:24:42 -07:00
31c2b29941
Sends both an EAH and a snapshot request from `set_root()` ( #28363 )
2022-10-14 11:00:04 -04:00
dd7fee8f32
Re-enqueues unhandled ABS requests ( #28362 )
2022-10-13 16:25:39 -04:00
9cbd00fdbc
Converts PendingAccountsPackage to a channel ( #28352 )
2022-10-13 12:47:36 -04:00
5a08eed82d
Cleans up debugging code in EAH tests ( #28324 )
2022-10-10 16:07:55 +00:00
27cd2c324e
Adds tests for EAH and snapshot interactions ( #28304 )
2022-10-10 10:16:13 -04:00
981c9d07a4
Rearranges eah TestEnvironment fields to ensure drop order ( #28270 )
2022-10-06 16:17:32 -04:00
2d936784dd
Ignore errors when joining background threads for EAH tests ( #28263 )
2022-10-06 18:43:56 +00:00
a8c6a9e5fc
Bank::freeze() waits for EAH calculation to complete ( #28170 )
2022-10-05 17:44:35 -04:00
6b17bee5a8
Remove the const default for RocksFifo ( #27965 )
...
#### Summary of Changes
Removes the constant default for ShredStorageType::RocksFifo
as the shred storage size is either user-specified or derived
from --limit-ledger-size in #27459 .
2022-10-01 15:10:54 -07:00
8877ac2aa9
Fix call to calculate_accounts_hash() ( #28169 )
2022-09-30 15:29:18 -04:00
2f8f6c6a31
Send Epoch Accounts Hash requests from set_root() ( #27764 )
2022-09-30 14:59:41 -04:00
97c9af4c6b
plumbs through flag to generate merkle variant of shreds
2022-09-23 16:45:18 +00:00
9a57c64f21
patches clippy errors from new rust nightly release ( #27996 )
2022-09-22 22:23:03 +00:00
ff71df4695
Remove unnecessary call to `set_startup_verification_complete()` ( #27986 )
2022-09-22 16:54:17 -04:00
1ee595ca9c
remove AccountsDb::initial_blockstore_processing_complete ( #27974 )
2022-09-22 13:52:04 -04:00
f2d6a7ecea
bank.initial_blockstore_processing_complete to avoid concurrent hash calculations ( #27776 )
...
* bank.initial_blockstore_processing_complete to avoid concurrent hash calculations
* Update runtime/src/bank.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/bank.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Rename TestValidator::set_startup_verification_complete()
* Initialize with `AtomicBool::new(false)` instead of `default()`
* snapshot tests: move where `initial_blockstore_processing_completed()` is called
* fixup bank_forks.rs calling `is_initial_blockstore_processing_complete()`
* only call initial_blockstore_processing_completed() in blockstore_processor
Co-authored-by: Brooks Prumo <brooks@prumo.org>
Co-authored-by: Brooks Prumo <brooks@solana.com>
2022-09-19 13:00:21 -07:00
765c628546
use exit signal for acct idx bg threads ( #27483 )
2022-09-12 11:51:12 -07:00
abd01553d5
tests: Keypair::new().pubkey() -> pubkey::new_rand ( #27705 )
2022-09-10 13:56:45 -07:00
6a322de845
Make Accounts Background Services aware of Epoch Accounts Hash ( #27626 )
2022-09-07 20:41:40 +00:00
a31d4a597d
serialize epoch_accounts_hash ( #27516 )
2022-09-07 10:07:00 -07:00
93a4f80a2c
Handling snapshot requests is now required ( #27537 )
2022-09-07 10:08:42 -04:00
3c7cd62030
Move pruned_banks_receiver into PrunedBanksRequestHandler ( #27445 )
2022-08-29 13:30:06 -04:00
225cddcffb
serialize incremental_snapshot_hash ( #26839 )
...
* serialize incremental_snapshot_hash
* pr feedback
2022-08-17 15:14:31 -05:00
69598ed4c0
Refactor: Add `RuntimeConfig` field to Bank ( #26946 )
...
* Refactor: Simplify arguments for bank constructor methods
* Refactor: Add RuntimeConfig to Bank fields
* Arc wrap runtime_config
* Arc wrap all runtime config usages
* Remove Copy trait derivation from RuntimeConfig
* Remove some arc wrapping
2022-08-05 20:49:00 +01:00
47716a5e01
async hash verify on load ( #26208 )
...
* verify accounts hash in bg on startup
* fix some tests and loading from genesis
* add extra state for when background thread has completed
2022-07-15 14:29:56 -05:00
602da5e51f
add accounts db config to bank tests ( #26517 )
2022-07-10 19:42:06 -05:00
9ec38a3191
Cleanup snapshot integration tests ( #26390 )
2022-07-05 09:23:23 -05:00
877fedadac
Remove StatusCacheRc type and use StatusCache directly ( #26184 )
2022-06-24 08:38:56 -05:00
23c50a2389
Add StatusCache::root_slot_deltas() and use it ( #26170 )
2022-06-23 15:19:06 -05:00
5b67960c76
(Refactor) Move blocktore options related stuff to blockstore_options.rs ( #25509 )
...
#### Problem
blockstore_db.rs has a mutual dependency between blockstore_metrics.rs.
#### Summary of Changes
This PR removes the mutual dependency by moving the option-related stuff
out from blockstore_db.rs to its new home --- blockstore_options.rs.
By doing this, we address the mutual dependency and also make the code cleaner.
2022-05-26 16:59:26 -07:00
2fa9bc3e70
Add options to store full and/or incremental snapshots in separate locations ( #24247 )
2022-05-10 16:37:41 -04:00
4e58b3870c
Update all BankForks methods to return owned values ( #24801 )
2022-04-28 18:51:00 +00:00
454ef38e43
caches StakeAccount instead of Delegation in Stakes
...
The commit makes values in stake_delegations map in Stakes struct
generic. Stakes<Delegation> is equivalent to the old code and is used
for backward compatibility in BankFieldsTo{Serialize,Deserialize}.
But banks cache Stakes<StakeAccount> which includes the entire stake
account and StakeState deserialized from account. Doing so, will remove
the need to load stake account from accounts-db when working with
stake-delegations.
2022-04-21 15:28:41 +00:00
0e2e0c8b7d
Extract most storage-related services from the Tvu abstraction
2022-04-19 19:23:58 -07:00
210f6a6fab
move hash calculation out of acct bg svc ( #23689 )
...
* move hash calculation out of acct bg svc
* pr feedback
2022-04-08 10:42:03 -05:00
c27150b1a3
reserialize_bank_fields_with_hash ( #23916 )
...
* reserialize_bank_with_new_accounts_hash
* Update runtime/src/serde_snapshot.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* pr feedback
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-07 14:05:57 -05:00
550ca7bf92
compare contents of serialized banks instead of exact file format ( #24141 )
...
* compare contents of serialized banks instead of exact file format
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* pr feedback
* get rid of clone
* pr feedback
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-06 21:55:44 -05:00
fddd162645
reserialize bank in ahv by first writing to temp file in abs ( #23947 )
2022-04-06 21:39:26 -05:00
c322842257
Replace channel with Mutex<Option> for AccountsPackage ( #24013 )
2022-04-06 05:47:19 -05:00
c24de17278
remove index hash calculation as an option ( #23928 )
2022-03-25 15:32:53 -05:00
db5d68f01f
HashCalc uses self.accounts_hash_cache_path ( #23882 )
2022-03-24 09:31:55 -05:00
ae75b1a25f
(LedgerStore) Add compression type ( #23578 )
...
This PR adds `--rocksdb-ledger-compression` as a hidden argument to the validator
for specifying the compression algorithm for TransactionStatus. Available compression
algorithms include `lz4`, `snappy`, `zlib`. The default value is `none`.
Experimental results show that with lz4 compression, we can achieve ~37% size-reduction
on the TransactionStatus column family, or ~8% size-reduction of the ledger store size.
2022-03-22 02:27:09 -07:00
f999eef452
(LedgerStore) Rename BlockstoreAdvancedOptions to LedgerColumnOptions ( #23764 )
...
This PR renames BlockstoreAdvancedOptions to LedgerColumnOptions, as we will
pass-down this struct to LedgerColumn to allow it to perform metric reporting.
2022-03-18 11:13:35 -07:00
1e20bd8f9a
(LedgerStore) Include storage type as a tag in RocksDB metric reporting ( #23523 )
...
#### Summary of Changes
This PR further enables group by operation on storage type in blockstore_rocksdb_cfs metrics.
Such group-by allows us to further compare the performance metrics between rocks-level and
rocks-fifo.
To make things extensible, this PR introduces BlockstoreAdvancedOptions and move shred_storage_type.
All fields in BlockstoreAdvancedOptions will support group-by operation in blockstore_rocksdb_cfs.
Dependency: #23580
2022-03-11 15:17:34 -08:00
9bbccbe27c
Use `AsRef<Path>` instead of `PathBuf` for parameters ( #23560 )
2022-03-09 16:08:33 -06:00
a1c45d5acb
typo ( #23564 )
2022-03-09 15:11:21 -06:00
181fffb916
rename status filename to be consistent ( #23501 )
2022-03-07 17:34:35 +00:00
b8b7163b66
(Ledger Store) Report RocksDB Column Family Metrics ( #22503 )
...
This PR enables blockstore to periodically report RocksDB column family properties.
The reported properties are under blockstore_rocksdb_cfs, and the properties also
support group by operation on cf_name.
2022-03-05 16:13:03 -08:00
4f0070a5c6
unittest for bind two consecutive ports ( #23008 )
...
* minor fix of comments in fork-selection tests
* fix doc link
* add unittest for bind_two_consecutive_in_range
2022-03-02 09:10:29 -06:00
8c872e9ce0
(LedgerStore/FIFO) Refactor FIFO options and sanity check. ( #23131 )
2022-02-19 00:58:38 -08:00
1add82aa9e
(Ledger Store Benchmark) Add flags for checking shred storage size. ( #22451 )
2022-02-18 19:35:28 -08:00
824446710b
(Ledger Store Benchmark) Enable FIFO compmaction in the benchmark. ( #22162 )
2022-02-12 00:43:25 -08:00
f8db314134
(Ledger Store Benchmark) Display storage size of all data shreds ( #22445 )
...
* (Ledger Store) APIs for obtaining physical size of all data and coding shreds
* (Ledger Store Benchmark) Display total data shred storage size.
2022-01-19 19:33:08 -08:00
8b66625c95
convert std::sync::mpsc to crossbeam_channel ( #22264 )
2022-01-11 02:44:46 -08:00
5771c36d3f
Rename open_with_access_type() to open_with_options() ( #22123 )
2022-01-07 12:11:43 -08:00
47b74e28ec
Add CLEANUP_SERVICE flag to ledger cleanup benchmark ( #22108 )
2022-01-05 23:46:02 -08:00
44d61465f1
(Ledger store benchmark - 3/N) Add comments about the benchmark and its arguments ( #22160 )
...
* Avoid shred generation in ledger_cleanup.rs
* Update comment for test_ledger_cleanup_compaction to include benchmark information.
2022-01-04 23:35:55 -10:00
e201b41341
Avoid shred generation in ledger_cleanup.rs ( #22091 )
2022-01-04 15:29:43 -10:00
557d35ec79
Remove duplicate code in ledger_cleanup_compaction_test ( #22204 )
2021-12-31 11:19:33 -05:00
f479ab7af2
ledger_cleanup test improvement (1/N) -- make the test lockless and simplify the logic ( #22090 )
2021-12-30 20:18:47 -10:00
e83ca4bb28
Clean up test_ledger_cleanup_compaction prints ( #21875 )
...
- Use info!()/warn!() over println!()/eprintln!()
- Make status prints consistent
- Add default RUST_LOG filter to see test printouts
- Adjust reported data to show shreds and rates we care about
2021-12-16 11:24:29 -06:00
4a9d7318d1
Rework test parameters to be shreds instead of entries ( #21780 )
...
The number of shreds that result from a given number of entries is
variable and in our test case, somewhat unintuitive to think about when
trying to determine how much data we're pushing into the blockstore. So,
this change converts the unit of test parameters from entries to shreds.
This change also cleans up some variable naming for clarity and prints.
2021-12-13 23:34:43 -06:00
65194c7ae8
Add NUM_WRITERS to ledger_cleanup to enable multiple writers. ( #21729 )
...
Summary:
* Add NUM_WRITERS to ledger_cleanup to enable multiple writers.
(Note that our insert_shreds() is still single threaded because
it has a lock that limits only one writer at a time.)
* Make pre-generated slots more performent by directly inserting
into the shared queue. Otherwise, the main-thread which
prepares the slots will be slower than the writers.
* Correct the shred insertion time -- before this diff it did not
wait for joining all writer threads.
2021-12-10 09:42:51 -08:00
behzad nouri
Brooks
Jeff Washington (jwash)
apfitzge
steviez
HaoranYi
Yueh-Hsuan Chiang
Jeff Washington (jwash)
Justin Starry
DimAn
Michael Vines
Jeff Biseda