* refactor out fault hash inject output AccountsHashVerifier
* refactor faught injector out of AccountHashVerifier
* use type alias
* Apply suggestions from code review
Co-authored-by: Brooks <brooks@prumo.org>
* move type alias
* rename
---------
Co-authored-by: Brooks <brooks@prumo.org>
Measure total time spent inside `confirm_slot_entries()`. Useful metric
in addition to `replay_elapsed` and
`poh_verify_elapsed`/`transaction_verify_elapsed`, as it shows how PoH
and transaction verification interact with the replay process.
Extracted time metrics related to transaction execution into a separate
structure. This allows me to call `process_entries_with_callback()`
without locking the whole instance of `ConfirmationTiming`, passing just
the `BatchExecutionTiming` part.
I want to add a new metric that starts at the beginning of the
`confirm_slot_entries()` call and ends until the very end. In order to
use a `scopeguard::defer`, I need to be able to have an excursive
reference to it for the whole body of `confirm_slot_entries()`.
Plus a few minor renamings to clarify which verifications and results
variables actually store. And corrected a few messages, that
incorrectly stated PoH verification, while they were actually issued
for transaction verification failures.
* Allow banking_stage to update prioritization_fee_cache
* Update core/src/banking_stage/committer.rs
Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>
* move use to top
---------
Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>
* Add --{replaying,banking}-backend flags (noop atm)
* Greatly simplify enums with strum
* Update programs/sbf/Cargo.lock...
* Rely on Display, removing Debug
* constify cli_names()
* Don't allow omitting bankend value
* Rename to --block-{verification,production}-method
* Use more specific name
* Actually support missing value....
* Remove strictly-unnecessary flags
* Use lazy_static! instead of abusing DefaultArgs...
`poh_verify` actually disables transaction signature, tick count and
built in program argument verifications as well. It is somewhat
confusing to call it `poh_verify`.
Shreds arriving at tvu/tvu_forward/repair sockets are each processed in
a separate thread, and since each thread has its own deduper, the
duplicates across these sockets are not filtered out.
Using a common deduper across these threads will require an RwLock
wrapper and may introduce lock contention.
The commit instead moves the shred-deduper to shred-sigverify-stage
where all these shreds arrive through the same channel.
If turbine_disabled is true, the commit discards turbine packets
earlier in the pipeline so that they won't interfere with the deduper
and the packets can get through once turbine is enabled again.
This is a prerequisite of:
https://github.com/solana-labs/solana/pull/30786
so that local-cluster tests pass.
For duplicate block detection, for each (slot, shred-index, shred-type)
we need to allow 2 different shreds to be retransmitted.
The commit implements this using two bloom-filter dedupers:
* Shreds are deduplicated using the 1st deduper.
* If a shred is not a duplicate, then we check if:
(slot, shred-index, shred-type, k)
is not a duplicate for either k = 0 or k = 1 using the 2nd deduper,
and if so then the shred is retransmitted.
This allows to achieve larger capactiy compared to current LRU-cache.
generalizes Deduper to work with any hashable type
Current Deduper is hard-coded only for Packet type. In order to use
Deduper in retransmit-stage, we need to dedup types other than Packet.
The commit generalizes Deduper to any hashable type.
* Add new vote state version that replaces Lockout with LandedVote to allow vote latency to be tracked in a future change. Includes a feature to be enabled which will when enabled cause the vote state to be written in the new form.
* Update feature set key to one owned by ashwin
---------
Co-authored-by: Ashwin Sekar <ashwin@solana.com>
removes the false_positive_rate field from the Deduper
Deduper.false_positive_rate field is misleading because it is not
enforced until maybe_reset is called. But then maybe_reset can be
invoked with an explicit argument.
* Clean up orphaned account snapshot hardlink dirs
* fix compilation issues
* debugged, now working. seeing the orphaned directories deleted
* change back to eprintln + exit for account_path error
* changed eprintln to panic for now
* add test_clean_orphaned_account_snapshot_dirs for codecov check
* address a few comments and nit isseus
* directly unzip, skipped the intermediate array of tuples
* let set_up_account_run_and_snapshot_paths return Result
* 'proper' typo, and comment on return
* use map_err
* use for loop in clean_orphaned_account_snapshot_dirs, removed panic
* add test_set_up_account_run_and_snapshot_paths
* minor, replace .for_each with .all
* rename set_up_account_run_and_snapshot_paths to create_all_accounts_run_and_snapshot_dirs
* remove unnecessary closure return type
* change to for loop
* change match to unwrap_or_else
* remove create_dir_all(&account_path) in create_all
* minor comment cleanup
Current Deduper implementation uses many bits per entry:
https://github.com/solana-labs/solana/blob/65cd55261/perf/src/deduper.rs#L70-L73
and may be saturated quickly. It also lacks api to specify desired false
positive rate.
The commit instead uses an atomic bloom filter with K hash functions.
The false positive rate is obtained by tracking popcount of bits.
* Add RewardsMessage enum
* Cache and update max_complete_rewards_slot
* Plumb max_complete_rewards_slot into JsonRpcRequestProcesseor
* Use max_complete_rewards_slot to check get_block requests
* Use max_complete_rewards_slot to limit Bigtable uploads
* Plumb max_complete_rewards_slot into RpcSubscriptions
* Use max_complete_rewards_slot to limit block subscriptions
* Nit: fix test
* Add helper function to create BlockstoreOptions from ValidatorConfig
* Make backup_and_clear_blockstore() honor ValidatorConfig options
backup_and_clear_blockstore() opens a Blockstore session; however, it
is currently using Blockstore::open(). This Blockstore method uses
BlockstoreOption::default() under the hood. As a result, any validator
args that adjust Blockstore settings are not considered in
backup_and_clear_blockstore().
This is especially problematic if the non-default value of
--rocksdb-shred-compaction is being used. In this case,
backup_and_clear_blockstore() was opening the wrong directory and
incorrectly finding an empty ledger.
This change plumbs any blockstore configuration to
backup_and_clear_blockstore().
* Move AdminRpcRequestMetadataPostInit to solana-core
* Move AdminRpcRequestMetadataPostInit write to just before wait_for_supermajority
* Pass AdminRpcRequestMetadataPostInit in TestValidatorGenesis
* Fixup local-cluster
* introduce workspace.package
* introduce workspace.dependencies
* read version from root cargo.toml
* pass check when version = { workspace = true }
* don't bump version when version = { workspace = true }
* including workspace Cargo.toml when bump version
* programs/sbf use workspace inheritance
* fix increasing cargo version ignore program/sbf/Cargo.toml
HaoranYi
Tao Zhu
Andrew Fitzgerald
behzad nouri
Ryo Onodera
Ashwin Sekar
Brennan
Tyera
steviez
Illia Bobyr
Brooks
Jeff Biseda
bji
Xiang Zhu
cavemanloverboy
ryleung-solana
Trent Nelson
sakridge
Jeff Washington (jwash)
Yihau Chen
Michael Vines