Fix SlotMeta is_connected tracking
The tracking of connected status was previously based upon an assumption
that would be practically false for all validators. The connected status
of slots played into whether Blockstore would signal ReplayStage that it
had new shreds ready to be replayed. Prior to the change, we would never
signal and ReplayStage would always wait the entire duration of a 100ms
timeout before restarting its' main processing loop.
This commit introduces a change where we mark snapshot slots as
connected. A validator may not have a path all the way back to genesis
itself; however, snapshots are taken at known roots so we extend the
connected status to these slots. Once a node has been bootstrapped once
to have is connected, the logic persists in Blockstore such that all
children on the main fork also get their connected status updated
properly.
There is some logic related to how timing values are collected that is not immediately obvious. It is better to document it, rather than requiring everyone interested to reverse engineer it from the code.
If turbine_disabled is true, the commit discards turbine packets
earlier in the pipeline so that they won't interfere with the deduper
and the packets can get through once turbine is enabled again.
This is a prerequisite of:
https://github.com/solana-labs/solana/pull/30786
so that local-cluster tests pass.
For duplicate block detection, for each (slot, shred-index, shred-type)
we need to allow 2 different shreds to be retransmitted.
The commit implements this using two bloom-filter dedupers:
* Shreds are deduplicated using the 1st deduper.
* If a shred is not a duplicate, then we check if:
(slot, shred-index, shred-type, k)
is not a duplicate for either k = 0 or k = 1 using the 2nd deduper,
and if so then the shred is retransmitted.
This allows to achieve larger capactiy compared to current LRU-cache.
#### Problem
As we start supporting new storage formats, alignment-related constants and macros
defined in append_vec.rs aren't only specific to AppendVec.
#### Summary of Changes
Move alignment-related constants/macros from append_vec.rs to accounts_file.rs
generalizes Deduper to work with any hashable type
Current Deduper is hard-coded only for Packet type. In order to use
Deduper in retransmit-stage, we need to dedup types other than Packet.
The commit generalizes Deduper to any hashable type.
* Add new vote state version that replaces Lockout with LandedVote to allow vote latency to be tracked in a future change. Includes a feature to be enabled which will when enabled cause the vote state to be written in the new form.
* Update feature set key to one owned by ashwin
---------
Co-authored-by: Ashwin Sekar <ashwin@solana.com>
removes the false_positive_rate field from the Deduper
Deduper.false_positive_rate field is misleading because it is not
enforced until maybe_reset is called. But then maybe_reset can be
invoked with an explicit argument.
* Clean up orphaned account snapshot hardlink dirs
* fix compilation issues
* debugged, now working. seeing the orphaned directories deleted
* change back to eprintln + exit for account_path error
* changed eprintln to panic for now
* add test_clean_orphaned_account_snapshot_dirs for codecov check
* address a few comments and nit isseus
* directly unzip, skipped the intermediate array of tuples
* let set_up_account_run_and_snapshot_paths return Result
* 'proper' typo, and comment on return
* use map_err
* use for loop in clean_orphaned_account_snapshot_dirs, removed panic
* add test_set_up_account_run_and_snapshot_paths
* minor, replace .for_each with .all
* rename set_up_account_run_and_snapshot_paths to create_all_accounts_run_and_snapshot_dirs
* remove unnecessary closure return type
* change to for loop
* change match to unwrap_or_else
* remove create_dir_all(&account_path) in create_all
* minor comment cleanup
Rust grammar allows trailing commas in most places where a list of
elements are accepted. It simplifies cases when the list is generated
by a macro, allowing the macro to avoid special cases for a one element
list vs longer lists.
As such, it is a common practice to allow trailing commas in macros as
well.
In most cases, for a given `Measure` value, a `stop()` call must always be followed by a call to one of the `as_*()` methods. Combining them into a single function call makes the API a bit simpler.
Declare constants at top of function with comment, and stringify the
constants when needed to be in line with how other defaults are done in
ledger-tool main function.
Current Deduper implementation uses many bits per entry:
https://github.com/solana-labs/solana/blob/65cd55261/perf/src/deduper.rs#L70-L73
and may be saturated quickly. It also lacks api to specify desired false
positive rate.
The commit instead uses an atomic bloom filter with K hash functions.
The false positive rate is obtained by tracking popcount of bits.
* ci: silence ci test output while recording in full
* shellcheck
* Adjust --color handling place
* Dump to stderr...
* Reduce too spammy solana_metrics logs
* Clean up
* Tweak
* Stash actual command's exit_code, not echo's
#### Problem
Accounts db currently use AppendVec::new_from_file() directly
to create a new AcountsFile instance from an existing file.
However, this method should be abstracted out to AccountsFile
so that an existing file can be opened correctly using the right format.
#### Summary of Changes
Add new_from_file() API to AccountsFile which will open an existing
file based on its accounts file format.
Currently, it only supports AppendVec.
steviez
Brooks
Alexander Meißner
Illia Bobyr
behzad nouri
Jeff Washington (jwash)
Nick Frostbutter
Yueh-Hsuan Chiang
HaoranYi
Tyera
bji
Xiang Zhu
Ryo Onodera
cavemanloverboy
Tao Zhu
ryleung-solana
Yihau Chen
Andrew Fitzgerald