Working towards LegacyContactInfo => ContactInfo migration, the commit
hides some implementation details of LegacyContactInfo and expands API
parity with the new ContactInfo.
* Fix bug where ReplayStage holds an Arc<Bank> for process lifetime
When ReplayStage::new() kicks off, it needs to do some setup with the
working bank prior to entering the main processing loop. This setup is
done before entering the main processing loop; however, a bug made it
such that an Arc<Bank> remained in scope after the processing loop had
been entered. The processing loop is only exited when the process exits,
so this means that Bank was being held for the lifetime of the process.
This is a waste of resources and prevents background cleanup.
* clippy
Add new vote state version that replaces Lockout with LandedVote to allow vote latency to be tracked in a future change.
Includes a feature to be enabled which will when enabled cause the vote state to be written in the new form.
* Add a type_select param for purge_old_bank_snapshots
* Use flags to make the function calls more readable
* Remove the extra purge calls
* replace select_type with filter_by_type
* Add test
* Use matches
* Fix CI test on reference
* use match and call do_purge once
* Let bank_snapshots_dir be TempDir
* remove account_paths in the test
* replace bank with _bank
* Remove create_snapshot_dirs_for_tests, will take the lastest from master
* Fix merge errors
* Let AHV hold and update last_snapshot_storages
* Clean up comment
* Move cloning after enqueued_time
* Minor positon change
* Remove type last_snapshot_storages annotation
* refactor out fault hash inject output AccountsHashVerifier
* refactor faught injector out of AccountHashVerifier
* use type alias
* Apply suggestions from code review
Co-authored-by: Brooks <brooks@prumo.org>
* move type alias
* rename
---------
Co-authored-by: Brooks <brooks@prumo.org>
Measure total time spent inside `confirm_slot_entries()`. Useful metric
in addition to `replay_elapsed` and
`poh_verify_elapsed`/`transaction_verify_elapsed`, as it shows how PoH
and transaction verification interact with the replay process.
Extracted time metrics related to transaction execution into a separate
structure. This allows me to call `process_entries_with_callback()`
without locking the whole instance of `ConfirmationTiming`, passing just
the `BatchExecutionTiming` part.
I want to add a new metric that starts at the beginning of the
`confirm_slot_entries()` call and ends until the very end. In order to
use a `scopeguard::defer`, I need to be able to have an excursive
reference to it for the whole body of `confirm_slot_entries()`.
Plus a few minor renamings to clarify which verifications and results
variables actually store. And corrected a few messages, that
incorrectly stated PoH verification, while they were actually issued
for transaction verification failures.
* Allow banking_stage to update prioritization_fee_cache
* Update core/src/banking_stage/committer.rs
Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>
* move use to top
---------
Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>
* Add --{replaying,banking}-backend flags (noop atm)
* Greatly simplify enums with strum
* Update programs/sbf/Cargo.lock...
* Rely on Display, removing Debug
* constify cli_names()
* Don't allow omitting bankend value
* Rename to --block-{verification,production}-method
* Use more specific name
* Actually support missing value....
* Remove strictly-unnecessary flags
* Use lazy_static! instead of abusing DefaultArgs...
`poh_verify` actually disables transaction signature, tick count and
built in program argument verifications as well. It is somewhat
confusing to call it `poh_verify`.
Shreds arriving at tvu/tvu_forward/repair sockets are each processed in
a separate thread, and since each thread has its own deduper, the
duplicates across these sockets are not filtered out.
Using a common deduper across these threads will require an RwLock
wrapper and may introduce lock contention.
The commit instead moves the shred-deduper to shred-sigverify-stage
where all these shreds arrive through the same channel.
If turbine_disabled is true, the commit discards turbine packets
earlier in the pipeline so that they won't interfere with the deduper
and the packets can get through once turbine is enabled again.
This is a prerequisite of:
https://github.com/solana-labs/solana/pull/30786
so that local-cluster tests pass.
For duplicate block detection, for each (slot, shred-index, shred-type)
we need to allow 2 different shreds to be retransmitted.
The commit implements this using two bloom-filter dedupers:
* Shreds are deduplicated using the 1st deduper.
* If a shred is not a duplicate, then we check if:
(slot, shred-index, shred-type, k)
is not a duplicate for either k = 0 or k = 1 using the 2nd deduper,
and if so then the shred is retransmitted.
This allows to achieve larger capactiy compared to current LRU-cache.
generalizes Deduper to work with any hashable type
Current Deduper is hard-coded only for Packet type. In order to use
Deduper in retransmit-stage, we need to dedup types other than Packet.
The commit generalizes Deduper to any hashable type.
* Add new vote state version that replaces Lockout with LandedVote to allow vote latency to be tracked in a future change. Includes a feature to be enabled which will when enabled cause the vote state to be written in the new form.
* Update feature set key to one owned by ashwin
---------
Co-authored-by: Ashwin Sekar <ashwin@solana.com>
removes the false_positive_rate field from the Deduper
Deduper.false_positive_rate field is misleading because it is not
enforced until maybe_reset is called. But then maybe_reset can be
invoked with an explicit argument.
* Clean up orphaned account snapshot hardlink dirs
* fix compilation issues
* debugged, now working. seeing the orphaned directories deleted
* change back to eprintln + exit for account_path error
* changed eprintln to panic for now
* add test_clean_orphaned_account_snapshot_dirs for codecov check
* address a few comments and nit isseus
* directly unzip, skipped the intermediate array of tuples
* let set_up_account_run_and_snapshot_paths return Result
* 'proper' typo, and comment on return
* use map_err
* use for loop in clean_orphaned_account_snapshot_dirs, removed panic
* add test_set_up_account_run_and_snapshot_paths
* minor, replace .for_each with .all
* rename set_up_account_run_and_snapshot_paths to create_all_accounts_run_and_snapshot_dirs
* remove unnecessary closure return type
* change to for loop
* change match to unwrap_or_else
* remove create_dir_all(&account_path) in create_all
* minor comment cleanup
Current Deduper implementation uses many bits per entry:
https://github.com/solana-labs/solana/blob/65cd55261/perf/src/deduper.rs#L70-L73
and may be saturated quickly. It also lacks api to specify desired false
positive rate.
The commit instead uses an atomic bloom filter with K hash functions.
The false positive rate is obtained by tracking popcount of bits.
* Add RewardsMessage enum
* Cache and update max_complete_rewards_slot
* Plumb max_complete_rewards_slot into JsonRpcRequestProcesseor
* Use max_complete_rewards_slot to check get_block requests
* Use max_complete_rewards_slot to limit Bigtable uploads
* Plumb max_complete_rewards_slot into RpcSubscriptions
* Use max_complete_rewards_slot to limit block subscriptions
* Nit: fix test
* Add helper function to create BlockstoreOptions from ValidatorConfig
* Make backup_and_clear_blockstore() honor ValidatorConfig options
backup_and_clear_blockstore() opens a Blockstore session; however, it
is currently using Blockstore::open(). This Blockstore method uses
BlockstoreOption::default() under the hood. As a result, any validator
args that adjust Blockstore settings are not considered in
backup_and_clear_blockstore().
This is especially problematic if the non-default value of
--rocksdb-shred-compaction is being used. In this case,
backup_and_clear_blockstore() was opening the wrong directory and
incorrectly finding an empty ledger.
This change plumbs any blockstore configuration to
backup_and_clear_blockstore().
* Move AdminRpcRequestMetadataPostInit to solana-core
* Move AdminRpcRequestMetadataPostInit write to just before wait_for_supermajority
* Pass AdminRpcRequestMetadataPostInit in TestValidatorGenesis
* Fixup local-cluster
* introduce workspace.package
* introduce workspace.dependencies
* read version from root cargo.toml
* pass check when version = { workspace = true }
* don't bump version when version = { workspace = true }
* including workspace Cargo.toml when bump version
* programs/sbf use workspace inheritance
* fix increasing cargo version ignore program/sbf/Cargo.toml
* measure! to measure_us!
* Consistent naming of transaction_recorder
* Remove outdated comment - Instant cannot be None
* use local
* Remove measure! import
* Add accounts hard-link files into the bank snapshot directory
* Small adjustments and fixes.
* Address some of the review issues
* Fix compilation issues
* Change the latest slot snapshot storage from VecDeque to Option
* IoWithSourceAndFile and expanded comments on accounts
* last_slot_snapshot_storages in return value
* Update comments following the review input
* rename dir_accounts_hard_links to hard_link_path
* Add dir_full_state flag for add_bank_snapshot
* Let appendvec files hardlinking work with multiple accounts paths across multiple partitions
* Fixes for rebasing
* fix tests which generates account_path without adding run/
* rebasing fixes
* fix account path test failures
* fix test test_concurrent_snapshot_packaging
* review comments. renamed the path setup function
* Addressed most of the review comments
* update with more review comments
* handle error from create_accounts_run_and_snapshot_dirs
* fix rebasing duplicate
* minor accounts_dir path cleanup
* minor cleanup, remove commented code
* misc review comments
* build error fix
* Fix test_incremental_snapshot_download_with_crossing_full_snapshot_interval_at_startup
* fix build error on MAX_BANK_SNAPSHOTS_TO_RETAIN
* rebase fix, update hardlink filename
* minor comment spelling fix
* rebasing fixes
* fix rebase issues; with_extension
* comments changes for review
* misc minor review issues
* bank.fill_bank_with_ticks_for_tests
* error handling on appendvec path
* fix use_jit
* minor comments refining
* Remove type AccountStorages
* get_account_path_from_appendvec_path return changed to Option
* removed appendvec_path.to_path_buf in create_accounts_run_and_snapshot_dirs
* add test_get_snapshot_accounts_hardlink_dir
* update last_snapshot_storages comment
* update last_snapshot_storages comment
* symlink map_err
* simplify test_get_snapshot_accounts_hardlink_dir with fake paths
* log last_snapshot_storages at the end of the loop
Snapshot construction and interaction code was a bit more manual than necessary, even causing a bug to slip past a review. Separated snapshot construction from the diffing of two snapshots.
This should make the logic clearer.
A recent change to this service to store the number of non-vote
transactions introduced a bug in the computation of the number of
transactions during the time interval. This resulted in bogus values
being stored in Blockstore and eventually getting served through RPC for
the TPS chart on explorer.