Commit Graph

3612 Commits

Author SHA1 Message Date
Brooks 965dd37924
Moves SnapshotGossipManager to its own file (#31147) 2023-04-11 15:51:52 -04:00
Brooks 453f272698
Rename IncrementalSnapshotHashes to SnapshotHashes (#31136) 2023-04-11 10:30:29 -04:00
Brooks f3083ad2e0
Rename SnapshotHashes to LegacySnapshotHashes (#31086) 2023-04-10 17:52:20 -04:00
behzad nouri ce21a58b65
reworks streamer::StakedNodes (#31082)
{min,max}_stake are computed but never assigned:
https://github.com/solana-labs/solana/blob/4564bcdc1/core/src/staked_nodes_updater_service.rs#L54-L57

The updater code is also inefficient and verbose.
2023-04-10 17:07:40 +00:00
Brooks f9276d1748
Uses MAX_ACCOUNTS_HASHES instead of MAX_SNAPSHOT_HASHES in accounts_hash_verifier.rs (#31114) 2023-04-10 10:44:40 -04:00
HaoranYi fcd1fe0959
Refactor fault hash injection into lambda (#31093)
* refactor out fault hash inject output AccountsHashVerifier

* refactor faught injector out of AccountHashVerifier

* use type alias

* Apply suggestions from code review

Co-authored-by: Brooks <brooks@prumo.org>

* move type alias

* rename

---------

Co-authored-by: Brooks <brooks@prumo.org>
2023-04-07 17:50:21 -05:00
Tao Zhu 9a7b6abc91
update benches after removing packet.sender_stake (#31110) 2023-04-07 14:27:29 -05:00
Andrew Fitzgerald 926bb0c794
MultiIteratorScanner::finalize returns (payload, already_processed) (#31054)
* finalize() returns (payload, already_processed)

* Additional testing around already_handled

* Return type wrapper and comment update
2023-04-07 11:17:36 -07:00
behzad nouri 466a9a2449
removes ip_stake_map field from streamer::StakedNodes (#31078) 2023-04-07 13:27:29 +00:00
Ryo Onodera f0432ec50f
Avoid overflow in ThreadSet::any() and nits (#31098)
Avoid overflow in ThreadSet::any and etc
2023-04-07 12:45:29 +09:00
behzad nouri 4d0abebe0e
removes Packet Meta.sender_stake and find_packet_sender_stake_stage (#31077)
Packet Meta.sender_stake is unused since
https://github.com/solana-labs/solana/pull/26512
removed sender_stake from banking-stage buffer prioritization.
2023-04-06 21:33:43 +00:00
Andrew Fitzgerald 00250819b8
ThreadAwareAccountLocks (#30422) 2023-04-06 10:12:03 -07:00
Ashwin Sekar 85dbd3d94d
Add stake breakdown to metrics for HeaviestForkFailures (#31067) 2023-04-05 20:35:12 -06:00
Brennan 60c4a718a5
enhance replay partition metrics (#31010)
* enhance replay partition metrics
2023-04-04 19:57:09 -07:00
Ashwin Sekar 9cbaf0e234
Rename DeadSlotAncestorRequestStatus -> AncestorRequestStatus (#31050) 2023-04-04 20:48:45 +00:00
Tyera 3442f184f7
Remove unneeded `clippy::new_ret_no_self` allows (#31035)
Remove unneeded allows
2023-04-03 20:35:20 -06:00
behzad nouri ff9a42a354
uses Duration type instead of untyped ..._ms: u64 (#30971) 2023-03-31 15:42:49 +00:00
steviez cc8e531a5d
Enforce a minimum of 1 on full and incremental snapshot retention (#30968) 2023-03-30 10:16:36 -05:00
Illia Bobyr fe5ae7733b
ledger: confirm_slot_entries(): confirmation_elapsed metric (#30807)
Measure total time spent inside `confirm_slot_entries()`.  Useful metric
in addition to `replay_elapsed` and
`poh_verify_elapsed`/`transaction_verify_elapsed`, as it shows how PoH
and transaction verification interact with the replay process.
2023-03-29 13:11:29 -07:00
Andrew Fitzgerald 8e910b494f
Forwarder: separate get_leader_and_addr (#30922) 2023-03-28 16:34:36 -07:00
Illia Bobyr 564f8c9b17
ledger: Extract `BatchExecutionTiming` (#30806)
Extracted time metrics related to transaction execution into a separate
structure.  This allows me to call `process_entries_with_callback()`
without locking the whole instance of `ConfirmationTiming`, passing just
the `BatchExecutionTiming` part.

I want to add a new metric that starts at the beginning of the
`confirm_slot_entries()` call and ends until the very end.  In order to
use a `scopeguard::defer`, I need to be able to have an excursive
reference to it for the whole body of `confirm_slot_entries()`.

Plus a few minor renamings to clarify which verifications and results
variables actually store.  And corrected a few messages, that
incorrectly stated PoH verification, while they were actually issued
for transaction verification failures.
2023-03-28 15:37:34 -07:00
Andrew Fitzgerald b72be0f086
Forwarder: clean up packet_vec filter (#30921) 2023-03-28 14:54:04 -07:00
Andrew Fitzgerald 2dfc46c71e
Forwarder: separate update_data_budget (#30920) 2023-03-28 12:58:39 -07:00
behzad nouri 75abfc79a6
removes unused dependencies (#30917) 2023-03-28 17:25:44 +00:00
behzad nouri 29f776c676
reports cluster-nodes metrics by stake (#30912) 2023-03-28 12:45:09 +00:00
Brooks d7ae05c3fd
Unifies logging of start/stop for background services (#30916) 2023-03-28 08:32:18 -04:00
behzad nouri 49b8ea771e
simplifies ServeRepair::run_orphan (#30908) 2023-03-27 23:19:55 +00:00
Andrew Fitzgerald a575ea2ee0
LeaderBankNotifier (#30395) 2023-03-27 08:17:17 -07:00
Brooks bf7fa02214
Add units to incremental accounts hash datapoint (#30894) 2023-03-24 21:39:20 +00:00
Andrew Fitzgerald f226a34f48
Only need bank reference for update (#30879) 2023-03-24 09:48:04 -07:00
Tao Zhu 52e63e2ffa
Allow banking_stage to update prioritization_fee_cache (#30853)
* Allow banking_stage to update prioritization_fee_cache

* Update core/src/banking_stage/committer.rs

Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>

* move use to top

---------

Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>
2023-03-24 00:05:54 +00:00
behzad nouri a36e8f559c
removes hash-map lookups when sorting vote-account keys (#30878) 2023-03-23 22:29:35 +00:00
Ryo Onodera 6c444df9e0
Add --block-{verification,production}-method flags (noop atm) (#30746)
* Add --{replaying,banking}-backend flags (noop atm)

* Greatly simplify enums with strum

* Update programs/sbf/Cargo.lock...

* Rely on Display, removing Debug

* constify cli_names()

* Don't allow omitting bankend value

* Rename to --block-{verification,production}-method

* Use more specific name

* Actually support missing value....

* Remove strictly-unnecessary flags

* Use lazy_static! instead of abusing DefaultArgs...
2023-03-23 12:57:28 +09:00
Jeff Biseda 04f0311aa1
check data budget before accessing blockstore (#30809) 2023-03-22 15:56:06 -07:00
Jeff Biseda 94b27d8f96
add metric for repair request blockstore misses (#30810) 2023-03-22 15:47:11 -07:00
Illia Bobyr 809041b151
poh_verify => run_verification: Rename to be more accurate (#30811)
`poh_verify` actually disables transaction signature, tick count and
built in program argument verifications as well.  It is somewhat
confusing to call it `poh_verify`.
2023-03-22 11:03:30 -07:00
Brooks 35437b8dad
Makes AccountsHashVerifier aware of Incremental Accounts Hash (#30820) 2023-03-22 10:20:16 -04:00
behzad nouri 25b7811869
moves shreds deduper to shred-sigverify stage (#30786)
Shreds arriving at tvu/tvu_forward/repair sockets are each processed in
a separate thread, and since each thread has its own deduper, the
duplicates across these sockets are not filtered out.
Using a common deduper across these threads will require an RwLock
wrapper and may introduce lock contention.
The commit instead moves the shred-deduper to shred-sigverify-stage
where all these shreds arrive through the same channel.
2023-03-22 13:19:16 +00:00
Andrew Fitzgerald ac8c31b5d6
PohRecorder::recorder -> new_recorder (#30838) 2023-03-22 13:19:20 +09:00
Brooks b64d0de771
Makes snapshot_utils aware of Incremental Accounts Hash (#30804) 2023-03-21 16:34:30 +00:00
steviez 5344a789d7
Revert "Add new vote state version that replaces Lockout with LandedV… (#30817)
Revert "Add new vote state version that replaces Lockout with LandedVote to a… (#29524)"

This reverts commit d77f0a22c7.
2023-03-21 22:54:13 +08:00
behzad nouri e66edeb180
moves turbine-disabled check to shred-fetch-stage (#30799)
If turbine_disabled is true, the commit discards turbine packets
earlier in the pipeline so that they won't interfere with the deduper
and the packets can get through once turbine is enabled again.

This is a prerequisite of:
https://github.com/solana-labs/solana/pull/30786
so that local-cluster tests pass.
2023-03-20 20:34:41 +00:00
behzad nouri c6e7aaf96c
removes lazy-static thread-pool from sigverify-shreds (#30787)
Instead the thread-pool is passed explicitly from higher in the call
stack so that
https://github.com/solana-labs/solana/pull/30786
can use the same thread-pool for shred deduplication.
2023-03-20 20:33:22 +00:00
behzad nouri 5d9aba5548
increases retransmit-stage deduper capacity and reset-cycle (#30758)
For duplicate block detection, for each (slot, shred-index, shred-type)
we need to allow 2 different shreds to be retransmitted.
The commit implements this using two bloom-filter dedupers:
* Shreds are deduplicated using the 1st deduper.
* If a shred is not a duplicate, then we check if:
      (slot, shred-index, shred-type, k)
  is not a duplicate for either k = 0  or k = 1 using the 2nd deduper,
  and if so then the shred is retransmitted.

This allows to achieve larger capactiy compared to current LRU-cache.
2023-03-20 20:32:23 +00:00
behzad nouri 9ad77485ce
generalizes deduper to work with any hashable type (#30753)
generalizes Deduper to work with any hashable type

Current Deduper is hard-coded only for Packet type. In order to use
Deduper in retransmit-stage, we need to dedup types other than Packet.
The commit generalizes Deduper to any hashable type.
2023-03-20 18:04:46 +00:00
behzad nouri 4de59881b7
filters out merkle shreds until feature activation (#30769)
In order to maintain backward compatibility, the commit reworks merkle
shreds feature gate to off by default until the feature activation.
2023-03-20 15:44:00 +00:00
behzad nouri 4b595ebaaf
adds metrics tracking deduper saturations (#30779) 2023-03-20 15:33:36 +00:00
bji d77f0a22c7
Add new vote state version that replaces Lockout with LandedVote to a… (#29524)
* Add new vote state version that replaces Lockout with LandedVote to allow vote latency to be tracked in a future change. Includes a feature to be enabled which will when enabled cause the vote state to be written in the new form.

* Update feature set key to one owned by ashwin

---------

Co-authored-by: Ashwin Sekar <ashwin@solana.com>
2023-03-20 08:31:46 -06:00
behzad nouri 46614c0e9f
removes false_positive_rate field from Deduper (#30788)
removes the false_positive_rate field from the Deduper

Deduper.false_positive_rate field is misleading because it is not
enforced until maybe_reset is called. But then maybe_reset can be
invoked with an explicit argument.
2023-03-20 13:16:52 +00:00
Brooks cd7fe76744
Removes writing BankIncrementalSnapshotPersistence in AccountsHashVerifier (#30792) 2023-03-19 21:45:13 -04:00