Commit Graph

3671 Commits

Author SHA1 Message Date
Tao Zhu 49f44f5ded
Refactor pass feature status to deserialized packet via packet meta (#31549)
Add a flag to packet, set its value by packet_deserializer when received by banking_stage with working_bank
2023-05-11 09:31:05 -05:00
Tyera 3f70ddb2c5
Add entry notification service for geyser (#31290)
* Move entry_notifier_interface

* Add EntryNotifierService

* Use descriptive struct in sender/receiver

* Optionally initialize EntryNotifierService in validator

* Plumb EntryNotfierSender into Tvu, blockstore_processor

* Plumb EntryNotfierSender into Tpu

* Only return one option when constructing EntryNotifierService
2023-05-10 17:20:51 -06:00
Ashwin Sekar c900ef8290
refactor: combine SlotStateUpdate impls (#31578) 2023-05-10 15:12:39 -06:00
steviez 18a118b438
Condense banking_stage counters into existing datapoint (#31564)
Counters incur additional overhead in sending points to the MetricsAgent
over a crossbeam channel. Additionally, some of these counters would be
submitted by non-voting nodes which is just extra overhead and noise.

This change condenses several updates of a counter into a field of the
existing BankingStageStats metrics struct.
2023-05-10 15:44:42 -05:00
Tao Zhu fb7ba97afc
refactor: move test and bench only code from main code to where they are needed (#31550)
* refactor: move test and bench only code to specific location
* remove inactive bench test
2023-05-09 16:39:23 -05:00
Brooks 3bb2e3b546
Purges incomplete snapshot dirs at startup (#31555) 2023-05-09 14:08:12 -04:00
steviez 4300d84c68
Remove counters from ReplayStage (#31532)
replay_stage-voted_empty_bank has been converted into a datapoint that
now includes slot number. replay_stage-replay_transactions has been
removed altogether as we can get similar information on a per-slot basis
from replay-slot-stats metric.
2023-05-09 11:44:02 -05:00
behzad nouri 6a4a0418a6
removes hard-coded QUIC_PORT_OFFSET from connection-cache (#31541)
New ContactInfo has api identifying QUIC vs UDP ports; no need to hard-code
port-offset deep in connection-cache.
2023-05-09 13:46:17 +00:00
Brooks 6e342ded42
clippy: Removes redundant async blocks (#31526) 2023-05-09 09:35:38 -04:00
behzad nouri 8e638b785a
removes feature gate code sending votes to tpu-vote-port (#31529) 2023-05-08 18:12:35 +00:00
Tao Zhu 1f91a90a53
Refactor remove unnecessary parameter (#31520)
Refactor: remove unnecessary parameter from DeserializedPacket constructor
2023-05-07 10:56:24 -05:00
Brooks 775639c058
Revert "SnapshotPackagerService purges old bank snapshots (#31511)" (#31524)
This reverts commit a6c39ded8e.
2023-05-06 09:18:03 -04:00
Tao Zhu b19cc03c9a
Refactor: remove test only public function, update tests (#31518) 2023-05-05 17:22:09 -05:00
Brooks a6c39ded8e
SnapshotPackagerService purges old bank snapshots (#31511) 2023-05-05 17:22:48 -04:00
Brooks c5e071c7fe
Upgrades nightly Rust to 2023-03-04 (#31487) 2023-05-05 08:28:23 -04:00
HaoranYi 0f4293914c
remove unnecessary-struct-initialization (#31486)
* remove unnecessary-struct-initialization

* more  remove unnecessary-struct-initialization

---------

Co-authored-by: haoran <haoran.yi@solana.com>
2023-05-04 17:48:33 -05:00
Brooks ef7470f50c
Removes needless borrows (#31489) 2023-05-04 18:09:17 +00:00
Andrew Fitzgerald 886aea21cb
Internal structs for ThreadAware AccountRead/WriteLocks (#31431) 2023-05-04 09:38:34 -07:00
Andrew Fitzgerald 18cd4311af
remove counters in hotpath (#31398) 2023-05-04 09:36:28 -07:00
Jeff Biseda 19319d5b70
Rationalize (Slot, Hash) in repair by removing SlotHash type (#31470) 2023-05-03 14:03:05 -07:00
Lijun Wang 7cf50e60fc
Fixed missing Root notifications via geyser plugin framework (#31180)
* Fixed missing Root notifications via geyser plugin framework

* Renamed a variable

* fmt issue

* Do not try the loop if no subscribers.

* Addressing some feedback -- passing parent roots from replay_stage to avoid race conditions

* clippy issue

* Address some reviewing findings

* Addressed some feedback from Carl

* fix a clippy issue

* Added comments on optimistically_confirmed_bank_tracker module to explain the workflow

* Addressed Trent's review
2023-05-03 18:50:00 +08:00
Brooks 90e1b00fb5
Uses get_bank_snapshot_dir() in `core/tests/snapshot.rs` (#31434) 2023-05-01 22:57:59 +00:00
Xiang Zhu 0a2e897f16
Clean up the outdated SnapshotPackage snapshot_links field (#31360)
* Remove snapshot_links

* Change the function name from snapshot_dir to bank_snapshot_dir

* Format fix

* Fix test_concurrent_snapshot_packaging

* Fix clippy error

* Fix nits

* Fix nits 2nd try

* Use get_bank_snapshots_dir

* Use slot_dir

* Revert "Use get_bank_snapshots_dir" because get_bank_snapshots_dir is private to crate

This reverts commit 1ed9b3b2c8e84689a918beee7159f63c56500a96.
2023-05-01 11:24:59 -07:00
steviez 427ad7b5bd
Combine AccountsHashVerifier metrics (#31420)
It is more efficient to submit the metrics together, and there is no
reason for them to be separate.
2023-04-30 21:39:44 -04:00
Jeff Biseda b5bb5c6da1
filter invalid repair requests by size (#30951) 2023-04-28 16:57:15 -07:00
Brennan e79b84ea70
Rework tx sig verify batching (#31355)
* Rework tx sig verify batching to eliminate special casing and increase the packet limit for sigverify from 2k to 5k
2023-04-28 09:21:12 -07:00
behzad nouri aafcac27d8
removes pubkey from LegacyContactInfo public interface (#31375)
Working towards LegacyContactInfo => ContactInfo migration, the commit
adds more api parity between the two.
2023-04-28 12:05:15 +00:00
Ryo Onodera a30830d7a9
ci: treewide: deny used_underscore_binding (#31319)
* Enforce used_underscore_binding

* Fix all

* Work around for cfg()-ed code...

* ci....

* Make clipply fixes more pleasant

* Clone exit signal while intentionally shadowing

* Use more verbose code to avoid any #[allow(...)]s
2023-04-27 10:10:16 +09:00
Xiang Zhu f3e94ca73c
AHV processes the snapshot dirs in place (#30978)
* AHV processes the snapshot dirs in place

Let account pacakge use the snapshot dir, so AHV computes the accounts hash and turns the pre snapshot dir into a post snapshot dir

* fix status cache path to maintain the archive layout for the in-place snapshot dir archiving

* fix test_package_snapshots

* Fix test_concurrent_snapshot_packaging

* Remove debug change.

* Fix snapshot_links path

* change to borrow for bank_snapshots_dir

* Reverted changes in create_and_verify_snapshot

* Fix param errors

* Fix rebase errors

* Remove NOTE 1

* Remove unwrap

* Remove the variables to make it apparent taht snapshot_links is the bank_snapshots_dir

* Use soft link instead of hard link for snapshot and status cache

* After switching to soft symlinking, the src path should be absolute
2023-04-26 11:48:48 -07:00
Andrew Fitzgerald 2d91c52373
derive Debug for multi_iterator_scanner::ProcessingDecision (#31312) 2023-04-26 08:53:31 -07:00
Brooks 32fb15655e
Upgrades fs_extra to v1.3.0 (#31341) 2023-04-25 18:29:19 -04:00
steviez 758bc1ca75
Make ReplayStage panic before dumping repeated-repair-attempt slots (#31333)
When ReplayStage repeatedly fails to compute the correct for a block
after purging and repairing, it panics on the assumption that something
is very wrong and will require human intervention.

If this is the case, there is typically something to be debugged, and
having the slot available locally is valuable. This change does the
retry check that will panic before purging the failure slot.
2023-04-25 11:50:47 -05:00
Xiang Zhu 16f3dcd5d2
Update fn create_and_verify_snapshot (#31245)
* only 1 snapshot per archive in create_and_verify_snapshot

* Update create_and_verify_snapshot with the newer funtion calls

* Fix test_package_snapshots

* Remove account path access change

* Rename slot to num_snapshots

* Remove unncessary purge_old_bank_snapshots in test

* Update non-deterministic format comment

* Cleanup unnecessary hash calls

* Use get_accounts_hash

* Remove extra declaration

* Remove rehash

* Remove clean_accounts

* Revert "Cleanup unnecessary hash calls"

This reverts commit 06b1457462cf6d7acf62e0e5531633caf5d9fc58.

Removing unncessary hash calls should be done for create_and_verify_snapshot,
not bank_to_full_snapshot_archive

* Fix typo appenedvecs

* Remove bank_snapshots_dir after rebasing
2023-04-24 18:52:50 -07:00
behzad nouri 1b08d01a80
removes shred_version from LegacyContactInfo public interface (#31304)
Working towards LegacyContactInfo => ContactInfo migration, the commit
adds more api parity between the two.
2023-04-24 15:19:33 +00:00
behzad nouri a88024e295
removes wallclock from LegacyContactInfo public interface (#31303) 2023-04-22 20:18:39 +00:00
Jeff Biseda 3cdd59e55f
check for prior discard in shred_fetch_stage (#31293) 2023-04-21 11:43:10 -07:00
behzad nouri cb65a785bc
makes sockets in LegacyContactInfo private (#31248)
Working towards LegacyContactInfo => ContactInfo migration, the commit
hides some implementation details of LegacyContactInfo and expands API
parity with the new ContactInfo.
2023-04-21 15:39:16 +00:00
Andrew Fitzgerald 7a393e479d
Scheduler Messages (#30976) 2023-04-19 15:14:47 -07:00
Andrew Fitzgerald 10d637d2e6
PohRecorder take Arc not &Arc for blockstore (#31234) 2023-04-19 11:41:18 -07:00
steviez 377ba53a31
Fix bug where ReplayStage holds an Arc<Bank> for process lifetime (#31267)
* Fix bug where ReplayStage holds an Arc<Bank> for process lifetime

When ReplayStage::new() kicks off, it needs to do some setup with the
working bank prior to entering the main processing loop. This setup is
done before entering the main processing loop; however, a bug made it
such that an Arc<Bank> remained in scope after the processing loop had
been entered. The processing loop is only exited when the process exits,
so this means that Bank was being held for the lifetime of the process.
This is a waste of resources and prevents background cleanup.

* clippy
2023-04-19 18:12:34 +00:00
Xiang Zhu a5275f8839
Remove bank_snapshots_dir param (#31249)
* Remove bank_snapshots_dir param

* Remove outdated comment

* Revert "Remove outdated comment"

This reverts commit e4441432bec57edb0dc22c4bacf4d48ce26ed818.

* Handle parent() error

* Fix format error
2023-04-19 09:37:46 -07:00
Andrew Fitzgerald 748220c9d3
Forwarder: Add common setup for tests (#31232) 2023-04-19 09:08:13 -07:00
Brooks ca1bde3591
Use Arc instead of &Arc in AccountsBackgroundService::new (#31268) 2023-04-19 11:10:41 -04:00
Brooks 80b27f3cd9
Use Arc instead of &Arc in AccountsHashVerifier::new (#31269) 2023-04-19 11:10:08 -04:00
Brooks 1d14156832
Use Arc instead of &Arc in SnapshotPackagerService::new (#31270) 2023-04-19 11:09:49 -04:00
bji a45710838d
Add new vote state version that replaces Lockout with LandedVote to a… (#30831)
Add new vote state version that replaces Lockout with LandedVote to allow vote latency to be tracked in a future change.

Includes a feature to be enabled which will when enabled cause the vote state to be written in the new form.
2023-04-18 20:27:38 -07:00
Trent Nelson f34a6bcfce
runtime: transpose `VoteAccount::vote_state()` return to improve ergonomics (#31256) 2023-04-18 14:48:52 -06:00
Brennan 2164a50d00
Move BankIncrementalSnapshotPersistence (#31236)
* Move BankIncrementalSnapshotPersistence

* Update bank serialize ABI digest
2023-04-18 11:18:17 -07:00
Xiang Zhu e74bc4e2e7
Add a filter_by_type param for purge_old_bank_snapshots (#31191)
* Add a type_select param for purge_old_bank_snapshots

* Use flags to make the function calls more readable

* Remove the extra purge calls

* replace select_type with filter_by_type

* Add test

* Use matches

* Fix CI test on reference

* use match and call do_purge once

* Let bank_snapshots_dir be TempDir

* remove account_paths in the test

* replace bank with _bank

* Remove create_snapshot_dirs_for_tests, will take the lastest from master

* Fix merge errors
2023-04-17 16:16:41 -07:00
Xiang Zhu 5747290d51
Move reference-holding last_snapshot_storages from ABS to AHV (#31175)
* Let AHV hold and update last_snapshot_storages

* Clean up comment

* Move cloning after enqueued_time

* Minor positon change

* Remove  type last_snapshot_storages annotation
2023-04-14 14:38:44 -07:00