Commit Graph

125 Commits

Author SHA1 Message Date
Andrew Fitzgerald e860019687
TransactionScheduler: Pipe BlockProductionMethod (#33217) 2023-09-18 10:05:27 -07:00
steviez 20fc3a5ded
Remove improper &Arc<Blockstore> instances (#32698)
Update to either &Blockstore if the function just needs a ref, or
Arc<Blockstore> if the function needs to hang onto a copy.
2023-08-03 15:10:25 -06:00
behzad nouri a3ada9c5ea
separates out turbine QUIC from TPU implementation (#32368)
* separates out turbine QUIC from TPU implementation

Turbine being tied to QUIC implementation for TPU hinders development
and makes it hard to optimize QUIC specifically for turbine.
The commit separates out turbine QUIC from TPU implementation.

* Update core/src/validator.rs

Co-authored-by: Jon Cinque <me@jonc.dev>

* Update turbine/src/retransmit_stage.rs

Co-authored-by: Jon Cinque <me@jonc.dev>

---------

Co-authored-by: Jon Cinque <me@jonc.dev>
2023-07-12 14:15:28 +00:00
behzad nouri f6e039b0b3
moves turbine to a separate crate out of solana/core (#32226) 2023-06-22 16:22:11 +00:00
behzad nouri ec0001ef85
adds code-path broadcasting shreds using QUIC (#31610)
adds quic connection cache to turbine

Working towards migrating turbine to QUIC.
2023-06-12 22:58:27 +00:00
Tyera 841609959c
Send messages to EntryNotifierService in Tpu, simpler (#31962)
* Add TpuEntryNotifier to send EntryNotifications from Tpu

* Optionally run TpuEntryNotifier to send out EntrySummarys alongside BroadcastStage messages

* Track entry index in TpuEntryNotifier

* Allow for leader slots that switch forks

* Exit if broadcast send fails
2023-06-06 00:19:17 +00:00
Illia Bobyr 4353ac6797
Pass Arc<AtomicBool> by value, not by reference. (#31916)
`Arc` is already a reference internally, so it does not seem to be
beneficial to pass a reference to it.  Just adds an extra layer of
indirection.

Functions that need to be able to increment `Arc` reference count need
to take `Arc<AtomicBool>`, but those that just want to read the
`AtomicBool` value can accept `&AtomicBool`, making them a bit more
generic.

This change focuses specifically on `Arc<AtomicBool>`.  There are other
uses of `&Arc<T>` in the code base that could be converted in a similar
manner.  But it would make the change even larger.
2023-06-01 17:25:48 -07:00
behzad nouri f1ebc5b5c3
separates out quic streamer connection stats from different servers (#31797) 2023-05-25 16:54:24 +00:00
Brennan a9b19f5b14
Add replay invalidator scaffolding upstream (#31567)
* Add replay invalidator scaffolding upstream
2023-05-16 13:08:39 -07:00
behzad nouri 4e34abbf3d
specifies protocol in contact-info get-socket api (#31602) 2023-05-12 16:16:20 +00:00
Tyera 3f70ddb2c5
Add entry notification service for geyser (#31290)
* Move entry_notifier_interface

* Add EntryNotifierService

* Use descriptive struct in sender/receiver

* Optionally initialize EntryNotifierService in validator

* Plumb EntryNotfierSender into Tvu, blockstore_processor

* Plumb EntryNotfierSender into Tpu

* Only return one option when constructing EntryNotifierService
2023-05-10 17:20:51 -06:00
behzad nouri ce21a58b65
reworks streamer::StakedNodes (#31082)
{min,max}_stake are computed but never assigned:
https://github.com/solana-labs/solana/blob/4564bcdc1/core/src/staked_nodes_updater_service.rs#L54-L57

The updater code is also inefficient and verbose.
2023-04-10 17:07:40 +00:00
behzad nouri 4d0abebe0e
removes Packet Meta.sender_stake and find_packet_sender_stake_stage (#31077)
Packet Meta.sender_stake is unused since
https://github.com/solana-labs/solana/pull/26512
removed sender_stake from banking-stage buffer prioritization.
2023-04-06 21:33:43 +00:00
behzad nouri ff9a42a354
uses Duration type instead of untyped ..._ms: u64 (#30971) 2023-03-31 15:42:49 +00:00
Tao Zhu 52e63e2ffa
Allow banking_stage to update prioritization_fee_cache (#30853)
* Allow banking_stage to update prioritization_fee_cache

* Update core/src/banking_stage/committer.rs

Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>

* move use to top

---------

Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>
2023-03-24 00:05:54 +00:00
ryleung-solana 0ed9f62602
Quic server batching (#30330) 2023-03-16 21:50:57 +08:00
behzad nouri ded457cd73
embeds the new gossip ContactInfo in ClusterInfo (#30022)
Working towards replacing the legacy gossip contact-info with the new
one, the commit updates the respective field in gossip cluster-info.
2023-02-10 20:07:45 +00:00
Ryo Onodera 40bbf99c74
Add fully-reproducible online tracer for banking (#29196)
* Add fully-reproducible online tracer for banking

* Don't use eprintln!()...

* Update programs/sbf/Cargo.lock...

* Remove meaningless assert_eq

* Group test-only code under aptly named mod

* Remove needless overflow handling in receive_until

* Delay stat aggregation as it's possible now

* Use Cow to avoid needless heap allocs

* Properly consume metrics action as soon as hold

* Trace UnprocessedTransactionStorage::len() instead

* Loosen joining api over type safety for replaystage

* Introce hash event to override these when simulating

* Use serde_with/serde_as instead of hacky workaround

* Update another Cargo.lock...

* Add detailed comment for Packet::buffer serialize

* Rename sender_overhead_minimized_receiver_loop()

* Use type interference for TraceError

* Another minor rename

* Retire now useless ForEach to simplify code

* Use type alias as much as possible

* Properly translate and propagate tracing errors

* Clarify --enable-banking-trace with better naming

* Consider unclean (signal-based) node restarts..

* Tweak logging and cli

* Remove Bank events as it's not needed anymore

* Make tpu own banking tracer thread

* Reduce diff a bit..

* Use latest serde_with

* Finally use the published rolling-file crate

* Make test code change more consistent

* Revive dead and non-terminating test code path...

* Dispose batches early now that possible

* Split off thread handle very early at ::new()

* Tweak message for TooSmallDirByteLimitl

* Remove too much of indirection

* Remove needless pub from ::channel()

* Clarify test comments

* Avoid needless event creation if tracer is disabled

* Write tests around file rotation and spill-over

* Remove unneeded PathBuf::clone()s...

* Introduce inner struct instead of tuple...

* Remove unused enum BankStatus...

* Avoid .unwrap() for the case of disabled tracer...
2023-01-25 21:54:38 +09:00
Ryo Onodera 4973fe18f1
Rename banking stage packet receivers consistently (#29752)
Rename banking stage batch receivers consistently
2023-01-19 10:04:55 +09:00
Lijun Wang ecea802fe6
Bidirectional quic communication support (#29155)
* Support bi-directional quic communication, use the same endpoint for the quic server and client
This is needed for supporting using quic for repair

* Added comments on the bi-directional communication tests

* Removed some debug logs

* clippy issue
2022-12-09 10:59:43 -08:00
Maximilian Schneider c8b0c3ede9
Update cost model to use requested_cu instead of estimated cu #27608 (#28281)
* Update cost model to use requested_cu instead of estimated cu #27608

* remove CostUpdate and CostModel from replay/tvu

* revive cost update service to send cost tracker stats

* CostModel is now static

* remove unused package

Co-authored-by: Tao Zhu <tao@solana.com>
2022-11-22 11:55:56 -06:00
Tyera c32377b5af
Split out quic- and udp-client definitions (#28762)
* Move ConnectionCache back to solana-client, and duplicate ThinClient, TpuClient there

* Dedupe thin_client modules

* Dedupe tpu_client modules

* Move TpuClient to TpuConnectionCache

* Move ThinClient to TpuConnectionCache

* Move TpuConnection and quic/udp trait implementations back to solana-client

* Remove enum_dispatch from solana-tpu-client

* Move udp-client to its own crate

* Move quic-client to its own crate
2022-11-18 12:21:45 -07:00
Lijun Wang f156bc12ca
Enforce stream receive timeout (#28513)
In the quic server handle_connection, when we timed out in receiving the chunks, we loop forever to wait for the chunk. If the client never provide another chunk, the server can hopelessly wait for that chunk and wasting server resources. Instead WAIT_FOR_CHUNK_TIMEOUT_MS is introduced to bound this to 10 seconds at maximum. The stream will be dropped if it times out.
2022-11-02 10:09:32 -07:00
Lijun Wang 7f223dc582
Added option to turn on UDP for TPU transaction and make UDP based TPU off by default (#27462)
--tpu-enable-udp is introduced. And when this is on, the transaction receive and transaction forward is enabled using udp.

Except for a few tests which was hard-coded sending transactions using udp, most tests are being done with udp based tpu disabled.
2022-09-07 13:19:14 -07:00
Tyera Eulberg b8b3d723da
Use new client crates (#27360)
* Update ancillary cli crates

* Update cli

* Update command-line tools

* Update rpc, etc

* Update client-test

* Update core, validator

* Update local-cluster
2022-08-24 10:47:02 -06:00
Michael Vines 3f4731b37f Standardize thread names
Tenets:
1. Limit thread names to 15 characters
2. Prefix all Solana-controlled threads with "sol"
3. Use Camel case. It's more character dense than Snake or Kebab case
2022-08-20 07:49:39 -07:00
Will Hickey dba2fd5a16
Enable QUIC client by default. Add arg to disable QUIC client. (Forward port #26927) (#27194)
Enable QUIC client by default. Add arg to disable QUIC client.

* Enable QUIC client by default. Add arg to disable QUIC client.
* Deprecate --disable-quic-servers arg
* Add #[ignore] annotation to failing tests
2022-08-19 09:15:15 -05:00
janlegner fc6cee9c06
allow staked nodes weight override (#26870)
* Allowed staked nodes weight override (#26407)

* Allowed staked nodes weight override, passing only HashMap over to core module

Co-authored-by: Ondra Chaloupka <chalda@chainkeepers.io>
2022-08-11 14:34:04 -07:00
Pankaj Garg fb922f613c
Compute maximum parallel QUIC streams using client stake (#26802)
* Compute maximum parallel QUIC streams using client stake

* clippy fixes

* Add unit test
2022-07-29 08:44:24 -07:00
Trent Nelson 2ee19f536a Revert "Revert "core: disable quic servers on mainnet-beta" (#26216)"
This reverts commit 4a7fb2a808.
2022-07-22 15:45:10 -06:00
sakridge 4a7fb2a808
Revert "core: disable quic servers on mainnet-beta" (#26216)
Enable QUIC server
2022-07-20 20:37:24 +02:00
Pankaj Garg 49a112ae74
Use pubkey of peer for active QUIC connection table (#26597)
* Use pubkey of peer for active QUIC connection table

* clippy

* update code
2022-07-13 09:59:01 -07:00
Nicholas Clarke ee0a40937e
Add validator argument log_messages_bytes_limit to change log truncation limit.
Add new cli argument log_messages_bytes_limit to solana-validator to control how long program logs can be before truncation
2022-07-11 10:53:18 -05:00
Tao Zhu c1d89ad749
forward packets by prioritization in desc order (#25406)
- Forward packets by prioritization in desc order
- Add support of cost-tracking by transaction requested compute units
- Hook up account buckets to forwarder
- Add metrics for forwardable batches count
- Remove redundant invalid packets filtering at end of slot since forwarder will do the same when batch forwardable packets
- Add bench test for forwarding
2022-07-05 23:24:58 -05:00
behzad nouri 61f0a7d9c3
replaces Mutex<PohRecorder> with RwLock<PohRecorder> (#26370)
Mutex causes superfluous lock contention when a read-only reference suffices.
2022-07-05 14:29:44 +00:00
HaoranYi b5d0c7b468
Revert "tvu and tpu timeout on joining its microservices (#24111)" (#26132)
This reverts commit e105547c14.
2022-06-22 10:57:46 -05:00
Lijun Wang 61946a49c3
Weight concurrent streams by stake (#25993)
Weight concurrent streams by stake for staked nodes
Ported changes from #25056 after address merge conflicts and some refactoring
2022-06-21 12:06:44 -07:00
Trent Nelson a5f290a66f core: disable quic servers on mainnet-beta 2022-06-17 20:04:05 -06:00
behzad nouri fe3c1d3d49
removes erroneous uses of &Arc<...> from broadcast-stage (#25962) 2022-06-15 13:44:24 +00:00
Jon Cinque 79a8ecd0ac
client: Remove static connection cache, plumb it instead (#25667)
* client: Remove static connection cache, plumb it instead

* Add TpuClient::new_with_connection_cache to not break downstream

* Refactor get_connection and RwLock into ConnectionCache

* Fix merge conflicts from new async TpuClient

* Remove `ConnectionCache::set_use_quic`

* Move DEFAULT_TPU_USE_QUIC to client, use ConnectionCache::default()
2022-06-08 13:57:12 +02:00
Pankaj Garg 1c2ae470c5
Fix forwarding of transactions over QUIC (#25674)
* Spawn QUIC server to receive forwarded txs

* Update validator port range

* forward votes using UDP

* no forwarding from unstaked nodes

* forwarding stats in banking stage

* fix test builds

* fix lifetime of forward sender
2022-06-02 11:14:58 -07:00
carllin 9651cdad99
Refactor Sigverify trait (#25359) 2022-05-24 16:01:41 -05:00
Brennan Watt 2fdc850176
Use Shared IP to Stake Map (#25377)
* Find packet sender stake stage use shared IP to stake map
2022-05-20 12:51:07 -07:00
sakridge 3d96a1ab76
Block packets in vote-only mode (#24906) 2022-05-14 17:53:37 +02:00
Christian Kamm cb6cd5d60f
FindPacketSenderStake: Improve metrics (#24971)
- separate names for vote and non-vote thread
- time unit postfixes (one is in ns!)
2022-05-06 21:16:13 +02:00
sakridge 5a430c15e2
Separate sigverify metrics for each verifier (#24744) 2022-04-28 01:16:17 -07:00
sakridge d71986cecf
Separate staked and un-staked on quic tpu port (#24339) 2022-04-16 10:54:22 +02:00
HaoranYi e105547c14
tvu and tpu timeout on joining its microservices (#24111)
* panic when test timeout

* nonblocking send when when droping banks

* debug log

* timeout for tvu

* unused varaible

* timeout for tpu

* Revert "debug log"

This reverts commit da780a3301a51d7c496141a85fcd35014fe6dff5.

* add timeout const

* fix typo

* Revert "nonblocking send when when droping banks".
I will create another pull request for this.

This reverts commit 088c98ec0facf825b5eca058fb860deba6d28888.

* Update core/src/tpu.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/tpu.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/tvu.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/tvu.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/validator.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
2022-04-07 20:20:13 -05:00
Tao Zhu c478fe2047 add timing metrics, some renaming 2022-03-17 19:31:28 -05:00
Tao Zhu fd515097d8 leader qos part 2: add stage to find sender stake, set to packet meta 2022-03-17 19:31:28 -05:00