Commit Graph

265 Commits

Author SHA1 Message Date
ryleung-solana f6c22e9796 Make the quic server connection table use an async lock, reducing thrashing (#293)
Make the quic server connection table use an async lock, reducing lock contention
2024-03-20 16:25:25 -05:00
steviez 4753dcae71 Rename and uniquify QUIC thread names (#28)
When viewing in various tools such as gdb and perf, it is not easy to
distinguish which threads are serving which function (TPU or TPU FWD)
2024-03-09 13:23:06 -06:00
steviez 7d6f1d5911
Give streamer::receiver() threads unique names (#35369)
The name was previously hard-coded to solReceiver. The use of the same
name makes it hard to figure out which thread is which when these
threads are handling many services (Gossip, Tvu, etc).
2024-03-01 13:36:08 -06:00
dependabot[bot] 7c59786f10
build(deps): bump indexmap from 2.1.0 to 2.2.2 (#35125)
* build(deps): bump indexmap from 2.1.0 to 2.2.2

Bumps [indexmap](https://github.com/indexmap-rs/indexmap) from 2.1.0 to 2.2.2.
- [Changelog](https://github.com/indexmap-rs/indexmap/blob/master/RELEASES.md)
- [Commits](https://github.com/indexmap-rs/indexmap/compare/2.1.0...2.2.2)

---
updated-dependencies:
- dependency-name: indexmap
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

* call swap_remove_entry directly

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: yihau <yihau.chen@icloud.com>
2024-02-07 19:02:20 +00:00
Lijun Wang 8fde8d26c7
don't sign X.509 certs (#34896)
This get rid of 3rd party components rcgen in the path of private key access to make the code more secure.
2024-01-28 16:17:46 -08:00
Pankaj Garg 9edf65b877
Do not reserve QUIC stream capacity for unstaked client on forward port (#34779) 2024-01-16 16:35:29 -08:00
Pankaj Garg f92275bcaa
Fix determination of staked QUIC connections (#34760)
* Fix determination of staked QUIC connections

* address review comments

* review comments

* treat connections with zero stake as unstaked
2024-01-13 18:38:31 -08:00
Pankaj Garg 22fcffeea8
Move EMA and stream throttling code to a new file (#34759) 2024-01-11 16:54:54 -08:00
Pankaj Garg 904700cc56
Use EMA to compute QUIC streamer load for staked connections (#34586)
* Use EMA to compute QUIC streamer load for staked connections

* change min load to 25% of max load

* reduce max PPS from 500K to 250K

* update ema_function to account for missing intervals

* replace f64 math with u128

* track streams across all connections for a peer

* u128  -> u64

* replace ' as ' type conversion to from and try_from

* add counter for u64 overflow

* reset recent stream load on ema interval

* do not use same counter for unstaked connections from a peer IP
2024-01-11 10:05:38 -08:00
Lijun Wang 1a001751dd
add metrics on throttled streams (#34579) 2023-12-22 16:49:01 -08:00
Pankaj Garg 6bbd3661e1
Throttle unstaked quic streams for a given connection (#34562)
* Throttle unstaked quic streams for a given connection

* Fix interval duration check

* move wait to handle_chunk

* set max unistreams to 0

* drop new streams

* cleanup

* some more cleanup

* fix tests

* update test and stop code

* fix bench-tps
2023-12-21 18:47:52 -08:00
ryleung-solana 132c910f81
Quic update identity (#33865)
Update the Quic transport layer keypair and identity when the Validator's identity keypair is updated
2023-12-08 14:53:19 +08:00
Jeff Biseda 3f805ad06d
improve batch_send error handling (#33936) 2023-10-31 23:39:26 -07:00
Alexander Meißner 9e703f85de
Upgrades Rust to 1.72.0 & nightly-2023-08-25 (#32961)
* allow pedantic invalid cast lint

* allow lint with false-positive triggered by `test-case` crate

* nightly `fmt` correction

* adapt to rust layout changes

* remove dubious test

* Use transmute instead of pointer cast and de/ref when check_aligned is false.

* Renames clippy::integer_arithmetic to clippy::arithmetic_side_effects.

* bump rust nightly to 2023-08-25

* Upgrades Rust to 1.72.0

---------

Co-authored-by: Trent Nelson <trent@solana.com>
2023-09-01 07:26:13 +00:00
behzad nouri 4ec5ea6f7b
replaces assert!(matches!(...)) with assert_matches!(...) (#33068)
assert_matches!(...) provides more informative error message when it
fails and it is part of nightly rust:
https://doc.rust-lang.org/std/assert_matches/macro.assert_matches.html
2023-08-30 13:48:27 -04:00
Trent Nelson b8dc5daedb
preliminaries for bumping nightly to 2023-08-25 (#33047)
* remove unnecessary hashes around raw string literals

* remove unncessary literal `unwrap()`s

* remove panicking `unwrap()`

* remove unnecessary `unwrap()`

* use `[]` instead of `vec![]` where applicable

* remove (more) unnecessary explicit `into_iter()` calls

* remove redundant pattern matching

* don't cast to same type and constness

* do not `cfg(any(...` a single item

* remove needless pass by `&mut`

* prefer `or_default()` to `or_insert_with(T::default())`

* `filter_map()` better written as `filter()`

* incorrect `PartialOrd` impl on `Ord` type

* replace "slow zero-filled `Vec` initializations"

* remove redundant local bindings

* add required lifetime to associated constant
2023-08-29 23:05:35 +00:00
Jon Cinque 0fe902ced7
Bump rand to 0.8, rand_chacha to 0.3, getrandom to 0.2 (#32871)
* sdk: Add concurrent support for rand 0.7 and 0.8

* Update rand, rand_chacha, and getrandom versions

* Run command to replace `gen_range`

Run `git grep -l gen_range | xargs sed -i'' -e 's/gen_range(\(\S*\), /gen_range(\1../'

* sdk: Fix users of older `gen_range`

* Replace `hash::new_rand` with `hash::new_with_thread_rng`

Run:
```
git grep -l hash::new_rand | xargs sed -i'' -e 's/hash::new_rand([^)]*/hash::new_with_thread_rng(/'
```

* perf: Use `Keypair::new()` instead of `generate`

* Use older rand version in zk-token-sdk

* program-runtime: Inline random key generation

* bloom: Fix clippy warnings in tests

* streamer: Scope rng usage correctly

* perf: Fix clippy warning

* accounts-db: Map to char to generate a random string

* Remove `from_secret_key_bytes`, it's just `keypair_from_seed`

* ledger: Generate keypairs by hand

* ed25519-tests: Use new rand

* runtime: Use new rand in all tests

* gossip: Clean up clippy and inline keypair generators

* core: Inline keypair generation for tests

* Push sbf lockfile change

* sdk: Sort dependencies correctly

* Remove `hash::new_with_thread_rng`, use `Hash::new_unique()`

* Use Keypair::new where chacha isn't used

* sdk: Fix build by marking rand 0.7 optional

* Hardcode secret key length, add static assertion

* Unify `getrandom` crate usage to fix linking errors

* bloom: Fix tests that require a random hash

* Remove some dependencies, try to unify others

* Remove unnecessary uses of rand and rand_core

* Update lockfiles

* Add back some dependencies to reduce rebuilds

* Increase max rebuilds from 14 to 15

* frozen-abi: Remove `getrandom`

* Bump rebuilds to 17

* Remove getrandom from zk-token-proof
2023-08-21 19:11:21 +02:00
Lijun Wang b44c9bca89
Reduce max staked streams count to avoid fragmentations (#32771)
Reduce max staked concurrent streams to 512 from 2048.
2023-08-15 12:02:58 -07:00
steviez 0dd4c208e6
Remove redundant inc_new_counter for num received packets (#32664)
The same value is reported as a field in StreamerReceiveStats in
/streamer/src/streamer.rs
2023-07-31 09:53:17 -06:00
behzad nouri 868e086d75
upgrades quinn and rustls crates (#32499) 2023-07-14 17:30:57 +00:00
behzad nouri d54b6204be
removes instances of clippy::manual_let_else (#32417) 2023-07-09 21:41:36 +00:00
behzad nouri 0da01270ef
removes redundant recycler clones (#32401) 2023-07-06 18:25:20 +00:00
behzad nouri 5a80dc0d73
adds QUIC endpoint specific for turbine connections (#32294)
Working towards separating out turbine QUIC from TPU.
2023-07-03 18:57:18 +00:00
Lijun Wang 689ca503e2
Remove a unnecessary sleep in run server (#32216)
remove sleep; and handle initializing connection as soon as available
2023-06-22 15:18:05 -07:00
ryleung-solana 36222a44d7
Use QUIC Retry packets during handshake (#31802)
Have the Quic server send a Retry packet to verify client control of the source IP
2023-06-06 14:23:23 -07:00
Illia Bobyr 4353ac6797
Pass Arc<AtomicBool> by value, not by reference. (#31916)
`Arc` is already a reference internally, so it does not seem to be
beneficial to pass a reference to it.  Just adds an extra layer of
indirection.

Functions that need to be able to increment `Arc` reference count need
to take `Arc<AtomicBool>`, but those that just want to read the
`AtomicBool` value can accept `&AtomicBool`, making them a bit more
generic.

This change focuses specifically on `Arc<AtomicBool>`.  There are other
uses of `&Arc<T>` in the code base that could be converted in a similar
manner.  But it would make the change even larger.
2023-06-01 17:25:48 -07:00
Lijun Wang 0426a2d96e
Flkay quic test in check_block_multiple_connections (#31871)
Flkay quic test -- put the test code in else condition
2023-05-30 12:43:07 -07:00
behzad nouri f1ebc5b5c3
separates out quic streamer connection stats from different servers (#31797) 2023-05-25 16:54:24 +00:00
Lijun Wang a8e2b82e38
Expect errors when opening 2nd stream due to connection limit (#31706) 2023-05-19 08:24:52 -07:00
behzad nouri cb65a785bc
makes sockets in LegacyContactInfo private (#31248)
Working towards LegacyContactInfo => ContactInfo migration, the commit
hides some implementation details of LegacyContactInfo and expands API
parity with the new ContactInfo.
2023-04-21 15:39:16 +00:00
behzad nouri 8556a40ee0
removes duplicate connection-table arguments (#31172)
prune_unstaked_connections_and_add_new_connection unnecessarily receives both
MutexGuard<ConnectionTable> and Arc<Mutex<ConnectionTable>>.
2023-04-12 20:39:29 +00:00
behzad nouri 34da001cda
passes through concrete QUIC connection errors up the call stack (#31168) 2023-04-12 19:53:25 +00:00
behzad nouri ce21a58b65
reworks streamer::StakedNodes (#31082)
{min,max}_stake are computed but never assigned:
https://github.com/solana-labs/solana/blob/4564bcdc1/core/src/staked_nodes_updater_service.rs#L54-L57

The updater code is also inefficient and verbose.
2023-04-10 17:07:40 +00:00
behzad nouri 466a9a2449
removes ip_stake_map field from streamer::StakedNodes (#31078) 2023-04-07 13:27:29 +00:00
behzad nouri 4d0abebe0e
removes Packet Meta.sender_stake and find_packet_sender_stake_stage (#31077)
Packet Meta.sender_stake is unused since
https://github.com/solana-labs/solana/pull/26512
removed sender_stake from banking-stage buffer prioritization.
2023-04-06 21:33:43 +00:00
sakridge e575650d47
Add connection error metrics (#31049) 2023-04-05 16:40:31 +02:00
behzad nouri 3cb22458f8
generalizes sample-size in quic::ConnectionTable::prune_random (#31011)
Even if there are many connections with stake less than the
threshold_stake, prune_random might still reject if both randomly
sampled connections have stake bigger than the threshold. A bigger
sample-size will make this less likely (at the cost of more
computations):
https://github.com/solana-labs/solana/blob/2cbd5d6c9/streamer/src/nonblocking/quic.rs#L958-L985

In order to benchmark for an optimal sample-size, the commit generalizes
the sample-size and makes it configurable.
2023-04-03 13:33:12 +00:00
behzad nouri 8a96b91515
simplifies quic::ConnectionTable::prune_oldest (#30995) 2023-03-31 18:24:37 +00:00
behzad nouri ff9a42a354
uses Duration type instead of untyped ..._ms: u64 (#30971) 2023-03-31 15:42:49 +00:00
ryleung-solana 721d8cb0ac
Add logging of the number of chunks handled by the quic server (#30954) 2023-03-31 01:51:32 +08:00
behzad nouri 4ed87112bc
simplifies streamer::nonblocking::quic::get_connection_stake (#30969) 2023-03-29 21:08:21 +00:00
ryleung-solana 92189d82b4
Quic server log data rate (#30892)
* Add more statistics to better track incoming data rate to the Quic server
2023-03-28 15:33:40 +08:00
ryleung-solana 0ed9f62602
Quic server batching (#30330) 2023-03-16 21:50:57 +08:00
Yihau Chen df3ef111f7
chore: workspace inheritance (#29893)
* introduce workspace.package

* introduce workspace.dependencies

* read version from root cargo.toml

* pass check when version = { workspace = true }

* don't bump version when version = { workspace = true }

* including workspace Cargo.toml when bump version

* programs/sbf use workspace inheritance

* fix increasing cargo version ignore program/sbf/Cargo.toml
2023-02-23 22:01:54 +08:00
Michael Vines 5136ed3448
Update homepage value for all crates (#30444) 2023-02-23 02:20:18 +00:00
dependabot[bot] 66dd9432f5
Bump futures-util from 0.3.25 to 0.3.26 (#30240)
* Bump futures-util from 0.3.25 to 0.3.26

Bumps [futures-util](https://github.com/rust-lang/futures-rs) from 0.3.25 to 0.3.26.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.25...0.3.26)

---
updated-dependencies:
- dependency-name: futures-util
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2023-02-10 13:46:16 -07:00
behzad nouri ded457cd73
embeds the new gossip ContactInfo in ClusterInfo (#30022)
Working towards replacing the legacy gossip contact-info with the new
one, the commit updates the respective field in gossip cluster-info.
2023-02-10 20:07:45 +00:00
Brennan b29a50a55e
Exclude loopback from valid IP addresses (#29992)
Excludes loopback IP addresses from being valid IP addresses unless private addresses are explicitly allowed
2023-02-01 11:16:15 -08:00
Will Hickey 04a6a631bc
Bump version to v1.16 (#30028) 2023-01-31 17:48:33 -06:00
joeaba a12bf8c003
Update maintainers references (#29997)
* update maintainers references

* chore: update maintainers reference
2023-01-31 08:07:13 -05:00