164c929b27
Cleanup QUIC single signed client cert code ( #29686 )
2023-01-12 15:24:02 -08:00
1d0a28c66a
Restrict QUIC to use single self signed client cert ( #29681 )
2023-01-12 10:05:56 -08:00
7c8b846344
Update quinn versions ( #29603 )
...
* chore: bump quinn-udp from 0.1.3 to 0.3.2
Bumps [quinn-udp](https://github.com/quinn-rs/quinn ) from 0.1.3 to 0.3.2.
- [Release notes](https://github.com/quinn-rs/quinn/releases )
- [Commits](https://github.com/quinn-rs/quinn/commits )
---
updated-dependencies:
- dependency-name: quinn-udp
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Try to use quinn 0.9.3 and quinn-proto 0.9.2
* Update streamer and client for quic to support qunn 0.9.3
* Update Cargo.lock
* Fixed unit test failure for quic tests
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-11 10:08:22 -08:00
dbf97daa78
Increase wait for chunk timeout in QUIC tests ( #29629 )
2023-01-11 07:16:41 -08:00
5c9beef498
fixes errors from clippy::useless_conversion ( #29534 )
...
https://rust-lang.github.io/rust-clippy/master/index.html#useless_conversion
2023-01-05 18:05:32 +00:00
d1cf4ced3d
quic test timeout fix ( #29260 )
...
Allow longer chunk receive timeout without impacting testing the stream exit condition for unit tests. The exit is periodically checked, we will break only when the total allowed chunk receive timed out. The start time is reset when a new chunk is received.
2022-12-16 14:26:04 -08:00
ecea802fe6
Bidirectional quic communication support ( #29155 )
...
* Support bi-directional quic communication, use the same endpoint for the quic server and client
This is needed for supporting using quic for repair
* Added comments on the bi-directional communication tests
* Removed some debug logs
* clippy issue
2022-12-09 10:59:43 -08:00
9524c9dbff
patches errors from clippy::uninlined_format_args
...
https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
2022-12-06 19:32:15 +00:00
b1340d77a2
sdk: Make Packet::meta private, use accessor functions ( #29092 )
...
sdk: Make packet meta private
2022-12-06 12:54:49 +01:00
16e2c5c0a2
Increase stream timeout to 1 sec to fix flaky tests ( #28983 )
2022-11-30 10:04:32 -08:00
516d27bc42
saturating max_concurrent_uni_streams and receive_window instead of overflow
2022-11-16 15:39:21 -06:00
ad24e37cc0
Limit the maximum staked streams to avoid excessive streams from staked nodes ( #27973 )
...
* Limit the maximum staked streams to avoid excessive streams from staked nodes.
* Fixed a clippy issue
2022-11-15 12:58:10 -08:00
8d34dfd881
Fixed flaky quic test ( #28789 )
2022-11-14 08:35:10 -08:00
d1ba42180d
clippy for rust 1.65.0 ( #28765 )
2022-11-09 19:39:38 +00:00
842fa993c8
Quic fix calculation of staked number of concurrent streams ( #28705 )
...
* Fix calculation of staked number of concurrent streams
2022-11-03 20:45:44 +08:00
f156bc12ca
Enforce stream receive timeout ( #28513 )
...
In the quic server handle_connection, when we timed out in receiving the chunks, we loop forever to wait for the chunk. If the client never provide another chunk, the server can hopelessly wait for that chunk and wasting server resources. Instead WAIT_FOR_CHUNK_TIMEOUT_MS is introduced to bound this to 10 seconds at maximum. The stream will be dropped if it times out.
2022-11-02 10:09:32 -07:00
8036f6f304
Address some comments from Ryan
2022-10-12 14:12:40 -07:00
56b9288f9e
Give better error code and reason for connection close
2022-10-12 14:12:40 -07:00
a8d5731e5b
Give the quic client and server thread the names
2022-10-06 16:55:21 -07:00
63c00e7f5e
move readable inside the loop of packets ( #27916 )
...
* Add recv_mmsg_exact function
* update tests
* address PR comments
2022-10-05 22:20:26 +02:00
9a57c64f21
patches clippy errors from new rust nightly release ( #27996 )
2022-09-22 22:23:03 +00:00
56de885dea
fix total_connections metric ( #27603 )
2022-09-06 18:36:55 +02:00
e4a7d01e10
Rust v1.63 ( #27303 )
...
* Upgrade to Rust v1.63.0
* Add nightly_clippy_allows
* Resolve some new clippy nightly lints
* Increase QUIC packets completion timeout
* Update quinn-udp crate
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-22 18:01:03 -07:00
3f4731b37f
Standardize thread names
...
Tenets:
1. Limit thread names to 15 characters
2. Prefix all Solana-controlled threads with "sol"
3. Use Camel case. It's more character dense than Snake or Kebab case
2022-08-20 07:49:39 -07:00
7573000d87
Revert "Rust v1.63.0 ( #27148 )" ( #27245 )
...
This reverts commit a2e7bdf50a
2022-08-19 09:19:44 +01:00
a2e7bdf50a
Rust v1.63.0 ( #27148 )
...
* Upgrade to Rust v1.63.0
* Add nightly_clippy_allows
* Resolve some new clippy nightly lints
* Increase QUIC packets completion timeout
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-17 15:48:33 -07:00
ddd660e2d3
Different staked vs unstaked chunks_received ( #27033 )
...
* Different staked vs unstaked chunks_received
* Suppress a clippy warning
2022-08-09 17:39:14 -07:00
a69470fd45
Set receive_window per quic connection ( #26936 )
...
This change sets the receive_window for non-staked node to 1 * PACKET_DATA_SIZE, and maps the staked nodes's connection's receive_window between 1.2 * PACKET_DATA_SIZE to 10 * PACKET_DATA_SIZE based on the stakes.
The changes is based on Quinn library change to support per connection receive_window tweak at the server side. quinn-rs/quinn#1393
2022-08-09 10:02:47 -07:00
3c15d26840
Refactor QUIC new connection handler function ( #26855 )
...
* Refactor QUIC new connection handler function
* cleanup setup_connection
* more cleanup
2022-08-05 11:24:07 -07:00
fb922f613c
Compute maximum parallel QUIC streams using client stake ( #26802 )
...
* Compute maximum parallel QUIC streams using client stake
* clippy fixes
* Add unit test
2022-07-29 08:44:24 -07:00
27866aeab4
Fix transaction chunking on QUIC batch send ( #26642 )
...
* Fix chunking of transaction at batch transmit via QUIC
* clippy fixes
2022-07-22 08:56:15 -07:00
9714cb3764
Remove some unchecked math from the quic chunk handling ( #26408 )
2022-07-16 17:03:31 +02:00
048b9f670b
Set solana-tpu ALPN protocol ID ( #26587 )
2022-07-13 20:55:13 +02:00
49a112ae74
Use pubkey of peer for active QUIC connection table ( #26597 )
...
* Use pubkey of peer for active QUIC connection table
* clippy
* update code
2022-07-13 09:59:01 -07:00
14ee87d793
Move QUIC TLS certificate code in its own file ( #26569 )
2022-07-12 13:34:37 -07:00
ea7448c568
Use client certs in QUIC to get peer's stake ( #26477 )
...
* Use client certs in QUIC to get peer's stake
* fixes to cert processing
* integrate the code
* clippy
* more cleanup
* sort cargo deps
* test fixes
* info -> debug
2022-07-11 18:06:40 +00:00
4772a93109
Quic handshake timeout ( #26306 )
...
* Implement timeout for the quic client and server handshake
2022-07-11 14:10:12 +08:00
94685e1222
Implement randomized pruning of QUIC connection from staked peers ( #26299 )
2022-06-30 17:56:15 -07:00
71087ac94f
Use correct QUIC connection table for removing the entry ( #26328 )
...
* Use correct QUIC connection table for removing the entry
* fix test
* address review comments
* address comments
2022-06-30 10:21:53 -07:00
e8ed7c1c46
Close QUIC connection before dropping the entry ( #26269 )
2022-06-28 13:46:56 -07:00
31d870fe1b
Do not unwrap the oldest IP in QUIC connectiont table ( #26272 )
2022-06-28 00:17:08 +00:00
2cc48a650b
Spawn for each stream ( #26086 )
2022-06-27 21:03:40 +02:00
974db28e4d
Increase the QUIC stream timeout duration to improve perf ( #26247 )
2022-06-27 17:01:39 +01:00
b26c79b77b
[quic]Decrement total_streams correctly ( #26158 )
2022-06-23 14:09:43 -07:00
17cf010ae6
Tune stream timeout and reading after timeouts ( #26133 )
...
* Tune stream timeout and reading after timeouts
* update sleep in test
2022-06-22 21:07:31 +00:00
d5dbfb67fd
QUIC stream timeouts if no data is received ( #26116 )
2022-06-21 18:56:47 -07:00
4ad32eb412
Perform QUIC handshake asynchronously to allow parallel handshakes ( #26043 )
...
* Perform QUIC handshake asynchronously to allow parallel handhskaes
* address comments
* update cargo.lock
* change sleep to 1ms
2022-06-21 23:22:11 +00:00
61946a49c3
Weight concurrent streams by stake ( #25993 )
...
Weight concurrent streams by stake for staked nodes
Ported changes from #25056 after address merge conflicts and some refactoring
2022-06-21 12:06:44 -07:00
84a1aae555
Fix incorrect total connection table size in the quic server when removing multiple connections with the same IP:Port ( #26073 )
...
* Fix incorrect total connection table size in the quic server when removing multiple connections with the same IP:Port
2022-06-21 15:13:35 +08:00
061dc5322a
streamer: Add nonblocking quic server ( #25806 )
2022-06-07 18:22:46 +02:00
5dbf7d8f91
removes raw indexing into packet data ( #25554 )
...
Packets are at the boundary of the system where, vast majority of the
time, they are received from an untrusted source. Raw indexing into the
data buffer can open attack vectors if the offsets are invalid.
Validating offsets beforehand is verbose and error prone.
The commit updates Packet::data() api to take a SliceIndex and always to
return an Option. The call-sites are so forced to explicitly handle the
case where the offsets are invalid.
2022-06-03 01:05:06 +00:00
1c2ae470c5
Fix forwarding of transactions over QUIC ( #25674 )
...
* Spawn QUIC server to receive forwarded txs
* Update validator port range
* forward votes using UDP
* no forwarding from unstaked nodes
* forwarding stats in banking stage
* fix test builds
* fix lifetime of forward sender
2022-06-02 11:14:58 -07:00
880684565c
limits read access into Packet data to Packet.meta.size ( #25484 )
...
Bytes past Packet.meta.size are not valid to read from.
The commit makes the buffer field private and instead provides two
methods:
* Packet::data() which returns an immutable reference to the underlying
buffer up to Packet.meta.size. The rest of the buffer is not valid to
read from.
* Packet::buffer_mut() which returns a mutable reference to the entirety
of the underlying buffer to write into. The caller is responsible to
update Packet.meta.size after writing to the buffer.
2022-05-25 16:52:54 +00:00
cad1c41ce2
Add Packet::deserialize_slice convenience method
2022-05-24 17:31:14 +08:00
ec7ca411dd
Make PacketBatch packets vector non-public ( #25413 )
...
Upcoming changes to PacketBatch to support variable sized packets will
modify the internals of PacketBatch. So, this change removes usage of
the internal packet struct and instead uses accessors (which are
currently just wrappers of Vector functions but will change down the
road).
2022-05-23 15:30:15 -05:00
c248fb3f51
renames Packet Meta::{,set_}addr methods to {,set_}socket_addr ( #25478 )
...
In order to distinguish between Meta.addr field which is an IpAddr and
the methods which refer to a SocketAddr.
2022-05-23 15:48:59 +00:00
b05c7d91ed
Fix derive_partial_eq_without_eq clippy lint
2022-05-22 22:22:21 -07:00
a56ce8283a
quic: Fix prune_oldest table size calculation ( #25431 )
2022-05-22 00:06:05 +02:00
773b2f23f4
Fix flakey streamer_send_test ( #25440 )
2022-05-22 01:09:26 +08:00
a5792885ca
streamer: Add nonblocking versions of sendmmsg / recvmmsg ( #25415 )
2022-05-20 22:40:40 +02:00
3d96a1ab76
Block packets in vote-only mode ( #24906 )
2022-05-14 17:53:37 +02:00
5be138826b
Add sender stake to quic packets ( #25054 )
2022-05-07 16:45:03 +02:00
7100f1c94b
Collect stats in streamer receiver and report fetch stage metrics ( #25010 )
2022-05-06 02:56:18 +08:00
0910d9a209
Increase worker thread limits in quic streamer runtime ( #24751 )
...
* Resurrect "Increase worker thread limits in quic streamer runtime (#24535 )""
* Fix test_quic_server_block_multiple_connections
* remove sleep
2022-04-27 20:50:08 +00:00
ded6ac6696
Swap quic counters with 0 while reporting ( #24754 )
2022-04-27 20:17:40 +00:00
2f62fb2031
Revert "Increase worker thread limits in quic streamer runtime ( #24535 )"
...
This reverts commit f244a2e141
2022-04-28 00:46:42 +08:00
f244a2e141
Increase worker thread limits in quic streamer runtime ( #24535 )
...
* Increase worker thread limits in quic streamer runtime
* fix Cargo.lock
* fix test, and use a constant for controlling thread count
2022-04-26 21:25:02 +00:00
564bd4b34b
Add more datapoints in QUIC streamer ( #24611 )
2022-04-25 10:12:11 -07:00
8d4e53d23c
chore: bump nix from 0.23.1 to 0.24.0 ( #24548 )
...
* chore: bump nix from 0.23.1 to 0.24.0
Bumps [nix](https://github.com/nix-rust/nix ) from 0.23.1 to 0.24.0.
- [Release notes](https://github.com/nix-rust/nix/releases )
- [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md )
- [Commits](https://github.com/nix-rust/nix/commits )
---
updated-dependencies:
- dependency-name: nix
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* allow deprecated
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2022-04-22 01:41:11 +00:00
5d0dcca1e7
Add some datapoints in quic for error conditions ( #24489 )
2022-04-19 13:05:02 -07:00
5c7060eaeb
fix test compilation error ( #24413 )
2022-04-16 14:41:33 +02:00
d71986cecf
Separate staked and un-staked on quic tpu port ( #24339 )
2022-04-16 10:54:22 +02:00
1b7d1f78de
Implement QUIC connection warmup service for future leaders ( #24054 )
...
* Increase connection timeouts
* Bump quic connection cache to 1024
* Use constant for quic connection timeout and add warm cache service
* Fixes to QUIC warmup service
* fix check failure
* fixes after rebase
* fix timeout test
Co-authored-by: Pankaj Garg <pankaj@solana.com>
2022-04-15 12:09:24 -07:00
82945ba973
Optimize TpuConnection and its implementations and refactor connection-cache to not use dyn in order to enable those changes ( #23877 )
2022-03-24 11:40:26 -04:00
976b138e76
Add tx weighting stage
2022-03-17 19:31:28 -05:00
17b00ad3a4
Add quic-client module ( #23166 )
...
* Add quic-client module to send transactions via quic, abstracted behind the TpuConnection trait (along with a legacy UDP implementation of TpuConnection) and change thin-client to use TpuConnection
2022-03-09 21:33:05 -05:00
7a9884c831
Quic limit connections ( #23283 )
...
* quic server limit connections
* bump per_ip
* Review comments
* Make the connections per port
2022-03-09 10:52:31 +01:00
d4292774c5
checks
2022-02-25 08:05:28 +00:00
70ebab2c82
Add rustfmt.toml and `cargo fmt` ( #23238 )
...
* fmt
* formatted
Co-authored-by: Lucas B <buffalu@jito.network>
2022-02-19 13:32:29 +08:00
c078ca3fb3
uses sendmmsg in streamer ( #23062 )
...
packet::send_to sends packets one by one:
https://github.com/solana-labs/solana/blob/9213fcb11/streamer/src/packet.rs#L63-L75
sendmmsg uses a single system call for multiple messages:
https://github.com/solana-labs/solana/blob/9213fcb11/streamer/src/sendmmsg.rs#L94
2022-02-11 13:36:50 +00:00
5a230f418d
Add quic port for accepting transactions ( #22753 )
...
using quinn library
streamer: Sign TLS cert with validator identity key
Handle multiple incoming chunks
2022-02-04 15:27:09 +01:00
6d5bbca630
Pacify clippy
2022-01-21 19:12:57 -08:00
8b66625c95
convert std::sync::mpsc to crossbeam_channel ( #22264 )
2022-01-11 02:44:46 -08:00
379feecae5
patches bug in recv_mmsg when npkts != nrecv
...
If recv_mmsg receives 2 packets where the first one is filtered out,
then it returns npkts == 1:
https://github.com/solana-labs/solana/blob/01a096adc/streamer/src/recvmmsg.rs#L104-L115
But then streamer::packet::recv_from will erroneously keep the 1st
packet and drop the 2nd one:
https://github.com/solana-labs/solana/blob/01a096adc/streamer/src/packet.rs#L34-L49
To avoid this bug, this commit updates recv_mmsg to always return total
number of received packets. If socket address cannot be correctly
obtained, it is left as the default value which is UNSPECIFIED:
https://github.com/solana-labs/solana/blob/01a096adc/sdk/src/packet.rs#L145
2022-01-04 21:06:59 +00:00
4b24499916
removes total-size from return value of recv_mmsg
2022-01-04 21:06:59 +00:00
73a7741c49
uses std::net::IpAddr type for Packet.Meta.addr
2022-01-04 13:53:40 +00:00
20b61e28b6
Flip iter operations to keep associated address/header/packets together ( #22245 )
...
Flip iter operations to keep associated address/header/packets together
Before this change, if cast_socket_addr() returned a None for any
address/header pair, the subsequent zip() would misalign the
address/header pair and packet. So, this change zips all three together,
then does filter_map() so keep things aligned.
Additionally, compute total_size inline to avoid running through packets
a second time.
2022-01-03 17:15:50 -06:00
97a1fa10a6
streamer send destination metrics for repair, gossip ( #21564 )
2021-12-17 15:21:05 -08:00
254ef3e7b6
Rename Packets to PacketBatch ( #21794 )
2021-12-11 09:44:15 -05:00
b8837c04ec
Reformat imports to a consistent style for imports
...
rustfmt.toml configuration:
imports_granularity = "One"
group_imports = "One"
2021-12-03 09:19:13 -08:00
7027d56064
Resolve nightly-2021-10-05 clippy complaints
2021-10-06 10:37:58 -07:00
94668c95c2
Prune sigverify queue ( #20331 )
2021-09-30 05:41:05 +02:00
640e93187c
periodically report sigverify_stage stats ( #19674 )
2021-09-21 10:37:58 -07:00
8229a4fbf6
removes raw indexing from streamer ( #19183 )
...
Raw indexing is verbose and error-prone. This same code had an indexing
bug causing validator nodes panic just a few months ago:
https://github.com/solana-labs/solana/commit/482b8c6be
2021-08-12 01:42:12 +00:00
0b7ed18cfa
recvmmsg IPv6 awareness ( #18957 )
2021-08-03 20:35:50 -07:00
049fb0417f
allows sendmmsg api taking owned values (as well as references) ( #18999 )
...
Current signature of api in sendmmsg requires a slice of inner
references:
https://github.com/solana-labs/solana/blob/fe1ee4980/streamer/src/sendmmsg.rs#L130-L152
That forces the call-site to convert owned values to references even
though doing so is redundant and adds an extra level of indirection:
https://github.com/solana-labs/solana/blob/fe1ee4980/core/src/repair_service.rs#L291
This commit expands the api using AsRef and Borrow traits to allow
calling the method with owned values (as well as references like
before).
2021-07-30 20:58:49 +00:00
d2d5f36a3c
adds validator flag to allow private ip addresses ( #18850 )
2021-07-23 15:25:03 +00:00
e316586516
excludes private ip addresses
2021-07-16 20:05:48 -06:00
ae5ad5cf9b
sendmmsg cleanup #18589
...
Rationalize usage of sendmmsg(2). Skip packets which failed to send and track failures.
2021-07-16 14:36:49 -07:00
4a12c715a3
Drop Error suffix from enum values to avoid the enum_variant_names clippy lint
2021-06-18 23:02:13 +00:00
Pankaj Garg
Lijun Wang
Brennan Watt
behzad nouri
Jon Cinque
Haoran Yi
Brooks Prumo
ryleung-solana
kirill lykov
Michael Vines
sakridge
Richard Patel
Justin Starry
Miles Obare
Jon Cinque
steviez
dependabot[bot]
Trent Nelson
buffalu
Jeff Biseda