Commit Graph

187 Commits

Author SHA1 Message Date
Will Hickey 4aa2a42cc7
Fix test_bench_tps_local_cluster_solana (#27448)
* Fix test_bench_tps_local_cluster_solana
* Remove #[ignore] annotations from dos tests (which are also fixed by this change)
* Remove #[ignore] annotations from local cluster tests (which are also fixed by this change)
2022-08-30 13:04:31 -05:00
Tyera Eulberg b8b3d723da
Use new client crates (#27360)
* Update ancillary cli crates

* Update cli

* Update command-line tools

* Update rpc, etc

* Update client-test

* Update core, validator

* Update local-cluster
2022-08-24 10:47:02 -06:00
Brennan Watt e4a7d01e10
Rust v1.63 (#27303)
* Upgrade to Rust v1.63.0

* Add nightly_clippy_allows

* Resolve some new clippy nightly lints

* Increase QUIC packets completion timeout

* Update quinn-udp crate

Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-22 18:01:03 -07:00
Michael Vines 3f4731b37f Standardize thread names
Tenets:
1. Limit thread names to 15 characters
2. Prefix all Solana-controlled threads with "sol"
3. Use Camel case. It's more character dense than Snake or Kebab case
2022-08-20 07:49:39 -07:00
Brennan Watt 7573000d87
Revert "Rust v1.63.0 (#27148)" (#27245)
This reverts commit a2e7bdf50a.
2022-08-19 09:19:44 +01:00
behzad nouri 6928b2a5af
adds hash domain to ping-pong protocol (#27193)
In order to maintain backward compatibility, for now the responding node
will hash the token both with and without domain so that the other node
will accept the response regardless of its upgrade status.
Once the cluster has upgraded to the new code, we will remove the legacy
domain = false case.
2022-08-18 22:39:31 +00:00
Brennan Watt a2e7bdf50a
Rust v1.63.0 (#27148)
* Upgrade to Rust v1.63.0

* Add nightly_clippy_allows

* Resolve some new clippy nightly lints

* Increase QUIC packets completion timeout

Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-17 15:48:33 -07:00
behzad nouri fea66c8b63
derives Error trait for ClusterInfoError and core::result::Error (#27208) 2022-08-17 22:01:51 +00:00
Brennan Watt 521c550ccd
Sleep between vote refreshes (#27115)
* Sleep between vote refreshes in unit test
2022-08-12 13:45:00 -07:00
Tyera Eulberg 45c0da8597
Fix quic client on TestValidator, alternative (#27046)
Add new method to enable custom offset
2022-08-10 15:27:12 +00:00
Jeff Biseda 857be1e237
sign repair requests (#26833) 2022-07-31 15:48:51 -07:00
behzad nouri 128226c6cc
patches flaky test_push_votes_with_tower (#26554)
cargo test --package solana-gossip --release test_push_votes_with_tower

occasionally fails because with --release all votes are generated at
the same wallclock (milliseconds resolution) and so the new ones will
not necessarily override existing entries in the table.

The commit ensures that the new vote is pushed with a wallclock later
than existing entries.
2022-07-11 12:56:31 +00:00
behzad nouri ba785cf8ab
removes erroneous uses of std::mem::swap (#26536)
All instances should be replace by std::mem::{replace,take},
or just plain assignment.
2022-07-11 11:33:15 +00:00
behzad nouri df616a0dda
removes redundant clone in gossip PruneData::signable_data (#26510)
PruneData::signable_data redundantly clones inner fields, while only
references suffice:
https://github.com/solana-labs/solana/blob/d1370f2c7/gossip/src/cluster_info.rs#L219-L233
2022-07-10 13:13:07 +00:00
Greg Cusack 032bee13ab
Add Gossip Loop metrics (#26195)
* add three gossip metrics measuring gossip loop times

* add 5 metrics

* rm space

* rm space

* Update SECURITY.md

- fix nav link
- add bounty split policy for duplicate reports

* Add transaction index in slot to geyser plugin TransactionInfo (#25688)

* Define shuffle to prep using same shuffle for multiple slices

* Determine transaction indexes and plumb to execute_batch

* Pair transaction_index with transaction in TransactionStatusService

* Add new ReplicaTransactionInfoVersion

* Plumb transaction_indexes through BankingStage

* Prepare BankingStage to receive transaction indexes from PohRecorder

* Determine transaction indexes in PohRecorder; add field to WorkingBank

* Add PohRecorder::record unit test

* Only pass starting_transaction_index around PohRecorder

* Add helper structs to simplify test DashMap

* Pass entry and starting-index into process_entries_with_callback together

* Add tx-index checks to test_rebatch_transactions

* Revert shuffle definition and use zip/unzip

* Only zip/unzip if randomize

* Add confirm_slot_entries test

* Review nits

* Add type alias to make sender docs more clear

* Update SECURITY.md

finish filling out the table....

* rpc: fix possible deadlock in rpc (#26051)

* Add StatusCache::root_slot_deltas() and use it (#26170)

* Remove InMemAccountsIndex::map() and use map_internal directly (#26189)

* [quic]Decrement total_streams correctly (#26158)

* remove comment

* alphabetical metrics. no abbreviations

* remove trailing white space

* cargo fmt to update code format/readability

Co-authored-by: Trent Nelson <trent@solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
Co-authored-by: Boqin Qin(秦 伯钦) <Bobbqqin@gmail.com>
Co-authored-by: Brooks Prumo <brooks@solana.com>
Co-authored-by: Miles Obare <bdhobare@gmail.com>
2022-06-29 11:55:41 -06:00
behzad nouri f534b8981b
maps number of data shreds to erasure batch size (#25917)
In prepration of
https://github.com/solana-labs/solana/pull/25807
which reworks erasure batch sizes, this commit:
* adds a helper function mapping the number of data shreds to the
  erasure batch size.
* adds ProcessShredsStats to Shredder::entries_to_shreds in order to
  replace and remove entries_to_data_shreds from the public interface.
2022-06-23 13:27:54 +00:00
Jon Cinque 79a8ecd0ac
client: Remove static connection cache, plumb it instead (#25667)
* client: Remove static connection cache, plumb it instead

* Add TpuClient::new_with_connection_cache to not break downstream

* Refactor get_connection and RwLock into ConnectionCache

* Fix merge conflicts from new async TpuClient

* Remove `ConnectionCache::set_use_quic`

* Move DEFAULT_TPU_USE_QUIC to client, use ConnectionCache::default()
2022-06-08 13:57:12 +02:00
HaoranYi 4223f82922
Fix format alignment for cluster info trace (#25741)
* double shrinking

* remove pre/post shrink time

* fix cluster info trace alignemnt

* add test

* format

* typo

* add checks in cluster info trace test

* update cargo lock

* clippy

* clippy

* move regex deps to dev deps

* cargo lock
2022-06-06 09:51:00 -05:00
Pankaj Garg 1c2ae470c5
Fix forwarding of transactions over QUIC (#25674)
* Spawn QUIC server to receive forwarded txs

* Update validator port range

* forward votes using UDP

* no forwarding from unstaked nodes

* forwarding stats in banking stage

* fix test builds

* fix lifetime of forward sender
2022-06-02 11:14:58 -07:00
behzad nouri 69cbbaf483
patches flaky gossip pull from entrypoint test (#25589)
test_pull_from_entrypoint_if_not_present relies on a deterministic
ordering for the entries when generating gossip pull requests.

https://github.com/solana-labs/solana/pull/25460
changed an intermediate type for gossip pull-requests from Vec to
HashMap, and so the entries are no longer deterministically ordered.
This causes the test to be flaky.

The commit updates the test so that it no longer relies on the ordering.
2022-05-26 18:58:06 +00:00
behzad nouri 1925b4f5cb
fans out gossip pull-requests to many randomly selected peers (#25460)
Each time a node generates gossip pull-requests, it sends out all the
requests to a single randomly selected peer:
https://github.com/solana-labs/solana/blob/fd7ad31ee/gossip/src/crds_gossip_pull.rs#L253-L266

This causes a burst of pull-requests at a single node at once. In order
to make gossip in-bound traffic less bursty, this commit fans out gossip
pull-requests to several randomly selected peers.

This should reduce spikes in inbound gossip traffic without changing the
average load which may help reduce number of times outbound data budget is
exhausted when responding to gossip pull-requests at the receiving node, and
reduce number of pull-requests dropped.
2022-05-26 12:45:53 +00:00
Justin Starry cad1c41ce2 Add Packet::deserialize_slice convenience method 2022-05-24 17:31:14 +08:00
steviez ec7ca411dd
Make PacketBatch packets vector non-public (#25413)
Upcoming changes to PacketBatch to support variable sized packets will
modify the internals of PacketBatch. So, this change removes usage of
the internal packet struct and instead uses accessors (which are
currently just wrappers of Vector functions but will change down the
road).
2022-05-23 15:30:15 -05:00
behzad nouri c248fb3f51
renames Packet Meta::{,set_}addr methods to {,set_}socket_addr (#25478)
In order to distinguish between Meta.addr field which is an IpAddr and
the methods which refer to a SocketAddr.
2022-05-23 15:48:59 +00:00
Michael Vines 3608801a54 Avoid clippy::significant_drop_in_scrutinee 2022-05-22 22:22:21 -07:00
Michael Vines b05c7d91ed Fix derive_partial_eq_without_eq clippy lint 2022-05-22 22:22:21 -07:00
Michael Vines c54e06355f
voteSubscribe pubsub notification now includes the vote transaction signature (#25291) 2022-05-19 18:28:46 -07:00
sakridge 3d96a1ab76
Block packets in vote-only mode (#24906) 2022-05-14 17:53:37 +02:00
Justin Starry 7100f1c94b
Collect stats in streamer receiver and report fetch stage metrics (#25010) 2022-05-06 02:56:18 +08:00
steviez b48fd4eec2
Construct PacketBatches from PongMessages directly (#24708)
Serialize pongs directly into PacketBatch to save copying the data from
intermediate packets into PacketBatch.
2022-04-26 21:30:00 -05:00
behzad nouri 12ae8d3be5
returns Error when Shred::sanitize fails (#24653)
Including the error in the output allows to debug when Shred::sanitize
fails.
2022-04-25 23:19:37 +00:00
behzad nouri 895f76a93c
hides implementation details of shred from its public interface (#24563)
Working towards embedding versioning into shreds binary, so that a new
variant of shred struct can include merkle tree hashes of the erasure
set.
2022-04-25 12:43:22 +00:00
behzad nouri 1d50832389
replaces counters with datapoints in gossip metrics (#24451) 2022-04-18 23:14:59 +00:00
HaoranYi 6d1b6bdd7c
typo (#24400) 2022-04-18 08:52:56 -05:00
Tyera Eulberg afeb1d3cca
Bump lru crate (#24150) 2022-04-06 16:18:42 -06:00
behzad nouri cd09390367
reduces gossip crds stats (#24132) 2022-04-06 15:35:25 +00:00
BG Zhu 22224127e0
Refactor thin_client::create_client (#24067)
Refactor the thin_client::create_client to take addresses separately instead of as a tuple

Co-authored-by: Bijie Zhu <bijiezhu@Bijies-MBP.cable.rcn.com>
2022-04-06 11:03:38 -04:00
ryleung-solana a38bd4acc8
Use LRU in connection-cache (#24109)
Switch to using LRU for connection-cache
2022-04-06 10:58:32 -04:00
behzad nouri db23295e1c
removes legacy weighted_shuffle and weighted_best methods (#24125)
Older weighted_shuffle is based on a heuristic which results in biased
samples as shown in:
https://github.com/solana-labs/solana/pull/18343
and can be replaced with WeightedShuffle.

Also, as described in:
https://github.com/solana-labs/solana/pull/13919
weighted_best can be replaced with rand::distributions::WeightedIndex,
or WeightdShuffle::first.
2022-04-05 19:19:22 +00:00
behzad nouri 2b718d00b0 removes legacy compatibility turbine peers shuffle code 2022-04-05 12:04:12 +00:00
behzad nouri 7cb3b6cbe2
demotes WeightedShuffle failures to error metrics (#24079)
Since call-sites are calling unwrap anyways, panicking seems too punitive
for our use cases.
2022-04-03 16:20:06 +00:00
ryleung-solana 8b72200afb
Thin client quic (#23973)
Change thin-client to use connection-cache
2022-03-31 15:47:00 -04:00
Michael Vines 7ef18f220a Update Version CrdsData on node identity changes 2022-03-28 15:57:16 -07:00
ryleung-solana 17b00ad3a4
Add quic-client module (#23166)
* Add quic-client module to send transactions via quic, abstracted behind the TpuConnection trait (along with a legacy UDP implementation of TpuConnection) and change thin-client to use TpuConnection
2022-03-09 21:33:05 -05:00
sakridge a4f4ac5279
add plumbing to allow for arbitrary tpu address in gossip (#22703)
* add plumbing to allow for arbitrary tpu address in gossip

* make clippy happy

* Review comments

Co-authored-by: CherryWorm <nico.gruendel@web.de>
2022-03-02 09:42:14 +01:00
behzad nouri 1282277126
bumps up crds-shards-bits (#23220)
The commit adjust CRDS_SHARDS_BITS up to be in-line with mask_bits in
gossip pull request. This will avoid redundant filtering of irrelevant
crds entries when responding to pull requests.
2022-03-01 15:14:11 +00:00
sakridge 514aab46d9
Search for consecutive ports (#22979) 2022-02-07 17:53:40 +01:00
sakridge 5a230f418d
Add quic port for accepting transactions (#22753)
using quinn library

streamer: Sign TLS cert with validator identity key

Handle multiple incoming chunks
2022-02-04 15:27:09 +01:00
behzad nouri e3b137066d
caches WeightedShuffle struct in ClusterNodes (#22877)
Instead of reconstructing WeightedShuffle struct for each shred
broadcast or retransmit, we can use the same struct with minimal
mutations.
2022-02-02 15:12:26 +00:00
behzad nouri 45e09664b8
removes Rng field from WeightedShuffle struct (#22850) 2022-02-01 15:27:23 +00:00