Commit Graph

82 Commits

Author SHA1 Message Date
steviez ec7ca411dd
Make PacketBatch packets vector non-public (#25413)
Upcoming changes to PacketBatch to support variable sized packets will
modify the internals of PacketBatch. So, this change removes usage of
the internal packet struct and instead uses accessors (which are
currently just wrappers of Vector functions but will change down the
road).
2022-05-23 15:30:15 -05:00
behzad nouri c248fb3f51
renames Packet Meta::{,set_}addr methods to {,set_}socket_addr (#25478)
In order to distinguish between Meta.addr field which is an IpAddr and
the methods which refer to a SocketAddr.
2022-05-23 15:48:59 +00:00
Michael Vines 3608801a54 Avoid clippy::significant_drop_in_scrutinee 2022-05-22 22:22:21 -07:00
Michael Vines c54e06355f
voteSubscribe pubsub notification now includes the vote transaction signature (#25291) 2022-05-19 18:28:46 -07:00
steviez b48fd4eec2
Construct PacketBatches from PongMessages directly (#24708)
Serialize pongs directly into PacketBatch to save copying the data from
intermediate packets into PacketBatch.
2022-04-26 21:30:00 -05:00
behzad nouri 895f76a93c
hides implementation details of shred from its public interface (#24563)
Working towards embedding versioning into shreds binary, so that a new
variant of shred struct can include merkle tree hashes of the erasure
set.
2022-04-25 12:43:22 +00:00
behzad nouri 1d50832389
replaces counters with datapoints in gossip metrics (#24451) 2022-04-18 23:14:59 +00:00
behzad nouri 7cb3b6cbe2
demotes WeightedShuffle failures to error metrics (#24079)
Since call-sites are calling unwrap anyways, panicking seems too punitive
for our use cases.
2022-04-03 16:20:06 +00:00
ryleung-solana 8b72200afb
Thin client quic (#23973)
Change thin-client to use connection-cache
2022-03-31 15:47:00 -04:00
Michael Vines 7ef18f220a Update Version CrdsData on node identity changes 2022-03-28 15:57:16 -07:00
sakridge a4f4ac5279
add plumbing to allow for arbitrary tpu address in gossip (#22703)
* add plumbing to allow for arbitrary tpu address in gossip

* make clippy happy

* Review comments

Co-authored-by: CherryWorm <nico.gruendel@web.de>
2022-03-02 09:42:14 +01:00
sakridge 514aab46d9
Search for consecutive ports (#22979) 2022-02-07 17:53:40 +01:00
sakridge 5a230f418d
Add quic port for accepting transactions (#22753)
using quinn library

streamer: Sign TLS cert with validator identity key

Handle multiple incoming chunks
2022-02-04 15:27:09 +01:00
behzad nouri 45e09664b8
removes Rng field from WeightedShuffle struct (#22850) 2022-02-01 15:27:23 +00:00
Michael Vines 6d5bbca630 Pacify clippy 2022-01-21 19:12:57 -08:00
Justin Starry 7f20c6149e
Refactor: move simple vote parsing to runtime (#22537) 2022-01-20 10:39:21 +08:00
anatoly yakovenko d343713f61
Optimize packet dedup (#22571)
* Use bloom filter to dedup packets

* dedup first

* Update bloom/src/bloom.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* fixup

* fixup

* fixup

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
2022-01-19 13:58:20 -08:00
Jeff Biseda 8b66625c95
convert std::sync::mpsc to crossbeam_channel (#22264) 2022-01-11 02:44:46 -08:00
behzad nouri c9c78622a8
discards serialized gossip crds votes if cannot parse tx (#22129) 2021-12-29 19:31:26 +00:00
carllin 7f6fb6937a
Ensure AncestorHashesSerice selects an open port (#21919) 2021-12-18 00:44:01 -05:00
Justin Starry 254ef3e7b6
Rename Packets to PacketBatch (#21794) 2021-12-11 09:44:15 -05:00
Ashwin Sekar f0acf7681e
Add vote instructions that directly update on chain vote state (#21531)
* Add vote state instructions

UpdateVoteState and UpdateVoteStateSwitch

* cargo tree

* extract vote state version conversion to common fn
2021-12-07 16:47:26 -08:00
Michael Vines b8837c04ec Reformat imports to a consistent style for imports
rustfmt.toml configuration:
  imports_granularity = "One"
  group_imports = "One"
2021-12-03 09:19:13 -08:00
behzad nouri 9886366977
exempts AccountsHashes from stake check (#21565)
Otherwise getHealth fails if account hashes are not propagated.
2021-12-02 18:01:32 +00:00
behzad nouri 57057f8d39 uses enum for shred type
Current code is using u8 which does not have any type-safety and can
contain invalid values:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L167

Checks for invalid shred-types are scattered through the code:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/blockstore.rs#L849-L851
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L346-L348

The commit uses enum for shred type with #[repr(u8)]. Backward
compatibility is maintained by implementing Serialize and Deserialize
compatible with u8, and adding a test to assert that.
2021-11-19 14:16:39 +00:00
carllin b30c94ce55
ClusterInfoVoteListener send only missing votes to BankingStage (#20873) 2021-11-18 15:20:41 -08:00
behzad nouri 43168e6365
doubles crds unique pubkey capacity (#20947) 2021-10-26 13:06:55 +00:00
behzad nouri 1297a13586
adds metrics tracking crds writes and votes (#20953) 2021-10-26 13:02:30 +00:00
behzad nouri 0c0384ec32
revises turbine peers shuffling order (#20480)
Turbine randomly shuffles cluster nodes on a broadcast tree for each
shred. This requires knowing the stakes and nodes' contact-infos (from
gossip).

However gossip is subject to partitioning and propogation delays.
Additionally unstaked nodes may join and leave the cluster at any
moment, changing the cluster view from one node to another.

This commit:
* Always arranges the unstaked nodes at the bottom of turbine broadcast
  tree.
* Staked nodes are always included regardless of if their contact-info
  is available in gossip or not.
* Uses the unbiased WeightedShuffle construct for shuffling nodes.
2021-10-14 15:09:36 +00:00
Brooks Prumo 1fcfbfccbb
Add fn to push IncrementalSnapshotHashes to cluster via gossip (#20395) 2021-10-08 08:20:35 -05:00
Brooks Prumo 57592e463e
Add get_incremental_snapshot_hash_for_node() to gossip (#20394) 2021-10-07 19:47:14 -05:00
Tao Zhu 177a375479
Tpu vote 1.7 (#20187) (#20494)
* Add separate vote processing tpu port

* Add feature to send to tpu vote port

* Add vote rejecting sigverify mode

* use packet.meta.is_simple_vote_tx in place of deserialization

* consolidate code that identifies vote tx atcommon path for cpu and gpu

* new key for feature set

* banking forward tpu vote

* add tpu vote port to dockerfile and other review changes

* Simplify thread id compare

* fix a test; updated cluster_info ABI change

Co-authored-by: Tao Zhu <tao@solana.com>

Co-authored-by: sakridge <sakridge@gmail.com>
2021-10-07 09:38:23 +00:00
Brooks Prumo 4e3818e5c1
Add CrdsData::IncrementalSnapshotHashes (#20374) 2021-10-05 09:57:46 -05:00
Brooks Prumo 5d141fe01d
Rename CRDS SnapshotHash to SnapshotHashes (#20421) 2021-10-04 19:03:28 -05:00
carllin ee8621a8bd
Add metric measuring number of successfully inserted push messages (#20275)
* Add number of successfully inserted push messages
2021-09-28 21:41:17 -07:00
sakridge 013e1d9d49
Limit transaction forwarding from banking_stage (#19940) 2021-09-21 08:49:41 -07:00
behzad nouri d7051b0d21
adds logs when push-vote panics with invalid vote-index (#19485)
In order to debug this panic on the clusters:

  panicked at 'assertion failed: (vote_index as usize) <
  MAX_LOCKOUT_HISTORY', core/src/cluster_info.rs:1012:9
2021-08-31 12:15:07 +00:00
behzad nouri 6909a79b6f
removes require-stake-for-gossip feature (#19476)
The feature is already activated on all clusters.
2021-08-27 21:17:15 +00:00
behzad nouri 3efccbffab sends shreds (instead of packets) to retransmit stage
Working towards channelling through shreds recovered from erasure codes
to retransmit stage.
2021-08-17 13:44:10 +00:00
behzad nouri 140abec6ef
exempts node-instances from shred-version check (#19190)
Clusters are kept separate using the shred-versions obtained from
contact-infos. However, this mechanism breaks if there are 2 instances
of the same identity key running on different clusters, because then one
of the two contact-infos have the right shred-version.
If a node has the contact-info with the matching shred-version, then it
will pass all associated crds values even if they belong to the other
instance. So the shred-version check breaks.
As a result we cannot support 2 instances of the same identity key
running on different clusters. To prevent that, this commit is exempting
node-instances from shred-version check so that they are always
propagated across clusters and halt one of the running duplicate
instances.
2021-08-14 00:47:44 +00:00
behzad nouri f302774cf7
implements copy-on-write for staked-nodes (#19090)
Bank::staked_nodes and Bank::epoch_staked_nodes redundantly clone
staked-nodes HashMap even though an immutable reference will suffice:
https://github.com/solana-labs/solana/blob/a9014cece/runtime/src/vote_account.rs#L77

This commit implements copy-on-write semantics for staked-nodes by
wrapping the underlying HashMap in Arc<...>.
2021-08-10 12:59:12 +00:00
Justin Starry 8817f59b6e
Version transaction message and add new message format (#18725)
* Version transaction message and add new message format

* Update abi digest due to message path change

* Update v0.rs

Fix comment

* Update original.rs

* Update message versions name and address map indexes field name

* s/original/legacy

* update comment

* cargo fmt

* Update abi digest due to legacy rename
2021-08-09 22:03:39 -07:00
behzad nouri 049fb0417f
allows sendmmsg api taking owned values (as well as references) (#18999)
Current signature of api in sendmmsg requires a slice of inner
references:
https://github.com/solana-labs/solana/blob/fe1ee4980/streamer/src/sendmmsg.rs#L130-L152

That forces the call-site to convert owned values to references even
though doing so is redundant and adds an extra level of indirection:
https://github.com/solana-labs/solana/blob/fe1ee4980/core/src/repair_service.rs#L291

This commit expands the api using AsRef and Borrow traits to allow
calling the method with owned values (as well as references like
before).
2021-07-30 20:58:49 +00:00
behzad nouri f1198fc6d5
filters crds values in parallel when responding to gossip pull-requests (#18877)
When responding to gossip pull-requests, filter_crds_values takes a lot of time
while holding onto read-lock:
https://github.com/solana-labs/solana/blob/f51d64868/gossip/src/crds_gossip_pull.rs#L509-L566

This commit will filter-crds-values in parallel using rayon thread-pools.
2021-07-26 17:13:11 +00:00
behzad nouri d2d5f36a3c
adds validator flag to allow private ip addresses (#18850) 2021-07-23 15:25:03 +00:00
behzad nouri bbd22f06f4
implements generic lookups into gossip crds table (#18765)
This commit adds CrdsEntry trait which allows generic lookups into crds
table. For example to get ContactInfo or LowestSlot associated with a
Pubkey, the lookup code would be respectively:
   crds.get::<&ContactInfo>(pubkey)
   crds.get::<&LowestSlot>(pubkey)
2021-07-21 12:16:26 +00:00
Justin Starry 207c90bd8b
Shorten long SerializeWith type paths in abi digest (#18734) 2021-07-20 08:59:50 -05:00
behzad nouri 8da261cf5c
locks crds only once in ClusterInfo::repair_peers (#18752)
ClusterInfo::repair_peers locks crds table twice, and shows performance
regression if the RwLock is not reader-preferred:
https://github.com/solana-labs/solana/blob/269028360/gossip/src/cluster_info.rs#L1188-L1210
2021-07-18 16:55:58 +00:00
behzad nouri e316586516 excludes private ip addresses 2021-07-16 20:05:48 -06:00
Jeff Biseda ae5ad5cf9b
sendmmsg cleanup #18589
Rationalize usage of sendmmsg(2). Skip packets which failed to send and track failures.
2021-07-16 14:36:49 -07:00