Commit Graph

33 Commits

Author SHA1 Message Date
behzad nouri 528a03f32a
removes outdated matches crate from dependencies (#33172)
removes outdated matches crate from the dependencies

std::matches has been stable since rust 1.42.0.
Other use-cases are covered by assert_matches crate.
2023-09-07 12:52:57 +00:00
behzad nouri 39615bd075
removes manual let...else (#33089)
https://rust-lang.github.io/rust-clippy/master/index.html#/manual_let_else
2023-08-31 22:35:47 +00:00
Jon Cinque 0fe902ced7
Bump rand to 0.8, rand_chacha to 0.3, getrandom to 0.2 (#32871)
* sdk: Add concurrent support for rand 0.7 and 0.8

* Update rand, rand_chacha, and getrandom versions

* Run command to replace `gen_range`

Run `git grep -l gen_range | xargs sed -i'' -e 's/gen_range(\(\S*\), /gen_range(\1../'

* sdk: Fix users of older `gen_range`

* Replace `hash::new_rand` with `hash::new_with_thread_rng`

Run:
```
git grep -l hash::new_rand | xargs sed -i'' -e 's/hash::new_rand([^)]*/hash::new_with_thread_rng(/'
```

* perf: Use `Keypair::new()` instead of `generate`

* Use older rand version in zk-token-sdk

* program-runtime: Inline random key generation

* bloom: Fix clippy warnings in tests

* streamer: Scope rng usage correctly

* perf: Fix clippy warning

* accounts-db: Map to char to generate a random string

* Remove `from_secret_key_bytes`, it's just `keypair_from_seed`

* ledger: Generate keypairs by hand

* ed25519-tests: Use new rand

* runtime: Use new rand in all tests

* gossip: Clean up clippy and inline keypair generators

* core: Inline keypair generation for tests

* Push sbf lockfile change

* sdk: Sort dependencies correctly

* Remove `hash::new_with_thread_rng`, use `Hash::new_unique()`

* Use Keypair::new where chacha isn't used

* sdk: Fix build by marking rand 0.7 optional

* Hardcode secret key length, add static assertion

* Unify `getrandom` crate usage to fix linking errors

* bloom: Fix tests that require a random hash

* Remove some dependencies, try to unify others

* Remove unnecessary uses of rand and rand_core

* Update lockfiles

* Add back some dependencies to reduce rebuilds

* Increase max rebuilds from 14 to 15

* frozen-abi: Remove `getrandom`

* Bump rebuilds to 17

* Remove getrandom from zk-token-proof
2023-08-21 19:11:21 +02:00
Greg Cusack 5c86f89bc7
Add from field for message tracking (#32725)
* we only want to report received message signatures on PUSH requests, not PULL requests

* woops accidently had it has LocalMessage not PushMessage

* switch from match to if let statement

* convert if let to matches macro

* add in from field in PushMessage for message tracking

* update with cargo fmt

* remove display for gossip route and add lifetime param to pubkey reference in gossiproute enum

* forgot to run fmt
2023-08-07 11:40:18 -07:00
Greg Cusack 849525735f
report received message signatures only on PUSH requests (#32708)
* we only want to report received message signatures on PUSH requests, not PULL requests

* woops accidently had it has LocalMessage not PushMessage

* switch from match to if let statement

* convert if let to matches macro
2023-08-06 20:36:23 -07:00
Greg Cusack 80f708298b
Report new received crds signatures and their respective origins to metrics (#32504)
* screwed up old branch and syncing with upstream branch

* add fixed size ring buff instead of variable sized vecdeque for reporting signatures

* modify difficulty to take in n 0 bits and check if signature ending ends in n 0 bits

* update to only push every 18 trailing zero bits. and clean up

* report origin with signature. and set trailing 0s to 8 for local testing

* change back to 18 trailing zeros and rm unused imports

* run cargo rmt

* run ./scripts/cargo-for-all-lock-files.sh tree

* allow integer arithmetic for bit comparison

* rm unused lifetime

* rm duplicate entry?

* re implement ring buf

* put ringbuf in sorted order

* ringbuf in cargo.toml now in sorted order

* rm ring buf, refactor, fix trailing zero bug

* fix bug in trailing zeros. was comparing wrong ones

* fix needless range loop bug

* fix bug

* change trailing zero checking to build in methods and only report first 8 bytes of signature and origin pubkey

* report full origin string and first 8 bytes of signature

* set SIGNATURE_SAMPLE_TRAILING_ZEROS back to 18

* forgot to run cargo tree

* avoid panic and change working

* rm bs58

* pass in Option<String> into datapoint_info

* shorten metric names
2023-07-20 14:33:40 -07:00
behzad nouri d54b6204be
removes instances of clippy::manual_let_else (#32417) 2023-07-09 21:41:36 +00:00
behzad nouri 1b08d01a80
removes shred_version from LegacyContactInfo public interface (#31304)
Working towards LegacyContactInfo => ContactInfo migration, the commit
adds more api parity between the two.
2023-04-24 15:19:33 +00:00
behzad nouri a88024e295
removes wallclock from LegacyContactInfo public interface (#31303) 2023-04-22 20:18:39 +00:00
behzad nouri cb65a785bc
makes sockets in LegacyContactInfo private (#31248)
Working towards LegacyContactInfo => ContactInfo migration, the commit
hides some implementation details of LegacyContactInfo and expands API
parity with the new ContactInfo.
2023-04-21 15:39:16 +00:00
Brooks 453f272698
Rename IncrementalSnapshotHashes to SnapshotHashes (#31136) 2023-04-11 10:30:29 -04:00
Brooks f3083ad2e0
Rename SnapshotHashes to LegacySnapshotHashes (#31086) 2023-04-10 17:52:20 -04:00
behzad nouri d4b30adffe
reworks gossip crds timeouts (#30468)
CrdsGossipPull::make_timeouts iterates over the stakes hashmap and
creates a new hashmap which is unnecessary:
https://github.com/solana-labs/solana/blob/c032dc275/gossip/src/crds_gossip_pull.rs#L517-L539

The commit instead keeps a reference to the stakes hashmap.
2023-03-27 21:52:48 +00:00
behzad nouri eb6d6e9bca
evicts old gossip entries by origin's wallclock (#30331)
Local-timestamp used in current gossip eviction code:
https://github.com/solana-labs/solana/blob/a36e1b211/gossip/src/crds.rs#L469-L511
does not indicate how old the entry is but how recently it was received.

The commit instead uses origin's wallclock to identify old values. In
order to avoid cases where the wallclock on the entry is bogus, it is
capped by local-timestamp.
2023-02-15 20:03:13 +00:00
Kevin Ji cd51499ab9
Use Ipv4Addr constants in socketaddr! (#30095) 2023-02-02 16:48:21 -07:00
behzad nouri d6fbf3fb17
adds new contact-info with forward compatible sockets (#29596)
The commit implement new ContactInfo where
* Ports and IP addresses are specified separately so that unique IP
  addresses can only be specified once.
* Different sockets (tvu, tpu, etc) are specified by opaque u8 tags so
  that adding and removing sockets is backward and forward compatible.
* solana_version::Version is also embedded in so that it won't need to
  be gossiped separately.
* NodeInstance is also rolled in by adding a field identifying when the
  instance was first created so that it won't need to be gossiped
  separately.

Update plan:
* Once the cluster is able to ingest the new type (i.e. this patch), a
  2nd patch will start gossiping the new ContactInfo along with the
  LegacyContactInfo.
* Once all nodes in the cluster gossip the new ContactInfo, a 3rd patch
  will start solely using the new ContactInfo while still gossiping the
  old LegacyContactInfo.
* Once all nodes in the cluster solely use the new ContactInfo, a 4th
  patch will stop gossiping the old LegacyContactInfo.
2023-01-26 17:02:18 +00:00
behzad nouri 590b75140f
removes legacy retransmit tests (#29817)
Retransmit code has moved to core/src/cluster_nodes.rs and has been
significantly revised.
gossip/tests/cluster_info.rs is testing the old code which is no longer
relevant.
2023-01-21 22:28:48 +00:00
Illia Bobyr e410d021ea
gossip: crds::test::test_update_timestamp: Remove hash comparison (#29567)
It was not immediately clear why the second `CrdsValue` insertion in the
test must always succeed.  Turns out the test was relying on hash values
having a specific relationship.  It is confusing to someone not deeply
familiar with the test.

As overwrite based on the hash value is not part of the behavior that we
consider valuable, we just remove that check.

Unified assertion between two checks into one.
2023-01-12 00:19:44 -08:00
behzad nouri 8c212f59ad
renames ContactInfo to LegacyContactInfo (#29566)
Working towards adding a new ContactInfo where new sockets can be
added in a backward compatible way.
2023-01-08 16:00:55 +00:00
behzad nouri 2d849a2eae
indexes duplicate-shreds in gossip crds table (#29317)
Also adding Crds::get_duplicate_shreds which retrieves all upserted
duplicate-shreds since a given cursor using the index.
2022-12-20 13:48:05 +00:00
behzad nouri b06656cbba tracks number of gossip push duplicates
The commit tracks number of times duplicates of a CRDS value is received
from gossip push. Following commits will utilize this metric to score
gossip nodes in terms of timeliness of their push messages, in order to
better pick which nodes to prune.
2022-12-15 13:28:27 +00:00
behzad nouri 9524c9dbff patches errors from clippy::uninlined_format_args
https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
2022-12-06 19:32:15 +00:00
Michael Vines b05c7d91ed Fix derive_partial_eq_without_eq clippy lint 2022-05-22 22:22:21 -07:00
behzad nouri 1282277126
bumps up crds-shards-bits (#23220)
The commit adjust CRDS_SHARDS_BITS up to be in-line with mask_bits in
gossip pull request. This will avoid redundant filtering of irrelevant
crds entries when responding to pull requests.
2022-03-01 15:14:11 +00:00
Michael Keleti b0ca335463
Rename "trusted" to "known" in `validators/` (#21197)
* Replaced trusted with known validator

* Format Convention
2021-11-12 11:57:55 -07:00
behzad nouri 1297a13586
adds metrics tracking crds writes and votes (#20953) 2021-10-26 13:02:30 +00:00
behzad nouri 0da661de62
adds metrics for number of nodes vs number of pubkeys (#20512) 2021-10-07 18:56:05 +00:00
Brooks Prumo 5d141fe01d
Rename CRDS SnapshotHash to SnapshotHashes (#20421) 2021-10-04 19:03:28 -05:00
behzad nouri bbd22f06f4
implements generic lookups into gossip crds table (#18765)
This commit adds CrdsEntry trait which allows generic lookups into crds
table. For example to get ContactInfo or LowestSlot associated with a
Pubkey, the lookup code would be respectively:
   crds.get::<&ContactInfo>(pubkey)
   crds.get::<&LowestSlot>(pubkey)
2021-07-21 12:16:26 +00:00
behzad nouri 9d983a34a0
debug logs when crds table trim failed (#18307)
reports of this error being possibly spammy:
https://discord.com/channels/428295358100013066/689412830075551748/859441080054710293

The commit changes the log level to debug.
Additionally adding a new metric to understand the frequency of this error.
2021-06-29 19:39:46 +00:00
behzad nouri 5a99fa3790
adds mapping from nodes pubkeys to their shred-version (#17940)
Crds values of nodes with different shred versions are creeping into
gossip table resulting in runtime issues as the one addressed in:
https://github.com/solana-labs/solana/pull/17899

This commit works towards enforcing more checks and filtering based on
shred version by adding necessary mapping and api to gossip table.
Once populated, pubkey->shred-version mapping persists as long as there
are any values associated with the pubkey.
2021-06-18 15:56:04 +00:00
behzad nouri cca46308bc
short cuts expiration check if origin's contact-info is still valid (#17918)
Crds::find_old_labels can skip checking values timestamps if the
origin's contact info hasn't expired yet:
https://github.com/solana-labs/solana/blob/985280ec0/gossip/src/crds.rs#L394-L408
2021-06-13 19:47:07 +00:00
Tyera Eulberg 9a5330b7eb
Move gossip modules into solana-gossip crate (#17352)
* Move gossip modules to solana-gossip

* Update Protocol abi digest due to move

* Move gossip benches and hook up CI

* Remove unneeded Result entries

* Single use statements
2021-05-26 09:15:46 -06:00