Commit Graph

32 Commits

Author SHA1 Message Date
Jeff Biseda 3f805ad06d
improve batch_send error handling (#33936) 2023-10-31 23:39:26 -07:00
behzad nouri 6470544ea2
adds metrics to turbine QUIC endpoint (#33819) 2023-10-24 14:59:15 +00:00
Pankaj Garg 9d42cd7efe
Initialize fork graph in program cache during bank_forks creation (#33810)
* Initialize fork graph in program cache during bank_forks creation

* rename BankForks::new to BankForks::new_rw_arc

* fix compilation

* no need to set fork_graph on insert()

* fix partition tests
2023-10-23 09:32:41 -07:00
steviez 56ccffdaa5
Replace get_tmp_ledger_path!() with self cleaning version (#33702)
This macro is used a lot for tests to create a ledger path in order to
open a Blockstore. Files will be left on disk unless the test remembers
to call Blockstore::destroy() on the directory. So, instead of requiring
this, use the get_tmp_ledger_path_auto_delete!() macro that creates a
TempDir (which automatically deletes itself when it goes out of scope).
2023-10-21 11:38:31 +02:00
behzad nouri e0b59a6f53
prunes turbine QUIC connections (#33663)
The commit implements lazy eviction for turbine QUIC connections.
The cache is allowed to grow to 2 x capacity at which point at least
half of the entries with lowest stake are evicted, resulting in an
amortized O(1) performance.
2023-10-20 21:52:37 +00:00
behzad nouri 8becb72b3e
separates out routing shreds from establishing connections (#33599)
Currently each outgoing shred will attempt to establish a connection if
one does not already exist. This is very wasteful and consumes many
tokio tasks if the remote node is down or unresponsive.

The commit decouples routing packets from establishing connections by
adding a buffering channel for each remote address. Outgoing packets are
always sent down this channel to be processed once the connection is
established. If connecting attempt fails, all packets already pushed to
the channel are dropped at once, reducing the number of attempts to make
a connection if the remote node is down or unresponsive.
2023-10-19 15:44:15 +00:00
behzad nouri 2151393271
adds more context why the first data shred is inserted synchronously (#33092) 2023-09-12 15:27:06 +00:00
behzad nouri 528a03f32a
removes outdated matches crate from dependencies (#33172)
removes outdated matches crate from the dependencies

std::matches has been stable since rust 1.42.0.
Other use-cases are covered by assert_matches crate.
2023-09-07 12:52:57 +00:00
behzad nouri 931665551e
rolls out merkle shreds to 100% of mainnet slots (#32979) 2023-09-01 16:58:32 +00:00
Alexander Meißner 9e703f85de
Upgrades Rust to 1.72.0 & nightly-2023-08-25 (#32961)
* allow pedantic invalid cast lint

* allow lint with false-positive triggered by `test-case` crate

* nightly `fmt` correction

* adapt to rust layout changes

* remove dubious test

* Use transmute instead of pointer cast and de/ref when check_aligned is false.

* Renames clippy::integer_arithmetic to clippy::arithmetic_side_effects.

* bump rust nightly to 2023-08-25

* Upgrades Rust to 1.72.0

---------

Co-authored-by: Trent Nelson <trent@solana.com>
2023-09-01 07:26:13 +00:00
behzad nouri 1431275328
removes outdated check for merkle shreds (#33088) 2023-08-31 20:01:54 +00:00
behzad nouri 7275caf708
simplifies sigverify_shreds::get_slot_leaders (#33079) 2023-08-31 19:49:23 +00:00
behzad nouri d5c2dacd07
revises turbine epoch stakes for shreds propagation (#32743)
Bank::get_leader_schedule_epoch returns: 1 + EpochSchedule::get_epoch.
As a result, at the epoch boundaries when the propagated shred is from
the next epoch, we are looking for epoch stakes for 2 epochs ahead of
the root or working bank's epoch.
However, the bank structure only contains epoch stakes for one epoch
ahead. This results in shreds propagated at epoch boundary having
unknown epoch stakes.
2023-08-28 21:31:51 +00:00
Jon Cinque 0fe902ced7
Bump rand to 0.8, rand_chacha to 0.3, getrandom to 0.2 (#32871)
* sdk: Add concurrent support for rand 0.7 and 0.8

* Update rand, rand_chacha, and getrandom versions

* Run command to replace `gen_range`

Run `git grep -l gen_range | xargs sed -i'' -e 's/gen_range(\(\S*\), /gen_range(\1../'

* sdk: Fix users of older `gen_range`

* Replace `hash::new_rand` with `hash::new_with_thread_rng`

Run:
```
git grep -l hash::new_rand | xargs sed -i'' -e 's/hash::new_rand([^)]*/hash::new_with_thread_rng(/'
```

* perf: Use `Keypair::new()` instead of `generate`

* Use older rand version in zk-token-sdk

* program-runtime: Inline random key generation

* bloom: Fix clippy warnings in tests

* streamer: Scope rng usage correctly

* perf: Fix clippy warning

* accounts-db: Map to char to generate a random string

* Remove `from_secret_key_bytes`, it's just `keypair_from_seed`

* ledger: Generate keypairs by hand

* ed25519-tests: Use new rand

* runtime: Use new rand in all tests

* gossip: Clean up clippy and inline keypair generators

* core: Inline keypair generation for tests

* Push sbf lockfile change

* sdk: Sort dependencies correctly

* Remove `hash::new_with_thread_rng`, use `Hash::new_unique()`

* Use Keypair::new where chacha isn't used

* sdk: Fix build by marking rand 0.7 optional

* Hardcode secret key length, add static assertion

* Unify `getrandom` crate usage to fix linking errors

* bloom: Fix tests that require a random hash

* Remove some dependencies, try to unify others

* Remove unnecessary uses of rand and rand_core

* Update lockfiles

* Add back some dependencies to reduce rebuilds

* Increase max rebuilds from 14 to 15

* frozen-abi: Remove `getrandom`

* Bump rebuilds to 17

* Remove getrandom from zk-token-proof
2023-08-21 19:11:21 +02:00
behzad nouri 7bd7410592
uses typed Pubkey instead of opaque [u8; 32] (#32842) 2023-08-18 23:28:08 +00:00
steviez a4c8cc3ce0
Remove improper uses of &Arc<Bank> (#32802)
In most cases, either a &Bank or an Arc<Bank> is more proper.
- &Bank is used if the function only needs a momentary reference
- Arc<Bank> is used if the function needs its' own copy

This PR leaves several instances of &Arc<Bank> around; these instances
are situations where a clone may only happen conditionally.
2023-08-18 16:46:34 -05:00
carllin d5faa6e8aa
Local Cluster Duplicate Switch Test (#32614)
* Add test for broken behavior in same batch

* tests

* redo test

* Important fixes to not immediately duplicate confirm by adding extra node

* Fixup merge

* PR comments

* Redo stakes

* clippy

* fixes

* Resolve conflicts

* add thread logging

* Fixup merge

* Fixup bugs

* Revert "add thread logging"

This reverts commit 9dc22401054b8f91f2b2aa3033e482996913febb.

* Hide scope

* Fixes

* Cleanup test_faulty_node

* More fixes

* Fixes

* Error logging

* Fix duplicate confirmed

* done

* PR comments

* Revert "Error logging"

This reverts commit 18953c36a5e865ecdd38bbf49b8d0502448087d2.

* PR comments

* nit
2023-08-08 19:29:39 -04:00
behzad nouri 0511753276
moves DATA_PLANE_FANOUT to turbine (#32744) 2023-08-07 20:53:49 +00:00
behzad nouri 207820879b
rolls out merkle shreds to ~50% of mainnet slots (#32678) 2023-08-04 20:33:08 +00:00
behzad nouri b0a4669336
demotes unknown_epoch_staked_nodes metric to debug (#32701) 2023-08-04 12:33:08 +00:00
behzad nouri 349f5e0846
read-locks Turbine QUIC connection cache for lookups (#32687)
Frequently accessed remote addresses are cached, making a shared-lock
sufficient which would reduce lock contention for the more common
cache-hit lookups.
2023-08-02 21:54:45 +00:00
behzad nouri 4e4b0a361e
uses RwLock instead of Mutex for cache entries in ClusterNodesCache (#32686)
For as long as cache entries have not expired yet, a shared lock is
enough and would avoid lock contention.
2023-08-02 18:08:18 +00:00
behzad nouri 9efa0eacac
rolls out merkle shreds to ~20% of mainnet slots (#32532) 2023-07-21 14:37:57 +00:00
behzad nouri 7f2f0136bd
enables merkle shreds for devnet and development clusters (#32533) 2023-07-20 12:47:28 +00:00
behzad nouri 868e086d75
upgrades quinn and rustls crates (#32499) 2023-07-14 17:30:57 +00:00
behzad nouri a3ada9c5ea
separates out turbine QUIC from TPU implementation (#32368)
* separates out turbine QUIC from TPU implementation

Turbine being tied to QUIC implementation for TPU hinders development
and makes it hard to optimize QUIC specifically for turbine.
The commit separates out turbine QUIC from TPU implementation.

* Update core/src/validator.rs

Co-authored-by: Jon Cinque <me@jonc.dev>

* Update turbine/src/retransmit_stage.rs

Co-authored-by: Jon Cinque <me@jonc.dev>

---------

Co-authored-by: Jon Cinque <me@jonc.dev>
2023-07-12 14:15:28 +00:00
behzad nouri d54b6204be
removes instances of clippy::manual_let_else (#32417) 2023-07-09 21:41:36 +00:00
behzad nouri 6068676c2a
removes redundant Mutex wrappers around Receiver channels (#32387)
Receiver channels are thread-safe and do not need a Mutex wrappers.
2023-07-05 22:19:59 +00:00
behzad nouri 5a80dc0d73
adds QUIC endpoint specific for turbine connections (#32294)
Working towards separating out turbine QUIC from TPU.
2023-07-03 18:57:18 +00:00
behzad nouri 43a5ed3139
rolls out merkle shreds to ~5% of mainnet slots (#32337) 2023-06-30 15:17:56 +00:00
behzad nouri b722ce4519
rolls out merkle shreds to 100% of testnet slots (#32316) 2023-06-29 18:46:00 +00:00
behzad nouri f6e039b0b3
moves turbine to a separate crate out of solana/core (#32226) 2023-06-22 16:22:11 +00:00