rolls out chained Merkle shreds to 100% of testnet slots (#2858)
(cherry picked from commit a15fef1b26)
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
extends Turbine fanout experiment to wider fanout values (#2373)
Based on previous Turbine fanout experiment, wider fanouts are more
effective in propagating shreds and reducing repairs:
https://discord.com/channels/428295358100013066/478692221441409024/1265782094211321897
In order to identify optimal fanout value, this commit extends the
experiment with wider fanout values.
(cherry picked from commit 57144b0cea)
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
rolls out chained Merkle shreds to ~50% of testnet slots (#2660)
(cherry picked from commit f2b7ef4aea)
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* customizes override logic for gossip ContactInfo (#2579)
If there are two running instances of the same node, we want the
ContactInfo with more recent start time to be propagated through
gossip regardless of wallclocks.
The commit adds custom override logic for ContactInfo to first compare
by outset timestamp.
* updates ContactInfo.outset when hot-swapping identity (#2613)
When hot-swapping identity, ContactInfo.outset should be updated so that
the new ContactInfo overrides older node with the same pubkey.
* patches bug causing false duplicate nodes error (#2666)
The bootstrap code during the validator start pushes a contact-info with
more recent timestamp to gossip. If the node is staked the contact-info
lingers in gossip causing false duplicate node instances when the fully
initialized node joins gossip later on.
The commit refreshes the timestamp on contact-info so that it overrides
the one pushed by bootstrap and avoid false duplicates error.
---------
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
rolls out chained Merkle shreds to ~21% of testnet slots (#2503)
(cherry picked from commit ea10d2e534)
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
The implementation of Blockstore::insert_shreds() holds a write lock for
the entirety of the function. Thus, there is no point in having more
than one thread in BroadcastStage insert shreds.
Looking up Merkle root of the last erasure batch for the parent block
can fail if the slot-meta is not yet available in blockstore.
This commit instead retains chained Merkle root across leader slots. If
the parent of the current block is the previous leader slot, then the
chained Merkle root is readily available and we can bypass blockstore
lookup.
```
error: assigning the result of `Clone::clone()` may be inefficient
--> bucket_map/src/bucket.rs:979:17
|
979 | hashed = hashed_raw.clone();
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: use `clone_from()`: `hashed.clone_from(&hashed_raw)`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#assigning_clones
= note: `-D clippy::assigning-clones` implied by `-D warnings`
= help: to override `-D warnings` add `#[allow(clippy::assigning_clones)]`
```
Merkle shreds sign the Merkle root of the erasure batch, so all shreds
within the same erasure batch have the same signature. The commit
improves shreds signature verification by adding an LRU cache.
We have a similar field for transaction sigverify, and knowing the
number of batches gives us insight into how packets are being pulled out
by the shred receive sockets
For duplicate blocks prevention we want to verify that the last erasure
batch was sufficiently propagated through turbine. This requires
additional bookkeeping because, depending on the erasure coding schema,
the entire batch might be recovered from only a few coding shreds.
In order to simplify above, this commit instead ensures that the last
erasure batch has >= 32 data shreds so that the batch cannot be
recovered unless 32+ shreds are received from turbine or repair.
Waiting on receiver.recv() can unnecessarily block while the connection is already closed.
The commit exits send_datagram_task if the connection is closed.
* Initialize fork graph in program cache during bank_forks creation
* rename BankForks::new to BankForks::new_rw_arc
* fix compilation
* no need to set fork_graph on insert()
* fix partition tests
This macro is used a lot for tests to create a ledger path in order to
open a Blockstore. Files will be left on disk unless the test remembers
to call Blockstore::destroy() on the directory. So, instead of requiring
this, use the get_tmp_ledger_path_auto_delete!() macro that creates a
TempDir (which automatically deletes itself when it goes out of scope).
The commit implements lazy eviction for turbine QUIC connections.
The cache is allowed to grow to 2 x capacity at which point at least
half of the entries with lowest stake are evicted, resulting in an
amortized O(1) performance.
Currently each outgoing shred will attempt to establish a connection if
one does not already exist. This is very wasteful and consumes many
tokio tasks if the remote node is down or unresponsive.
The commit decouples routing packets from establishing connections by
adding a buffering channel for each remote address. Outgoing packets are
always sent down this channel to be processed once the connection is
established. If connecting attempt fails, all packets already pushed to
the channel are dropped at once, reducing the number of attempts to make
a connection if the remote node is down or unresponsive.
removes outdated matches crate from the dependencies
std::matches has been stable since rust 1.42.0.
Other use-cases are covered by assert_matches crate.
* allow pedantic invalid cast lint
* allow lint with false-positive triggered by `test-case` crate
* nightly `fmt` correction
* adapt to rust layout changes
* remove dubious test
* Use transmute instead of pointer cast and de/ref when check_aligned is false.
* Renames clippy::integer_arithmetic to clippy::arithmetic_side_effects.
* bump rust nightly to 2023-08-25
* Upgrades Rust to 1.72.0
---------
Co-authored-by: Trent Nelson <trent@solana.com>
mergify[bot]
behzad nouri
steviez
Illia Bobyr
Yihau Chen
Brooks
Lijun Wang
GoodDaisy
Ashwin Sekar
Lucas Steuernagel
Brooks
Jeff Biseda
Pankaj Garg
steviez
Alexander Meißner