7da2df7d8c
adds methods to obtain shreds' erasure coded block and index ( #21325 )
2021-11-19 20:01:15 +00:00
57057f8d39
uses enum for shred type
...
Current code is using u8 which does not have any type-safety and can
contain invalid values:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L167
Checks for invalid shred-types are scattered through the code:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/blockstore.rs#L849-L851
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L346-L348
The commit uses enum for shred type with #[repr(u8)]. Backward
compatibility is maintained by implementing Serialize and Deserialize
compatible with u8, and adding a test to assert that.
2021-11-19 14:16:39 +00:00
5fb0ab9d00
removes redundant args from Shredder::try_recovery ( #21226 )
...
Shredder::try_recovery is already taking a Vec<Shred> as an argument. All the
other arguments are embedded in the shreds, and are so redundant.
2021-11-10 21:19:03 +00:00
11a53de0e3
rewrites Shredder::try_recovery ( #21082 )
...
For every missing data-shred, Shredder::try_recovery calls into
new_empty_data_shred which does a redundant serialization into payload
buffer which is then immediately overwritten by the erasure recovery:
https://github.com/solana-labs/solana/blob/696501500/ledger/src/shred.rs#L372-L417
Additionally, the implementation is unnecessary complex hindering
upcoming changes to erasure coding generation
https://github.com/solana-labs/solana/blob/696501500/ledger/src/shred.rs#L814-L938
The commit simplifies the Shredder::try_recovery implementation.
2021-11-04 16:15:51 +00:00
4642a2c856
Optimize test_recovery_with_expanded_coding_shreds ( #20849 )
2021-10-28 11:38:08 +02:00
7027d56064
Resolve nightly-2021-10-05 clippy complaints
2021-10-06 10:37:58 -07:00
6d9818b8e4
skips retransmit for shreds with unknown slot leader ( #19472 )
...
Shreds' signatures should be verified before they reach retransmit
stage, and if the leader is unknown they should fail signature check.
Therefore retransmit-stage can as well expect to know who the slot
leader is and otherwise just skip the shred.
Blockstore checking signature of recovered shreds before sending them to
retransmit stage:
https://github.com/solana-labs/solana/blob/4305d4b7b/ledger/src/blockstore.rs#L884-L930
Shred signature verifier:
https://github.com/solana-labs/solana/blob/4305d4b7b/core/src/sigverify_shreds.rs#L41-L57
https://github.com/solana-labs/solana/blob/4305d4b7b/ledger/src/sigverify_shreds.rs#L105
2021-09-01 15:44:26 +00:00
7f2254225e
Move entry/poh to own crate to speed up poh bench build ( #18225 )
2021-07-14 14:16:29 +02:00
a86ced0bac
generate deterministic seeds for shreds ( #17950 )
...
* generate shred seed from leader pubkey
* clippy
* clippy
* review
* review 2
* fmt
* review
* check
* review
* cleanup
* fmt
2021-07-07 08:21:12 -07:00
84b9de8c18
Shredder no longer holds a keypair
2021-06-21 21:29:52 -07:00
6514096a67
chore: cargo +nightly clippy --fix -Z unstable-options
2021-06-18 10:42:46 -07:00
81ad795d46
removes position field in coding-shred-header
...
CodingShredHeader.position is equal to
ShredCommonHeader.index - ShredCommonHeader.fec_set_index
and is so redundant. The extra position field can add bugs if not
consistent with index and fec_set_index.
2021-05-10 13:20:56 +00:00
bc31378797
Trim extra shred bytes in blockstore ( #16602 )
...
Strip the zero-padding off of data shreds before insertion into blockstore
Co-authored-by: Stephen Akridge <sakridge@gmail.com>
Co-authored-by: Nathan Hawkins <utsl@utsl.org>
2021-04-27 17:40:41 -05:00
0f3ac51cf1
limits to data_header.size when combining shreds' payloads ( #16708 )
...
Shredder::deshred is ignoring data_header.size when combining shreds' payloads:
https://github.com/solana-labs/solana/blob/37b8587d4/ledger/src/shred.rs#L940-L961
Also adding more sanity checks on the alignment of data shreds indices.
2021-04-27 12:04:44 +00:00
03194145c0
removes first_coding_index from erasure recovery code ( #16646 )
...
first_coding_index is the same as the set_index and is so redundant:
https://github.com/solana-labs/solana/blob/37b8587d4/ledger/src/blockstore_meta.rs#L49-L60
2021-04-23 12:00:37 +00:00
37b8587d4e
expands number of erasure coding shreds in the last batch in slots ( #16484 )
...
Number of parity coding shreds is always less than the number of data
shreds in FEC blocks:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L719
Data shreds are batched in chunks of 32 shreds each:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L714
However the very last batch of data shreds in a slot can be small, in
which case the loss rate can be exacerbated.
This commit expands the number of coding shreds in the last FEC block in
slots to: 64 - number of data shreds; so that FEC blocks are always 64
data and parity coding shreds each.
As a consequence of this, the last FEC block has more parity coding
shreds than data shreds. So for some shred indices we will have a coding
shred but no data shreds. This should not cause any kind of overlapping
FEC blocks as in:
https://github.com/solana-labs/solana/pull/10095
since this is done only for the very last batch in a slot, and the next
slot will reset the shred index.
2021-04-21 12:47:50 +00:00
bb24318ef0
Document shreds ( #16514 )
...
No functionality changes from this commit
2021-04-16 14:04:46 -05:00
4f82b897bc
buffers data shreds to make larger erasure coded sets ( #15849 )
...
Broadcast stage batches up to 8 entries:
https://github.com/solana-labs/solana/blob/79280b304/core/src/broadcast_stage/broadcast_utils.rs#L26-L29
which will be serialized into some number of shreds and chunked into FEC
sets of at most 32 shreds each:
https://github.com/solana-labs/solana/blob/79280b304/ledger/src/shred.rs#L576-L597
So depending on the size of entries, FEC sets can be small, which may
aggravate loss rate.
For example 16 FEC sets of 2:2 data/code shreds each have higher loss
rate than one 32:32 set.
This commit broadcasts data shreds immediately, but also buffers them
until it has a batch of 32 data shreds, at which point 32 coding shreds
are generated and broadcasted.
2021-03-23 14:52:38 +00:00
3b85cbc504
renames is_last_in_fec_set back to is_last_data ( #15848 )
...
https://github.com/solana-labs/solana/pull/10095
renamed is_last_data to is_last_in_fec_set. However, the code shows that
this is actually meant to indicate where the serialized data is
complete:
https://github.com/solana-labs/solana/blob/420174d3d/ledger/src/shred.rs#L599-L600
https://github.com/solana-labs/solana/blob/420174d3d/ledger/src/shred.rs#L229-L231
There are multiple FEC sets for each `&[Entry]` serialized and this flag
does not represent shreds last in FEC sets (only the very last one by
overlap). So the name is wrong and confusing
2021-03-16 10:09:16 +00:00
918d04e3f0
Add more slot update notifications ( #15734 )
...
* Add more slot update notifications
* fix merge
* Address feedback and add integration test
* switch to datapoint
* remove unused shred method
* fix clippy
* new thread for rpc completed slots
* remove extra constant
* fixes
* rely on channel closing
* fix check
2021-03-12 21:44:06 +08:00
6a3797e164
adds crds-value for broadcasting duplicate shreds through gossip ( #14133 )
...
In gossip, the header overhead we get from:
https://github.com/solana-labs/solana/blob/de9ac43eb/core/src/cluster_info.rs#L434-L435
https://github.com/solana-labs/solana/blob/de9ac43eb/core/src/crds_value.rs#L31-L36
https://github.com/solana-labs/solana/blob/de9ac43eb/core/src/crds_value.rs#L73
already exceeds SIZE_OF_NONCE in shreds. We also need aditional
meta-data (wallclock, source pubkey, ...). Which means that given the
SHRED_PAYLOAD_SIZE, we cannot fit all these in PACKET_DATA_SIZE:
https://github.com/solana-labs/solana/blob/de9ac43eb/ledger/src/shred.rs#L80
On top of that, we need 2 shred payloads as the proof of duplicate. So
each DuplicateShred crds value includes only a chunk of the payload,
along with the meta-data to reconstruct the full payload from the chunks
on the receiving end.
2020-12-18 14:32:43 +00:00
d4a174fb7c
Partial shred deserialize cleanup and shred type differentiation ( #14094 )
...
* Partial shred deserialize cleanup and shred type differentiation in retransmit
* consolidate packet hashing logic
2020-12-15 16:50:40 -08:00
7143aaa89b
Clippy
2020-12-14 08:03:29 -08:00
c5fe076432
Better dupe detection ( #13992 )
2020-12-09 23:14:31 -08:00
afc1b59475
Fix assertion failure ( #13626 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-16 21:30:38 -08:00
b4cf968e14
Add back shredding broadcast stats ( #13463 )
2020-11-09 23:04:27 -08:00
841ecfd927
chore(deps): bump bincode from 1.2.1 to 1.3.1 ( #10867 )
...
* chore(deps): bump bincode from 1.2.1 to 1.3.1
Bumps [bincode](https://github.com/servo/bincode ) from 1.2.1 to 1.3.1.
- [Release notes](https://github.com/servo/bincode/releases )
- [Commits](https://github.com/servo/bincode/commits )
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
* [auto-commit] Update all Cargo lock files
* Switch from deprecated method
* Add options to maintain behavior with bincode::options()
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-07-09 00:08:05 +00:00
0550b893b0
Fix typos ( #10675 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2020-06-17 20:54:52 -07:00
439fd30840
Fix erasure ( #10095 )
...
* Fix bad FEC blocks
* Add test
Co-authored-by: Carl <carl@solana.com>
2020-05-19 16:13:12 -07:00
97f2bcff69
master: Add nonce to shreds repairs, add shred data size to header ( #10109 )
...
* Add nonce to shreds/repairs
* Add data shred size to header
Co-authored-by: Carl <carl@solana.com>
2020-05-19 12:38:18 -07:00
f562ed4cc8
Distinguish between shred type in shred fetch stage duplicate filter ( #10068 )
...
* Shred type check
* Test
2020-05-15 13:23:56 -07:00
58ef02f02b
9951 clippy errors in the test suite ( #10030 )
...
automerge
2020-05-15 09:35:43 -07:00
a5419fe79e
Error for invalid shred. ( #9588 )
2020-04-19 21:15:09 -07:00
30b3862770
Don't unwrap on session new
2020-04-16 08:25:45 -07:00
73063544bd
Move shred_version module to sdk/
2020-02-24 14:46:12 -07:00
ab361a8073
Rename KeypairUtil to Signer ( #8360 )
...
automerge
2020-02-20 13:28:55 -08:00
ed5101b031
Generate max coding shreds when necessary ( #8099 )
...
* Generate max coding shreds when necessary
* test
2020-02-04 15:45:01 -08:00
a2f2c46f87
Ensure shred version is never 0
2020-01-24 17:41:20 -07:00
37d1daf58e
Revert "Generate MAX_DATA_SHREDS_PER_FEC_BLOCK coding shreds for each FEC block ( #7474 )" ( #7898 )
...
automerge
2020-01-21 11:48:09 -08:00
2dd8ab197d
Remove redundant threadpools in sigverify ( #7888 )
...
* Limit the number of thread pools sigverify creates
* Name local threadpools
2020-01-20 20:08:19 -08:00
87b2525e03
Limit maximum number of shreds in a slot to 32K ( #7584 )
...
* Limit maximum number of shreds in a slot to 32K
* mark dead slot replay as fatal error
2019-12-30 07:42:09 -08:00
97589f77f8
Pipeline broadcast socket transmit and blocktree record ( #7481 )
...
automerge
2019-12-16 17:11:18 -08:00
7adc721d96
Generate MAX_DATA_SHREDS_PER_FEC_BLOCK coding shreds for each FEC block ( #7474 )
...
* Generate MAX_DATA_SHREDS_PER_FEC_BLOCK coding shreds for each FEC block
* fix tests, and detangle ErasureMeta::status
2019-12-13 14:50:22 -08:00
d94041e98d
Allow coding shred index to be different than data shred index ( #7438 )
...
* Allow coding shred index to be different than data shred index
* move fec_set_index to shred's common header
* fix bench
2019-12-12 16:50:29 -08:00
e082418e4a
Cleanup the sign shreds interface ( #7456 )
...
automerge
2019-12-12 13:27:33 -08:00
f526c424c5
Move slow shred multi_fec test to integration tests folder ( #7426 )
2019-12-11 11:10:21 -08:00
6796b08909
Migrate to thiserror ( #7177 )
...
* Migrate to thiserror
* Discourage the use of other modules' Result alias
`io::Result` set a bad precedent. Don't import other `Result`
aliases.
2019-12-02 15:42:05 -07:00
6bfe0fca1f
Add a version field to shreds ( #7023 )
...
* Add a version field to shreds
* Clippy
* Fix Chacha Golden
* Fix shredder bench compile
* Fix blocktree bench compile
2019-11-18 18:05:02 -08:00
7c32640a9b
Set index and set data should write into shred data ( #6995 )
2019-11-16 02:41:59 -08:00
6009801c5f
More granular timings in shred generation ( #6900 )
2019-11-13 16:30:12 -08:00
behzad nouri
sakridge
Michael Vines
jbiseda
Alexander Meißner
steviez
Justin Starry
carllin
dependabot-preview[bot]
Greg Fitzgerald
Kristofer Peterson
Tyera Eulberg
Pankaj Garg
Sagar Dhawan
anatoly yakovenko
sakridge