96d977fd05
test_shred_fec_set_index fix ( #24540 )
2022-04-20 16:26:44 -07:00
705ea53353
moves sign_shred and new_coding_shred_header out of Shredder ( #24487 )
2022-04-19 20:00:05 +00:00
3bbfaae7b6
moves shred stats to a separate file ( #24484 )
2022-04-19 18:25:09 +00:00
855801cc95
removes deterministic-shred-seed feature
2022-04-05 12:04:12 +00:00
cda3d66b21
uses first_coding_index for erasure meta obtained from coding shreds ( #23974 )
...
Now that nodes correctly populate position field in coding shreds, and
first_coding_index in erasure meta, the old code to maintain backward
compatibility can be removed.
The commit is working towards changing erasure coding schema to 32:64.
2022-03-30 13:55:11 +00:00
fc2d1a61f3
signing_coding_time is not used ( #23655 )
2022-03-14 17:16:30 -05:00
6d5bbca630
Pacify clippy
2022-01-21 19:12:57 -08:00
65d59f4ef0
tracks erasure coding shreds' indices explicitly ( #21822 )
...
The indices for erasure coding shreds are tied to data shreds:
https://github.com/solana-labs/solana/blob/90f41fd9b/ledger/src/shred.rs#L921
However with the upcoming changes to erasure schema, there will be more
erasure coding shreds than data shreds and we can no longer infer coding
shreds indices from data shreds.
The commit adds constructs to track coding shreds indices explicitly.
2021-12-19 22:37:55 +00:00
89d66c3210
removes next_shred_index from return value of entries to shreds api ( #21961 )
...
next-shred-index is already readily available from returned data shreds.
The commit simplifies the api for upcoming changes to erasure coding
schema which will require explicit tracking of indices for coding shreds
as well as data shreds.
2021-12-17 15:01:55 +00:00
8183f28636
adds ErasureSetId identifying erasure coding sets of shreds ( #21928 )
2021-12-16 14:18:55 +00:00
4ceb2689f5
adds ShredId uniquely identifying each shred ( #21820 )
2021-12-14 17:34:02 +00:00
e19c7923c3
chore: fix typo in shred.rs ( #21890 )
...
begining -> beginning
2021-12-14 11:19:27 -05:00
49ba09b333
adds back ErasureMeta::first_coding_index field ( #21623 )
...
https://github.com/solana-labs/solana/pull/16646
removed first_coding_index since the field is currently redundant and
always equal to fec_set_index.
However, with upcoming changes to erasure coding schema, this will no
longer be the same as fec_set_index and so requires a separate field to
represent.
2021-12-10 20:08:04 +00:00
8063273d09
adds more sanity checks to shreds ( #21675 )
2021-12-09 16:43:57 +00:00
cd17f63d81
adds back position field to coding-shred-header ( #21600 )
...
https://github.com/solana-labs/solana/pull/17004
removed position field from coding-shred-header because as it stands the
field is redundant and unused.
However, with the upcoming changes to erasure coding schema this field
will no longer be redundant and needs to be populated.
2021-12-05 14:42:09 +00:00
b8837c04ec
Reformat imports to a consistent style for imports
...
rustfmt.toml configuration:
imports_granularity = "One"
group_imports = "One"
2021-12-03 09:19:13 -08:00
dd338b6c9f
changes Shred::parent return type to Option<Slot> ( #21370 )
...
Shred::parent can return garbage if the struct fields are invalid:
https://github.com/solana-labs/solana/blob/8a50b6302/ledger/src/shred.rs#L446-L453
The commit adds more sanity checks and changes the return type to Option<Slot>.
2021-11-23 14:45:26 +00:00
7da2df7d8c
adds methods to obtain shreds' erasure coded block and index ( #21325 )
2021-11-19 20:01:15 +00:00
57057f8d39
uses enum for shred type
...
Current code is using u8 which does not have any type-safety and can
contain invalid values:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L167
Checks for invalid shred-types are scattered through the code:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/blockstore.rs#L849-L851
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L346-L348
The commit uses enum for shred type with #[repr(u8)]. Backward
compatibility is maintained by implementing Serialize and Deserialize
compatible with u8, and adding a test to assert that.
2021-11-19 14:16:39 +00:00
5fb0ab9d00
removes redundant args from Shredder::try_recovery ( #21226 )
...
Shredder::try_recovery is already taking a Vec<Shred> as an argument. All the
other arguments are embedded in the shreds, and are so redundant.
2021-11-10 21:19:03 +00:00
11a53de0e3
rewrites Shredder::try_recovery ( #21082 )
...
For every missing data-shred, Shredder::try_recovery calls into
new_empty_data_shred which does a redundant serialization into payload
buffer which is then immediately overwritten by the erasure recovery:
https://github.com/solana-labs/solana/blob/696501500/ledger/src/shred.rs#L372-L417
Additionally, the implementation is unnecessary complex hindering
upcoming changes to erasure coding generation
https://github.com/solana-labs/solana/blob/696501500/ledger/src/shred.rs#L814-L938
The commit simplifies the Shredder::try_recovery implementation.
2021-11-04 16:15:51 +00:00
4642a2c856
Optimize test_recovery_with_expanded_coding_shreds ( #20849 )
2021-10-28 11:38:08 +02:00
7027d56064
Resolve nightly-2021-10-05 clippy complaints
2021-10-06 10:37:58 -07:00
6d9818b8e4
skips retransmit for shreds with unknown slot leader ( #19472 )
...
Shreds' signatures should be verified before they reach retransmit
stage, and if the leader is unknown they should fail signature check.
Therefore retransmit-stage can as well expect to know who the slot
leader is and otherwise just skip the shred.
Blockstore checking signature of recovered shreds before sending them to
retransmit stage:
https://github.com/solana-labs/solana/blob/4305d4b7b/ledger/src/blockstore.rs#L884-L930
Shred signature verifier:
https://github.com/solana-labs/solana/blob/4305d4b7b/core/src/sigverify_shreds.rs#L41-L57
https://github.com/solana-labs/solana/blob/4305d4b7b/ledger/src/sigverify_shreds.rs#L105
2021-09-01 15:44:26 +00:00
7f2254225e
Move entry/poh to own crate to speed up poh bench build ( #18225 )
2021-07-14 14:16:29 +02:00
a86ced0bac
generate deterministic seeds for shreds ( #17950 )
...
* generate shred seed from leader pubkey
* clippy
* clippy
* review
* review 2
* fmt
* review
* check
* review
* cleanup
* fmt
2021-07-07 08:21:12 -07:00
84b9de8c18
Shredder no longer holds a keypair
2021-06-21 21:29:52 -07:00
6514096a67
chore: cargo +nightly clippy --fix -Z unstable-options
2021-06-18 10:42:46 -07:00
81ad795d46
removes position field in coding-shred-header
...
CodingShredHeader.position is equal to
ShredCommonHeader.index - ShredCommonHeader.fec_set_index
and is so redundant. The extra position field can add bugs if not
consistent with index and fec_set_index.
2021-05-10 13:20:56 +00:00
bc31378797
Trim extra shred bytes in blockstore ( #16602 )
...
Strip the zero-padding off of data shreds before insertion into blockstore
Co-authored-by: Stephen Akridge <sakridge@gmail.com>
Co-authored-by: Nathan Hawkins <utsl@utsl.org>
2021-04-27 17:40:41 -05:00
0f3ac51cf1
limits to data_header.size when combining shreds' payloads ( #16708 )
...
Shredder::deshred is ignoring data_header.size when combining shreds' payloads:
https://github.com/solana-labs/solana/blob/37b8587d4/ledger/src/shred.rs#L940-L961
Also adding more sanity checks on the alignment of data shreds indices.
2021-04-27 12:04:44 +00:00
03194145c0
removes first_coding_index from erasure recovery code ( #16646 )
...
first_coding_index is the same as the set_index and is so redundant:
https://github.com/solana-labs/solana/blob/37b8587d4/ledger/src/blockstore_meta.rs#L49-L60
2021-04-23 12:00:37 +00:00
37b8587d4e
expands number of erasure coding shreds in the last batch in slots ( #16484 )
...
Number of parity coding shreds is always less than the number of data
shreds in FEC blocks:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L719
Data shreds are batched in chunks of 32 shreds each:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L714
However the very last batch of data shreds in a slot can be small, in
which case the loss rate can be exacerbated.
This commit expands the number of coding shreds in the last FEC block in
slots to: 64 - number of data shreds; so that FEC blocks are always 64
data and parity coding shreds each.
As a consequence of this, the last FEC block has more parity coding
shreds than data shreds. So for some shred indices we will have a coding
shred but no data shreds. This should not cause any kind of overlapping
FEC blocks as in:
https://github.com/solana-labs/solana/pull/10095
since this is done only for the very last batch in a slot, and the next
slot will reset the shred index.
2021-04-21 12:47:50 +00:00
bb24318ef0
Document shreds ( #16514 )
...
No functionality changes from this commit
2021-04-16 14:04:46 -05:00
4f82b897bc
buffers data shreds to make larger erasure coded sets ( #15849 )
...
Broadcast stage batches up to 8 entries:
https://github.com/solana-labs/solana/blob/79280b304/core/src/broadcast_stage/broadcast_utils.rs#L26-L29
which will be serialized into some number of shreds and chunked into FEC
sets of at most 32 shreds each:
https://github.com/solana-labs/solana/blob/79280b304/ledger/src/shred.rs#L576-L597
So depending on the size of entries, FEC sets can be small, which may
aggravate loss rate.
For example 16 FEC sets of 2:2 data/code shreds each have higher loss
rate than one 32:32 set.
This commit broadcasts data shreds immediately, but also buffers them
until it has a batch of 32 data shreds, at which point 32 coding shreds
are generated and broadcasted.
2021-03-23 14:52:38 +00:00
3b85cbc504
renames is_last_in_fec_set back to is_last_data ( #15848 )
...
https://github.com/solana-labs/solana/pull/10095
renamed is_last_data to is_last_in_fec_set. However, the code shows that
this is actually meant to indicate where the serialized data is
complete:
https://github.com/solana-labs/solana/blob/420174d3d/ledger/src/shred.rs#L599-L600
https://github.com/solana-labs/solana/blob/420174d3d/ledger/src/shred.rs#L229-L231
There are multiple FEC sets for each `&[Entry]` serialized and this flag
does not represent shreds last in FEC sets (only the very last one by
overlap). So the name is wrong and confusing
2021-03-16 10:09:16 +00:00
918d04e3f0
Add more slot update notifications ( #15734 )
...
* Add more slot update notifications
* fix merge
* Address feedback and add integration test
* switch to datapoint
* remove unused shred method
* fix clippy
* new thread for rpc completed slots
* remove extra constant
* fixes
* rely on channel closing
* fix check
2021-03-12 21:44:06 +08:00
6a3797e164
adds crds-value for broadcasting duplicate shreds through gossip ( #14133 )
...
In gossip, the header overhead we get from:
https://github.com/solana-labs/solana/blob/de9ac43eb/core/src/cluster_info.rs#L434-L435
https://github.com/solana-labs/solana/blob/de9ac43eb/core/src/crds_value.rs#L31-L36
https://github.com/solana-labs/solana/blob/de9ac43eb/core/src/crds_value.rs#L73
already exceeds SIZE_OF_NONCE in shreds. We also need aditional
meta-data (wallclock, source pubkey, ...). Which means that given the
SHRED_PAYLOAD_SIZE, we cannot fit all these in PACKET_DATA_SIZE:
https://github.com/solana-labs/solana/blob/de9ac43eb/ledger/src/shred.rs#L80
On top of that, we need 2 shred payloads as the proof of duplicate. So
each DuplicateShred crds value includes only a chunk of the payload,
along with the meta-data to reconstruct the full payload from the chunks
on the receiving end.
2020-12-18 14:32:43 +00:00
d4a174fb7c
Partial shred deserialize cleanup and shred type differentiation ( #14094 )
...
* Partial shred deserialize cleanup and shred type differentiation in retransmit
* consolidate packet hashing logic
2020-12-15 16:50:40 -08:00
7143aaa89b
Clippy
2020-12-14 08:03:29 -08:00
c5fe076432
Better dupe detection ( #13992 )
2020-12-09 23:14:31 -08:00
afc1b59475
Fix assertion failure ( #13626 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-16 21:30:38 -08:00
b4cf968e14
Add back shredding broadcast stats ( #13463 )
2020-11-09 23:04:27 -08:00
841ecfd927
chore(deps): bump bincode from 1.2.1 to 1.3.1 ( #10867 )
...
* chore(deps): bump bincode from 1.2.1 to 1.3.1
Bumps [bincode](https://github.com/servo/bincode ) from 1.2.1 to 1.3.1.
- [Release notes](https://github.com/servo/bincode/releases )
- [Commits](https://github.com/servo/bincode/commits )
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
* [auto-commit] Update all Cargo lock files
* Switch from deprecated method
* Add options to maintain behavior with bincode::options()
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-07-09 00:08:05 +00:00
0550b893b0
Fix typos ( #10675 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2020-06-17 20:54:52 -07:00
439fd30840
Fix erasure ( #10095 )
...
* Fix bad FEC blocks
* Add test
Co-authored-by: Carl <carl@solana.com>
2020-05-19 16:13:12 -07:00
97f2bcff69
master: Add nonce to shreds repairs, add shred data size to header ( #10109 )
...
* Add nonce to shreds/repairs
* Add data shred size to header
Co-authored-by: Carl <carl@solana.com>
2020-05-19 12:38:18 -07:00
f562ed4cc8
Distinguish between shred type in shred fetch stage duplicate filter ( #10068 )
...
* Shred type check
* Test
2020-05-15 13:23:56 -07:00
58ef02f02b
9951 clippy errors in the test suite ( #10030 )
...
automerge
2020-05-15 09:35:43 -07:00
a5419fe79e
Error for invalid shred. ( #9588 )
2020-04-19 21:15:09 -07:00
Jeff Biseda
behzad nouri
HaoranYi
Michael Vines
Ikko Ashimine
sakridge
Alexander Meißner
steviez
Justin Starry
carllin
dependabot-preview[bot]
Greg Fitzgerald
Kristofer Peterson