blockworks-foundation
/
solana
mirror of https://github.com/blockworks-foundation/solana.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
19,073 Commits 174 Branches 497 Tags 541 MiB
Commit Graph

99 Commits

Author SHA1 Message Date
steviez bc31378797
Trim extra shred bytes in blockstore (#16602) 
Strip the zero-padding off of data shreds before insertion into blockstore

Co-authored-by: Stephen Akridge <sakridge@gmail.com>
Co-authored-by: Nathan Hawkins <utsl@utsl.org>
 2021-04-27 17:40:41 -05:00
behzad nouri 0f3ac51cf1
limits to data_header.size when combining shreds' payloads (#16708) 
Shredder::deshred is ignoring data_header.size when combining shreds' payloads:
https://github.com/solana-labs/solana/blob/37b8587d4/ledger/src/shred.rs#L940-L961

Also adding more sanity checks on the alignment of data shreds indices.
 2021-04-27 12:04:44 +00:00
behzad nouri 03194145c0
removes first_coding_index from erasure recovery code (#16646) 
first_coding_index is the same as the set_index and is so redundant:
https://github.com/solana-labs/solana/blob/37b8587d4/ledger/src/blockstore_meta.rs#L49-L60
 2021-04-23 12:00:37 +00:00
behzad nouri 37b8587d4e
expands number of erasure coding shreds in the last batch in slots (#16484) 
Number of parity coding shreds is always less than the number of data
shreds in FEC blocks:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L719

Data shreds are batched in chunks of 32 shreds each:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L714

However the very last batch of data shreds in a slot can be small, in
which case the loss rate can be exacerbated.

This commit expands the number of coding shreds in the last FEC block in
slots to: 64 - number of data shreds; so that FEC blocks are always 64
data and parity coding shreds each.

As a consequence of this, the last FEC block has more parity coding
shreds than data shreds. So for some shred indices we will have a coding
shred but no data shreds. This should not cause any kind of overlapping
FEC blocks as in:
https://github.com/solana-labs/solana/pull/10095
since this is done only for the very last batch in a slot, and the next
slot will reset the shred index.
 2021-04-21 12:47:50 +00:00
steviez bb24318ef0
Document shreds (#16514) 
No functionality changes from this commit
 2021-04-16 14:04:46 -05:00
behzad nouri 4f82b897bc
buffers data shreds to make larger erasure coded sets (#15849) 
Broadcast stage batches up to 8 entries:
https://github.com/solana-labs/solana/blob/79280b304/core/src/broadcast_stage/broadcast_utils.rs#L26-L29
which will be serialized into some number of shreds and chunked into FEC
sets of at most 32 shreds each:
https://github.com/solana-labs/solana/blob/79280b304/ledger/src/shred.rs#L576-L597
So depending on the size of entries, FEC sets can be small, which may
aggravate loss rate.
For example 16 FEC sets of 2:2 data/code shreds each have higher loss
rate than one 32:32 set.

This commit broadcasts data shreds immediately, but also buffers them
until it has a batch of 32 data shreds, at which point 32 coding shreds
are generated and broadcasted.
 2021-03-23 14:52:38 +00:00
behzad nouri 3b85cbc504
renames is_last_in_fec_set back to is_last_data (#15848) 
https://github.com/solana-labs/solana/pull/10095
renamed is_last_data to is_last_in_fec_set. However, the code shows that
this is actually meant to indicate where the serialized data is
complete:
https://github.com/solana-labs/solana/blob/420174d3d/ledger/src/shred.rs#L599-L600
https://github.com/solana-labs/solana/blob/420174d3d/ledger/src/shred.rs#L229-L231

There are multiple FEC sets for each `&[Entry]` serialized and this flag
does not represent shreds last in FEC sets (only the very last one by
overlap). So the name is wrong and confusing
 2021-03-16 10:09:16 +00:00
Justin Starry 918d04e3f0
Add more slot update notifications (#15734) 
* Add more slot update notifications

* fix merge

* Address feedback and add integration test

* switch to datapoint

* remove unused shred method

* fix clippy

* new thread for rpc completed slots

* remove extra constant

* fixes

* rely on channel closing

* fix check
 2021-03-12 21:44:06 +08:00
behzad nouri 6a3797e164
adds crds-value for broadcasting duplicate shreds through gossip (#14133) 
In gossip, the header overhead we get from:
https://github.com/solana-labs/solana/blob/de9ac43eb/core/src/cluster_info.rs#L434-L435
https://github.com/solana-labs/solana/blob/de9ac43eb/core/src/crds_value.rs#L31-L36
https://github.com/solana-labs/solana/blob/de9ac43eb/core/src/crds_value.rs#L73
already exceeds SIZE_OF_NONCE in shreds. We also need aditional
meta-data (wallclock, source pubkey, ...). Which means that given the
SHRED_PAYLOAD_SIZE, we cannot fit all these in PACKET_DATA_SIZE:
https://github.com/solana-labs/solana/blob/de9ac43eb/ledger/src/shred.rs#L80

On top of that, we need 2 shred payloads as the proof of duplicate. So
each DuplicateShred crds value includes only a chunk of the payload,
along with the meta-data to reconstruct the full payload from the chunks
on the receiving end.
 2020-12-18 14:32:43 +00:00
sakridge d4a174fb7c
Partial shred deserialize cleanup and shred type differentiation (#14094) 
* Partial shred deserialize cleanup and shred type differentiation in retransmit

* consolidate packet hashing logic
 2020-12-15 16:50:40 -08:00
Michael Vines 7143aaa89b Clippy 2020-12-14 08:03:29 -08:00
sakridge c5fe076432
Better dupe detection (#13992) 2020-12-09 23:14:31 -08:00
carllin afc1b59475
Fix assertion failure (#13626) 
Co-authored-by: Carl Lin <carl@solana.com>
 2020-11-16 21:30:38 -08:00
sakridge b4cf968e14
Add back shredding broadcast stats (#13463) 2020-11-09 23:04:27 -08:00
dependabot-preview[bot] 841ecfd927
chore(deps): bump bincode from 1.2.1 to 1.3.1 (#10867) 
* chore(deps): bump bincode from 1.2.1 to 1.3.1

Bumps [bincode](https://github.com/servo/bincode) from 1.2.1 to 1.3.1.
- [Release notes](https://github.com/servo/bincode/releases)
- [Commits](https://github.com/servo/bincode/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

* Switch from deprecated method

* Add options to maintain behavior with bincode::options()

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2020-07-09 00:08:05 +00:00
Greg Fitzgerald 0550b893b0
Fix typos (#10675) 
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2020-06-17 20:54:52 -07:00
carllin 439fd30840
Fix erasure (#10095) 
* Fix bad FEC blocks

* Add test

Co-authored-by: Carl <carl@solana.com>
 2020-05-19 16:13:12 -07:00
carllin 97f2bcff69
master: Add nonce to shreds repairs, add shred data size to header (#10109) 
* Add nonce to shreds/repairs

* Add data shred size to header

Co-authored-by: Carl <carl@solana.com>
 2020-05-19 12:38:18 -07:00
sakridge f562ed4cc8 Distinguish between shred type in shred fetch stage duplicate filter (#10068) 
* Shred type check

* Test
 2020-05-15 13:23:56 -07:00
Kristofer Peterson 58ef02f02b
9951 clippy errors in the test suite (#10030) 
automerge
 2020-05-15 09:35:43 -07:00
sakridge a5419fe79e
Error for invalid shred. (#9588) 2020-04-19 21:15:09 -07:00
Stephen Akridge 30b3862770 Don't unwrap on session new 2020-04-16 08:25:45 -07:00
Michael Vines 73063544bd Move shred_version module to sdk/ 2020-02-24 14:46:12 -07:00
Tyera Eulberg ab361a8073
Rename KeypairUtil to Signer (#8360) 
automerge
 2020-02-20 13:28:55 -08:00
Pankaj Garg ed5101b031
Generate max coding shreds when necessary (#8099) 
* Generate max coding shreds when necessary

* test
 2020-02-04 15:45:01 -08:00
Michael Vines a2f2c46f87 Ensure shred version is never 0 2020-01-24 17:41:20 -07:00
Sagar Dhawan 37d1daf58e Revert "Generate MAX_DATA_SHREDS_PER_FEC_BLOCK coding shreds for each FEC block (#7474)" (#7898) 
automerge
 2020-01-21 11:48:09 -08:00
Sagar Dhawan 2dd8ab197d
Remove redundant threadpools in sigverify (#7888) 
* Limit the number of thread pools sigverify creates

* Name local threadpools
 2020-01-20 20:08:19 -08:00
Pankaj Garg 87b2525e03
Limit maximum number of shreds in a slot to 32K (#7584) 
* Limit maximum number of shreds in a slot to 32K

* mark dead slot replay as fatal error
 2019-12-30 07:42:09 -08:00
anatoly yakovenko 97589f77f8 Pipeline broadcast socket transmit and blocktree record (#7481) 
automerge
 2019-12-16 17:11:18 -08:00
Pankaj Garg 7adc721d96
Generate MAX_DATA_SHREDS_PER_FEC_BLOCK coding shreds for each FEC block (#7474) 
* Generate MAX_DATA_SHREDS_PER_FEC_BLOCK coding shreds for each FEC block

* fix tests, and detangle ErasureMeta::status
 2019-12-13 14:50:22 -08:00
Pankaj Garg d94041e98d
Allow coding shred index to be different than data shred index (#7438) 
* Allow coding shred index to be different than data shred index

* move fec_set_index to shred's common header

* fix bench
 2019-12-12 16:50:29 -08:00
anatoly yakovenko e082418e4a Cleanup the sign shreds interface (#7456) 
automerge
 2019-12-12 13:27:33 -08:00
sakridge f526c424c5
Move slow shred multi_fec test to integration tests folder (#7426) 2019-12-11 11:10:21 -08:00
Greg Fitzgerald 6796b08909
Migrate to thiserror (#7177) 
* Migrate to thiserror

* Discourage the use of other modules' Result alias

`io::Result` set a bad precedent. Don't import other `Result`
aliases.
 2019-12-02 15:42:05 -07:00
Sagar Dhawan 6bfe0fca1f
Add a version field to shreds (#7023) 
* Add a version field to shreds

* Clippy

* Fix Chacha Golden

* Fix shredder bench compile

* Fix blocktree bench compile
 2019-11-18 18:05:02 -08:00
carllin 7c32640a9b
Set index and set data should write into shred data (#6995) 2019-11-16 02:41:59 -08:00
sakridge 6009801c5f
More granular timings in shred generation (#6900) 2019-11-13 16:30:12 -08:00
Pankaj Garg ccac35fc01 Increase FEC ratio to 32:32 (#6800) 
automerge
 2019-11-07 16:38:06 -08:00
Sagar Dhawan 67d1e2903c
Upgrade Repair be more intelligent and agressive (#6789) 
* Upgrade Repair be more intelligent and agressive

* Fix u64 casts

* Fix missing bracket

* Add 1 second delay to test to allow repair to kick in
 2019-11-07 11:08:09 -08:00
Pankaj Garg 0ace79939b
Add reference tick to data shreds (#6772) 
* Add reference tick to data shreds

* fix tests
 2019-11-06 13:27:58 -08:00
carllin 7ff2a44a63 Make last shred for an interrupted slot signed + typed (#6760) 2019-11-06 08:25:17 -07:00
Jack May 9614d17024
Limit deserialization of data coming off the wire (#6751) 
* Limit deserialization of data coming off the wire

* Feedback and cleanup
 2019-11-06 00:07:57 -08:00
Michael Vines 50a17fc00b Use Slot and Epoch type aliases instead of raw u64 (#6693) 
automerge
 2019-11-02 00:38:30 -07:00
Justin Starry e8e5ddc55d
Verify number of hashes for each block of entries (#6262) 
* Verify number of hashes for each block of entries

* Fix blocktree processor tick check

* Rebase once more
 2019-10-31 16:38:50 -04:00
anatoly yakovenko 243fa6cf63
Shred gpu sigverify (#6520) 
Implement APIs for verifying shred signatures on the GPU.
 2019-10-28 10:29:38 -07:00
Pankaj Garg 84e911361a
Use constants instead of lazy_static for shred header sizes (#6472) 2019-10-21 12:46:16 -07:00
Pankaj Garg badeb4d31a
Rework shred headers to fix position of signature (#6451) 
* Rework shred headers to fix position of signature

* fix clippy
 2019-10-18 22:55:59 -07:00
Greg Fitzgerald 5468be2ef9 Add solana-ledger crate (#6415) 
automerge
 2019-10-18 09:28:51 -07:00