behzad nouri
4ceb2689f5
adds ShredId uniquely identifying each shred ( #21820 )
2021-12-14 17:34:02 +00:00
Justin Starry
254ef3e7b6
Rename Packets to PacketBatch ( #21794 )
2021-12-11 09:44:15 -05:00
behzad nouri
8063273d09
adds more sanity checks to shreds ( #21675 )
2021-12-09 16:43:57 +00:00
behzad nouri
cd17f63d81
adds back position field to coding-shred-header ( #21600 )
...
https://github.com/solana-labs/solana/pull/17004
removed position field from coding-shred-header because as it stands the
field is redundant and unused.
However, with the upcoming changes to erasure coding schema this field
will no longer be redundant and needs to be populated.
2021-12-05 14:42:09 +00:00
Michael Vines
b8837c04ec
Reformat imports to a consistent style for imports
...
rustfmt.toml configuration:
imports_granularity = "One"
group_imports = "One"
2021-12-03 09:19:13 -08:00
behzad nouri
dd338b6c9f
changes Shred::parent return type to Option<Slot> ( #21370 )
...
Shred::parent can return garbage if the struct fields are invalid:
https://github.com/solana-labs/solana/blob/8a50b6302/ledger/src/shred.rs#L446-L453
The commit adds more sanity checks and changes the return type to Option<Slot>.
2021-11-23 14:45:26 +00:00
behzad nouri
48dfdfb4d5
changes Blockstore::is_shred_duplicate arg type to ShredType
2021-11-19 14:16:39 +00:00
behzad nouri
57057f8d39
uses enum for shred type
...
Current code is using u8 which does not have any type-safety and can
contain invalid values:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L167
Checks for invalid shred-types are scattered through the code:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/blockstore.rs#L849-L851
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L346-L348
The commit uses enum for shred type with #[repr(u8)]. Backward
compatibility is maintained by implementing Serialize and Deserialize
compatible with u8, and adding a test to assert that.
2021-11-19 14:16:39 +00:00
steviez
e6280fc1fa
Add additional checks for should_retransmit_and_persist() ( #20672 )
...
Add additional checks to should_retransmit_and_persist()
- Check invalid shred index
- Update cases that check if node was leader
- Some comments and variable rename for clarity
2021-11-03 02:01:07 -05:00
behzad nouri
0f03971c3c
adds counters for errors in window-service run_insert ( #20670 )
2021-10-15 14:13:26 +00:00
sakridge
588168b99d
Add check for shred data header size ( #20668 )
2021-10-14 05:56:14 +02:00
behzad nouri
7a8807b8bb
retransmits shreds recovered from erasure codes
...
Shreds recovered from erasure codes have not been received from turbine
and have not been retransmitted to other nodes downstream. This results
in more repairs across the cluster which is slower.
This commit channels through recovered shreds to retransmit stage in
order to further broadcast the shreds to downstream nodes in the tree.
2021-08-17 13:44:10 +00:00
behzad nouri
3efccbffab
sends shreds (instead of packets) to retransmit stage
...
Working towards channelling through shreds recovered from erasure codes
to retransmit stage.
2021-08-17 13:44:10 +00:00
behzad nouri
8198a7eae1
adds packet/shred count stats to window-service
...
Adding back these metrics from the earlier commit which removed them
from retransmit stage.
2021-08-17 13:44:10 +00:00
behzad nouri
b64eeb7729
removes erroneous uses of &Arc<...> from window-service
2021-08-13 17:26:31 +00:00
behzad nouri
d57398a959
removes repeated bank-forks locking in window-service
...
Window service is repeatedly locking bank-forks to look-up working-bank
for every single shred:
https://github.com/solana-labs/solana/blob/5fde4ee3a/core/src/window_service.rs#L597-L606
This commit updates shred_filter signature in recv_window so that where
we already obtain the lock on bank-forks, we can also look-up
working-bank once for all packets:
https://github.com/solana-labs/solana/blob/5fde4ee3a/core/src/window_service.rs#L256-L277
2021-08-13 17:26:31 +00:00
Jeff Washington (jwash)
14361906ca
for all tests, bank::new -> bank::new_for_tests ( #19064 )
2021-08-05 08:42:38 -05:00
Jeff Biseda
9255ae334d
drop outstanding_requests lock before sending repair requests ( #18893 )
2021-07-28 19:30:43 -07:00
carllin
c0704d4ec9
Plumb signal from replay to ancestor hashes service ( #18880 )
2021-07-26 20:59:00 -07:00
carllin
1ee64afb12
Introduce AncestorHashesService ( #18812 )
2021-07-23 16:54:47 -07:00
behzad nouri
d2d5f36a3c
adds validator flag to allow private ip addresses ( #18850 )
2021-07-23 15:25:03 +00:00
behzad nouri
7d56fa8363
sends packets in batches from sigverify-stage ( #18446 )
...
sigverify-stage is breaking batches to single-item vectors before
sending them down the channel:
https://github.com/solana-labs/solana/blob/d451363dc/core/src/sigverify_stage.rs#L88-L92
Also simplifying window-service code, reducing number of nested branches.
2021-07-22 14:49:21 +00:00
carllin
588c0464b8
Add sampling logic and DuplicateSlotRepairStatus module ( #18721 )
2021-07-21 11:15:08 -07:00
carllin
8a846b048e
Add AncestorHashesRepair type ( #18681 )
2021-07-15 19:29:53 -07:00
sakridge
7f2254225e
Move entry/poh to own crate to speed up poh bench build ( #18225 )
2021-07-14 14:16:29 +02:00
jbiseda
a86ced0bac
generate deterministic seeds for shreds ( #17950 )
...
* generate shred seed from leader pubkey
* clippy
* clippy
* review
* review 2
* fmt
* review
* check
* review
* cleanup
* fmt
2021-07-07 08:21:12 -07:00
sakridge
8d9a6deda4
Add repair number per slot ( #18082 )
2021-06-30 18:20:07 +02:00
Michael Vines
84b9de8c18
Shredder no longer holds a keypair
2021-06-21 21:29:52 -07:00
Michael Vines
4a12c715a3
Drop Error suffix from enum values to avoid the enum_variant_names clippy lint
2021-06-18 23:02:13 +00:00
Alexander Meißner
6514096a67
chore: cargo +nightly clippy --fix -Z unstable-options
2021-06-18 10:42:46 -07:00
carllin
52dccc656a
Purge slots greater than new last index ( #16071 )
2021-05-26 16:12:57 -07:00
Tyera Eulberg
9a5330b7eb
Move gossip modules into solana-gossip crate ( #17352 )
...
* Move gossip modules to solana-gossip
* Update Protocol abi digest due to move
* Move gossip benches and hook up CI
* Remove unneeded Result entries
* Single use statements
2021-05-26 09:15:46 -06:00
Tao Zhu
0781fe1b4f
Upgrade Rust to 1.52.0 ( #17096 )
...
* Upgrade Rust to 1.52.0
update nightly_version to newly pushed docker image
fix clippy lint errors
1.52 comes with grcov 0.8.0, include this version to script
* upgrade to Rust 1.52.1
* disabling Serum from downstream projects until it is upgraded to Rust 1.52.1
2021-05-19 09:31:47 -05:00
behzad nouri
81ad795d46
removes position field in coding-shred-header
...
CodingShredHeader.position is equal to
ShredCommonHeader.index - ShredCommonHeader.fec_set_index
and is so redundant. The extra position field can add bugs if not
consistent with index and fec_set_index.
2021-05-10 13:20:56 +00:00
behzad nouri
37b8587d4e
expands number of erasure coding shreds in the last batch in slots ( #16484 )
...
Number of parity coding shreds is always less than the number of data
shreds in FEC blocks:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L719
Data shreds are batched in chunks of 32 shreds each:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L714
However the very last batch of data shreds in a slot can be small, in
which case the loss rate can be exacerbated.
This commit expands the number of coding shreds in the last FEC block in
slots to: 64 - number of data shreds; so that FEC blocks are always 64
data and parity coding shreds each.
As a consequence of this, the last FEC block has more parity coding
shreds than data shreds. So for some shred indices we will have a coding
shred but no data shreds. This should not cause any kind of overlapping
FEC blocks as in:
https://github.com/solana-labs/solana/pull/10095
since this is done only for the very last batch in a slot, and the next
slot will reset the shred index.
2021-04-21 12:47:50 +00:00
sakridge
8e69dd42c1
Add non-default repair nonce values ( #16512 )
...
* Track outstanding nonces in repair
* Rework outstanding requests to use lru cache and randomize nonces
Co-authored-by: Carl <carl@solana.com>
2021-04-20 09:37:33 -07:00
carllin
52703badfa
Setup ReplayStage confirmation scaffolding for duplicate slots ( #9698 )
2021-03-24 23:41:52 -07:00
behzad nouri
491b059755
broadcasts duplicate shreds through gossip ( #14699 )
2021-01-24 15:47:43 +00:00
sakridge
aa2751e614
Check shred type in is_duplicate ( #14050 )
2020-12-10 18:20:08 -08:00
Michael Vines
7bc073defe
Run `codemod --extensions rs Pubkey::new_rand solana_sdk::pubkey::new_rand`
2020-10-21 19:08:13 -07:00
sakridge
ce98088457
Track inserted repair shreds ( #12455 )
2020-09-29 14:13:21 -07:00
carllin
1c1a3f979d
Detect and notify when deserializable shreds are available ( #11816 )
...
* Add logic to check for complete data ranges
* Add RPC signature notification
Co-authored-by: Carl <carl@solana.com>
2020-09-01 22:06:06 -07:00
carllin
7a14e359d7
Switch to using weighted repair in RepairService ( #10735 )
...
* Plumb votes into repair service
* Remove refactoring
* Fix tests
* Switch to using RepairWeight for generating repairs
* Revert "Weight repair slots based on vote stake (#10741 )"
This reverts commit cabd0a09c3
.
* Update logging
Co-authored-by: Carl <carl@solana.com>
2020-07-10 05:52:54 +00:00
sakridge
cabd0a09c3
Weight repair slots based on vote stake ( #10741 )
...
* Weight repair slots based on vote stake
* Add test
2020-06-22 20:27:45 -07:00
Greg Fitzgerald
6ee222363e
Move BankForks to solana_runtime ( #10637 )
...
* Move BankForks to solana_runtime
* Update imports
2020-06-17 15:27:03 +00:00
Kristofer Peterson
e23340d89e
Clippy cleanup for all targets and nighly rust (also support 1.44.0) ( #10445 )
...
* address warnings from 'rustup run beta cargo clippy --workspace'
minor refactoring in:
- cli/src/cli.rs
- cli/src/offline/blockhash_query.rs
- logger/src/lib.rs
- runtime/src/accounts_db.rs
expect some performance improvement AccountsDB::clean_accounts()
* address warnings from 'rustup run beta cargo clippy --workspace --tests'
* address warnings from 'rustup run nightly cargo clippy --workspace --all-targets'
* rustfmt
* fix warning stragglers
* properly fix clippy warnings test_vote_subscribe()
replace ref-to-arc with ref parameters where arc not cloned
* Remove lock around JsonRpcRequestProcessor (#10417 )
automerge
* make ancestors parameter optional to avoid forcing construction of empty hash maps
Co-authored-by: Greg Fitzgerald <greg@solana.com>
2020-06-09 09:38:14 +09:00
carllin
97f2bcff69
master: Add nonce to shreds repairs, add shred data size to header ( #10109 )
...
* Add nonce to shreds/repairs
* Add data shred size to header
Co-authored-by: Carl <carl@solana.com>
2020-05-19 12:38:18 -07:00
Kristofer Peterson
58ef02f02b
9951 clippy errors in the test suite ( #10030 )
...
automerge
2020-05-15 09:35:43 -07:00
Jack May
eb1acaf927
Remove archiver and storage program ( #9992 )
...
automerge
2020-05-14 18:22:47 -07:00
carllin
bab3502260
Push down cluster_info lock ( #9594 )
...
* Push down cluster_info lock
* Rework budget decrement
Co-authored-by: Carl <carl@solana.com>
2020-04-21 12:54:45 -07:00