6 Commits

Author	SHA1	Message	Date
behzad nouri	604ca9316c	includes zero weighted entries in WeightedShuffle (#22829) ... Current WeightedShuffle implementation excludes zero weighted entries from the shuffle: https://github.com/solana-labs/solana/blob/13e631dcf/gossip/src/weighted_shuffle.rs#L29-L30 Though mathematically this might make more sense, for our use-cases (turbine specifically), this results in less efficient code: https://github.com/solana-labs/solana/blob/13e631dcf/core/src/cluster_nodes.rs#L409-L430 This commit changes the implementation so that zero weighted indices are also included in the shuffle but appear only at the end after non-zero weighted indices.	2022-01-31 16:23:50 +00:00
Michael Vines	b8837c04ec	Reformat imports to a consistent style for imports ... rustfmt.toml configuration: imports_granularity = "One" group_imports = "One"	2021-12-03 09:19:13 -08:00
behzad nouri	0c0384ec32	revises turbine peers shuffling order (#20480) ... Turbine randomly shuffles cluster nodes on a broadcast tree for each shred. This requires knowing the stakes and nodes' contact-infos (from gossip). However gossip is subject to partitioning and propogation delays. Additionally unstaked nodes may join and leave the cluster at any moment, changing the cluster view from one node to another. This commit: * Always arranges the unstaked nodes at the bottom of turbine broadcast tree. * Staked nodes are always included regardless of if their contact-info is available in gossip or not. * Uses the unbiased WeightedShuffle construct for shuffling nodes.	2021-10-14 15:09:36 +00:00
carllin	588c0464b8	Add sampling logic and DuplicateSlotRepairStatus module (#18721)	2021-07-21 11:15:08 -07:00
behzad nouri	dba42c57b4	implements an unbiased weighted shuffle using binary indexed tree (#18343) ... Current implementation of weighted_shuffle: https://github.com/solana-labs/solana/blob/b08f8bd1b/gossip/src/weighted_shuffle.rs#L11-L37 uses a heuristic which results in biased samples. For example, if the weights are [1, 10, 100], then the 3rd index should come first 100 times more often than the 1st index. However, weighted_shuffle is picking the 3rd index 200+ times more often than the 1st index, showing a disproportional bias in favor of higher weights. This commit implements weighted shuffle using binary indexed tree to maintain cumulative sum of weights while sampling. The resulting samples are demonstrably unbiased and precisely proportional to the weights. Additionally the iterator interface allows to skip computations when not all indices are processed. Of the use cases of weighted_shuffle, changing turbine code requires feature-gating to keep the cluster in sync. That is not updated in this commit, but can be done together with future updates to turbine.	2021-07-07 14:14:43 +00:00
Tyera Eulberg	9a5330b7eb	Move gossip modules into solana-gossip crate (#17352) ... * Move gossip modules to solana-gossip * Update Protocol abi digest due to move * Move gossip benches and hook up CI * Remove unneeded Result entries * Single use statements	2021-05-26 09:15:46 -06:00