crds_gossip tests start large networks, which with large thread-pools
will exhaust system resources, causing failures in ci tests:
https://buildkite.com/solana-labs/solana/builds/31953
The commit limits size of thread-pools in the test.
Based on run-time profiles, the majority time of new_pull_requests is
spent building bloom filters, in hashing and bit-vec ops.
This commit builds crds filters in parallel using rayon constructs. The
added benchmark shows ~5x speedup (4-core machine, 8 threads).
* Batch process pull responses
* Generate pull requests at 1/2 rate
* Do filtering work of process_pull_response in read lock
Only take write lock to insert if needed.
* filter messages that are likely to be pushed from the response
* tests
* tests
* wait to start filtering responses, and push stats to influx
* wait to start filtering responses, and push stats to influx
* reduce the timers to match the publish self timeout
* fmt
* fmt
* Add CrdsValue timeout checks on Pull Responses
* Allow older values to enter Crds as long as a ContactInfo exists
* Allow staked contact infos to be inserted into crds if they haven't expired
* Try and handle oveflows
* Fix test
* Some comments
* Fix compile
* fix test deadlock
* Add a test for processing timed out values received via pull response
* Remove Blobs and switch to Packets
* Fix some gossip messages not respecting MTU size
* Failure to serialize is not fatal
* Add log macros
* Remove unused extern
* Apparently macro use is required
* Explicitly scope macro
* Fix test compile
* Coalesce gossip pull requests and serve them in batches
* batch all filters and immediately respond to messages in gossip
* Fix tests
* make download_from_replicator perform a greedy recv
* fixed bloom filter math
* Add split each pull request into multiple pulls with different filters
* Rework CrdsFilter to generate all possible masks to cover the keyspace
* Limit the bloom sizes such that each pull request is no larger than mtu
* Be able to create bank snapshots
* fix clippy
* load snapshot on start
* regenerate account index from the storage
* Remove rc feature dependency
* cleanup
* save snapshot for slot 0
* Add Epoch Slots to gossip
* Add new gossip structure to support Repair
* remove unnecessary clones
* Setup dummy fast repair in repair_service
* PR comments
* move core tests to core
* remove window
* fix up flaky tests
* test_entryfication needs a singly-threaded banking_stage
* move core benches to core
* remove unnecessary dependencies
* remove core as a member for now, test it like runtime
* stop running tests twice
* remove duplicate runs of tests in perf
behzad nouri
Michael Vines
anatoly yakovenko
sakridge
Kristofer Peterson
Sagar Dhawan
Pankaj Garg
Justin Starry
Sathish
Rob Walker
carllin
Greg Fitzgerald
