blockworks-foundation
/
solana
mirror of https://github.com/blockworks-foundation/solana.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
22,253 Commits 168 Branches 497 Tags 539 MiB
Commit Graph

126 Commits

Author SHA1 Message Date
Trent Nelson d4292774c5 checks 2022-02-25 08:05:28 +00:00
buffalu 70ebab2c82
Add rustfmt.toml and `cargo fmt` (#23238) 
* fmt

* formatted

Co-authored-by: Lucas B <buffalu@jito.network>
 2022-02-19 13:32:29 +08:00
anatoly yakovenko 83d31c9e65
shrink batches when over 80% of the space is wasted (#23066) 
* shrink batches when over 80% of the space is wasted
 2022-02-16 08:18:17 -08:00
sakridge 2e56c59bcb
Handle already discarded packets in gpu sigverify path (#22680) 2022-01-24 14:35:47 +01:00
sakridge 7569f282c6
Move discard check before generate offsets (#22684) 2022-01-24 12:59:47 +00:00
anatoly yakovenko d6011ba14d
Dedup bloom filter is too slow (#22607) 
* Faster dedup

* use ahash

* fixup

* single threaded

* use duration type

* remove the count

* fixup
 2022-01-21 20:23:48 -07:00
Justin Starry a2d251ce1e
Speed up packet dedup and fix benches (#22592) 
* Speed up packet dedup and fix benches

* fix tests

* allow int arithmetic in bench
 2022-01-20 13:59:16 -07:00
anatoly yakovenko d343713f61
Optimize packet dedup (#22571) 
* Use bloom filter to dedup packets

* dedup first

* Update bloom/src/bloom.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* fixup

* fixup

* fixup

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
 2022-01-19 13:58:20 -08:00
behzad nouri 01a096adc8 adds bitflags to Packet.Meta 
Instead of a separate bool type for each flag, all the flags can be
encoded in a type-safe bitflags encoded in a single u8:
https://github.com/solana-labs/solana/blob/d6ec103be/sdk/src/packet.rs#L19-L31
 2022-01-04 13:53:40 +00:00
behzad nouri c9c78622a8
discards serialized gossip crds votes if cannot parse tx (#22129) 2021-12-29 19:31:26 +00:00
Justin Starry b1d9a2e60e
Don't forward packets received from TPU forwards port (#22078) 
* Don't forward packets received from TPU forwards port

* Add banking stage test
 2021-12-29 19:34:31 +01:00
Justin Starry 254ef3e7b6
Rename Packets to PacketBatch (#21794) 2021-12-11 09:44:15 -05:00
Michael Vines b8837c04ec Reformat imports to a consistent style for imports 
rustfmt.toml configuration:
  imports_granularity = "One"
  group_imports = "One"
 2021-12-03 09:19:13 -08:00
Anton Lazarev 3c8b33eaef don't check x86 features on non-x86 architectures 2021-12-02 08:02:48 -08:00
Michael Vines dd12d90eac Upgrade to Rust 2021 2021-11-30 20:43:46 -08:00
ryleung-solana 8cf36e5cb0
Add GPU sigverify for verify path (#20851) 
Allows the use of GPU acceleration in verifying the signatures in Entry's after deserialization in the replay stage

Co-authored-by: Stephen Akridge <sakridge@gmail.com>
Co-authored-by: Ryan Leung <ryan.leung@solana.com>
 2021-11-30 21:16:13 -05:00
Ivan Mironov 18b1baa3c1 Add function for changing thread's nice value 
Linux only.
 2021-11-04 17:16:46 -06:00
Michael Vines 2b76ea51b4 Reduce visibility of `Hash` struct contents 2021-10-19 09:30:47 -07:00
Michael Vines c16510152e Rework AVX/AVX2 detection again 2021-10-10 12:22:10 -07:00
Tao Zhu 177a375479
Tpu vote 1.7 (#20187) (#20494) 
* Add separate vote processing tpu port

* Add feature to send to tpu vote port

* Add vote rejecting sigverify mode

* use packet.meta.is_simple_vote_tx in place of deserialization

* consolidate code that identifies vote tx atcommon path for cpu and gpu

* new key for feature set

* banking forward tpu vote

* add tpu vote port to dockerfile and other review changes

* Simplify thread id compare

* fix a test; updated cluster_info ABI change

Co-authored-by: Tao Zhu <tao@solana.com>

Co-authored-by: sakridge <sakridge@gmail.com>
 2021-10-07 09:38:23 +00:00
Michael Vines 7027d56064 Resolve nightly-2021-10-05 clippy complaints 2021-10-06 10:37:58 -07:00
Justin Starry 6f08f9decd fix clippy 2021-09-26 08:57:29 -04:00
Tao Zhu 46757238de allow vote to have 1 or 2 sigs (#20082) 2021-09-26 08:57:29 -04:00
Tao Zhu 198929fe8b sigverify to identify and mark simple vote transaction 2021-09-26 08:57:29 -04:00
sakridge 013e1d9d49
Limit transaction forwarding from banking_stage (#19940) 2021-09-21 08:49:41 -07:00
Justin Starry fd33f523ab
Generate packet offsets for versioned messages (#19138) 
* Add support for generating packet offsets for new versioned message

* Remove outdated test

* match over version
 2021-08-16 17:55:15 -07:00
sakridge f4aa5c5d8d
Sigverify refactor (#18871) 2021-07-23 21:44:21 +02:00
Alexander Meißner 6514096a67 chore: cargo +nightly clippy --fix -Z unstable-options 2021-06-18 10:42:46 -07:00
Jon Cinque 1b1d34da59
Refactor stake program into solana_program (#17906) 
* Move stake state / instructions into solana_program

* Update account-decoder

* Update cli and runtime

* Update all other parts

* Commit Cargo.lock changes in programs/bpf

* Update cli stake instruction import

* Allow integer arithmetic

* Update ABI digest

* Bump rust mem instruction count

* Remove useless structs

* Move stake::id() -> stake::program::id()

* Re-export from solana_sdk and mark deprecated

* Address feedback

* Run cargo fmt
 2021-06-15 18:04:00 +02:00
sakridge eeee75c5be
Don't use pinned memory when unnecessary (#17832) 
Reports of excessive GPU memory usage and errors
from cudaHostRegister. There are some cases where pinning is
not required.
 2021-06-14 16:10:04 +02:00
Tyera Eulberg a66566e75b
Remove budget program (#17816) 2021-06-08 01:20:17 +00:00
Tao Zhu 0781fe1b4f
Upgrade Rust to 1.52.0 (#17096) 
* Upgrade Rust to 1.52.0
update nightly_version to newly pushed docker image
fix clippy lint errors
1.52 comes with grcov 0.8.0, include this version to script

* upgrade to Rust 1.52.1

* disabling Serum from downstream projects until it is upgraded to Rust 1.52.1
 2021-05-19 09:31:47 -05:00
Michael Vines a911ae00ba clippy 2021-04-18 20:55:02 -07:00
behzad nouri cfe7a4340b adds a shrink policy to the recycler without an allocation limit 
https://github.com/solana-labs/solana/pull/15320
added an allocation limit to the recycler, which has been the source of a
number of bugs. For example the code bellow panics by simply cloning packets:

    const RECYCLER_LIMIT: usize = 8;
    let recycler = PacketsRecycler::new_with_limit("", RECYCLER_LIMIT as u32);
    let packets = Packets::new_with_recycler(recycler.clone(), 1).unwrap();
    for _ in 0..RECYCLER_LIMIT {
        let _ = packets.clone();
    }
    Packets::new_with_recycler(recycler.clone(), 1);

The implementation also fails to account for instances where objects are
consumed. Having the allocation limit in the recycler also seems out of place,
as higher level code has better context to impose allocation limits (e.g. by
using bounded channels to rate-limit), whereas the recycler would be simpler
and more efficient if it just do the recycling.

This commit:
* Reverts https://github.com/solana-labs/solana/pull/15320
* Adds a shrink policy to the recycler without an allocation limit.
 2021-04-18 19:29:24 +00:00
behzad nouri e405747409 Revert "Add limit and shrink policy for recycler (#15320)" 
This reverts commit c2e8814dce.
 2021-04-18 19:29:24 +00:00
behzad nouri 22a18a68e3
stops consuming pinned vectors with a recycler (#16441) 
If the vector is pinned and has a recycler, From<PinnedVec>
implementation of Vec should clone (instead of consuming) the underlying
vector so that the next allocation of a PinnedVec will recycle an
already pinned one.
 2021-04-09 16:55:24 +00:00
behzad nouri 0e262aab3d
simplifies PinnedVec implementation (#16382) 2021-04-08 10:40:30 +00:00
Kristofer Peterson 6bc858a888 Refactored ShortU16Visitor::visit_seq() to reject overflows, extra leading zeros and ensure one-to-one encoding. 2021-03-24 21:53:35 -06:00
Trent Nelson 482c027d3b sq: optimize 2021-03-22 12:01:57 -06:00
Trent Nelson 834fae684b perf: use saturating/checked integer arithmetic 2021-03-22 12:01:57 -06:00
carllin 2bee9435f3
Add tracer key for tracing transaction path through the network (#15732) 2021-03-08 19:31:00 -08:00
sakridge 830be855dc
Forward and hold packets (#15634) 2021-03-03 10:23:05 -08:00
carllin c2e8814dce
Add limit and shrink policy for recycler (#15320) 2021-02-24 00:15:58 -08:00
Trent Nelson 7f7370c306 Re-allow clippy::integer_arithmetic at crate-level 2021-02-17 13:55:08 -07:00
Ryo Onodera d6873b82ab
Remove potentially too costly Packets::default() (#14821) 
* Remove potentially too costly Packets::default()

* Fix test...

* Restore Packets::default()

* Restore Packets::default() more
 2021-01-29 09:32:38 +09:00
Ryo Onodera 015058e0b7
Reduce ~2 GBs mem by avoiding another overalloc. (#14806) 
* Reduce few GBs mem by avoiding another overalloc.

* Use x.len() for the last item from chunks()
 2021-01-25 04:27:47 +00:00
Michael Vines cbffab7850 Upgrade to Rust v1.49.0 2021-01-23 19:16:36 -08:00
Michael Vines 7143aaa89b Clippy 2020-12-14 08:03:29 -08:00
behzad nouri 1ffab5de77
breaks prunes data into chunks to fit into packets (#13613) 
Validator logs show that prune messages are dropped because they exceed
packet data size:
https://github.com/solana-labs/solana/blob/f25c969ad/perf/src/packet.rs#L90-L92
This can exacerbate gossip traffic by redundantly increasing push
messages across network. The workaround is to break prunes into smaller
chunks and send over in multiple messages.
 2020-11-19 16:38:01 +00:00
behzad nouri 7f4debdad5
drops older gossip packets when load shedding (#13364) 
Gossip drops incoming packets when overloaded:
https://github.com/solana-labs/solana/blob/f6a73098a/core/src/cluster_info.rs#L2462-L2475
However newer packets are dropped in favor of the older ones.
This is probably not ideal as newer packets are more likely to contain
more recent data, so dropping them will keep the validator state
lagging.
 2020-11-05 17:14:28 +00:00
behzad nouri 75d62ca095
improves threads' utilization in processing gossip packets (#12962) 
ClusterInfo::process_packets handles incoming packets in a thread_pool:
https://github.com/solana-labs/solana/blob/87311cce7/core/src/cluster_info.rs#L2118-L2134

However, profiling runtime shows that threads are not well utilized and
a lot of the processing is done sequentially.

This commit redistributes the work done in parallel. Testing on a gce
cluster shows 20%+ improvement in processing gossip packets with much
smaller variations.
 2020-10-19 19:03:38 +00:00
Justin Starry f6cda2579f Fix off-by-one max payload checks 2020-09-16 12:46:06 -07:00
Trent Nelson 30dbe257cf Reimplement decode_len() with ShortU16 vistor helper 2020-08-14 08:17:52 -06:00
dependabot-preview[bot] 841ecfd927
chore(deps): bump bincode from 1.2.1 to 1.3.1 (#10867) 
* chore(deps): bump bincode from 1.2.1 to 1.3.1

Bumps [bincode](https://github.com/servo/bincode) from 1.2.1 to 1.3.1.
- [Release notes](https://github.com/servo/bincode/releases)
- [Commits](https://github.com/servo/bincode/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

* Switch from deprecated method

* Add options to maintain behavior with bincode::options()

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2020-07-09 00:08:05 +00:00
Tyera Eulberg 1a6bbd2867
Limited Deserialize isn't limiting anything (#10952) 
* Add failing test

* Use deserialize_from to enable limit
 2020-07-08 15:54:42 -06:00
sakridge 0de6c444d6
Simd poh (#10604) 
* Simd poh

* Fix poh verify bench
 2020-06-16 23:03:26 -07:00
Kristofer Peterson 58ef02f02b
9951 clippy errors in the test suite (#10030) 
automerge
 2020-05-15 09:35:43 -07:00
Jack May eb1acaf927
Remove archiver and storage program (#9992) 
automerge
 2020-05-14 18:22:47 -07:00
sakridge d5c889d6b0
Re-enable gpu sigverify (#9870) 
Add sigverify fuzz and scalar test
 2020-05-06 15:44:55 -07:00
Stephen Akridge eb24f3df84 Update dalek version 2020-04-28 21:02:47 -06:00
Michael Vines 7d27be2a73 Upgrade to Rust 1.41.1 2020-02-28 10:10:42 -07:00
Tyera Eulberg ab361a8073
Rename KeypairUtil to Signer (#8360) 
automerge
 2020-02-20 13:28:55 -08:00
Pankaj Garg ac1d075d73
Drop packet if destination is unspecified (0.0.0.0/0) (#8321) 2020-02-18 16:14:20 -08:00
carllin 0b263f8714
Fix larger than necessary allocations in streamer (#8187) 2020-02-10 11:49:07 -08:00
Sagar Dhawan 2dd8ab197d
Remove redundant threadpools in sigverify (#7888) 
* Limit the number of thread pools sigverify creates

* Name local threadpools
 2020-01-20 20:08:19 -08:00
Michael Vines 1d01777a13 Prefer CUDA_HOME environment variable 2020-01-15 09:03:52 -07:00
sakridge dd54fff978
Use pinned memory for entry verify (#7440) 2019-12-12 10:36:27 -08:00
anatoly yakovenko bec5835289 Automatically call `.recycle` in Drop (#7429) 
automerge
 2019-12-11 11:58:40 -08:00
Jack May 3415db9739
Merge api/program into single units (#7061) 2019-11-20 16:32:19 -08:00
anatoly yakovenko b150da837a Use epoch as the gossip purge timeout for staked nodes. (#7005) 
automerge
 2019-11-20 11:25:18 -08:00
sakridge d4f336db40 Fix unpin argument (#7057) 
automerge
 2019-11-20 10:22:26 -08:00
anatoly yakovenko 0ca943f49b RecyclerCache for shred_sigverify (#6986) 
automerge
 2019-11-15 12:16:56 -08:00
Sagar Dhawan f108f483b7
Remove Blobs and switch to Packets (#6937) 
* Remove Blobs and switch to Packets

* Fix some gossip messages not respecting MTU size

* Failure to serialize is not fatal

* Add log macros

* Remove unused extern

* Apparently macro use is required

* Explicitly scope macro

* Fix test compile
 2019-11-14 10:24:53 -08:00
sakridge 8e81bc1b49
Fix pinning (#6604) 
Remove Deref implementations and add more pass-throughs to the PinnedVec
wrapper.
Warm recyclers
set_pinnable
 2019-11-07 19:48:33 -08:00
anatoly yakovenko 67f636545a Refactor sigverify to stage for signing shreds on the GPU (#6635) 
automerge
 2019-11-06 10:52:30 -08:00
anatoly yakovenko b825d04597 Pull perf into a separate module. (#6718) 
automerge
 2019-11-04 20:13:43 -08:00