Commit Graph

57 Commits

Author SHA1 Message Date
Lijun Wang 344e466e12
Async connection creation in connection cache (#33302)
If there is a connection in the cache available, use it and create the additional connection asynchronously.
2023-09-25 18:17:47 -07:00
Alexander Meißner 9e703f85de
Upgrades Rust to 1.72.0 & nightly-2023-08-25 (#32961)
* allow pedantic invalid cast lint

* allow lint with false-positive triggered by `test-case` crate

* nightly `fmt` correction

* adapt to rust layout changes

* remove dubious test

* Use transmute instead of pointer cast and de/ref when check_aligned is false.

* Renames clippy::integer_arithmetic to clippy::arithmetic_side_effects.

* bump rust nightly to 2023-08-25

* Upgrades Rust to 1.72.0

---------

Co-authored-by: Trent Nelson <trent@solana.com>
2023-09-01 07:26:13 +00:00
Jon Cinque 2124f0a13c
client: Speed up send-and-confirm-parallel (#33032)
* Rename blockheight -> block height

* Clean up comments

* Simplify tpu-client implementation

* client: Destructure more in match arm

* client: Avoid passing Arc where it isn't needed

* Refactor sending into a new function

* Chain work after send to make last part cleaner

* Do all sending at once in the same future

* Sleep before sending to avoid overwhelming
2023-08-29 18:29:10 +02:00
Jon Cinque b42249fffb
tpu-client: Speed up performance by awaiting all futures at once (#32945)
* tpu-client: Await all futures at once

* Add timeout when sending to not waste time on down nodes

* Update comment to make it clearer that we're not spiking
2023-08-24 13:04:00 +02:00
Jon Cinque e1972f07fa
rpc-client: Encapsulate `set_message_for_confirmed_transactions` (#32941) 2023-08-23 11:46:17 +02:00
Jon Cinque 0fe902ced7
Bump rand to 0.8, rand_chacha to 0.3, getrandom to 0.2 (#32871)
* sdk: Add concurrent support for rand 0.7 and 0.8

* Update rand, rand_chacha, and getrandom versions

* Run command to replace `gen_range`

Run `git grep -l gen_range | xargs sed -i'' -e 's/gen_range(\(\S*\), /gen_range(\1../'

* sdk: Fix users of older `gen_range`

* Replace `hash::new_rand` with `hash::new_with_thread_rng`

Run:
```
git grep -l hash::new_rand | xargs sed -i'' -e 's/hash::new_rand([^)]*/hash::new_with_thread_rng(/'
```

* perf: Use `Keypair::new()` instead of `generate`

* Use older rand version in zk-token-sdk

* program-runtime: Inline random key generation

* bloom: Fix clippy warnings in tests

* streamer: Scope rng usage correctly

* perf: Fix clippy warning

* accounts-db: Map to char to generate a random string

* Remove `from_secret_key_bytes`, it's just `keypair_from_seed`

* ledger: Generate keypairs by hand

* ed25519-tests: Use new rand

* runtime: Use new rand in all tests

* gossip: Clean up clippy and inline keypair generators

* core: Inline keypair generation for tests

* Push sbf lockfile change

* sdk: Sort dependencies correctly

* Remove `hash::new_with_thread_rng`, use `Hash::new_unique()`

* Use Keypair::new where chacha isn't used

* sdk: Fix build by marking rand 0.7 optional

* Hardcode secret key length, add static assertion

* Unify `getrandom` crate usage to fix linking errors

* bloom: Fix tests that require a random hash

* Remove some dependencies, try to unify others

* Remove unnecessary uses of rand and rand_core

* Update lockfiles

* Add back some dependencies to reduce rebuilds

* Increase max rebuilds from 14 to 15

* frozen-abi: Remove `getrandom`

* Bump rebuilds to 17

* Remove getrandom from zk-token-proof
2023-08-21 19:11:21 +02:00
galactus dbac50dcc8
New method to send large number of transactions to the cluster (#32388)
* Adding a method which sends and confirm a large number of transactions in parallel and using it to deploy programs

* Using the new method to deploy programs

* Minor changes, break when transaction map is empty

* Updating cargo for all lock files

* Sorting dependencies

* Changes after tnelson's comment

* confirm recently expired transactions and improve tracking of transaction sending

* Minor changes

* more changes after tnelson's comments

* Adding serialized transaction in TransactionData

* Update client/src/send_and_confirm_transactions_in_parallel.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update client/src/send_and_confirm_transactions_in_parallel.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* nit changes from trent

* Fixing kirills comments, adding some unit tests

* fixing ci, and minor change

* addressing Kirills nit comments

---------

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
2023-07-25 12:08:01 -06:00
behzad nouri 5760390d3b
removes redundant NewConnectionConfig trait (#31979)
Working towards removing hard-coded TPU specific configurations from
QUIC code; NewConnectionConfig is redundant and gets in the way.
2023-06-06 21:28:29 +00:00
behzad nouri 9281ab7d97
separates out connection-cache metrics for different protocols (#31803) 2023-05-25 14:48:22 +00:00
behzad nouri 6a4a0418a6
removes hard-coded QUIC_PORT_OFFSET from connection-cache (#31541)
New ContactInfo has api identifying QUIC vs UDP ports; no need to hard-code
port-offset deep in connection-cache.
2023-05-09 13:46:17 +00:00
Brooks 6e342ded42
clippy: Removes redundant async blocks (#31526) 2023-05-09 09:35:38 -04:00
blackghost1987 9f9d6f8a89
Relax `Sized` bound for `Signers` in `Transaction` and `Client` (#30568)
* relax Sized bound for Signers in Transaction and Client

* also relax Sized bounds in client, runtime, thin-client and tpu-client

* add tests for using non-sized transaction signers

* fix macro for thin_client vs ?Sized

* move tests to transactions, add Sized relaxation to mut macro

* fix clippy warning

* get rid of unnecessary imports
2023-03-30 13:12:48 +00:00
Kevin Ji 3713bb32b5
tpu-client: Fix counter for expired blockhash retries (#30360)
tpu-client: Fix counter for expired blockhash retries

The original counter would expire after saying "1 retry left", which was
misleading. We can safely change the comparison to >= 0 as our counter
is an i32.
2023-03-28 22:48:52 -05:00
Yihau Chen 2c842e9932
chore: workspace inheritance (#30939)
* client/Cargo.toml

* udp-client/Cargo.toml

* tpu-client/Cargo.toml

* thin-client/Cargo.toml

* storage-proto/Cargo.toml

* quic-client/Cargo.toml

* dos/Cargo.toml

* entry/Cargo.toml

* perf/Cargo.toml

* program-runtime/Cargo.toml

* program-test/Cargo.toml

* programs/address-lookup-table/Cargo.toml

* programs/loader-v3/Cargo.toml

* connection-cache/Cargo.toml

* programs/sbf/rust/big_mod_exp/Cargo.toml

* programs/zk-token-proof-tests/Cargo.toml
2023-03-29 03:28:56 +00:00
behzad nouri 75abfc79a6
removes unused dependencies (#30917) 2023-03-28 17:25:44 +00:00
Yihau Chen df3ef111f7
chore: workspace inheritance (#29893)
* introduce workspace.package

* introduce workspace.dependencies

* read version from root cargo.toml

* pass check when version = { workspace = true }

* don't bump version when version = { workspace = true }

* including workspace Cargo.toml when bump version

* programs/sbf use workspace inheritance

* fix increasing cargo version ignore program/sbf/Cargo.toml
2023-02-23 22:01:54 +08:00
Michael Vines 5136ed3448
Update homepage value for all crates (#30444) 2023-02-23 02:20:18 +00:00
Lijun Wang d49b48100d
Cleanup tempoary pub modules (#30268)
Clean up temporary_pub_modules in tpu_client and thin_client
2023-02-10 20:35:09 -08:00
dependabot[bot] 66dd9432f5
Bump futures-util from 0.3.25 to 0.3.26 (#30240)
* Bump futures-util from 0.3.25 to 0.3.26

Bumps [futures-util](https://github.com/rust-lang/futures-rs) from 0.3.25 to 0.3.26.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.25...0.3.26)

---
updated-dependencies:
- dependency-name: futures-util
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2023-02-10 13:46:16 -07:00
behzad nouri 1ad69cfc38
removes dynamic cast and dynamic dispatch from connection-cache (#30128)
Dynamic dispatch forces heap allocation and adds extra overhead.
Dynamic casting as in the ones below, lacks compile-time type safety:
https://github.com/solana-labs/solana/blob/eeb622c4e/quic-client/src/lib.rs#L172-L175
https://github.com/solana-labs/solana/blob/eeb622c4e/udp-client/src/lib.rs#L52-L55

The commit removes all instances of Any, Box<dyn ...>, and Arc<dyn ...>,
and instead uses generic and associated types.

There are only two protocols QUIC and UDP; and the code which has to
work with both protocols can use a trivial thin enum wrapper.

With respect to connection-cache specifically:
* connection-cache/ConnectionCache is a single protocol cache which
  allows to use either QUIC or UDP without any build dependency on the
  other protocol.
* client/ConnectionCache is an enum wrapper around both protocols and
  can be used in the code which has to work with both QUIC and UDP.

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2023-02-09 00:50:44 +00:00
Lijun Wang ada6136a6c
Refactor connection cache to support generic msgs (#29774)
tpu-client/tpu_connection_cache is refactored out the module and moved to connection-cache/connection_cache and the logic in client/connection_cache is consolidated to connection-cache/connection_cache as well. client/connection_cache only has a thin wrapper which forward calls to connection-cache/connection_cache and deal with constructions of quic/udp connection cache for clients using them both.2.

The TpuConnection is refactored to ClientConnection to make it generic and functions renamed to be proper for other workflows. eg. tpu_addr -> server_addr, send_transaction --> send_data and etc...

The enum dispatch is removed so that we can make the bulk of code of quic and udp agnostic of each other. The client is possible to load quic or udp only into its runtime.

The generic type parameter in the tpu-client/tpu_connection_cache is removed in order to create both quic and udp connection cache and use the object to send transactions with multiple branching when sending data. The generic type parameters and associated types are dropped in other types in order to make the trait "object safe" for this purpose.

I have annotated the code explaining the reasoning and the refactoring source -> destination.

There is no functional changes

bench-tps has been performed for rpc-client, thin-client and tpu-client. And it is found the performance number largely match the ones before the refactoring.
2023-02-01 18:10:06 -08:00
Will Hickey 04a6a631bc
Bump version to v1.16 (#30028) 2023-01-31 17:48:33 -06:00
joeaba a12bf8c003
Update maintainers references (#29997)
* update maintainers references

* chore: update maintainers reference
2023-01-31 08:07:13 -05:00
Yihau Chen 9193b4221d
Revert "chore: workspace inheritance (#29509)" (#29892)
This reverts commit a67d239dde.
2023-01-25 15:50:41 +08:00
Yihau Chen a67d239dde
chore: workspace inheritance (#29509)
* introduce workspace.package

* introduce workspace.dependencies

* read version from root cargo.toml

* pass check when version = { workspace = true }

* don't bump version when version = { workspace = true }

* including workspace Cargo.toml when bump version

* programs/sbf use workspace inheritance

* fix increasing cargo version ignore program/sbf/Cargo.toml
2023-01-25 13:59:59 +08:00
Kevin Ji dd92f225bb
Use Ipv4Addr::{LOCALHOST, UNSPECIFIED} constants (#29813) 2023-01-23 16:49:51 -06:00
behzad nouri 5c9beef498
fixes errors from clippy::useless_conversion (#29534)
https://rust-lang.github.io/rust-clippy/master/index.html#useless_conversion
2023-01-05 18:05:32 +00:00
dependabot[bot] 65391a70cb
chore: bump futures-util from 0.3.24 to 0.3.25 (#29337)
* chore: bump futures-util from 0.3.24 to 0.3.25

Bumps [futures-util](https://github.com/rust-lang/futures-rs) from 0.3.24 to 0.3.25.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.24...0.3.25)

---
updated-dependencies:
- dependency-name: futures-util
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-12-22 22:42:18 -07:00
behzad nouri 9524c9dbff patches errors from clippy::uninlined_format_args
https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
2022-12-06 19:32:15 +00:00
Pankaj Garg 07d21d4114
Use estimated current slot in tpu client (#28997) 2022-11-30 11:36:26 -08:00
Pankaj Garg a369b4a070
TPU client fix for leader schedule cache lookup (#28954)
TPU client fix for leader schedule cache
2022-11-28 13:44:08 -08:00
Tyera c32377b5af
Split out quic- and udp-client definitions (#28762)
* Move ConnectionCache back to solana-client, and duplicate ThinClient, TpuClient there

* Dedupe thin_client modules

* Dedupe tpu_client modules

* Move TpuClient to TpuConnectionCache

* Move ThinClient to TpuConnectionCache

* Move TpuConnection and quic/udp trait implementations back to solana-client

* Remove enum_dispatch from solana-tpu-client

* Move udp-client to its own crate

* Move quic-client to its own crate
2022-11-18 12:21:45 -07:00
Tyera Eulberg 8b63f73100
Remove Default impls for TpuConnectionCache and ConnectionPool clients (#28788)
* Remove TpuConnectionCache impl Default

* Add fallible ctors for Quic/UdpClients

* Add Quic/TpuClient ctor trait

* Remove Quic/UdpClient impl Default
2022-11-17 20:13:43 -07:00
Brooks Prumo d1ba42180d
clippy for rust 1.65.0 (#28765) 2022-11-09 19:39:38 +00:00
ryleung-solana 842fa993c8
Quic fix calculation of staked number of concurrent streams (#28705)
* Fix calculation of staked number of concurrent streams
2022-11-03 20:45:44 +08:00
Lijun Wang f156bc12ca
Enforce stream receive timeout (#28513)
In the quic server handle_connection, when we timed out in receiving the chunks, we loop forever to wait for the chunk. If the client never provide another chunk, the server can hopelessly wait for that chunk and wasting server resources. Instead WAIT_FOR_CHUNK_TIMEOUT_MS is introduced to bound this to 10 seconds at maximum. The stream will be dropped if it times out.
2022-11-02 10:09:32 -07:00
Lijun Wang f10ef763dc
Time out async sends to avoid slackers stuck in the queue for too long (#28545)
* Time out async sends to avoid slackers stuck in the queue for too long

* Fixed a clippy error

* Added stats for timeout counts

* Link with stats correctly
2022-10-31 13:46:45 -07:00
Lijun Wang 03f4204297 Check the boundary condition correctly. 2022-10-19 16:46:36 -07:00
Lijun Wang ab85ab7a5b handle semaphore locally without impacting the interface 2022-10-19 16:46:36 -07:00
Lijun Wang a6884d682c Restore the dev-dependency 2022-10-19 16:46:36 -07:00
Lijun Wang a378284621 removed crossbeam 2022-10-19 16:46:36 -07:00
Lijun Wang 6a7f78028d Removed unnecessary dependency 2022-10-19 16:46:36 -07:00
Lijun Wang f829f3583e FMT code 2022-10-19 16:46:36 -07:00
Lijun Wang 062c7f828c Fixed tests 2022-10-19 16:46:36 -07:00
Lijun Wang 5fbbad991e FMT code 2022-10-19 16:46:36 -07:00
Lijun Wang 8a269246ac Limit the number async tasks spawned to the runtime to avoid congestions. 2022-10-19 16:46:36 -07:00
Lijun Wang a8d5731e5b Give the quic client and server thread the names 2022-10-06 16:55:21 -07:00
Tyera Eulberg 10db560278
Add TpuConnectionCache and begin disentangling quic and udp (#28080)
* Add new empty crates and tpu_connection_cache module

* Add BaseTpuConnection trait and impl for udp and quic

* Add ConnectionPool trait and impl for udp and quic (quic-client doesn't build)

* Add a couple quic-specific apis to QuicConfig (from ConnectionCache)

* Re-export quic- and udp-client modules to prep for move

* Move ConnectionCacheStats to new module

* Move consts

* Duplicate ConnectionCache into tpu_connection_cache (doesn't build)

* Move methods to QuicConfig and remove unneeded methods (doesn't build)

* Genericize new ConnectionCache

* Rename new struct

* Copy unit tests (tests don't build)

* Fixup unit tests

* Move quic-specific test to quic-client crate

* DRY by using ConnectionPool method

* Prevent evicting pool about to be used
2022-09-30 19:51:50 -06:00
behzad nouri 9ee53e594d
patches clippy errors from new rust nightly release (#28028) 2022-09-23 20:57:27 +00:00
dependabot[bot] e1a49f7766
chore: bump indicatif from 0.17.0 to 0.17.1 (#27968)
* chore: bump indicatif from 0.17.0 to 0.17.1

Bumps [indicatif](https://github.com/console-rs/indicatif) from 0.17.0 to 0.17.1.
- [Release notes](https://github.com/console-rs/indicatif/releases)
- [Commits](https://github.com/console-rs/indicatif/compare/0.17.0...0.17.1)

---
updated-dependencies:
- dependency-name: indicatif
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-09-21 23:32:11 +00:00