6.0 KiB

Raw Permalink Blame History

Welcome to the Lite RPC QUIC Forward Proxy Sub-Project!

Project Info

Status: experimental - Feedback welcome (Solana-Discord: grooviegermanikus)
Lite RPC QUIC Forward Proxy
Lite RPC

Purpose

This component (quic-forward-proxy) can be optionally used along with one or multiple Lite RPC micro-service instances to optimized and simplify sending transactions to Solana TPUs.

Benefits:

the quic-forward-proxy can batch transactions from multiple source and send them efficiently to the Solana validator
the quic-forward-proxy can be optionally configured with a validator identity keypair:
- connections to TPU will be privileged (staked connections)
- keypair can be kept in one place while the Lite RPC instances can benefit from the staked connection

Configuration

Prepare: choose a proxy port (e.g. 11111) and bind address of appropriate (minimal) network interface (e.g. 127.0.0.1)

run quic proxy

# unstaked
solana-lite-rpc-quic-forward-proxy --proxy-listen-addr 127.0.0.1:11111
# staked
solana-lite-rpc-quic-forward-proxy --proxy-listen-addr 127.0.0.1:11111 --identity-keypair /pathto/validator-keypair.json

run lite-rpc

lite-rpc --experimental-quic-proxy-addr 127.0.0.1:11111

Architecture Overview

 +------------+          +------------+          +------------+          +------------+
 |            |          |            |          |            |          |            |
 |   client   | ---1---> |  lite-rpc  | ---2---> |   proxy    | ---3---> |  validator |
 |            |          |            |          |            |          |            |
 +------------+          +------------+          +------------+          +------------+
 
 1. rpc request
 2. tpu forward proxy request (QUIC): transactions, tpu address and tpu identity
 3. tpu call (QUIC), transactions:
 
 * client: RPC client to lite-rpc
 * proxy: QUIC forward proxy service (one instance)
 * lite-rpc: N lite-rpc services 
 * validator: solana validator (TPU) according to the leader schedule

Local Development / Testing

Rust Integration Test

Use integrated testing in quic_proxy_tpu_integrationtest.rs for fast feedback.

Local Setup With Solana Test Validator

run test-validator (tested with 1.16.1)

RUST_LOG="error,solana_streamer::nonblocking::quic=debug" solana-test-validator --log

run quic proxy

# unstaked
RUST_LOG=debug cargo run --bin solana-lite-rpc-quic-forward-proxy -- --proxy-listen-addr 0.0.0.0:11111
# staked
RUST_LOG=debug cargo run --bin solana-lite-rpc-quic-forward-proxy -- --proxy-listen-addr 0.0.0.0:11111 --identity-keypair /pathto-test-ledger/validator-keypair.json

run lite-rpc

RUST_LOG=debug cargo run --bin lite-rpc -- --quic-proxy-addr 127.0.0.1:11111

run rust bench tool in lite-rpc
```
cd bench; cargo run -- --tx-count=10
```

Implementation Details

proxy expects clients to send transactions via QUIC using the proxy request format:
- array of transactions (signature and raw bytes)
- array of tpu target nodes (address:port and identity public key)
the proxy is designed to be light-weight and stateless (no persistence)
note: only one instance of the proxy should talk to the TPU nodes at a time to be able to correctly comply with the validator quic policy
outbound connections (to TPU):
- proxy tries to maintain active quic connections to recent TPU nodes using transparent reconnect
- proxy will maintain multiple quic connection per TPU according to the Solana validator quic policy
- proxy will use lightweight quic streams to send the transactions
inbound traffic (from Lite RPC)
- client-proxy-communication is done via QUIC using a custom wire format
- proxy supports only quic ATM but that could be extended to support other protocols
- proxy should perform client authentication by TLS (see issue)
proxy uses a single queue (channel) for buffering the transactions from any inbound connection
TPU selection / Leader Schedule
- the proxy will not perform any TPU selection; the TPU target nodes MUST be selected by the client (Lite RPC) and not by the proxy
- pitfall: the TPU target node list might become stale if the transactions are not sent out fast enough

Example Output from Solana Validator:

(note: the peer type is Staked)

[2023-06-26T15:16:18.430602000Z INFO  solana_streamer::nonblocking::quic] Got a connection 127.0.0.1:8058
[2023-06-26T15:16:18.430633000Z DEBUG solana_streamer::nonblocking::quic] Peer public key is EPLzGRhibYmZ7qysF9BiPmSTRaL8GiLhrQdFTfL8h2fy
[2023-06-26T15:16:18.430839000Z DEBUG solana_streamer::nonblocking::quic] Peer type: Staked, stake 999999997717120, total stake 999999997717120, max streams 2048 receive_window Ok(12320) from peer 127.0.0.1:8058
[2023-06-26T15:16:18.430850000Z DEBUG solana_streamer::nonblocking::quic] quic new connection 127.0.0.1:8058 streams: 0 connections: 1
[2023-06-26T15:16:18.430854000Z DEBUG solana_streamer::nonblocking::quic] stream error: ApplicationClosed(ApplicationClose { error_code: 0, reason: b"done" })

Solana Validator QUIC Policy

TPU has complex logic to assign connection capacity to TPU clients (see Solana quic.rs)

it purges connections
it limits number of parallel quic streams
it considers stake vs unstaked connections (validator identity signature in QUIC TLS certificate, see Solana get_remote_pubkey+get_pubkey_from_tls_certificate)
it keeps connections on a per peer address / peer identity basis
...

License & Copyright

Licensed under the AGPL-3.0 license

6.0 KiB Raw Permalink Blame History