* Connection pool in connection cache and handle connection errors
1. The connection not has a pool of connections per address, configurable, default 4
2. The connections per address share a lazy initialized endpoint
3. Handle connection issues better, avoid race conditions
4. Various log improvement for help debug connection issues
* client: Remove static connection cache, plumb it instead
* Add TpuClient::new_with_connection_cache to not break downstream
* Refactor get_connection and RwLock into ConnectionCache
* Fix merge conflicts from new async TpuClient
* Remove `ConnectionCache::set_use_quic`
* Move DEFAULT_TPU_USE_QUIC to client, use ConnectionCache::default()
1. Move logics related to creating endpoint, creating new connection and retry 0rtt connections into a wrapper construct QuicNewConnection to put the logic together.
2. get_or_add_connection: logic is much simplified to allow manage the connection cache -- the QUIC connection is lazy constructed. get_or_add_connection should no longer be a global hotspot.
3. Per connection stats update is moved out of get_or_add_connection to avoid bad connection impacting good connections.
* Use RwLock instead of Mutex in QUIC connection cache
* replace LruCache with HashMap
* fix tests
* fix tests
* refactor
* add cache eviction for a random connection on reaching upperbound
* cleanup
* Increase connection timeouts
* Bump quic connection cache to 1024
* Use constant for quic connection timeout and add warm cache service
* Fixes to QUIC warmup service
* fix check failure
* fixes after rebase
* fix timeout test
Co-authored-by: Pankaj Garg <pankaj@solana.com>
Add an interface send_wire_transaction_batch_async to TpuConnection to allow for sending batches without waiting for completion
Co-authored-by: Anatoly Yakovenko <anatoly@solana.com>
Pankaj Garg
Lijun Wang
Jon Cinque
ryleung-solana
Brennan Watt
sakridge
anatoly yakovenko