* Add helper function to create BlockstoreOptions from ValidatorConfig
* Make backup_and_clear_blockstore() honor ValidatorConfig options
backup_and_clear_blockstore() opens a Blockstore session; however, it
is currently using Blockstore::open(). This Blockstore method uses
BlockstoreOption::default() under the hood. As a result, any validator
args that adjust Blockstore settings are not considered in
backup_and_clear_blockstore().
This is especially problematic if the non-default value of
--rocksdb-shred-compaction is being used. In this case,
backup_and_clear_blockstore() was opening the wrong directory and
incorrectly finding an empty ledger.
This change plumbs any blockstore configuration to
backup_and_clear_blockstore().
* Move AdminRpcRequestMetadataPostInit to solana-core
* Move AdminRpcRequestMetadataPostInit write to just before wait_for_supermajority
* Pass AdminRpcRequestMetadataPostInit in TestValidatorGenesis
* Fixup local-cluster
* introduce workspace.package
* introduce workspace.dependencies
* read version from root cargo.toml
* pass check when version = { workspace = true }
* don't bump version when version = { workspace = true }
* including workspace Cargo.toml when bump version
* programs/sbf use workspace inheritance
* fix increasing cargo version ignore program/sbf/Cargo.toml
* measure! to measure_us!
* Consistent naming of transaction_recorder
* Remove outdated comment - Instant cannot be None
* use local
* Remove measure! import
* Add accounts hard-link files into the bank snapshot directory
* Small adjustments and fixes.
* Address some of the review issues
* Fix compilation issues
* Change the latest slot snapshot storage from VecDeque to Option
* IoWithSourceAndFile and expanded comments on accounts
* last_slot_snapshot_storages in return value
* Update comments following the review input
* rename dir_accounts_hard_links to hard_link_path
* Add dir_full_state flag for add_bank_snapshot
* Let appendvec files hardlinking work with multiple accounts paths across multiple partitions
* Fixes for rebasing
* fix tests which generates account_path without adding run/
* rebasing fixes
* fix account path test failures
* fix test test_concurrent_snapshot_packaging
* review comments. renamed the path setup function
* Addressed most of the review comments
* update with more review comments
* handle error from create_accounts_run_and_snapshot_dirs
* fix rebasing duplicate
* minor accounts_dir path cleanup
* minor cleanup, remove commented code
* misc review comments
* build error fix
* Fix test_incremental_snapshot_download_with_crossing_full_snapshot_interval_at_startup
* fix build error on MAX_BANK_SNAPSHOTS_TO_RETAIN
* rebase fix, update hardlink filename
* minor comment spelling fix
* rebasing fixes
* fix rebase issues; with_extension
* comments changes for review
* misc minor review issues
* bank.fill_bank_with_ticks_for_tests
* error handling on appendvec path
* fix use_jit
* minor comments refining
* Remove type AccountStorages
* get_account_path_from_appendvec_path return changed to Option
* removed appendvec_path.to_path_buf in create_accounts_run_and_snapshot_dirs
* add test_get_snapshot_accounts_hardlink_dir
* update last_snapshot_storages comment
* update last_snapshot_storages comment
* symlink map_err
* simplify test_get_snapshot_accounts_hardlink_dir with fake paths
* log last_snapshot_storages at the end of the loop
Snapshot construction and interaction code was a bit more manual than necessary, even causing a bug to slip past a review. Separated snapshot construction from the diffing of two snapshots.
This should make the logic clearer.
A recent change to this service to store the number of non-vote
transactions introduced a bug in the computation of the number of
transactions during the time interval. This resulted in bogus values
being stored in Blockstore and eventually getting served through RPC for
the TPS chart on explorer.
Dynamic dispatch forces heap allocation and adds extra overhead.
Dynamic casting as in the ones below, lacks compile-time type safety:
https://github.com/solana-labs/solana/blob/eeb622c4e/quic-client/src/lib.rs#L172-L175https://github.com/solana-labs/solana/blob/eeb622c4e/udp-client/src/lib.rs#L52-L55
The commit removes all instances of Any, Box<dyn ...>, and Arc<dyn ...>,
and instead uses generic and associated types.
There are only two protocols QUIC and UDP; and the code which has to
work with both protocols can use a trivial thin enum wrapper.
With respect to connection-cache specifically:
* connection-cache/ConnectionCache is a single protocol cache which
allows to use either QUIC or UDP without any build dependency on the
other protocol.
* client/ConnectionCache is an enum wrapper around both protocols and
can be used in the code which has to work with both QUIC and UDP.
Co-authored-by: Tyera Eulberg <tyera@solana.com>
tpu-client/tpu_connection_cache is refactored out the module and moved to connection-cache/connection_cache and the logic in client/connection_cache is consolidated to connection-cache/connection_cache as well. client/connection_cache only has a thin wrapper which forward calls to connection-cache/connection_cache and deal with constructions of quic/udp connection cache for clients using them both.2.
The TpuConnection is refactored to ClientConnection to make it generic and functions renamed to be proper for other workflows. eg. tpu_addr -> server_addr, send_transaction --> send_data and etc...
The enum dispatch is removed so that we can make the bulk of code of quic and udp agnostic of each other. The client is possible to load quic or udp only into its runtime.
The generic type parameter in the tpu-client/tpu_connection_cache is removed in order to create both quic and udp connection cache and use the object to send transactions with multiple branching when sending data. The generic type parameters and associated types are dropped in other types in order to make the trait "object safe" for this purpose.
I have annotated the code explaining the reasoning and the refactoring source -> destination.
There is no functional changes
bench-tps has been performed for rpc-client, thin-client and tpu-client. And it is found the performance number largely match the ones before the refactoring.
* Add run parent directory for accounts files
* fix test test_concurrent_snapshot_packaging
* review comments. renamed the path setup function
* Addressed most of the review comments
* remove explict type def for map result
* handle create_accounts_run_and_snapshot_dirs error with expect
* update with more review comments
* minor fixes from review comments
* simplify account_filename option assignment
* handle error from create_accounts_run_and_snapshot_dirs
* use then instead of then_some for lazy evaluation
* Clean up files in the old account_path before trasitioning to the new run path
* try_exist and accounts_dir removing extra
* sync rmdir, is_dir check
* handle the account_path not deletable case
* Add fully-reproducible online tracer for banking
* Don't use eprintln!()...
* Update programs/sbf/Cargo.lock...
* Remove meaningless assert_eq
* Group test-only code under aptly named mod
* Remove needless overflow handling in receive_until
* Delay stat aggregation as it's possible now
* Use Cow to avoid needless heap allocs
* Properly consume metrics action as soon as hold
* Trace UnprocessedTransactionStorage::len() instead
* Loosen joining api over type safety for replaystage
* Introce hash event to override these when simulating
* Use serde_with/serde_as instead of hacky workaround
* Update another Cargo.lock...
* Add detailed comment for Packet::buffer serialize
* Rename sender_overhead_minimized_receiver_loop()
* Use type interference for TraceError
* Another minor rename
* Retire now useless ForEach to simplify code
* Use type alias as much as possible
* Properly translate and propagate tracing errors
* Clarify --enable-banking-trace with better naming
* Consider unclean (signal-based) node restarts..
* Tweak logging and cli
* Remove Bank events as it's not needed anymore
* Make tpu own banking tracer thread
* Reduce diff a bit..
* Use latest serde_with
* Finally use the published rolling-file crate
* Make test code change more consistent
* Revive dead and non-terminating test code path...
* Dispose batches early now that possible
* Split off thread handle very early at ::new()
* Tweak message for TooSmallDirByteLimitl
* Remove too much of indirection
* Remove needless pub from ::channel()
* Clarify test comments
* Avoid needless event creation if tracer is disabled
* Write tests around file rotation and spill-over
* Remove unneeded PathBuf::clone()s...
* Introduce inner struct instead of tuple...
* Remove unused enum BankStatus...
* Avoid .unwrap() for the case of disabled tracer...
* introduce workspace.package
* introduce workspace.dependencies
* read version from root cargo.toml
* pass check when version = { workspace = true }
* don't bump version when version = { workspace = true }
* including workspace Cargo.toml when bump version
* programs/sbf use workspace inheritance
* fix increasing cargo version ignore program/sbf/Cargo.toml
* Add run parent directory for accounts files
* fix test test_concurrent_snapshot_packaging
* review comments. renamed the path setup function
* Addressed most of the review comments
* remove explict type def for map result
* handle create_accounts_run_and_snapshot_dirs error with expect
* update with more review comments
* minor fixes from review comments
* simplify account_filename option assignment
* handle error from create_accounts_run_and_snapshot_dirs
* use then instead of then_some for lazy evaluation
Brooks
Trent Nelson
behzad nouri
Andrew Fitzgerald
Ashwin Sekar
Jeff Biseda
HaoranYi
steviez
sakridge
Tyera
Brennan
Jeff Washington (jwash)
Yihau Chen
Michael Vines
Xiang Zhu
Illia Bobyr
Proph3t
Tao Zhu
Wen
Pankaj Garg
Lijun Wang
dependabot[bot]
Will Hickey
carllin
joeaba
Kirill Fomichev
Ryo Onodera