Commit Graph

385 Commits

Author SHA1 Message Date
Tyera 7b1d446001
Admin RPC Service: move post-init activation to before wait-for-supermajority (#30544)
* Move AdminRpcRequestMetadataPostInit to solana-core

* Move AdminRpcRequestMetadataPostInit write to just before wait_for_supermajority

* Pass AdminRpcRequestMetadataPostInit in TestValidatorGenesis

* Fixup local-cluster
2023-03-01 19:38:11 -07:00
HaoranYi 16db984cb5
improve supermajority waiting logging (#30479)
make logging for supermajority waiting and stake percent from gossip in sync
2023-03-01 08:57:42 -06:00
Yihau Chen df3ef111f7
chore: workspace inheritance (#29893)
* introduce workspace.package

* introduce workspace.dependencies

* read version from root cargo.toml

* pass check when version = { workspace = true }

* don't bump version when version = { workspace = true }

* including workspace Cargo.toml when bump version

* programs/sbf use workspace inheritance

* fix increasing cargo version ignore program/sbf/Cargo.toml
2023-02-23 22:01:54 +08:00
Michael Vines 5136ed3448
Update homepage value for all crates (#30444) 2023-02-23 02:20:18 +00:00
Brooks 0a79a7882c
local-cluster test: Do not delete files that are in use by node (#30389) 2023-02-17 17:15:08 +00:00
Brooks 4d289222c9
Fixes race in local cluster incremental snapshot test (#30340) 2023-02-15 20:54:48 +00:00
Xiang Zhu 4909267c88
Add accounts hard-link files into the bank snapshot directory (#29496)
* Add accounts hard-link files into the bank snapshot directory

* Small adjustments and fixes.

* Address some of the review issues

* Fix compilation issues

* Change the latest slot snapshot storage from VecDeque to Option

* IoWithSourceAndFile and expanded comments on accounts

* last_slot_snapshot_storages in return value

* Update comments following the review input

* rename dir_accounts_hard_links to hard_link_path

* Add dir_full_state flag for add_bank_snapshot

* Let appendvec files hardlinking work with multiple accounts paths across multiple partitions

* Fixes for rebasing

* fix tests which generates account_path without adding run/

* rebasing fixes

* fix account path test failures

* fix test test_concurrent_snapshot_packaging

* review comments.  renamed the path setup function

* Addressed most of the review comments

* update with more review comments

* handle error from create_accounts_run_and_snapshot_dirs

* fix rebasing duplicate

* minor accounts_dir path cleanup

* minor cleanup, remove commented code

* misc review comments

* build error fix

* Fix test_incremental_snapshot_download_with_crossing_full_snapshot_interval_at_startup

* fix build error on MAX_BANK_SNAPSHOTS_TO_RETAIN

* rebase fix, update hardlink filename

* minor comment spelling fix

* rebasing fixes

* fix rebase issues; with_extension

* comments changes for review

* misc minor review issues

* bank.fill_bank_with_ticks_for_tests

* error handling on appendvec path

* fix use_jit

* minor comments refining

* Remove type AccountStorages

* get_account_path_from_appendvec_path return changed to Option

* removed appendvec_path.to_path_buf in create_accounts_run_and_snapshot_dirs

* add test_get_snapshot_accounts_hardlink_dir

* update last_snapshot_storages comment

* update last_snapshot_storages comment

* symlink map_err

* simplify test_get_snapshot_accounts_hardlink_dir with fake paths

* log last_snapshot_storages at the end of the loop
2023-02-15 09:52:07 -08:00
behzad nouri ded457cd73
embeds the new gossip ContactInfo in ClusterInfo (#30022)
Working towards replacing the legacy gossip contact-info with the new
one, the commit updates the respective field in gossip cluster-info.
2023-02-10 20:07:45 +00:00
Brennan da34f2edd0
Fix duplicate broadcast test (#30209) 2023-02-09 11:58:54 -08:00
Lijun Wang ada6136a6c
Refactor connection cache to support generic msgs (#29774)
tpu-client/tpu_connection_cache is refactored out the module and moved to connection-cache/connection_cache and the logic in client/connection_cache is consolidated to connection-cache/connection_cache as well. client/connection_cache only has a thin wrapper which forward calls to connection-cache/connection_cache and deal with constructions of quic/udp connection cache for clients using them both.2.

The TpuConnection is refactored to ClientConnection to make it generic and functions renamed to be proper for other workflows. eg. tpu_addr -> server_addr, send_transaction --> send_data and etc...

The enum dispatch is removed so that we can make the bulk of code of quic and udp agnostic of each other. The client is possible to load quic or udp only into its runtime.

The generic type parameter in the tpu-client/tpu_connection_cache is removed in order to create both quic and udp connection cache and use the object to send transactions with multiple branching when sending data. The generic type parameters and associated types are dropped in other types in order to make the trait "object safe" for this purpose.

I have annotated the code explaining the reasoning and the refactoring source -> destination.

There is no functional changes

bench-tps has been performed for rpc-client, thin-client and tpu-client. And it is found the performance number largely match the ones before the refactoring.
2023-02-01 18:10:06 -08:00
Will Hickey 04a6a631bc
Bump version to v1.16 (#30028) 2023-01-31 17:48:33 -06:00
carllin b345d97f67
Add local cluster test for optimistic confirmation with malformed votes (#29822) 2023-01-31 14:19:45 -06:00
joeaba a12bf8c003
Update maintainers references (#29997)
* update maintainers references

* chore: update maintainers reference
2023-01-31 08:07:13 -05:00
Xiang Zhu 856598969c
Account path add run parent with old path cleanup (#29942)
* Add run parent directory for accounts files

* fix test test_concurrent_snapshot_packaging

* review comments.  renamed the path setup function

* Addressed most of the review comments

* remove explict type def for map result

* handle create_accounts_run_and_snapshot_dirs error with expect

* update with more review comments

* minor fixes from review comments

* simplify account_filename option assignment

* handle error from create_accounts_run_and_snapshot_dirs

* use then instead of then_some for lazy evaluation

* Clean up files in the old account_path before trasitioning to the new run path

* try_exist and accounts_dir removing extra

* sync rmdir, is_dir check

* handle the account_path not deletable case
2023-01-30 10:26:43 -08:00
Xiang Zhu 4ebcacb4a3
Revert "Add run parent directory for accounts files (#29794)" (#29899)
This PR is causing OOM on master.  Reverting it for now.

This reverts commit 74f89d1494.
2023-01-25 10:03:01 -08:00
Ryo Onodera 40bbf99c74
Add fully-reproducible online tracer for banking (#29196)
* Add fully-reproducible online tracer for banking

* Don't use eprintln!()...

* Update programs/sbf/Cargo.lock...

* Remove meaningless assert_eq

* Group test-only code under aptly named mod

* Remove needless overflow handling in receive_until

* Delay stat aggregation as it's possible now

* Use Cow to avoid needless heap allocs

* Properly consume metrics action as soon as hold

* Trace UnprocessedTransactionStorage::len() instead

* Loosen joining api over type safety for replaystage

* Introce hash event to override these when simulating

* Use serde_with/serde_as instead of hacky workaround

* Update another Cargo.lock...

* Add detailed comment for Packet::buffer serialize

* Rename sender_overhead_minimized_receiver_loop()

* Use type interference for TraceError

* Another minor rename

* Retire now useless ForEach to simplify code

* Use type alias as much as possible

* Properly translate and propagate tracing errors

* Clarify --enable-banking-trace with better naming

* Consider unclean (signal-based) node restarts..

* Tweak logging and cli

* Remove Bank events as it's not needed anymore

* Make tpu own banking tracer thread

* Reduce diff a bit..

* Use latest serde_with

* Finally use the published rolling-file crate

* Make test code change more consistent

* Revive dead and non-terminating test code path...

* Dispose batches early now that possible

* Split off thread handle very early at ::new()

* Tweak message for TooSmallDirByteLimitl

* Remove too much of indirection

* Remove needless pub from ::channel()

* Clarify test comments

* Avoid needless event creation if tracer is disabled

* Write tests around file rotation and spill-over

* Remove unneeded PathBuf::clone()s...

* Introduce inner struct instead of tuple...

* Remove unused enum BankStatus...

* Avoid .unwrap() for the case of disabled tracer...
2023-01-25 21:54:38 +09:00
Yihau Chen 9193b4221d
Revert "chore: workspace inheritance (#29509)" (#29892)
This reverts commit a67d239dde.
2023-01-25 15:50:41 +08:00
Yihau Chen a67d239dde
chore: workspace inheritance (#29509)
* introduce workspace.package

* introduce workspace.dependencies

* read version from root cargo.toml

* pass check when version = { workspace = true }

* don't bump version when version = { workspace = true }

* including workspace Cargo.toml when bump version

* programs/sbf use workspace inheritance

* fix increasing cargo version ignore program/sbf/Cargo.toml
2023-01-25 13:59:59 +08:00
Xiang Zhu 74f89d1494
Add run parent directory for accounts files (#29794)
* Add run parent directory for accounts files

* fix test test_concurrent_snapshot_packaging

* review comments.  renamed the path setup function

* Addressed most of the review comments

* remove explict type def for map result

* handle create_accounts_run_and_snapshot_dirs error with expect

* update with more review comments

* minor fixes from review comments

* simplify account_filename option assignment

* handle error from create_accounts_run_and_snapshot_dirs

* use then instead of then_some for lazy evaluation
2023-01-24 16:44:35 -08:00
Brennan Watt a72c2ec459
Reduce TPU connection pool size for test (#29831) 2023-01-23 17:00:53 -08:00
Kevin Ji dd92f225bb
Use Ipv4Addr::{LOCALHOST, UNSPECIFIED} constants (#29813) 2023-01-23 16:49:51 -06:00
behzad nouri 8c212f59ad
renames ContactInfo to LegacyContactInfo (#29566)
Working towards adding a new ContactInfo where new sockets can be
added in a backward compatible way.
2023-01-08 16:00:55 +00:00
Brennan Watt 5a969ce3f1
Snapshot download test (#29474) 2023-01-02 17:51:28 -08:00
Brennan Watt e43957d5f9
Fix snapshot download test (#29457)
snapshot download tests will now attempt to load the downloaded snapshot
2023-01-02 14:11:07 -08:00
Ashwin Sekar 17b64005d3
Add more logging and documentation to flaky optimistic confirmation tests (#29418)
* Revert "add retry for flakey local cluster test (#29228)"

This reverts commit 7a97121747.

* Add logging for repair
2022-12-27 10:47:45 -07:00
Jeff Biseda a44ea779bd
add support for a repair protocol whitelist (#29161) 2022-12-15 19:24:23 -08:00
HaoranYi 7a97121747
add retry for flakey local cluster test (#29228) 2022-12-14 08:55:51 -06:00
Jeff Washington (jwash) 560143a267
remove ValidatorConfig.caching_enabled (#29172) 2022-12-09 11:31:55 -08:00
HaoranYi 33b15240ac
Revert #28945 (#29127)
revert #28945
2022-12-06 17:08:56 -06:00
steviez aeb6b53502
Remove unused Option<> around ValidatorConfig's SnapshotConfig (#29090)
Remove Option<> around ValidatorConfig's SnapshotConfig

The SnapshotConfig is required and is currently hard-coded to be a
Some().
2022-12-06 22:47:55 +00:00
behzad nouri 9524c9dbff patches errors from clippy::uninlined_format_args
https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
2022-12-06 19:32:15 +00:00
Haoran Yi e1ba5a2a63 add monitoring for open file descriptors stat 2022-12-06 13:31:50 -06:00
Jeff Washington (jwash) 20d8b5e98b
default some tests to write cache = true (#28917) 2022-11-21 15:53:39 -08:00
Tyera c32377b5af
Split out quic- and udp-client definitions (#28762)
* Move ConnectionCache back to solana-client, and duplicate ThinClient, TpuClient there

* Dedupe thin_client modules

* Dedupe tpu_client modules

* Move TpuClient to TpuConnectionCache

* Move ThinClient to TpuConnectionCache

* Move TpuConnection and quic/udp trait implementations back to solana-client

* Remove enum_dispatch from solana-tpu-client

* Move udp-client to its own crate

* Move quic-client to its own crate
2022-11-18 12:21:45 -07:00
Brooks Prumo d1ba42180d
clippy for rust 1.65.0 (#28765) 2022-11-09 19:39:38 +00:00
steviez 2272fd807e
Remove Blockstore manual compaction code (#28409)
The manual Blockstore compaction that was being initiated from
LedgerCleanupService has been disabled for quite some time in favor of
several optimizations.

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2022-10-28 10:39:00 +02:00
Ashwin Sekar f207af765e
Split out voting and banking threads in banking stage (#27931)
* Split out voting and banking threads in banking stage

Additionally this allows us to aggressively prune the buffer for voting threads
as with the new vote state only the latest vote from each validator is
necessary.

* Update local cluster test to use new Vote ix

* Encapsulate transaction storage filtering better

* Address pr comments

* Commit cargo lock change

* clippy

* Remove unsafe impls

* pr comments

* compute_sanitized_transaction -> build_sanitized_transaction

* &Arc -> Arc

* Move test

* Refactor metrics enums

* clippy
2022-10-20 21:10:48 +00:00
Brooks Prumo a8c6a9e5fc
Bank::freeze() waits for EAH calculation to complete (#28170) 2022-10-05 17:44:35 -04:00
Brooks Prumo 2f8f6c6a31
Send Epoch Accounts Hash requests from set_root() (#27764) 2022-09-30 14:59:41 -04:00
Brooks Prumo 48143a78af
local-cluster: Add timeout to wait_for_next_snapshot() (#27941) 2022-09-28 11:21:37 -04:00
behzad nouri 9a57c64f21
patches clippy errors from new rust nightly release (#27996) 2022-09-22 22:23:03 +00:00
Brooks Prumo 1ee595ca9c
remove AccountsDb::initial_blockstore_processing_complete (#27974) 2022-09-22 13:52:04 -04:00
dependabot[bot] c4fa849844
chore: bump itertools from 0.10.3 to 0.10.5 (#27962)
* chore: bump itertools from 0.10.3 to 0.10.5

Bumps [itertools](https://github.com/rust-itertools/itertools) from 0.10.3 to 0.10.5.
- [Release notes](https://github.com/rust-itertools/itertools/releases)
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-itertools/itertools/commits)

---
updated-dependencies:
- dependency-name: itertools
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-09-21 15:28:49 -06:00
Will Hickey c0e4379f43
Whickey/version v1.15 (#27739)
* Bump version to v1.13.0
* Bump version to v1.14.0
* Bump version to v1.15.0
2022-09-13 09:06:15 -05:00
Jeff Washington (jwash) 765c628546
use exit signal for acct idx bg threads (#27483) 2022-09-12 11:51:12 -07:00
Christian Kamm 90b8a3a44d
Remove KeypairInsecureClone trait and add insecure_clone() instead (#27396)
See discussion in #26248
2022-09-12 14:59:41 +00:00
Michael Vines 83d4d128c2 Add --process-ledger-before-service flag to solana-validator 2022-09-11 07:58:42 -07:00
Lijun Wang 7f223dc582
Added option to turn on UDP for TPU transaction and make UDP based TPU off by default (#27462)
--tpu-enable-udp is introduced. And when this is on, the transaction receive and transaction forward is enabled using udp.

Except for a few tests which was hard-coded sending transactions using udp, most tests are being done with udp based tpu disabled.
2022-09-07 13:19:14 -07:00
Brooks Prumo 93a4f80a2c
Handling snapshot requests is now required (#27537) 2022-09-07 10:08:42 -04:00
Lijun Wang fc054331e9
Fixed the local-cluster test case test_optimistic_confirmation_violation_detection (#27580)
* Fixed the local-cluster test case test_optimistic_confirmation_violation_detection, which is an existing test issue exposed when we disable the udp based tpu port. In the test, when we restart the node, we restarted the entry point validator. And when the gossip port of the restarted node changes, the two nodes are not able to talk to each other via gossip. I have changed the restart the logic to make the second validator the heavier node and restart it instead of the entypoint one (the first node).

* Addressed a review comment from Carl
2022-09-06 12:38:46 -07:00