Commit Graph

78 Commits

Author SHA1 Message Date
Pankaj Garg 9d42cd7efe
Initialize fork graph in program cache during bank_forks creation (#33810)
* Initialize fork graph in program cache during bank_forks creation

* rename BankForks::new to BankForks::new_rw_arc

* fix compilation

* no need to set fork_graph on insert()

* fix partition tests
2023-10-23 09:32:41 -07:00
steviez 56ccffdaa5
Replace get_tmp_ledger_path!() with self cleaning version (#33702)
This macro is used a lot for tests to create a ledger path in order to
open a Blockstore. Files will be left on disk unless the test remembers
to call Blockstore::destroy() on the directory. So, instead of requiring
this, use the get_tmp_ledger_path_auto_delete!() macro that creates a
TempDir (which automatically deletes itself when it goes out of scope).
2023-10-21 11:38:31 +02:00
steviez 8bd0e4cd95
Change getHealth to compare optimistically confirmed slots (#33651)
The current getHealth mechanism checks a local accounts hash slot vs.
those of other nodes as specified by --known-validator. This is a
very coarse comparison given that the default for this value is 100
slots. More so, any nodes using a value larger than the default
(ie --incremental-snapshot-interval 500) will likely see getHealth
return status behind at some point.

Change the underlying mechanism of how health is computed. Instead of
using the accounts hash slots published in gossip, use the latest
optimistically confirmed slot from the cluster. Even when a node is
behind, it is able to observe cluster optimistically confirmed by slots
by viewing votes published in gossip.

Thus, the latest cluster optimistically confirmed slot can be compared
against the latest optimistically confirmed bank from replay to
determine health. This new comparison is much more granular, and not
needing to depend on individual known validators is also a plus.
2023-10-16 11:21:33 -05:00
steviez 3eae980293
Minor cleanup on some snapshot related tests (#33485) 2023-10-02 20:25:30 +02:00
Illia Bobyr 4353ac6797
Pass Arc<AtomicBool> by value, not by reference. (#31916)
`Arc` is already a reference internally, so it does not seem to be
beneficial to pass a reference to it.  Just adds an extra layer of
indirection.

Functions that need to be able to increment `Arc` reference count need
to take `Arc<AtomicBool>`, but those that just want to read the
`AtomicBool` value can accept `&AtomicBool`, making them a bit more
generic.

This change focuses specifically on `Arc<AtomicBool>`.  There are other
uses of `&Arc<T>` in the code base that could be converted in a similar
manner.  But it would make the change even larger.
2023-06-01 17:25:48 -07:00
behzad nouri 9281ab7d97
separates out connection-cache metrics for different protocols (#31803) 2023-05-25 14:48:22 +00:00
Tyera 929fe015c5
Remove redundant clone (#31690) 2023-05-17 21:32:02 +09:00
Tyera 2cdb43ff1b
Eradicate zombie RPC threads (#31688)
* Ensure jsonrpc server has closed when joining rpc_service thread

* Use same exit bool as other services

* Remove redundant registered exit line
2023-05-17 03:09:44 +00:00
behzad nouri 4e34abbf3d
specifies protocol in contact-info get-socket api (#31602) 2023-05-12 16:16:20 +00:00
behzad nouri 6a4a0418a6
removes hard-coded QUIC_PORT_OFFSET from connection-cache (#31541)
New ContactInfo has api identifying QUIC vs UDP ports; no need to hard-code
port-offset deep in connection-cache.
2023-05-09 13:46:17 +00:00
Tyera 03c1744e1d
Register SendTransactionService exit (#31261)
* Pass exit into SendTransactionService

* Abort SendTransactionService with BanksService

* Register SendTransactionService exit as part of RpcService validator Exit

* Improve test, ensure receiver has been dropped
2023-04-20 18:23:14 -06:00
Brooks e8ea722061
Uses AccountsHashes type for AccountsHashes CrdsData variant (#31003) 2023-04-03 16:42:21 -04:00
Tyera b389d509a8
Track max_complete_rewards_slot for use in rpc, bigtable (#30698)
* Add RewardsMessage enum

* Cache and update max_complete_rewards_slot

* Plumb max_complete_rewards_slot into JsonRpcRequestProcesseor

* Use max_complete_rewards_slot to check get_block requests

* Use max_complete_rewards_slot to limit Bigtable uploads

* Plumb max_complete_rewards_slot into RpcSubscriptions

* Use max_complete_rewards_slot to limit block subscriptions

* Nit: fix test
2023-03-14 12:08:48 -06:00
mvines a791e1b541
rpc_service: Rename variable for consistency with the rest of the code base (#30694)
Rename variable for consistency with the rest of the code base
2023-03-13 19:37:06 +00:00
behzad nouri ded457cd73
embeds the new gossip ContactInfo in ClusterInfo (#30022)
Working towards replacing the legacy gossip contact-info with the new
one, the commit updates the respective field in gossip cluster-info.
2023-02-10 20:07:45 +00:00
Brooks daea6722f6
Improves RPC path sanitation (#29931) 2023-01-26 23:49:02 -05:00
behzad nouri 1c7662a37f
asserts that cluster-info keypair is consistent with contact-info id (#29818) 2023-01-24 16:57:55 +00:00
Kevin Ji dd92f225bb
Use Ipv4Addr::{LOCALHOST, UNSPECIFIED} constants (#29813) 2023-01-23 16:49:51 -06:00
behzad nouri 8c212f59ad
renames ContactInfo to LegacyContactInfo (#29566)
Working towards adding a new ContactInfo where new sockets can be
added in a backward compatible way.
2023-01-08 16:00:55 +00:00
Tyera fd13323f4b
Split rpc_accounts api to identify calls that scan accountsDB (#28968)
* Update stale comment

* Collect RPC endpoints that perform accounts scans
2022-11-29 10:08:19 -07:00
Tyera c32377b5af
Split out quic- and udp-client definitions (#28762)
* Move ConnectionCache back to solana-client, and duplicate ThinClient, TpuClient there

* Dedupe thin_client modules

* Dedupe tpu_client modules

* Move TpuClient to TpuConnectionCache

* Move ThinClient to TpuConnectionCache

* Move TpuConnection and quic/udp trait implementations back to solana-client

* Remove enum_dispatch from solana-tpu-client

* Move udp-client to its own crate

* Move quic-client to its own crate
2022-11-18 12:21:45 -07:00
Tyera Eulberg 9b8bed86f9
Add getRecentPrioritizationFees RPC endpoint (#27278)
* Plumb priority_fee_cache into rpc

* Add PrioritizationFeeCache api

* Add getRecentPrioritizationFees rpc endpoint

* Use MAX_TX_ACCOUNT_LOCKS to limit input keys

* Remove unused cache apis

* Map fee data by slot, and make rpc account inputs optional

* Add priority_fee_cache to rpc test framework, and add test

* Add endpoint to jsonrpc docs

* Update docs/src/developing/clients/jsonrpc-api.md

* Update docs/src/developing/clients/jsonrpc-api.md
2022-09-01 23:12:12 +00:00
Tyera Eulberg b8b3d723da
Use new client crates (#27360)
* Update ancillary cli crates

* Update cli

* Update command-line tools

* Update rpc, etc

* Update client-test

* Update core, validator

* Update local-cluster
2022-08-24 10:47:02 -06:00
Tyera Eulberg c24eaa36f8
Split up solana-client (#27246)
* Move thin-client to new crate

* Move tpu client and varieties to new crate

* Move pubsub-client to new crate

* Move rpc-client to new crate

* Add client-common crate to avoid circular dependencies

* Move rpc_cache and make less pub

* Remove unused unpub modules

* Add nonce-client

* Remove unused dependencies

* Fix rpc_client docs

* Move spinner to calling clients

* Rename client-common to rpc-client-api

* Remove unnecessary rpc_ prefix

* Remove unused ClientErrorKind variant

* Remove unnecessary Client prefix

* Move mod declarations into lib.rs and remove unnecessary files

* Rename nonce-client and remove redundant module name

* Restore mock_sender_for_cli in solana-client
2022-08-24 05:34:35 +00:00
Michael Vines 3f4731b37f Standardize thread names
Tenets:
1. Limit thread names to 15 characters
2. Prefix all Solana-controlled threads with "sol"
3. Use Camel case. It's more character dense than Snake or Kebab case
2022-08-20 07:49:39 -07:00
Jeff Biseda e50013acdf
Handle JsonRpcService startup failure (#27075) 2022-08-11 23:25:20 -07:00
Ahmad ad0acaa6fa
added arg --rpc-max-request-payload-size to validator (#26377)
* added ability to pass --rpc-max-request-payload-size to validator

* fixed lint errors

* more lint fix

* patch

Co-authored-by: ultd <ultd>
Co-authored-by: Justin Starry <justin@solana.com>
2022-08-08 13:50:05 +01:00
behzad nouri 61f0a7d9c3
replaces Mutex<PohRecorder> with RwLock<PohRecorder> (#26370)
Mutex causes superfluous lock contention when a read-only reference suffices.
2022-07-05 14:29:44 +00:00
Jeff Washington (jwash) 557bf6e656
allow initial hash calc to occur in bg (#26271)
* allow initial hash calc to occur in bg

* validator_initialized -> startup_verification_complete

* add infos for leader and vote

* rework snapshot for startup verification

* change to assert
2022-06-29 16:48:33 -05:00
Jeff Biseda bafdb7dd62
Revert handle start_http failure in rpc_service (#25400) (#26130)
* revert e263be2000
2022-06-22 10:52:27 -07:00
Dan Coombs 32a58dd9e0
Add ability to use a non-default app profile id in bigtable requests (#25968)
* Add ability to use a non-default app profile id in bigtable requests

* Only run subcommand once when getting global configs

* Remove unneded scoping on option type
2022-06-16 00:58:16 +00:00
Jon Cinque 79a8ecd0ac
client: Remove static connection cache, plumb it instead (#25667)
* client: Remove static connection cache, plumb it instead

* Add TpuClient::new_with_connection_cache to not break downstream

* Refactor get_connection and RwLock into ConnectionCache

* Fix merge conflicts from new async TpuClient

* Remove `ConnectionCache::set_use_quic`

* Move DEFAULT_TPU_USE_QUIC to client, use ConnectionCache::default()
2022-06-08 13:57:12 +02:00
Jeff Biseda e263be2000
handle start_http failure in rpc_service (#25400) 2022-05-20 17:59:23 -07:00
Justin Starry 4690956849
RPC: Add minContextSlot param to ensure consistent responses (#25161)
* RPC: Add minContextSlot param to ensure consistent responses

* revert changes to deprecated endpoints

* update docs
2022-05-17 14:46:02 +08:00
Tyera Eulberg bc005e3408
Add configurable limit to number of blocks to check before Bigtable upload (#24716)
* Add ConfirmedBlockUploadConfig, no behavior changes

* Add comment

* A little DRY cleanup

* Add configurable limit to number of blocks to check in Blockstore and Bigtable before uploading

* Limit blockstore and bigtable look-ahead

* Exit iterator early when reach ending_slot

* Use rooted_slot_iterator instead of slot_meta_iterator

* Only check blocks in the ledger
2022-05-13 07:34:02 +00:00
DimAn 2fa9bc3e70
Add options to store full and/or incremental snapshots in separate locations (#24247) 2022-05-10 16:37:41 -04:00
Justin Starry a61652104b
Avoid holding lock guards in match expressions (#24805)
* Avoid holding bank forks read lock for RPC requests

* Avoid using lock guards in temporaries

* revert fetch stage change
2022-04-29 16:32:46 +08:00
Rachael Pai aea17c35ae
Add a stringified credential option for LedgerStorage (#24314)
* add a stringified credential option for LedgerStorage

* fix clippy::useless-format warning

* change CredentialOption to enum CredentialType

* rename credential_option to credential_type

* restore LedgerStorage new fn signature

* fmt

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2022-04-13 14:35:06 -06:00
Trent Nelson 9b32b72990 bigtable: allow custom instance names 2022-03-21 11:26:49 -06:00
Trent Nelson f513195468 bigtable: add a config ctor for `LedgerStorage` 2022-03-21 11:26:49 -06:00
Tyera Eulberg 63ee00e647 Refactor validator bigtable config 2022-03-21 11:26:49 -06:00
DimAn 739e43ba58
Add ability to get the latest incremental snapshot via RPC (#23788) 2022-03-21 11:48:49 -05:00
HaoranYi 0c684721d8
Separate remotely downloaded snapshot archives (#23510)
* seperate remotely downloaded snapshot archives

* add str const for snapshot download dir

* only walk remote sub directory

* move directory creation outside of loop

* move is_remote to traits

* clippy simplify

* clippy

* clippy

* add unittest

* fix local cluster tests

* look for remote snapshot archive in remote foler

* create remote dir in tests

* use snapshot download dir constant

* extract build_remote_dir fn

* fix build

* code review - walking snapshot archived dirs explicitly

* fix build

* fix build

* fix comments

* Update runtime/src/snapshot_utils.rs

Co-authored-by: Brooks Prumo <brooks@prumo.org>

* clippy

* borrow to avoid copy

Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-03-14 14:03:59 -05:00
Trent Nelson eac4a6df68 rpc: use minimal mode by default 2022-02-01 19:00:06 -07:00
Jeff Biseda 8b66625c95
convert std::sync::mpsc to crossbeam_channel (#22264) 2022-01-11 02:44:46 -08:00
pieceofr d9220652ad
[ledger-tool]compare_blocks (#22229)
* 1.made load_credentials accept credential path as a parameter. 2.partial implement bigtable comparasion function

* finding missing blocks in bigtables in a specified range

* refactor compare-blocks,add unit test for missing_blocks and fmt

* compare-block fix last block bug

* refactor compare-block and improve wording

* Update ledger-tool/src/bigtable.rs

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* update compare-block command-line description

* style:improve wording/naming/code style

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2022-01-06 14:36:03 +08:00
Michael Vines b8837c04ec Reformat imports to a consistent style for imports
rustfmt.toml configuration:
  imports_granularity = "One"
  group_imports = "One"
2021-12-03 09:19:13 -08:00
Tyera Eulberg 9e043d2055
Use max_complete_transaction_status_slot in BigTableUploadService (#21401)
* Use max_complete_transaction_status_slot in BigTableUploadService

* Use node root to limit BigTableUploadService
2021-11-23 17:36:13 -07:00
Michael Keleti b0ca335463
Rename "trusted" to "known" in `validators/` (#21197)
* Replaced trusted with known validator

* Format Convention
2021-11-12 11:57:55 -07:00
Ivan Mironov 0e751ef7df Add validator option to change niceness of RPC server threads
Fixes https://github.com/solana-labs/solana/issues/14556
2021-11-04 17:16:46 -06:00