Commit Graph

2753 Commits

Author SHA1 Message Date
apfitzge 079bf561b0
Clean_up/upb_push_comment (#27707) 2022-09-12 18:59:41 -05:00
Jeff Washington (jwash) 765c628546
use exit signal for acct idx bg threads (#27483) 2022-09-12 11:51:12 -07:00
behzad nouri 4f22ee8f9b uses varint encoding for vote-state lockout offsets
The commit removes CompactVoteStateUpdate and instead reduces serialized
size of VoteStateUpdate using varint encoding for vote-state lockout
offsets.
2022-09-12 16:31:20 +00:00
Christian Kamm 90b8a3a44d
Remove KeypairInsecureClone trait and add insecure_clone() instead (#27396)
See discussion in #26248
2022-09-12 14:59:41 +00:00
Michael Vines 83d4d128c2 Add --process-ledger-before-service flag to solana-validator 2022-09-11 07:58:42 -07:00
Jeff Washington (jwash) 1f00b468e5
add enable_rehashing to AccountsPackage (#27644) 2022-09-08 09:25:25 -07:00
apfitzge a9c5adbf88
UnprocessedPacketBatches pop_max fn are only used in tests (#27645) 2022-09-08 11:01:14 -05:00
Maximilian Schneider cc58968b76
add new leader slot metric to track account contention throttling (#27654) 2022-09-08 09:22:58 -05:00
Xiang Zhu 4308c300b4
In ledger-tool delete the account files in the async way (#27622)
* In ledger-tool delete the account files in the async way

* format changes by ./cargo nightly fmt --all
2022-09-07 14:35:06 -07:00
Brooks Prumo 6a322de845
Make Accounts Background Services aware of Epoch Accounts Hash (#27626) 2022-09-07 20:41:40 +00:00
Lijun Wang 7f223dc582
Added option to turn on UDP for TPU transaction and make UDP based TPU off by default (#27462)
--tpu-enable-udp is introduced. And when this is on, the transaction receive and transaction forward is enabled using udp.

Except for a few tests which was hard-coded sending transactions using udp, most tests are being done with udp based tpu disabled.
2022-09-07 13:19:14 -07:00
apfitzge c04747dd66
cluster_slot_state_verifier: clippy nightly fixes (#27521)
clippy nightly fixes
2022-09-07 15:04:56 -05:00
apfitzge 1465ec947d
replay_stage: clippy nightly fixes (#27520)
clippy nightly fixes
2022-09-07 15:04:46 -05:00
apfitzge d6a1e7498f
Add tests for deserialize_and_collect_packets (#27623) 2022-09-07 12:52:18 -05:00
Jeff Washington (jwash) 22007a3c96
allow accounts hash calc to specify enable_rehashing (#27615) 2022-09-07 10:16:52 -07:00
Jeff Washington (jwash) a31d4a597d
serialize epoch_accounts_hash (#27516) 2022-09-07 10:07:00 -07:00
Brooks Prumo 93a4f80a2c
Handling snapshot requests is now required (#27537) 2022-09-07 10:08:42 -04:00
Jeff Biseda 269eb519dd
track time to coalesce entries in recv_slot_entries (#27525) 2022-09-06 16:07:17 -07:00
apfitzge a67d56f462
refactor: add function for deserializing and collecting packets - separate from channel receive (#27548) 2022-09-06 15:54:31 -05:00
Brooks Prumo 6684c62280
Add SnapshotUsage to SnapshotConfig (#27508) 2022-09-02 08:56:23 -04:00
Brennan Watt 242c9cb442
RPC Notifier Signal when Setup Complete (#27481)
* RPC notifier signal when ready
2022-09-01 16:39:55 -07:00
Tyera Eulberg 9b8bed86f9
Add getRecentPrioritizationFees RPC endpoint (#27278)
* Plumb priority_fee_cache into rpc

* Add PrioritizationFeeCache api

* Add getRecentPrioritizationFees rpc endpoint

* Use MAX_TX_ACCOUNT_LOCKS to limit input keys

* Remove unused cache apis

* Map fee data by slot, and make rpc account inputs optional

* Add priority_fee_cache to rpc test framework, and add test

* Add endpoint to jsonrpc docs

* Update docs/src/developing/clients/jsonrpc-api.md

* Update docs/src/developing/clients/jsonrpc-api.md
2022-09-01 23:12:12 +00:00
apfitzge 3bdc5b3f2b
separate packet_deserializer inside banking_stage (#27120)
* separate packet_deserializer inside banking_stage

* Make ReceivePacketResults into a struct with named fields
2022-09-01 10:00:48 -05:00
Tao Zhu 8bb039d08d
collect min prioritization fees when replaying sanitized transactions (#26709)
* Collect blocks' minimum prioritization fees when replaying sanitized transactions

* Limits block min-fee metrics reporting to top 10 writable accounts

* Add service thread to asynchronously update and finalize prioritization fee cache

* Add bench test for prioritization_fee_cache

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2022-08-31 08:00:55 -05:00
Haoran Yi 5b64107626 make pruned_bank channel unbonded.
In kin-sim, we found that bounded channel causes halt for account
background services. As the number of accounts grows, the time for
pruning and cleaning increases, which would leads to longer intervals
between the pruning of deaded bank slots. With 1.7B accounts, we will
exceed the 10K bounded channel threshold that causes halt of account
back ground services. Without pruning, the node will eventually run out
of memory.
2022-08-29 19:06:30 -05:00
Brooks Prumo 3c7cd62030
Move pruned_banks_receiver into PrunedBanksRequestHandler (#27445) 2022-08-29 13:30:06 -04:00
Brennan Watt 46a48760db
Switch concurrent replay from feature to param (#27401)
* Switch concurrent replay from feature to param
2022-08-26 12:36:02 -07:00
Will Hickey 5eefc256d6
Fix startup panic if removing accounts directory fails (#27386)
* Remove contents of accounts directory if deleting the directory fails.
2022-08-25 20:35:12 -05:00
Trent Nelson b1cff5d740 make fatal log message sound fatal 2022-08-25 21:49:12 +00:00
Jeff Biseda d1522fc790
coalesce entries in recv_slot_entries to target byte count (#27321) 2022-08-25 13:51:55 -07:00
Jeff Washington (jwash) 2da93bd45a
add text to assert (#27377) 2022-08-24 14:11:53 -05:00
Tyera Eulberg b8b3d723da
Use new client crates (#27360)
* Update ancillary cli crates

* Update cli

* Update command-line tools

* Update rpc, etc

* Update client-test

* Update core, validator

* Update local-cluster
2022-08-24 10:47:02 -06:00
Ashwin Sekar efa6201eda
Check overflow on vote tx compaction boundary (#27185)
* Check overflow on vote tx compaction boundary

Check for overflow during the conversion between VoteStateUpdate and
CompactVoteStateUpdate.

* Try removing clippy supress
2022-08-23 22:29:03 -07:00
Xiang Zhu 827d8e4bc0
Fallback to synchronous rm_dir call if path moving fails (#27306)
Remove some log lines, as suggested in PR #26910
2022-08-22 22:47:39 -07:00
Brennan Watt e4a7d01e10
Rust v1.63 (#27303)
* Upgrade to Rust v1.63.0

* Add nightly_clippy_allows

* Resolve some new clippy nightly lints

* Increase QUIC packets completion timeout

* Update quinn-udp crate

Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-22 18:01:03 -07:00
Michael Vines 7bdeea10ad Assign custom names to the Rayon global thread pool 2022-08-22 17:56:55 +00:00
Michael Vines 7c01c1ecc6
Update delete_path thread name 2022-08-22 09:01:26 -07:00
Jeff Washington (jwash) fc1a4dd11a
run hash calc with index on failure (#27279) 2022-08-22 10:58:04 -05:00
Michael Vines 3f4731b37f Standardize thread names
Tenets:
1. Limit thread names to 15 characters
2. Prefix all Solana-controlled threads with "sol"
3. Use Camel case. It's more character dense than Snake or Kebab case
2022-08-20 07:49:39 -07:00
Xiang Zhu c54824e4f5
Account files remove (#26910)
* Create a new function cleanup_accounts_paths, a trivial change

* Remove account files asynchronously

* Update and simplify the implementation after the validator test runs.

* Fixes after testing on the dev device

* Discard tokio.  Use thread instead

* Fix comments format

* Fix config type to pass the github test

* Fix failed tests.  Handle the case of non-existing path

* Final cleanup, addressing the review comments
Avoided OsString.
Made the function more generic with "impl AsRef<Path>"

Co-authored-by: Jeff Washington <jeff.washington@solana.com>
2022-08-19 23:56:52 -07:00
apfitzge eb06bb61e8
banking stage: actually aggregate tracer packet stats (#27118)
* aggregated_tracer_packet_stats_option was alwasys None

* Actually accumulate tracer packet stats
2022-08-19 15:16:56 -05:00
Will Hickey dba2fd5a16
Enable QUIC client by default. Add arg to disable QUIC client. (Forward port #26927) (#27194)
Enable QUIC client by default. Add arg to disable QUIC client.

* Enable QUIC client by default. Add arg to disable QUIC client.
* Deprecate --disable-quic-servers arg
* Add #[ignore] annotation to failing tests
2022-08-19 09:15:15 -05:00
Brennan Watt 7573000d87
Revert "Rust v1.63.0 (#27148)" (#27245)
This reverts commit a2e7bdf50a.
2022-08-19 09:19:44 +01:00
behzad nouri 6928b2a5af
adds hash domain to ping-pong protocol (#27193)
In order to maintain backward compatibility, for now the responding node
will hash the token both with and without domain so that the other node
will accept the response regardless of its upgrade status.
Once the cluster has upgraded to the new code, we will remove the legacy
domain = false case.
2022-08-18 22:39:31 +00:00
Brennan Watt a2e7bdf50a
Rust v1.63.0 (#27148)
* Upgrade to Rust v1.63.0

* Add nightly_clippy_allows

* Resolve some new clippy nightly lints

* Increase QUIC packets completion timeout

Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-17 15:48:33 -07:00
behzad nouri fea66c8b63
derives Error trait for ClusterInfoError and core::result::Error (#27208) 2022-08-17 22:01:51 +00:00
Jeff Washington (jwash) 225cddcffb
serialize incremental_snapshot_hash (#26839)
* serialize incremental_snapshot_hash

* pr feedback
2022-08-17 15:14:31 -05:00
behzad nouri 3b87aa9227
reverts wide fanout in broadcast when the root node is down (#26359)
A change included in
https://github.com/solana-labs/solana/pull/20480
was that when the root node in turbine broadcast tree is down, the
leader will broadcast the shred to all nodes in the first layer.
The intention was to mitigate the impact of dead nodes on shreds
propagation, because if the root node is down, then the entire cluster
will miss out the shred.
On the other hand, if x% of stake is down, this will cause 200*x% + 1
packets/shreds ratio at the broadcast stage which might contribute to
line-rate saturation and packet drop.
To avoid this bandwidth saturation issue, this commit reverts that logic
and always broadcasts shreds from the leader only to the root node.
As before we rely on erasure codes to recover shreds lost due to staked
nodes being offline.
2022-08-16 19:40:06 +00:00
Justin Starry bdce208fe5
clean feature: `request_units_deprecated` (#27102)
clean feature: request_units_deprecated
2022-08-13 13:12:35 +01:00
Jeff Biseda e50013acdf
Handle JsonRpcService startup failure (#27075) 2022-08-11 23:25:20 -07:00