dd81af9fff
increase fd report interval to 3 minutes
2022-12-06 13:31:50 -06:00
0ce507c20d
refactor SystemMonitorReportConfig
2022-12-06 13:31:50 -06:00
914f7bd85d
fix
2022-12-06 13:31:50 -06:00
1635b99486
add mmmap file count
2022-12-06 13:31:50 -06:00
e1ba5a2a63
add monitoring for open file descriptors stat
2022-12-06 13:31:50 -06:00
7ed22f7b18
Remove gate from accepting packets for forwarding ( #29049 )
2022-12-06 12:13:01 -06:00
b1340d77a2
sdk: Make Packet::meta private, use accessor functions ( #29092 )
...
sdk: Make packet meta private
2022-12-06 12:54:49 +01:00
fd3b5d08d7
Refactor/banking_stage_make_decision_consume_bank ( #28946 )
2022-12-02 10:07:01 -06:00
5850af5316
Refactor to remove requested_cu from cost_trarcker ( #29015 )
...
* refactor cost tracker by removing requested_cu from it, call sites to use cost_model forr consistency
* review fix
2022-12-02 00:25:09 +00:00
3c42c87098
Remove obsoleted return value from Blockstore insert shred method ( #28992 )
2022-12-01 11:17:46 -06:00
b6dce6cf3b
Move BlockstoreInsertionMetrics field update to blockstore.rs ( #28991 )
...
The num_repair field is only blockstore insertion metric being updated
outside of Blockstore::insert() call chain; move the update to insert()
with the rest of the fields in BlockstoreInsertionMetrics struct.
2022-11-30 11:46:35 -06:00
edacd3c411
Add dump_node to update stake for heaviest subtrees ( #28827 )
...
* Add dump_node to update stake for heaviest subtrees
Additionally refactor subtrees to store children as a hashset
* Add a more complicated forks test
* chose -> choose
* remove is_dumped flag and reuse latest_invalid_ancestor instead
2022-11-30 09:26:13 -08:00
4d338ed882
Bugfix/mi_remove_never_entries ( #28978 )
2022-11-29 16:00:21 -06:00
0d0a491f27
More documentation + small refactor for RepairService ( #28933 )
2022-11-28 19:46:06 -08:00
9f370475d4
remove obsoleted comment ( #28960 )
2022-11-28 13:39:40 -06:00
7d99cddb9f
dedups turbine retransmit peers by tvu socket addresses ( #28944 )
...
No need to send duplicate shreds if several nodes have the same tvu
socket address because they are behind a relayer or whatever.
2022-11-28 19:23:02 +00:00
7e87998091
reduce memory usage report freq to 1 per 5s ( #28327 )
2022-11-28 19:08:06 +00:00
bdd162492c
Feature/multi-iterator-scanner-read-locks ( #28862 )
2022-11-28 11:23:04 -06:00
9327658007
Promotes accounts hash to a strong type ( #28930 )
2022-11-28 10:09:47 -05:00
638b26ea65
Renames EAH test fn ( #28939 )
2022-11-23 05:18:50 +00:00
38f7122605
separate make_decision in BankingStage ( #28884 )
2022-11-22 19:01:09 -06:00
c8b0c3ede9
Update cost model to use requested_cu instead of estimated cu #27608 ( #28281 )
...
* Update cost model to use requested_cu instead of estimated cu #27608
* remove CostUpdate and CostModel from replay/tvu
* revive cost update service to send cost tracker stats
* CostModel is now static
* remove unused package
Co-authored-by: Tao Zhu <tao@solana.com>
2022-11-22 11:55:56 -06:00
637e8a937b
clean up: remove my_pubkey arg from consume_buffered_packets ( #28888 )
2022-11-22 11:40:04 -06:00
20d8b5e98b
default some tests to write cache = true ( #28917 )
2022-11-21 15:53:39 -08:00
dd723210ca
remove unnecessary clippy attributes ( #28891 )
2022-11-21 12:54:54 -06:00
d43b001189
rolls out merkle shreds to ~20% of testnet ( #28905 )
2022-11-21 16:20:02 +00:00
c6927151ef
Sort offline/wrong-shred nodes by stake weight while waiting for supermajority ( #28872 )
2022-11-18 15:26:21 -08:00
f22104d46b
use write cache by default in some tests ( #28876 )
2022-11-18 14:35:52 -08:00
a636038fff
Clean up: banking_stage_prepare_sanitized_batch ( #28841 )
...
Use measure! for bank.prepare_sanitized_batch_with_results
2022-11-18 14:04:44 -06:00
c32377b5af
Split out quic- and udp-client definitions ( #28762 )
...
* Move ConnectionCache back to solana-client, and duplicate ThinClient, TpuClient there
* Dedupe thin_client modules
* Dedupe tpu_client modules
* Move TpuClient to TpuConnectionCache
* Move ThinClient to TpuConnectionCache
* Move TpuConnection and quic/udp trait implementations back to solana-client
* Remove enum_dispatch from solana-tpu-client
* Move udp-client to its own crate
* Move quic-client to its own crate
2022-11-18 12:21:45 -07:00
88e6ea37d9
refactor: move more BankingStage cost_model stuff into qos_service ( #28840 )
2022-11-17 14:03:17 -06:00
ee2f760d3d
MultiIteratorScanner - improve banking stage performance with high contention
2022-11-17 10:54:12 -06:00
2bafb0cb12
Requires EAH state cannot be Invalid ( #28817 )
2022-11-17 11:01:01 -05:00
ae0bb44401
Fixes test_snapshots_with_background_services ( #28848 )
2022-11-17 00:49:28 -05:00
17ee3349f8
limit repairs to top staked requests in batch ( #28673 )
2022-11-16 16:30:41 -08:00
ddf4ff2d26
Repair service documentation ( #28592 )
...
* repair doc update
* tree_root rename
* remove extra todo
2022-11-16 02:38:07 +00:00
e10d958352
signed repair request test fixes/cleanup ( #28691 )
2022-11-15 16:46:17 -08:00
d798e751a0
Disables EAH with short epochs ( #28803 )
2022-11-15 13:26:19 -05:00
d4cf18421d
Use 400 slots-per-epoch in EAH tests ( #28801 )
2022-11-14 17:49:20 -05:00
0bfea02056
Snapshots wait for EAH calculations to complete ( #28777 )
2022-11-14 11:34:44 -06:00
e5ae0b3371
check is_forwarded packet earlier ( #28159 )
...
* check and filter is_forwarded packet earlier
* review fix: renaming; and rebase
2022-11-11 23:32:03 +00:00
4d6653598b
Upgrades to Rust 1.65.0 ( #28741 )
2022-11-09 17:15:03 -05:00
d1ba42180d
clippy for rust 1.65.0 ( #28765 )
2022-11-09 19:39:38 +00:00
9e1cdc7e60
Enables not taking a bank snapshot ( #28756 )
2022-11-09 12:43:33 -05:00
d4c2900590
Removes `snapshot_bank()` wrapper fn ( #28753 )
2022-11-07 15:09:31 +00:00
0b9426e734
Simplifies AHV's `test_max_hashes()` ( #28754 )
2022-11-07 02:32:33 +00:00
064cfc70d2
Removes cluster_type from AccountsPackage ( #28725 )
2022-11-02 18:21:13 -04:00
d0f639745a
Uses AccountsPackage::default_for_tests() in AHV tests ( #28723 )
2022-11-02 14:13:35 -04:00
f156bc12ca
Enforce stream receive timeout ( #28513 )
...
In the quic server handle_connection, when we timed out in receiving the chunks, we loop forever to wait for the chunk. If the client never provide another chunk, the server can hopelessly wait for that chunk and wasting server resources. Instead WAIT_FOR_CHUNK_TIMEOUT_MS is introduced to bound this to 10 seconds at maximum. The stream will be dropped if it times out.
2022-11-02 10:09:32 -07:00
59bf1809fe
Uses SnapshotHash type in snapshot archive fields ( #28681 )
2022-10-31 14:28:35 -04:00
34865d032c
chore: update Solana docs and code comments that specify "BPF" to "SBF"
2022-10-31 14:14:25 -04:00
37507a2de6
Removes EAH parameter from serde_snapshot::reserialize_bank() ( #28669 )
2022-10-31 09:43:17 -04:00
340ad68223
Banking stage refactor commit transactions ( #28660 )
...
* Refactor commit transactions step
* Cleanup token pre-balances
* Collect prebalances together
* Collect pre/post balances in separate function
* Fix clippy
2022-10-29 21:36:57 +02:00
6b93d05c37
Add LedgerCleanupService::find_slots_to_clean() test ( #28656 )
...
Add a test to better exercise find_slots_to_clean(), as well as a minor
bug fix to this method that was found as a result of writing test.
2022-10-29 00:55:21 +02:00
22ce49ae7f
Maintain original queue capacity for unprocessed packet buffer ( #28661 )
2022-10-28 16:37:21 -05:00
0a148b2bf7
remove unused handle_retryable_packets_elapsed ( #28355 )
2022-10-28 16:36:41 -05:00
5a3d252899
Renames fn to Bank::update_accounts_hash_for_tests() ( #28620 )
2022-10-28 14:33:05 -04:00
2272fd807e
Remove Blockstore manual compaction code ( #28409 )
...
The manual Blockstore compaction that was being initiated from
LedgerCleanupService has been disabled for quite some time in favor of
several optimizations.
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2022-10-28 10:39:00 +02:00
ae557a9eb5
Exit when stuck in an unrecoverable repair/purge loop ( #28596 )
...
* Exit when stuck in an unrecoverable repair/purge loop
* add tests
2022-10-27 20:06:06 -07:00
340d3b5468
rename and change capacity on unprocessed transaction storage - max_receive_size ( #28586 )
2022-10-26 10:03:47 -05:00
f158bab0ef
Tracks how long background requests wait before processing ( #28581 )
2022-10-25 12:10:53 -04:00
bc02789c43
Renames fn to calculate_accounts_hash_from_storages() ( #28566 )
2022-10-24 21:07:00 -04:00
2354a0a343
Renames fn to calculate_accounts_hash_from_index() ( #28568 )
2022-10-24 19:20:08 -04:00
9eafad467c
Add convenience methods to VoteInstruction to distinguish vote types ( #28526 )
...
* Add convenience methods to VoteInstruction to distinguish vote types
* use matches! macro instead
2022-10-21 14:17:40 -06:00
f207af765e
Split out voting and banking threads in banking stage ( #27931 )
...
* Split out voting and banking threads in banking stage
Additionally this allows us to aggressively prune the buffer for voting threads
as with the new vote state only the latest vote from each validator is
necessary.
* Update local cluster test to use new Vote ix
* Encapsulate transaction storage filtering better
* Address pr comments
* Commit cargo lock change
* clippy
* Remove unsafe impls
* pr comments
* compute_sanitized_transaction -> build_sanitized_transaction
* &Arc -> Arc
* Move test
* Refactor metrics enums
* clippy
2022-10-20 21:10:48 +00:00
0df4be06a0
enable repair ping/pong cache ( #28408 )
2022-10-19 14:55:55 -07:00
12f3e8c9cc
Ignores errors when joining background threads in snapshot tests ( #28480 )
2022-10-19 16:54:59 -04:00
274d9ea607
Check for valid address in broadcast ( #28432 )
...
Check for valid address
2022-10-19 14:49:22 -05:00
d81d2bba59
comments out print in test ( #28475 )
2022-10-19 10:25:11 -05:00
1cc9cf927c
Supports warping with Epoch Accounts Hash ( #28459 )
2022-10-19 10:37:14 -04:00
e283461d99
enforces hash domain for ping-pong protocol ( #28433 )
...
https://github.com/solana-labs/solana/pull/27193
added hash domain to ping-pong protocol.
For backward compatibility responses both with and without domain were
generated and accepted.
Now that all clusters are upgraded, this commit enforces the hash domain
by removing the response without the domain.
2022-10-18 18:17:12 +00:00
28a89a1d99
remove expected rent collection and rehashing completely ( #28422 )
2022-10-17 07:24:42 -07:00
39fa297bf6
Report total_transactions in replay-slot-stats ( #28382 )
...
We have transactions counted in replay-slot-end-to-end-stats, but that
metric is broken down to report things per thread.
So, report total_transactions for the entire slot (all threads) in
replay-slot-stats.
2022-10-15 14:07:03 +01:00
31c2b29941
Sends both an EAH and a snapshot request from `set_root()` ( #28363 )
2022-10-14 11:00:04 -04:00
dd7fee8f32
Re-enqueues unhandled ABS requests ( #28362 )
2022-10-13 16:25:39 -04:00
9cbd00fdbc
Converts PendingAccountsPackage to a channel ( #28352 )
2022-10-13 12:47:36 -04:00
e2fc9d51de
Increase cpu metric reporting interval from 1s to 10s
2022-10-11 10:44:59 -05:00
15050b14b9
use signed repair request variants ( #28283 )
2022-10-10 14:09:45 -07:00
5a08eed82d
Cleans up debugging code in EAH tests ( #28324 )
2022-10-10 16:07:55 +00:00
27cd2c324e
Adds tests for EAH and snapshot interactions ( #28304 )
2022-10-10 10:16:13 -04:00
50985f79a1
Correctly mark packets as forwarded ( #28161 )
...
Only mark packets accepted for forwarding as `forwarded`
2022-10-07 11:50:57 -05:00
0324573667
report additional transaction errors to metrics ( #28285 )
2022-10-07 10:36:22 -05:00
981c9d07a4
Rearranges eah TestEnvironment fields to ensure drop order ( #28270 )
2022-10-06 16:17:32 -04:00
2d936784dd
Ignore errors when joining background threads for EAH tests ( #28263 )
2022-10-06 18:43:56 +00:00
a8c6a9e5fc
Bank::freeze() waits for EAH calculation to complete ( #28170 )
2022-10-05 17:44:35 -04:00
c899ededfc
Minor refactoring and cleaning of cpuid code
2022-10-05 11:43:27 -05:00
3b2ab313de
Use num-enum crate to make everything typesafe
2022-10-05 11:43:27 -05:00
1e1455688d
Convert magic numbers to named constants
2022-10-05 11:43:27 -05:00
fac772ff90
Update naming, style after PR review comments
2022-10-05 11:43:27 -05:00
13b095b4ab
Fix a fmt problem, I think. Shows up in the git check, but not when I run here
2022-10-05 11:43:27 -05:00
c8584b0cdd
Cargo fmt applied
2022-10-05 11:43:27 -05:00
d841286c21
Add cpuid calls and metric reporting; change cpu info sampling interval from 1s to 10s
2022-10-05 11:43:27 -05:00
e3e888c0e0
stats for staked/unstaked repair requests ( #28215 )
2022-10-04 17:37:24 -07:00
9e7a0e7420
rolls out merkle shreds to ~5% of testnet ( #28199 )
2022-10-04 19:36:16 +00:00
14a415ccf3
Consensus Logging ( #28176 )
2022-10-03 20:45:55 -05:00
c4aab3f178
typo
2022-10-03 09:41:15 -05:00
c2bb2b8e60
Allow validators to reset to the slot which matches their last voted slot ( #28172 )
...
* Add failing test
* Allow resetting to duplicate was confirmed
* feedback
* feedback
* bump
* simplify change
* Revert "simplify change"
This reverts commit 72e5de3e5bdac595f71dc7fc01650ca3bc7da98e.
* update comment
* Update core/src/replay_stage.rs
2022-10-03 16:49:47 +08:00
6b17bee5a8
Remove the const default for RocksFifo ( #27965 )
...
#### Summary of Changes
Removes the constant default for ShredStorageType::RocksFifo
as the shred storage size is either user-specified or derived
from --limit-ledger-size in #27459 .
2022-10-01 15:10:54 -07:00
8877ac2aa9
Fix call to calculate_accounts_hash() ( #28169 )
2022-09-30 15:29:18 -04:00
2f8f6c6a31
Send Epoch Accounts Hash requests from set_root() ( #27764 )
2022-09-30 14:59:41 -04:00
cfc124c825
acct idx can no longer use write cache ( #28150 )
2022-09-30 10:55:27 -07:00
82558226f7
ImmutableDeserializedPacket rc to arc ( #28145 )
2022-09-30 12:07:48 -05:00
82e65593ee
Batch filtering invalid transactions before forwarding ( #26798 )
...
- Batch filtering invalid transactions (fail to sanitize, too old or already processed) before forwarding
- Combine packet filtering and forwarding to share sanitized transactions
- `iter_desc` is no longer needed, remove it;
- Add a method to share the logic of removing packets from buffer after they were removed from MinMaxHeap
- Add test coverage for forward_packet_batches_by_accounts
- rebase, resolve conflicts
2022-09-29 16:33:40 -05:00
84acef007c
Add bench test for voting threads ( #28031 )
2022-09-27 12:12:22 -07:00
8b0f9b4917
make ping cache rate limit delay configurable ( #27955 )
2022-09-26 14:16:56 -07:00
f49beb0cbc
caches reed-solomon encoder/decoder instance ( #27510 )
...
ReedSolomon::new(...) initializes a matrix and a data-decode-matrix cache:
https://github.com/rust-rse/reed-solomon-erasure/blob/273ebbced/src/core.rs#L460-L466
In order to cache this computation, this commit caches the reed-solomon
encoder/decoder instance for each (data_shards, parity_shards) pair.
2022-09-25 18:09:47 +00:00
9816c94d7e
metrics to distinguish why repair packets are dropped ( #27960 )
2022-09-24 23:20:05 -07:00
8b43215ddd
count unsigned repair requests ( #27953 )
2022-09-24 12:56:02 -07:00
e51cf46d6b
Remove priority from vote transactions ( #28030 )
...
vote transactions have same priority fee
2022-09-24 00:31:50 +00:00
9ee53e594d
patches clippy errors from new rust nightly release ( #28028 )
2022-09-23 20:57:27 +00:00
d9b31fd6b0
ahv: Add debug logging for EAH ( #27998 )
2022-09-23 14:04:48 -04:00
206cc9407b
allow unsigned repair requests ( #27910 )
2022-09-23 10:11:08 -07:00
97c9af4c6b
plumbs through flag to generate merkle variant of shreds
2022-09-23 16:45:18 +00:00
e4affb9fea
Add Blockstore::highest_slot() method ( #27981 )
2022-09-23 04:53:43 -05:00
9a57c64f21
patches clippy errors from new rust nightly release ( #27996 )
2022-09-22 22:23:03 +00:00
ff71df4695
Remove unnecessary call to `set_startup_verification_complete()` ( #27986 )
2022-09-22 16:54:17 -04:00
1ee595ca9c
remove AccountsDb::initial_blockstore_processing_complete ( #27974 )
2022-09-22 13:52:04 -04:00
c4fa849844
chore: bump itertools from 0.10.3 to 0.10.5 ( #27962 )
...
* chore: bump itertools from 0.10.3 to 0.10.5
Bumps [itertools](https://github.com/rust-itertools/itertools ) from 0.10.3 to 0.10.5.
- [Release notes](https://github.com/rust-itertools/itertools/releases )
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-itertools/itertools/commits )
---
updated-dependencies:
- dependency-name: itertools
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-09-21 15:28:49 -06:00
11b7c45bff
chore: bump systemstat from 0.1.11 to 0.2.0 ( #27967 )
...
Bumps [systemstat](https://github.com/unrelentingtech/systemstat ) from 0.1.11 to 0.2.0.
- [Release notes](https://github.com/unrelentingtech/systemstat/releases )
- [Commits](https://github.com/unrelentingtech/systemstat/compare/v0.1.11...v0.2.0 )
---
updated-dependencies:
- dependency-name: systemstat
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-21 15:28:31 -06:00
f2d6a7ecea
bank.initial_blockstore_processing_complete to avoid concurrent hash calculations ( #27776 )
...
* bank.initial_blockstore_processing_complete to avoid concurrent hash calculations
* Update runtime/src/bank.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/bank.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Rename TestValidator::set_startup_verification_complete()
* Initialize with `AtomicBool::new(false)` instead of `default()`
* snapshot tests: move where `initial_blockstore_processing_completed()` is called
* fixup bank_forks.rs calling `is_initial_blockstore_processing_complete()`
* only call initial_blockstore_processing_completed() in blockstore_processor
Co-authored-by: Brooks Prumo <brooks@prumo.org>
Co-authored-by: Brooks Prumo <brooks@solana.com>
2022-09-19 13:00:21 -07:00
abfb996135
tracks number of staked/stale/dead nodes in turbine cluster-nodes ( #27915 )
2022-09-19 18:16:04 +00:00
9119dc13ec
Add structure to house unprocessed transactions in banking_stage ( #27777 )
...
Separate storage for voting and transaction threads:
- Voting threads utilize a shared reference in order to dedup extraneous
votes
- Transactions have thread local storage like before
2022-09-14 10:40:44 -07:00
c74df830b1
Add structure to collect and coalesce vote packets ( #27558 )
...
* Add structure to collect and coalesce vote packets
Will be used in banking stage to throw out extraneous vote packets
before processing
* pr comments
* Update inner lock to arc to improve performance
2022-09-14 00:44:26 -07:00
c0e4379f43
Whickey/version v1.15 ( #27739 )
...
* Bump version to v1.13.0
* Bump version to v1.14.0
* Bump version to v1.15.0
2022-09-13 09:06:15 -05:00
079bf561b0
Clean_up/upb_push_comment ( #27707 )
2022-09-12 18:59:41 -05:00
765c628546
use exit signal for acct idx bg threads ( #27483 )
2022-09-12 11:51:12 -07:00
4f22ee8f9b
uses varint encoding for vote-state lockout offsets
...
The commit removes CompactVoteStateUpdate and instead reduces serialized
size of VoteStateUpdate using varint encoding for vote-state lockout
offsets.
2022-09-12 16:31:20 +00:00
90b8a3a44d
Remove KeypairInsecureClone trait and add insecure_clone() instead ( #27396 )
...
See discussion in #26248
2022-09-12 14:59:41 +00:00
83d4d128c2
Add --process-ledger-before-service flag to solana-validator
2022-09-11 07:58:42 -07:00
abd01553d5
tests: Keypair::new().pubkey() -> pubkey::new_rand ( #27705 )
2022-09-10 13:56:45 -07:00
1f00b468e5
add enable_rehashing to AccountsPackage ( #27644 )
2022-09-08 09:25:25 -07:00
a9c5adbf88
UnprocessedPacketBatches pop_max fn are only used in tests ( #27645 )
2022-09-08 11:01:14 -05:00
cc58968b76
add new leader slot metric to track account contention throttling ( #27654 )
2022-09-08 09:22:58 -05:00
f338aa62ba
chore: bump serde from 1.0.143 to 1.0.144 ( #27511 )
...
* chore: bump serde from 1.0.143 to 1.0.144
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.143 to 1.0.144.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.143...v1.0.144 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-09-07 16:54:44 -06:00
4308c300b4
In ledger-tool delete the account files in the async way ( #27622 )
...
* In ledger-tool delete the account files in the async way
* format changes by ./cargo nightly fmt --all
2022-09-07 14:35:06 -07:00
6a322de845
Make Accounts Background Services aware of Epoch Accounts Hash ( #27626 )
2022-09-07 20:41:40 +00:00
7f223dc582
Added option to turn on UDP for TPU transaction and make UDP based TPU off by default ( #27462 )
...
--tpu-enable-udp is introduced. And when this is on, the transaction receive and transaction forward is enabled using udp.
Except for a few tests which was hard-coded sending transactions using udp, most tests are being done with udp based tpu disabled.
2022-09-07 13:19:14 -07:00
c04747dd66
cluster_slot_state_verifier: clippy nightly fixes ( #27521 )
...
clippy nightly fixes
2022-09-07 15:04:56 -05:00
1465ec947d
replay_stage: clippy nightly fixes ( #27520 )
...
clippy nightly fixes
2022-09-07 15:04:46 -05:00
452866dbcf
shredder: clippy nightly fixes ( #27522 )
...
clippy nightly fixes
2022-09-07 15:04:32 -05:00
d6a1e7498f
Add tests for deserialize_and_collect_packets ( #27623 )
2022-09-07 12:52:18 -05:00
22007a3c96
allow accounts hash calc to specify enable_rehashing ( #27615 )
2022-09-07 10:16:52 -07:00
a31d4a597d
serialize epoch_accounts_hash ( #27516 )
2022-09-07 10:07:00 -07:00
93a4f80a2c
Handling snapshot requests is now required ( #27537 )
2022-09-07 10:08:42 -04:00
269eb519dd
track time to coalesce entries in recv_slot_entries ( #27525 )
2022-09-06 16:07:17 -07:00
a67d56f462
refactor: add function for deserializing and collecting packets - separate from channel receive ( #27548 )
2022-09-06 15:54:31 -05:00
6684c62280
Add SnapshotUsage to SnapshotConfig ( #27508 )
2022-09-02 08:56:23 -04:00
242c9cb442
RPC Notifier Signal when Setup Complete ( #27481 )
...
* RPC notifier signal when ready
2022-09-01 16:39:55 -07:00
9b8bed86f9
Add getRecentPrioritizationFees RPC endpoint ( #27278 )
...
* Plumb priority_fee_cache into rpc
* Add PrioritizationFeeCache api
* Add getRecentPrioritizationFees rpc endpoint
* Use MAX_TX_ACCOUNT_LOCKS to limit input keys
* Remove unused cache apis
* Map fee data by slot, and make rpc account inputs optional
* Add priority_fee_cache to rpc test framework, and add test
* Add endpoint to jsonrpc docs
* Update docs/src/developing/clients/jsonrpc-api.md
* Update docs/src/developing/clients/jsonrpc-api.md
2022-09-01 23:12:12 +00:00
3bdc5b3f2b
separate packet_deserializer inside banking_stage ( #27120 )
...
* separate packet_deserializer inside banking_stage
* Make ReceivePacketResults into a struct with named fields
2022-09-01 10:00:48 -05:00
66717ff87d
chore: bump chrono from 0.4.21 to 0.4.22 ( #27509 )
...
* chore: bump chrono from 0.4.21 to 0.4.22
Bumps [chrono](https://github.com/chronotope/chrono ) from 0.4.21 to 0.4.22.
- [Release notes](https://github.com/chronotope/chrono/releases )
- [Changelog](https://github.com/chronotope/chrono/blob/v0.4.22/CHANGELOG.md )
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.21...v0.4.22 )
---
updated-dependencies:
- dependency-name: chrono
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-08-31 22:39:12 +00:00
8bb039d08d
collect min prioritization fees when replaying sanitized transactions ( #26709 )
...
* Collect blocks' minimum prioritization fees when replaying sanitized transactions
* Limits block min-fee metrics reporting to top 10 writable accounts
* Add service thread to asynchronously update and finalize prioritization fee cache
* Add bench test for prioritization_fee_cache
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2022-08-31 08:00:55 -05:00
7bd08ad3ae
Featurize spinner in rpc-client and tpu-client ( #27381 )
...
* Add spinner features to rpc-client and tpu-client, and disable where unneeded
* Add doc comment
2022-08-30 09:56:56 -06:00
5b64107626
make pruned_bank channel unbonded.
...
In kin-sim, we found that bounded channel causes halt for account
background services. As the number of accounts grows, the time for
pruning and cleaning increases, which would leads to longer intervals
between the pruning of deaded bank slots. With 1.7B accounts, we will
exceed the 10K bounded channel threshold that causes halt of account
back ground services. Without pruning, the node will eventually run out
of memory.
2022-08-29 19:06:30 -05:00
3c7cd62030
Move pruned_banks_receiver into PrunedBanksRequestHandler ( #27445 )
2022-08-29 13:30:06 -04:00
46a48760db
Switch concurrent replay from feature to param ( #27401 )
...
* Switch concurrent replay from feature to param
2022-08-26 12:36:02 -07:00
5eefc256d6
Fix startup panic if removing accounts directory fails ( #27386 )
...
* Remove contents of accounts directory if deleting the directory fails.
2022-08-25 20:35:12 -05:00
b1cff5d740
make fatal log message sound fatal
2022-08-25 21:49:12 +00:00
d1522fc790
coalesce entries in recv_slot_entries to target byte count ( #27321 )
2022-08-25 13:51:55 -07:00
2da93bd45a
add text to assert ( #27377 )
2022-08-24 14:11:53 -05:00
b8b3d723da
Use new client crates ( #27360 )
...
* Update ancillary cli crates
* Update cli
* Update command-line tools
* Update rpc, etc
* Update client-test
* Update core, validator
* Update local-cluster
2022-08-24 10:47:02 -06:00
efa6201eda
Check overflow on vote tx compaction boundary ( #27185 )
...
* Check overflow on vote tx compaction boundary
Check for overflow during the conversion between VoteStateUpdate and
CompactVoteStateUpdate.
* Try removing clippy supress
2022-08-23 22:29:03 -07:00
827d8e4bc0
Fallback to synchronous rm_dir call if path moving fails ( #27306 )
...
Remove some log lines, as suggested in PR #26910
2022-08-22 22:47:39 -07:00
e4a7d01e10
Rust v1.63 ( #27303 )
...
* Upgrade to Rust v1.63.0
* Add nightly_clippy_allows
* Resolve some new clippy nightly lints
* Increase QUIC packets completion timeout
* Update quinn-udp crate
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-22 18:01:03 -07:00
7bdeea10ad
Assign custom names to the Rayon global thread pool
2022-08-22 17:56:55 +00:00
7c01c1ecc6
Update delete_path thread name
2022-08-22 09:01:26 -07:00
fc1a4dd11a
run hash calc with index on failure ( #27279 )
2022-08-22 10:58:04 -05:00
3f4731b37f
Standardize thread names
...
Tenets:
1. Limit thread names to 15 characters
2. Prefix all Solana-controlled threads with "sol"
3. Use Camel case. It's more character dense than Snake or Kebab case
2022-08-20 07:49:39 -07:00
c54824e4f5
Account files remove ( #26910 )
...
* Create a new function cleanup_accounts_paths, a trivial change
* Remove account files asynchronously
* Update and simplify the implementation after the validator test runs.
* Fixes after testing on the dev device
* Discard tokio. Use thread instead
* Fix comments format
* Fix config type to pass the github test
* Fix failed tests. Handle the case of non-existing path
* Final cleanup, addressing the review comments
Avoided OsString.
Made the function more generic with "impl AsRef<Path>"
Co-authored-by: Jeff Washington <jeff.washington@solana.com>
2022-08-19 23:56:52 -07:00
eb06bb61e8
banking stage: actually aggregate tracer packet stats ( #27118 )
...
* aggregated_tracer_packet_stats_option was alwasys None
* Actually accumulate tracer packet stats
2022-08-19 15:16:56 -05:00
dba2fd5a16
Enable QUIC client by default. Add arg to disable QUIC client. (Forward port #26927 ) ( #27194 )
...
Enable QUIC client by default. Add arg to disable QUIC client.
* Enable QUIC client by default. Add arg to disable QUIC client.
* Deprecate --disable-quic-servers arg
* Add #[ignore] annotation to failing tests
2022-08-19 09:15:15 -05:00
7573000d87
Revert "Rust v1.63.0 ( #27148 )" ( #27245 )
...
This reverts commit a2e7bdf50a
2022-08-19 09:19:44 +01:00
6928b2a5af
adds hash domain to ping-pong protocol ( #27193 )
...
In order to maintain backward compatibility, for now the responding node
will hash the token both with and without domain so that the other node
will accept the response regardless of its upgrade status.
Once the cluster has upgraded to the new code, we will remove the legacy
domain = false case.
2022-08-18 22:39:31 +00:00
a2e7bdf50a
Rust v1.63.0 ( #27148 )
...
* Upgrade to Rust v1.63.0
* Add nightly_clippy_allows
* Resolve some new clippy nightly lints
* Increase QUIC packets completion timeout
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-17 15:48:33 -07:00
fea66c8b63
derives Error trait for ClusterInfoError and core::result::Error ( #27208 )
2022-08-17 22:01:51 +00:00
225cddcffb
serialize incremental_snapshot_hash ( #26839 )
...
* serialize incremental_snapshot_hash
* pr feedback
2022-08-17 15:14:31 -05:00
3b87aa9227
reverts wide fanout in broadcast when the root node is down ( #26359 )
...
A change included in
https://github.com/solana-labs/solana/pull/20480
was that when the root node in turbine broadcast tree is down, the
leader will broadcast the shred to all nodes in the first layer.
The intention was to mitigate the impact of dead nodes on shreds
propagation, because if the root node is down, then the entire cluster
will miss out the shred.
On the other hand, if x% of stake is down, this will cause 200*x% + 1
packets/shreds ratio at the broadcast stage which might contribute to
line-rate saturation and packet drop.
To avoid this bandwidth saturation issue, this commit reverts that logic
and always broadcasts shreds from the leader only to the root node.
As before we rely on erasure codes to recover shreds lost due to staked
nodes being offline.
2022-08-16 19:40:06 +00:00
a0d1f4ef88
chore: bump serial_test from 0.8.0 to 0.9.0 ( #27097 )
...
Bumps [serial_test](https://github.com/palfrey/serial_test ) from 0.8.0 to 0.9.0.
- [Release notes](https://github.com/palfrey/serial_test/releases )
- [Commits](https://github.com/palfrey/serial_test/compare/v0.8.0...v0.9.0 )
---
updated-dependencies:
- dependency-name: serial_test
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-15 15:53:12 -06:00
bdce208fe5
clean feature: `request_units_deprecated` ( #27102 )
...
clean feature: request_units_deprecated
2022-08-13 13:12:35 +01:00
e50013acdf
Handle JsonRpcService startup failure ( #27075 )
2022-08-11 23:25:20 -07:00
fc6cee9c06
allow staked nodes weight override ( #26870 )
...
* Allowed staked nodes weight override (#26407 )
* Allowed staked nodes weight override, passing only HashMap over to core module
Co-authored-by: Ondra Chaloupka <chalda@chainkeepers.io>
2022-08-11 14:34:04 -07:00
f641d3bad6
chore: bump chrono from 0.4.19 to 0.4.21 ( #27076 )
...
* chore: bump chrono from 0.4.19 to 0.4.21
Bumps [chrono](https://github.com/chronotope/chrono ) from 0.4.19 to 0.4.21.
- [Release notes](https://github.com/chronotope/chrono/releases )
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md )
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.19...v0.4.21 )
---
updated-dependencies:
- dependency-name: chrono
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2022-08-11 09:28:50 -06:00
ac91cdab74
removes buffering when generating coding shreds in broadcast ( #25807 )
...
Given the 32:32 erasure recovery schema, current implementation requires
exactly 32 data shreds to generate coding shreds for the batch (except
for the final erasure batch in each slot).
As a result, when serializing ledger entries to data shreds, if the
number of data shreds is not a multiple of 32, the coding shreds for the
last batch cannot be generated until there are more data shreds to
complete the batch to 32 data shreds. This adds latency in generating
and broadcasting coding shreds.
In addition, with Merkle variants for shreds, data shreds cannot be
signed and broadcasted until coding shreds are also generated. As a
result *both* code and data shreds will be delayed before broadcast if
we still require exactly 32 data shreds for each batch.
This commit instead always generates and broadcast coding shreds as soon
as there any number of data shreds available. When serializing entries
to shreds:
* if the number of resulting data shreds is less than 32, then more
coding shreds will be generated so that the resulting erasure batch
has the same recovery probabilities as a 32:32 batch.
* if the number of data shreds is more than 32, then the data shreds are
split uniformly into erasure batches with _at least_ 32 data shreds in
each batch. Each erasure batch will have the same number of code and
data shreds.
For example:
* If there are 19 data shreds, 27 coding shreds are generated. The
resulting 19(data):27(code) erasure batch has the same recovery
probabilities as a 32:32 batch.
* If there are 107 data shreds, they are split into 3 batches of 36:36,
36:36 and 35:35 data:code shreds each.
A consequence of this change is that code and data shreds indices will
no longer align as there will be more coding shreds than data shreds
(not only in the last batch in each slot but also in the intermediate
ones);
2022-08-11 12:44:27 +00:00
4e79d78629
`solana-validator monitor` how displays slot and gossip stake % while waiting for supermajority
2022-08-10 11:13:25 -07:00
e3a8d2ecdd
chore: bump serde_json from 1.0.81 to 1.0.83 ( #27036 )
...
* chore: bump serde_json from 1.0.81 to 1.0.83
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.81 to 1.0.83.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.81...v1.0.83 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-08-10 09:45:42 -06:00
c03f3b1436
Separate file for ImmutableDeserializedPacket type ( #26951 )
2022-08-09 22:39:01 -07:00
ae5b680c6f
chore: bump serde from 1.0.138 to 1.0.143 ( #27015 )
...
* chore: bump serde from 1.0.138 to 1.0.143
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.138 to 1.0.143.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.138...v1.0.143 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-08-10 05:08:43 +00:00
370de8129e
ancestor hashes socket ping/pong support ( #26866 )
2022-08-09 21:39:55 -07:00
ccfbc54195
Move vote program state and instructions to solana-program
2022-08-09 20:52:47 -07:00
c2455e7aa4
Fix typo in test function ( #27031 )
2022-08-09 12:39:22 -07:00
a69470fd45
Set receive_window per quic connection ( #26936 )
...
This change sets the receive_window for non-staked node to 1 * PACKET_DATA_SIZE, and maps the staked nodes's connection's receive_window between 1.2 * PACKET_DATA_SIZE to 10 * PACKET_DATA_SIZE based on the stakes.
The changes is based on Quinn library change to support per connection receive_window tweak at the server side. quinn-rs/quinn#1393
2022-08-09 10:02:47 -07:00
e2a2d271f2
adds number of coding shreds to broadcast metrics ( #27006 )
2022-08-09 13:59:40 +00:00
b6d38aad69
tracer-packet-stats reporting should not reset id ( #27012 )
2022-08-09 06:38:08 -07:00
99ef2184cc
Delete files older than the lowest_cleanup_slot in LedgerCleanupService::cleanup_ledger ( #26651 )
...
#### Problem
LedgerCleanupService requires compactions to propagate & digest range-delete tombstones
to eventually reclaim disk space.
#### Summary of Changes
This PR makes LedgerCleanupService::cleanup_ledger delete any file whose slot-range is
older than the lowest_cleanup_slot. This allows us to reclaim disk space more often with
fewer IOps. Experimental results on mainnet validators show that the PR can effectively
reduce 33% to 40% ledger disk size.
2022-08-09 00:48:06 +08:00
ed8c224374
Bump version to v1.12 ( #26967 )
2022-08-06 13:20:30 -05:00
cf58640937
Keypair: implement clone() ( #26248 )
...
* Keypair: implement clone()
This was not implemented upstream in ed25519-dalek to force everyone to
think twice before creating another copy of a potentially sensitive
private key in memory.
See https://github.com/dalek-cryptography/ed25519-dalek/issues/76
However, there are now 9 instances of
Keypair::from_bytes(&keypair.to_bytes())
in the solana codebase and it would be preferable to have a function.
In particular since this also comes up when writing programs and can
cause users to either start messing with lifetimes or discover the
from_bytes() workaround themselves.
This patch opts to not implement the Clone trait. This avoids automatic
use in order to preserve some of the original "let developers think
twice about this" intention.
* Use Keypair::clone
2022-08-06 11:54:38 -06:00
270315a7f6
transaction-status, storage-proto: add compute_units_consumed ( #26528 )
...
* transaction-status, storage-proto: add compute_units_consumed
* fix bpf test
Co-authored-by: Justin Starry <justin@solana.com>
2022-08-06 17:14:31 +00:00
69598ed4c0
Refactor: Add `RuntimeConfig` field to Bank ( #26946 )
...
* Refactor: Simplify arguments for bank constructor methods
* Refactor: Add RuntimeConfig to Bank fields
* Arc wrap runtime_config
* Arc wrap all runtime config usages
* Remove Copy trait derivation from RuntimeConfig
* Remove some arc wrapping
2022-08-05 20:49:00 +01:00
5bc81a6c35
Io stats v2 ( #26898 )
...
* Use sysfs instead of procfs for disk stats
* Filter map to filter dmcrypt and mdraid volumes
* Unit test cover different kernel formats
2022-08-05 10:38:49 -07:00
2dca239480
Remove runtime dependency from solana-transaction-status ( #26930 )
...
* Move RewardType out of runtime
* Move collect_token_balances to solana-ledger
* Remove solana-runtime dependency
2022-08-05 00:20:27 -06:00
haoran
Tao Zhu
Jon Cinque
apfitzge
steviez
Ashwin Sekar
behzad nouri
Brooks Prumo
Maximilian Schneider
Jeff Washington (jwash)
Michael Vines
Tyera
Jeff Biseda
Lijun Wang
Dmitri Makarov
sakridge
carllin
Jason Davis
haoran
Justin Starry
Yueh-Hsuan Chiang
dependabot[bot]
Will Hickey
Christian Kamm
apfitzge
Xiang Zhu
Brennan Watt
Tyera Eulberg
Trent Nelson
Jeff Washington (jwash)
janlegner
Richard Patel