8c793da7d0
BankingStage Refactor: Move decision making functions to new module ( #29788 )
...
Move decision making functions to new module
2023-01-20 10:10:47 -08:00
5fc83a3d19
BankingStage Refactor: Separate Next Leader Functions ( #29401 )
...
Separate next_leader functions
2023-01-20 10:02:29 -08:00
64c13b74d8
errors out when retransmit loopbacks to the slot leader ( #29789 )
...
When broadcasting shreds, turbine excludes the slot leader from the
random shuffle. Doing so, shreds should never loopback to the leader.
If shreds reaching retransmit stage are from the node's own leader slots
they should not be retransmited to any nodes.
2023-01-20 17:20:51 +00:00
b36791956e
Ingest duplicate proofs sent through Gossip ( #29227 )
...
* First draft of ingesting duplicate proofs in Gossip into blockstore.
* Add more unittests.
* Add more unittests for bad cases.
* Fix lint errors for tests.
* More linter fixes for tests.
* Lint fixes
* Rename get_entries, move location of comment.
* Some renaming changes and comment fixes.
* Fix compile warning, this enum is not used.
* Fix lint errors.
* Slow down cleanup because this could potentially be expensive.
* Forgot to reset cleanup count.
* Add protection against attackers when constructing chunk map when
we ingest Gossip proofs.
* Use duplicate shred index instead of get_entries.
* Rename ClusterInfoDuplicateShredListener and fix a few small problems.
* Use into_shreds to piece together the proof.
* Remove redundant code.
* Address a few small errors.
* Discard slots too advanced in the future.
* - Use oldest proof for each pubkey
- Limit number of pubkeys in each slot to 100
* Disable duplicate shred handling for now.
* Revert "Disable duplicate shred handling for now."
This reverts commit c3fcf403876cfbf90afe4d2265a826f21a5e24ab.
2023-01-19 13:00:56 -08:00
2c347ac0a5
BankingStage Refactor: Move packet receiving and buffering functions to separate module ( #29761 )
...
Move packet receiving and buffering functions to separate module
2023-01-19 08:52:32 -08:00
c4e43f1de4
vote: encapsulate `Lockout` ( #29753 )
2023-01-18 19:28:28 -07:00
4973fe18f1
Rename banking stage packet receivers consistently ( #29752 )
...
Rename banking stage batch receivers consistently
2023-01-19 10:04:55 +09:00
55d743c49a
Rename remaining ones to replay_vote_{sender,receiver} ( #29716 )
...
* Rename remaining ones to replay_vote_{sender,receiver}
* Fix typo...
2023-01-18 14:14:04 +09:00
f9062718c4
prioritize repair requests by stake ( #29730 )
2023-01-17 18:38:10 -08:00
aa40c2b712
Increase turbine propagation const ( #29742 )
...
* Increase turbine propagation const
Value is used as a delay threshold for issuing shred repairs and analysis is showing we are overly aggressive in requesting repairs. Shreds show up via turbine before the repair completes the vast majority of the time
* Use Duration type for MAX_TURBINE_PROPAGATION
2023-01-17 15:01:00 -08:00
f6fcb14a3e
adjust normalized stake calculation in compute_weight ( #29694 )
2023-01-17 11:27:57 -08:00
156454c980
Remove PacketDeserializer's extra overflow guard ( #29715 )
2023-01-17 14:21:17 +09:00
0db14ad39c
Removes full_snapshot from CalcAccountsHashConfig ( #29722 )
2023-01-16 16:22:46 -05:00
80a39bd6a5
adds feature to (temporarily) drop merkle shreds from testnet ( #29711 )
2023-01-15 15:41:58 +00:00
5b5a3ebce8
adds metrics for num merkle shreds on the receiving end ( #29710 )
2023-01-14 23:07:42 +00:00
59fde130d6
ledger/blockstore: PerfSampleV2: num_non_vote_transactions ( #29404 )
...
Store non-vote transaction counts that are now recorded by the banks
into the `blockstore`.
`SamplePerformanceService` now populates `PerfSampleV2` with counts from
the banks.
2023-01-12 19:14:04 -08:00
544b9745c2
snapshot storage path uses 1 append vec per slot ( #29627 )
2023-01-11 12:05:15 -08:00
8c212f59ad
renames ContactInfo to LegacyContactInfo ( #29566 )
...
Working towards adding a new ContactInfo where new sockets can be
added in a backward compatible way.
2023-01-08 16:00:55 +00:00
43a0745b37
Fix doc warnings ( #29537 )
2023-01-07 09:24:50 +00:00
283a2b1540
removes #[allow(clippy::same_item_push)] ( #29543 )
2023-01-06 17:32:26 +00:00
12da2da389
fixes errors from clippy::redundant_clone ( #29536 )
...
https://rust-lang.github.io/rust-clippy/master/index.html#redundant_clone
2023-01-05 18:42:19 +00:00
5c9beef498
fixes errors from clippy::useless_conversion ( #29534 )
...
https://rust-lang.github.io/rust-clippy/master/index.html#useless_conversion
2023-01-05 18:05:32 +00:00
1e8a8e07b6
Stream the executed transaction count in the block notification ( #29272 )
...
Problem
The plugins need to know when all transactions for a block have been all notified to serve getBlock request correctly. As block and transaction notifications are sent asynchronously to each other it will be difficult.
Summary of Changes
Include the executed transaction count in block notification which can be used to check if all transactions have been notified.
2023-01-05 09:36:19 -08:00
832302485e
require repair request signature, ping/pong for Testnet, Development clusters ( #29351 )
2023-01-04 14:54:19 -08:00
d7bd1bf970
bank: Record non-vote transaction count ( #29383 )
...
A subsequent change to `SamplePerformanceService` introduces non-vote transaction counts, which `bank`s need to store.
Part of work on https://github.com/solana-labs/solana/issues/29159
2023-01-03 14:46:20 -08:00
3363c08ac0
Move async remove to snapshot_utils.rs ( #29406 )
2023-01-03 06:15:32 -08:00
754ecf467b
generalizes the return type of Shred::get_signed_data ( #29446 )
...
The commit adds an associated SignedData type to Shred trait so that
merkle and legacy shreds can return different types for signed_data
method.
This would allow legacy shreds to point to a section of the shred
payload, whereas merkle shreds would compute and return the merkle root.
Ultimately this would allow to remove the merkle root from the shreds
binary.
2022-12-31 17:08:25 +00:00
17b64005d3
Add more logging and documentation to flaky optimistic confirmation tests ( #29418 )
...
* Revert "add retry for flakey local cluster test (#29228 )"
This reverts commit 7a97121747
2022-12-27 10:47:45 -07:00
456d06785d
experiments different turbine fanouts for propagating shreds ( #29393 )
...
The commit allocates 2% of slots to running experiments with different
turbine fanouts based on the slot number.
The experiment is feature gated with an additional feature to disable
the experiment.
2022-12-26 14:18:56 +00:00
f2ba16ee87
Plumb dumps from replay_stage to repair ( #29058 )
...
* Plumb dumps from replay_stage to repair
When dumping a slot from replay_stage as a result of duplicate or
ancestor hashes, properly update repair subtrees to keep weighting and
forks view accurate.
* add test
* pr comments
2022-12-25 09:58:30 -07:00
558292466b
rolls back merkle shreds on testnet ( #29340 )
...
https://github.com/solana-labs/solana/pull/29339
adds hash domain to merkle shreds. In order to merge that change, need
to temporarily disable merkle shreds on testnet.
2022-12-20 18:33:48 +00:00
053775ad77
Elides unnecessary lifetimes ( #29299 )
2022-12-20 12:44:17 -05:00
c657f42d77
remove a wrapper function ( #29305 )
2022-12-19 16:10:16 +00:00
86b2e545e1
Prune redundant const SLOT_MS ( #29278 )
...
* Alias redundant const SLOT_MS to DEFAULT_MS_PER_SLOT
* Slate SLOT_MS for deprecation
* Add doc comments
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-12-16 08:05:09 -08:00
a44ea779bd
add support for a repair protocol whitelist ( #29161 )
2022-12-15 19:24:23 -08:00
dca5d7f9b4
chore: bump test-case from 2.1.0 to 2.2.2 ( #28184 )
...
Bumps [test-case](https://github.com/frondeus/test-case ) from 2.1.0 to 2.2.2.
- [Release notes](https://github.com/frondeus/test-case/releases )
- [Changelog](https://github.com/frondeus/test-case/blob/master/CHANGELOG.md )
- [Commits](https://github.com/frondeus/test-case/compare/v2.1.0...v2.2.2 )
---
updated-dependencies:
- dependency-name: test-case
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-13 16:07:50 +00:00
b95835143e
remove AccountsBackgroundService::new(caching_enabled) ( #29234 )
2022-12-13 07:18:02 -08:00
bb0bfc4214
remove bank_from_latest_snapshot_archives(caching_enabled) ( #29238 )
2022-12-13 07:16:24 -08:00
fec8f61566
remove ProcessOptions::accounts_db_caching_enabled ( #29217 )
2022-12-12 20:25:00 -08:00
2c2324f4ea
remove caching_enabled from Bank::new_with_paths_for_tests ( #29214 )
2022-12-12 15:30:46 -08:00
1b0aaf1607
Makes a new PathBuf instead of moving the test's TempDir ( #29220 )
2022-12-12 18:29:36 -05:00
249607dbfe
Use a different tempdir for unpacking snapshots ( #29219 )
2022-12-12 17:26:52 -06:00
391f68da61
Uses Storages to calculate accounts hash in EAH warp tests ( #29192 )
2022-12-12 13:30:23 -05:00
88a8f40bd2
apply [limit repairs to top staked... #28673 ] to non-MainnetBeta clusters ( #29163 )
2022-12-11 15:52:41 -08:00
4ee318b2b2
fixes rust code formatting in core/src/consensus.rs ( #29204 )
2022-12-11 23:20:52 +00:00
631a98a3b6
warp_from_parents works with write_cache enabled ( #29185 )
2022-12-09 14:28:18 -08:00
cd9f1f1862
Typo/filter_and_forward_with_account_limits ( #29183 )
2022-12-09 16:22:25 -06:00
560143a267
remove ValidatorConfig.caching_enabled ( #29172 )
2022-12-09 11:31:55 -08:00
ecea802fe6
Bidirectional quic communication support ( #29155 )
...
* Support bi-directional quic communication, use the same endpoint for the quic server and client
This is needed for supporting using quic for repair
* Added comments on the bi-directional communication tests
* Removed some debug logs
* clippy issue
2022-12-09 10:59:43 -08:00
6a90abd056
remove handle_snapshot_requests.caching_enabled ( #29174 )
2022-12-09 10:51:44 -08:00
45ba5ef6fd
remove bank_from_snapshot_archives caching_enabled ( #29171 )
2022-12-09 10:45:21 -08:00
ec5098a723
remove bank_test_config_caching_enabled ( #29170 )
2022-12-09 08:28:02 -08:00
ca5d8c4b4d
refactor sysmonitor config ( #29132 )
2022-12-09 07:43:03 -06:00
049fb3d725
Remove an unnecessary clone of a PohConfig inside Validator::new
2022-12-07 13:03:14 -06:00
8f24ceffbd
Removed Arcs from PohConfig parameters and pass the struct by reference only
2022-12-07 10:52:07 -06:00
582397ad48
fix solRptLdgrStat thread hang ( #29118 )
2022-12-06 17:09:56 -06:00
33b15240ac
Revert #28945 ( #29127 )
...
revert #28945
2022-12-06 17:08:56 -06:00
aeb6b53502
Remove unused Option<> around ValidatorConfig's SnapshotConfig ( #29090 )
...
Remove Option<> around ValidatorConfig's SnapshotConfig
The SnapshotConfig is required and is currently hard-coded to be a
Some().
2022-12-06 22:47:55 +00:00
df7fd8ae5f
patches rust code formatting in core/src/replay_stage.rs ( #29123 )
2022-12-06 22:09:57 +00:00
9524c9dbff
patches errors from clippy::uninlined_format_args
...
https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
2022-12-06 19:32:15 +00:00
9433c06745
patches errors from clippy::unchecked_duration_subtraction
...
https://rust-lang.github.io/rust-clippy/master/index.html#unchecked_duration_subtraction
2022-12-06 19:32:15 +00:00
bbd49acb2f
fix merge error
2022-12-06 13:31:50 -06:00
7a512d7f27
report number of open files
2022-12-06 13:31:50 -06:00
412cf3df27
sort deps
2022-12-06 13:31:50 -06:00
f716cad4af
don't use procfs as it is not supported on mac and windows.
...
make open_fd stats only on linux platform
2022-12-06 13:31:50 -06:00
fc97d818b6
share code
2022-12-06 13:31:50 -06:00
36dc3a457f
get mmap with wc-l
2022-12-06 13:31:50 -06:00
dd81af9fff
increase fd report interval to 3 minutes
2022-12-06 13:31:50 -06:00
0ce507c20d
refactor SystemMonitorReportConfig
2022-12-06 13:31:50 -06:00
914f7bd85d
fix
2022-12-06 13:31:50 -06:00
1635b99486
add mmmap file count
2022-12-06 13:31:50 -06:00
e1ba5a2a63
add monitoring for open file descriptors stat
2022-12-06 13:31:50 -06:00
7ed22f7b18
Remove gate from accepting packets for forwarding ( #29049 )
2022-12-06 12:13:01 -06:00
b1340d77a2
sdk: Make Packet::meta private, use accessor functions ( #29092 )
...
sdk: Make packet meta private
2022-12-06 12:54:49 +01:00
fd3b5d08d7
Refactor/banking_stage_make_decision_consume_bank ( #28946 )
2022-12-02 10:07:01 -06:00
5850af5316
Refactor to remove requested_cu from cost_trarcker ( #29015 )
...
* refactor cost tracker by removing requested_cu from it, call sites to use cost_model forr consistency
* review fix
2022-12-02 00:25:09 +00:00
3c42c87098
Remove obsoleted return value from Blockstore insert shred method ( #28992 )
2022-12-01 11:17:46 -06:00
b6dce6cf3b
Move BlockstoreInsertionMetrics field update to blockstore.rs ( #28991 )
...
The num_repair field is only blockstore insertion metric being updated
outside of Blockstore::insert() call chain; move the update to insert()
with the rest of the fields in BlockstoreInsertionMetrics struct.
2022-11-30 11:46:35 -06:00
edacd3c411
Add dump_node to update stake for heaviest subtrees ( #28827 )
...
* Add dump_node to update stake for heaviest subtrees
Additionally refactor subtrees to store children as a hashset
* Add a more complicated forks test
* chose -> choose
* remove is_dumped flag and reuse latest_invalid_ancestor instead
2022-11-30 09:26:13 -08:00
4d338ed882
Bugfix/mi_remove_never_entries ( #28978 )
2022-11-29 16:00:21 -06:00
0d0a491f27
More documentation + small refactor for RepairService ( #28933 )
2022-11-28 19:46:06 -08:00
9f370475d4
remove obsoleted comment ( #28960 )
2022-11-28 13:39:40 -06:00
7d99cddb9f
dedups turbine retransmit peers by tvu socket addresses ( #28944 )
...
No need to send duplicate shreds if several nodes have the same tvu
socket address because they are behind a relayer or whatever.
2022-11-28 19:23:02 +00:00
7e87998091
reduce memory usage report freq to 1 per 5s ( #28327 )
2022-11-28 19:08:06 +00:00
bdd162492c
Feature/multi-iterator-scanner-read-locks ( #28862 )
2022-11-28 11:23:04 -06:00
9327658007
Promotes accounts hash to a strong type ( #28930 )
2022-11-28 10:09:47 -05:00
638b26ea65
Renames EAH test fn ( #28939 )
2022-11-23 05:18:50 +00:00
38f7122605
separate make_decision in BankingStage ( #28884 )
2022-11-22 19:01:09 -06:00
c8b0c3ede9
Update cost model to use requested_cu instead of estimated cu #27608 ( #28281 )
...
* Update cost model to use requested_cu instead of estimated cu #27608
* remove CostUpdate and CostModel from replay/tvu
* revive cost update service to send cost tracker stats
* CostModel is now static
* remove unused package
Co-authored-by: Tao Zhu <tao@solana.com>
2022-11-22 11:55:56 -06:00
637e8a937b
clean up: remove my_pubkey arg from consume_buffered_packets ( #28888 )
2022-11-22 11:40:04 -06:00
20d8b5e98b
default some tests to write cache = true ( #28917 )
2022-11-21 15:53:39 -08:00
dd723210ca
remove unnecessary clippy attributes ( #28891 )
2022-11-21 12:54:54 -06:00
d43b001189
rolls out merkle shreds to ~20% of testnet ( #28905 )
2022-11-21 16:20:02 +00:00
c6927151ef
Sort offline/wrong-shred nodes by stake weight while waiting for supermajority ( #28872 )
2022-11-18 15:26:21 -08:00
f22104d46b
use write cache by default in some tests ( #28876 )
2022-11-18 14:35:52 -08:00
a636038fff
Clean up: banking_stage_prepare_sanitized_batch ( #28841 )
...
Use measure! for bank.prepare_sanitized_batch_with_results
2022-11-18 14:04:44 -06:00
c32377b5af
Split out quic- and udp-client definitions ( #28762 )
...
* Move ConnectionCache back to solana-client, and duplicate ThinClient, TpuClient there
* Dedupe thin_client modules
* Dedupe tpu_client modules
* Move TpuClient to TpuConnectionCache
* Move ThinClient to TpuConnectionCache
* Move TpuConnection and quic/udp trait implementations back to solana-client
* Remove enum_dispatch from solana-tpu-client
* Move udp-client to its own crate
* Move quic-client to its own crate
2022-11-18 12:21:45 -07:00
88e6ea37d9
refactor: move more BankingStage cost_model stuff into qos_service ( #28840 )
2022-11-17 14:03:17 -06:00
ee2f760d3d
MultiIteratorScanner - improve banking stage performance with high contention
2022-11-17 10:54:12 -06:00
2bafb0cb12
Requires EAH state cannot be Invalid ( #28817 )
2022-11-17 11:01:01 -05:00
apfitzge
behzad nouri
Wen
Trent Nelson
Ryo Onodera
Jeff Biseda
Brennan Watt
Brooks
Illia Bobyr
Jeff Washington (jwash)
Brian Anderson
Lijun Wang
Xiang Zhu
Ashwin Sekar
Tao Zhu
dependabot[bot]
HaoranYi
Jason Davis
steviez
Jon Cinque
Maximilian Schneider
Michael Vines
Tyera