c0e4379f43
Whickey/version v1.15 ( #27739 )
...
* Bump version to v1.13.0
* Bump version to v1.14.0
* Bump version to v1.15.0
2022-09-13 09:06:15 -05:00
079bf561b0
Clean_up/upb_push_comment ( #27707 )
2022-09-12 18:59:41 -05:00
765c628546
use exit signal for acct idx bg threads ( #27483 )
2022-09-12 11:51:12 -07:00
4f22ee8f9b
uses varint encoding for vote-state lockout offsets
...
The commit removes CompactVoteStateUpdate and instead reduces serialized
size of VoteStateUpdate using varint encoding for vote-state lockout
offsets.
2022-09-12 16:31:20 +00:00
90b8a3a44d
Remove KeypairInsecureClone trait and add insecure_clone() instead ( #27396 )
...
See discussion in #26248
2022-09-12 14:59:41 +00:00
83d4d128c2
Add --process-ledger-before-service flag to solana-validator
2022-09-11 07:58:42 -07:00
abd01553d5
tests: Keypair::new().pubkey() -> pubkey::new_rand ( #27705 )
2022-09-10 13:56:45 -07:00
1f00b468e5
add enable_rehashing to AccountsPackage ( #27644 )
2022-09-08 09:25:25 -07:00
a9c5adbf88
UnprocessedPacketBatches pop_max fn are only used in tests ( #27645 )
2022-09-08 11:01:14 -05:00
cc58968b76
add new leader slot metric to track account contention throttling ( #27654 )
2022-09-08 09:22:58 -05:00
f338aa62ba
chore: bump serde from 1.0.143 to 1.0.144 ( #27511 )
...
* chore: bump serde from 1.0.143 to 1.0.144
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.143 to 1.0.144.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.143...v1.0.144 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-09-07 16:54:44 -06:00
4308c300b4
In ledger-tool delete the account files in the async way ( #27622 )
...
* In ledger-tool delete the account files in the async way
* format changes by ./cargo nightly fmt --all
2022-09-07 14:35:06 -07:00
6a322de845
Make Accounts Background Services aware of Epoch Accounts Hash ( #27626 )
2022-09-07 20:41:40 +00:00
7f223dc582
Added option to turn on UDP for TPU transaction and make UDP based TPU off by default ( #27462 )
...
--tpu-enable-udp is introduced. And when this is on, the transaction receive and transaction forward is enabled using udp.
Except for a few tests which was hard-coded sending transactions using udp, most tests are being done with udp based tpu disabled.
2022-09-07 13:19:14 -07:00
c04747dd66
cluster_slot_state_verifier: clippy nightly fixes ( #27521 )
...
clippy nightly fixes
2022-09-07 15:04:56 -05:00
1465ec947d
replay_stage: clippy nightly fixes ( #27520 )
...
clippy nightly fixes
2022-09-07 15:04:46 -05:00
452866dbcf
shredder: clippy nightly fixes ( #27522 )
...
clippy nightly fixes
2022-09-07 15:04:32 -05:00
d6a1e7498f
Add tests for deserialize_and_collect_packets ( #27623 )
2022-09-07 12:52:18 -05:00
22007a3c96
allow accounts hash calc to specify enable_rehashing ( #27615 )
2022-09-07 10:16:52 -07:00
a31d4a597d
serialize epoch_accounts_hash ( #27516 )
2022-09-07 10:07:00 -07:00
93a4f80a2c
Handling snapshot requests is now required ( #27537 )
2022-09-07 10:08:42 -04:00
269eb519dd
track time to coalesce entries in recv_slot_entries ( #27525 )
2022-09-06 16:07:17 -07:00
a67d56f462
refactor: add function for deserializing and collecting packets - separate from channel receive ( #27548 )
2022-09-06 15:54:31 -05:00
6684c62280
Add SnapshotUsage to SnapshotConfig ( #27508 )
2022-09-02 08:56:23 -04:00
242c9cb442
RPC Notifier Signal when Setup Complete ( #27481 )
...
* RPC notifier signal when ready
2022-09-01 16:39:55 -07:00
9b8bed86f9
Add getRecentPrioritizationFees RPC endpoint ( #27278 )
...
* Plumb priority_fee_cache into rpc
* Add PrioritizationFeeCache api
* Add getRecentPrioritizationFees rpc endpoint
* Use MAX_TX_ACCOUNT_LOCKS to limit input keys
* Remove unused cache apis
* Map fee data by slot, and make rpc account inputs optional
* Add priority_fee_cache to rpc test framework, and add test
* Add endpoint to jsonrpc docs
* Update docs/src/developing/clients/jsonrpc-api.md
* Update docs/src/developing/clients/jsonrpc-api.md
2022-09-01 23:12:12 +00:00
3bdc5b3f2b
separate packet_deserializer inside banking_stage ( #27120 )
...
* separate packet_deserializer inside banking_stage
* Make ReceivePacketResults into a struct with named fields
2022-09-01 10:00:48 -05:00
66717ff87d
chore: bump chrono from 0.4.21 to 0.4.22 ( #27509 )
...
* chore: bump chrono from 0.4.21 to 0.4.22
Bumps [chrono](https://github.com/chronotope/chrono ) from 0.4.21 to 0.4.22.
- [Release notes](https://github.com/chronotope/chrono/releases )
- [Changelog](https://github.com/chronotope/chrono/blob/v0.4.22/CHANGELOG.md )
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.21...v0.4.22 )
---
updated-dependencies:
- dependency-name: chrono
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-08-31 22:39:12 +00:00
8bb039d08d
collect min prioritization fees when replaying sanitized transactions ( #26709 )
...
* Collect blocks' minimum prioritization fees when replaying sanitized transactions
* Limits block min-fee metrics reporting to top 10 writable accounts
* Add service thread to asynchronously update and finalize prioritization fee cache
* Add bench test for prioritization_fee_cache
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2022-08-31 08:00:55 -05:00
7bd08ad3ae
Featurize spinner in rpc-client and tpu-client ( #27381 )
...
* Add spinner features to rpc-client and tpu-client, and disable where unneeded
* Add doc comment
2022-08-30 09:56:56 -06:00
5b64107626
make pruned_bank channel unbonded.
...
In kin-sim, we found that bounded channel causes halt for account
background services. As the number of accounts grows, the time for
pruning and cleaning increases, which would leads to longer intervals
between the pruning of deaded bank slots. With 1.7B accounts, we will
exceed the 10K bounded channel threshold that causes halt of account
back ground services. Without pruning, the node will eventually run out
of memory.
2022-08-29 19:06:30 -05:00
3c7cd62030
Move pruned_banks_receiver into PrunedBanksRequestHandler ( #27445 )
2022-08-29 13:30:06 -04:00
46a48760db
Switch concurrent replay from feature to param ( #27401 )
...
* Switch concurrent replay from feature to param
2022-08-26 12:36:02 -07:00
5eefc256d6
Fix startup panic if removing accounts directory fails ( #27386 )
...
* Remove contents of accounts directory if deleting the directory fails.
2022-08-25 20:35:12 -05:00
b1cff5d740
make fatal log message sound fatal
2022-08-25 21:49:12 +00:00
d1522fc790
coalesce entries in recv_slot_entries to target byte count ( #27321 )
2022-08-25 13:51:55 -07:00
2da93bd45a
add text to assert ( #27377 )
2022-08-24 14:11:53 -05:00
b8b3d723da
Use new client crates ( #27360 )
...
* Update ancillary cli crates
* Update cli
* Update command-line tools
* Update rpc, etc
* Update client-test
* Update core, validator
* Update local-cluster
2022-08-24 10:47:02 -06:00
efa6201eda
Check overflow on vote tx compaction boundary ( #27185 )
...
* Check overflow on vote tx compaction boundary
Check for overflow during the conversion between VoteStateUpdate and
CompactVoteStateUpdate.
* Try removing clippy supress
2022-08-23 22:29:03 -07:00
827d8e4bc0
Fallback to synchronous rm_dir call if path moving fails ( #27306 )
...
Remove some log lines, as suggested in PR #26910
2022-08-22 22:47:39 -07:00
e4a7d01e10
Rust v1.63 ( #27303 )
...
* Upgrade to Rust v1.63.0
* Add nightly_clippy_allows
* Resolve some new clippy nightly lints
* Increase QUIC packets completion timeout
* Update quinn-udp crate
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-22 18:01:03 -07:00
7bdeea10ad
Assign custom names to the Rayon global thread pool
2022-08-22 17:56:55 +00:00
7c01c1ecc6
Update delete_path thread name
2022-08-22 09:01:26 -07:00
fc1a4dd11a
run hash calc with index on failure ( #27279 )
2022-08-22 10:58:04 -05:00
3f4731b37f
Standardize thread names
...
Tenets:
1. Limit thread names to 15 characters
2. Prefix all Solana-controlled threads with "sol"
3. Use Camel case. It's more character dense than Snake or Kebab case
2022-08-20 07:49:39 -07:00
c54824e4f5
Account files remove ( #26910 )
...
* Create a new function cleanup_accounts_paths, a trivial change
* Remove account files asynchronously
* Update and simplify the implementation after the validator test runs.
* Fixes after testing on the dev device
* Discard tokio. Use thread instead
* Fix comments format
* Fix config type to pass the github test
* Fix failed tests. Handle the case of non-existing path
* Final cleanup, addressing the review comments
Avoided OsString.
Made the function more generic with "impl AsRef<Path>"
Co-authored-by: Jeff Washington <jeff.washington@solana.com>
2022-08-19 23:56:52 -07:00
eb06bb61e8
banking stage: actually aggregate tracer packet stats ( #27118 )
...
* aggregated_tracer_packet_stats_option was alwasys None
* Actually accumulate tracer packet stats
2022-08-19 15:16:56 -05:00
dba2fd5a16
Enable QUIC client by default. Add arg to disable QUIC client. (Forward port #26927 ) ( #27194 )
...
Enable QUIC client by default. Add arg to disable QUIC client.
* Enable QUIC client by default. Add arg to disable QUIC client.
* Deprecate --disable-quic-servers arg
* Add #[ignore] annotation to failing tests
2022-08-19 09:15:15 -05:00
7573000d87
Revert "Rust v1.63.0 ( #27148 )" ( #27245 )
...
This reverts commit a2e7bdf50a
2022-08-19 09:19:44 +01:00
6928b2a5af
adds hash domain to ping-pong protocol ( #27193 )
...
In order to maintain backward compatibility, for now the responding node
will hash the token both with and without domain so that the other node
will accept the response regardless of its upgrade status.
Once the cluster has upgraded to the new code, we will remove the legacy
domain = false case.
2022-08-18 22:39:31 +00:00
a2e7bdf50a
Rust v1.63.0 ( #27148 )
...
* Upgrade to Rust v1.63.0
* Add nightly_clippy_allows
* Resolve some new clippy nightly lints
* Increase QUIC packets completion timeout
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-17 15:48:33 -07:00
fea66c8b63
derives Error trait for ClusterInfoError and core::result::Error ( #27208 )
2022-08-17 22:01:51 +00:00
225cddcffb
serialize incremental_snapshot_hash ( #26839 )
...
* serialize incremental_snapshot_hash
* pr feedback
2022-08-17 15:14:31 -05:00
3b87aa9227
reverts wide fanout in broadcast when the root node is down ( #26359 )
...
A change included in
https://github.com/solana-labs/solana/pull/20480
was that when the root node in turbine broadcast tree is down, the
leader will broadcast the shred to all nodes in the first layer.
The intention was to mitigate the impact of dead nodes on shreds
propagation, because if the root node is down, then the entire cluster
will miss out the shred.
On the other hand, if x% of stake is down, this will cause 200*x% + 1
packets/shreds ratio at the broadcast stage which might contribute to
line-rate saturation and packet drop.
To avoid this bandwidth saturation issue, this commit reverts that logic
and always broadcasts shreds from the leader only to the root node.
As before we rely on erasure codes to recover shreds lost due to staked
nodes being offline.
2022-08-16 19:40:06 +00:00
a0d1f4ef88
chore: bump serial_test from 0.8.0 to 0.9.0 ( #27097 )
...
Bumps [serial_test](https://github.com/palfrey/serial_test ) from 0.8.0 to 0.9.0.
- [Release notes](https://github.com/palfrey/serial_test/releases )
- [Commits](https://github.com/palfrey/serial_test/compare/v0.8.0...v0.9.0 )
---
updated-dependencies:
- dependency-name: serial_test
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-15 15:53:12 -06:00
bdce208fe5
clean feature: `request_units_deprecated` ( #27102 )
...
clean feature: request_units_deprecated
2022-08-13 13:12:35 +01:00
e50013acdf
Handle JsonRpcService startup failure ( #27075 )
2022-08-11 23:25:20 -07:00
fc6cee9c06
allow staked nodes weight override ( #26870 )
...
* Allowed staked nodes weight override (#26407 )
* Allowed staked nodes weight override, passing only HashMap over to core module
Co-authored-by: Ondra Chaloupka <chalda@chainkeepers.io>
2022-08-11 14:34:04 -07:00
f641d3bad6
chore: bump chrono from 0.4.19 to 0.4.21 ( #27076 )
...
* chore: bump chrono from 0.4.19 to 0.4.21
Bumps [chrono](https://github.com/chronotope/chrono ) from 0.4.19 to 0.4.21.
- [Release notes](https://github.com/chronotope/chrono/releases )
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md )
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.19...v0.4.21 )
---
updated-dependencies:
- dependency-name: chrono
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2022-08-11 09:28:50 -06:00
ac91cdab74
removes buffering when generating coding shreds in broadcast ( #25807 )
...
Given the 32:32 erasure recovery schema, current implementation requires
exactly 32 data shreds to generate coding shreds for the batch (except
for the final erasure batch in each slot).
As a result, when serializing ledger entries to data shreds, if the
number of data shreds is not a multiple of 32, the coding shreds for the
last batch cannot be generated until there are more data shreds to
complete the batch to 32 data shreds. This adds latency in generating
and broadcasting coding shreds.
In addition, with Merkle variants for shreds, data shreds cannot be
signed and broadcasted until coding shreds are also generated. As a
result *both* code and data shreds will be delayed before broadcast if
we still require exactly 32 data shreds for each batch.
This commit instead always generates and broadcast coding shreds as soon
as there any number of data shreds available. When serializing entries
to shreds:
* if the number of resulting data shreds is less than 32, then more
coding shreds will be generated so that the resulting erasure batch
has the same recovery probabilities as a 32:32 batch.
* if the number of data shreds is more than 32, then the data shreds are
split uniformly into erasure batches with _at least_ 32 data shreds in
each batch. Each erasure batch will have the same number of code and
data shreds.
For example:
* If there are 19 data shreds, 27 coding shreds are generated. The
resulting 19(data):27(code) erasure batch has the same recovery
probabilities as a 32:32 batch.
* If there are 107 data shreds, they are split into 3 batches of 36:36,
36:36 and 35:35 data:code shreds each.
A consequence of this change is that code and data shreds indices will
no longer align as there will be more coding shreds than data shreds
(not only in the last batch in each slot but also in the intermediate
ones);
2022-08-11 12:44:27 +00:00
4e79d78629
`solana-validator monitor` how displays slot and gossip stake % while waiting for supermajority
2022-08-10 11:13:25 -07:00
e3a8d2ecdd
chore: bump serde_json from 1.0.81 to 1.0.83 ( #27036 )
...
* chore: bump serde_json from 1.0.81 to 1.0.83
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.81 to 1.0.83.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.81...v1.0.83 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-08-10 09:45:42 -06:00
c03f3b1436
Separate file for ImmutableDeserializedPacket type ( #26951 )
2022-08-09 22:39:01 -07:00
ae5b680c6f
chore: bump serde from 1.0.138 to 1.0.143 ( #27015 )
...
* chore: bump serde from 1.0.138 to 1.0.143
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.138 to 1.0.143.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.138...v1.0.143 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-08-10 05:08:43 +00:00
370de8129e
ancestor hashes socket ping/pong support ( #26866 )
2022-08-09 21:39:55 -07:00
ccfbc54195
Move vote program state and instructions to solana-program
2022-08-09 20:52:47 -07:00
c2455e7aa4
Fix typo in test function ( #27031 )
2022-08-09 12:39:22 -07:00
a69470fd45
Set receive_window per quic connection ( #26936 )
...
This change sets the receive_window for non-staked node to 1 * PACKET_DATA_SIZE, and maps the staked nodes's connection's receive_window between 1.2 * PACKET_DATA_SIZE to 10 * PACKET_DATA_SIZE based on the stakes.
The changes is based on Quinn library change to support per connection receive_window tweak at the server side. quinn-rs/quinn#1393
2022-08-09 10:02:47 -07:00
e2a2d271f2
adds number of coding shreds to broadcast metrics ( #27006 )
2022-08-09 13:59:40 +00:00
b6d38aad69
tracer-packet-stats reporting should not reset id ( #27012 )
2022-08-09 06:38:08 -07:00
99ef2184cc
Delete files older than the lowest_cleanup_slot in LedgerCleanupService::cleanup_ledger ( #26651 )
...
#### Problem
LedgerCleanupService requires compactions to propagate & digest range-delete tombstones
to eventually reclaim disk space.
#### Summary of Changes
This PR makes LedgerCleanupService::cleanup_ledger delete any file whose slot-range is
older than the lowest_cleanup_slot. This allows us to reclaim disk space more often with
fewer IOps. Experimental results on mainnet validators show that the PR can effectively
reduce 33% to 40% ledger disk size.
2022-08-09 00:48:06 +08:00
ed8c224374
Bump version to v1.12 ( #26967 )
2022-08-06 13:20:30 -05:00
cf58640937
Keypair: implement clone() ( #26248 )
...
* Keypair: implement clone()
This was not implemented upstream in ed25519-dalek to force everyone to
think twice before creating another copy of a potentially sensitive
private key in memory.
See https://github.com/dalek-cryptography/ed25519-dalek/issues/76
However, there are now 9 instances of
Keypair::from_bytes(&keypair.to_bytes())
in the solana codebase and it would be preferable to have a function.
In particular since this also comes up when writing programs and can
cause users to either start messing with lifetimes or discover the
from_bytes() workaround themselves.
This patch opts to not implement the Clone trait. This avoids automatic
use in order to preserve some of the original "let developers think
twice about this" intention.
* Use Keypair::clone
2022-08-06 11:54:38 -06:00
270315a7f6
transaction-status, storage-proto: add compute_units_consumed ( #26528 )
...
* transaction-status, storage-proto: add compute_units_consumed
* fix bpf test
Co-authored-by: Justin Starry <justin@solana.com>
2022-08-06 17:14:31 +00:00
69598ed4c0
Refactor: Add `RuntimeConfig` field to Bank ( #26946 )
...
* Refactor: Simplify arguments for bank constructor methods
* Refactor: Add RuntimeConfig to Bank fields
* Arc wrap runtime_config
* Arc wrap all runtime config usages
* Remove Copy trait derivation from RuntimeConfig
* Remove some arc wrapping
2022-08-05 20:49:00 +01:00
5bc81a6c35
Io stats v2 ( #26898 )
...
* Use sysfs instead of procfs for disk stats
* Filter map to filter dmcrypt and mdraid volumes
* Unit test cover different kernel formats
2022-08-05 10:38:49 -07:00
2dca239480
Remove runtime dependency from solana-transaction-status ( #26930 )
...
* Move RewardType out of runtime
* Move collect_token_balances to solana-ledger
* Remove solana-runtime dependency
2022-08-05 00:20:27 -06:00
300666dce7
Make `solana-ledger-tool` run AccountsBackgroundService ( #26914 )
...
Prior to this change, long running commands like `solana-ledger-tool
verify` would OOM due to AccountsDb cleanup not happening.
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-04 15:44:31 -05:00
83a0f5da0f
core: fix double-readlock in replay_stage ( #26052 )
2022-08-04 10:45:31 -04:00
457f9ef739
Reduce severity level of log in replay ( #26893 )
...
* Reduce active banks log severity from warn to trace
2022-08-03 13:51:16 -07:00
fbf1bf6d86
Bump Version to 1.11.6 ( #26906 )
...
Co-authored-by: willhickey <willhickey@users.noreply.github.com>
2022-08-03 12:48:43 -05:00
f3b760dd91
Add IO metrics ( #26804 )
...
* Add Disk IO metrics
2022-08-02 14:29:53 -07:00
ec36f0c5df
removes redundant Option<&Arc<...>> wrapper for Blockstore in serve-repair
2022-08-02 15:30:53 +00:00
6423da0218
removes redundant Arc<RwLock<...>> wrapper off ServeRepair
2022-08-02 15:30:53 +00:00
ded9a35cd6
mark repair ping packets for discard only after successful signature verification ( #26878 )
2022-08-01 16:17:19 -07:00
857be1e237
sign repair requests ( #26833 )
2022-07-31 15:48:51 -07:00
fbfcc3febf
Bugfix: VoteProcessingTiming reset both counters ( #26843 )
2022-07-29 12:56:04 -05:00
fb922f613c
Compute maximum parallel QUIC streams using client stake ( #26802 )
...
* Compute maximum parallel QUIC streams using client stake
* clippy fixes
* Add unit test
2022-07-29 08:44:24 -07:00
467cb5def5
Concurrent slot replay ( #26465 )
...
* Concurrent replay slots
* Split out concurrent and single bank replay paths
* Sub function processing of replay results for readability
* Add feature switch for concurrent replay
2022-07-28 11:33:19 -07:00
817f65bb50
add full_snapshot to hash config ( #26811 )
2022-07-28 09:46:34 -05:00
8d69e8d447
Compact vote state updates to reduce block size ( #26616 )
...
* Compact vote state updates to reduce block size
* Add rpc transaction tests
2022-07-27 13:23:44 -06:00
ed539d65b4
Only take the latest vote for each validator in gossip ( #25934 )
...
* Only take the latest vote for each validator in gossip
Since the new vote updates are no longer incremental, there
is no value in storing intermediate votes.
* Address pr feedback
* Handle potential downgrade path, FullTowerVote -> Incremental
* Rename sent to bank -> gossip slot
* Handle downgrade case properly
* Only downgrade for newer votes and feature flag, ignore incremental votes otherwise
* Update test
2022-07-26 16:38:30 -06:00
5d038b9d2a
Bump Version to 1.11.5 ( #26758 )
...
Co-authored-by: willhickey <willhickey@users.noreply.github.com>
2022-07-25 13:05:14 -06:00
f6d5b253fb
Enforce a 12MB limit on outbound repair ( #26493 )
2022-07-24 20:44:22 -05:00
d9c7bc7e78
Revert "cleanup feature: default units per instruction ( #26684 )" ( #26750 )
...
This reverts commit 39a34db52a
2022-07-23 11:03:46 -05:00
a603c8b0bc
Enable QUIC servers by default
2022-07-22 15:45:10 -06:00
2ee19f536a
Revert "Revert "core: disable quic servers on mainnet-beta" ( #26216 )"
...
This reverts commit 4a7fb2a808
2022-07-22 15:45:10 -06:00
a6215c1b92
Remove unnecessary poh_recorder read lock acquire ( #26743 )
...
Remove unnecessary acquiring of poh_recorder read lock
2022-07-22 15:23:05 -05:00
333a48e4e2
Add comment for forwarder buffer iteration behavior ( #26721 )
...
Add comment of forwarder buffer iteration behavior
2022-07-22 01:15:17 +00:00
932abe98a7
Switch UDP stats from usize to u64 ( #26700 )
2022-07-20 15:20:30 -07:00
39a34db52a
cleanup feature: default units per instruction ( #26684 )
2022-07-20 19:13:34 +00:00
502f249904
Add proc net dev metrics to net stats ( #26603 )
...
* Add proc net dev metrics to net stats
2022-07-20 11:44:36 -07:00
4a7fb2a808
Revert "core: disable quic servers on mainnet-beta" ( #26216 )
...
Enable QUIC server
2022-07-20 20:37:24 +02:00
263911e7fd
save off what we find when calculating hash ( #26663 )
2022-07-19 09:55:52 -05:00
2dd8573287
removes erroneous allow(dead_code) annotations from core ( #26660 )
2022-07-18 17:15:47 +00:00
22d465cd57
Share function to get priority details from various transaction types ( #26643 )
2022-07-15 18:17:22 -05:00
47716a5e01
async hash verify on load ( #26208 )
...
* verify accounts hash in bg on startup
* fix some tests and loading from genesis
* add extra state for when background thread has completed
2022-07-15 14:29:56 -05:00
f13b5c832d
Remove obsoleted metrics reporting to reduce lock contention on cost_model ( #26608 )
...
remove obsoleted metrics reporting to reduce lock contention on cost_model
2022-07-14 23:02:49 -05:00
49a112ae74
Use pubkey of peer for active QUIC connection table ( #26597 )
...
* Use pubkey of peer for active QUIC connection table
* clippy
* update code
2022-07-13 09:59:01 -07:00
bf14440895
clean up and optimize account hash verify ( #26560 )
...
* remove unused code
* extract test related fault hash inject fn
* use rotate to optimize hashes removal
* use rotate to optimize snapshot hashes removal
* address code reveiw feedbacks
* revise comments
* inline
2022-07-12 19:27:28 +00:00
fd5df1cf25
Bump Version to 1.11.4 ( #26578 )
...
Co-authored-by: willhickey <willhickey@users.noreply.github.com>
2022-07-11 23:30:38 -05:00
ea7448c568
Use client certs in QUIC to get peer's stake ( #26477 )
...
* Use client certs in QUIC to get peer's stake
* fixes to cert processing
* integrate the code
* clippy
* more cleanup
* sort cargo deps
* test fixes
* info -> debug
2022-07-11 18:06:40 +00:00
a3b094300b
Remove sender stakes from banking_stage buffer prioritization ( #26512 )
...
* remove sender stakes from banking_stage buffer prioritization
2022-07-11 12:46:15 -05:00
ee0a40937e
Add validator argument log_messages_bytes_limit to change log truncation limit.
...
Add new cli argument log_messages_bytes_limit to solana-validator to control how long program logs can be before truncation
2022-07-11 10:53:18 -05:00
ba785cf8ab
removes erroneous uses of std::mem::swap ( #26536 )
...
All instances should be replace by std::mem::{replace,take},
or just plain assignment.
2022-07-11 11:33:15 +00:00
275e47f931
do this right: add 2nd pass at hash calc when failure seen ( #26392 ) ( #26538 )
2022-07-10 23:10:22 -05:00
602da5e51f
add accounts db config to bank tests ( #26517 )
2022-07-10 19:42:06 -05:00
734fedea4c
Create a more compact vote state update transaction ( #26092 )
...
* Create a more compact vote state update transaction
* pr comments
* change root to not be an option and update abi
2022-07-07 22:29:02 -07:00
5bffee248c
Cleanup repair logging ( #26461 )
2022-07-07 15:02:43 -05:00
9d937fb8a0
Bump Version to 1.11.3 ( #26481 )
...
Co-authored-by: willhickey <willhickey@users.noreply.github.com>
2022-07-07 14:39:46 -05:00
5e8e1beeb5
chore: bump serial_test from 0.6.0 to 0.8.0 ( #26463 )
...
Bumps [serial_test](https://github.com/palfrey/serial_test ) from 0.6.0 to 0.8.0.
- [Release notes](https://github.com/palfrey/serial_test/releases )
- [Commits](https://github.com/palfrey/serial_test/compare/v0.6.0...v0.8.0 )
---
updated-dependencies:
- dependency-name: serial_test
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-07 10:52:20 -06:00
6f4838719b
decouples shreds sig-verify from tpu vote and transaction packets ( #26300 )
...
Shreds have different workload and traffic pattern from TPU vote and
transaction packets. Some of recent changes to SigVerifyStage are not
suitable or at least optimal for shreds sig-verify; e.g. random discard,
dedup with false positives, discard excess by IP-address, ...
SigVerifier trait is meant to abstract out the distinctions between the
two pipelines, but in practice it has led to more verbose and convoluted
code.
This commit discards SigVerifier implementation for shreds sig-verify
and instead provides a standalone stage for verifying shreds signatures.
2022-07-07 11:13:13 +00:00
d33c548660
bypasses window-service stage before retransmitting shreds ( #26291 )
...
With recent patches, window-service recv-window does not do much other
than redirecting packets/shreds to downstream channels.
The commit removes window-service recv-window and instead sends
packets/shreds directly from sigverify to retransmit-stage and
window-service insert thread.
2022-07-06 11:49:58 +00:00
37f4621c06
chore: bump serde from 1.0.137 to 1.0.138 ( #26421 )
...
* chore: bump serde from 1.0.137 to 1.0.138
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.137 to 1.0.138.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.137...v1.0.138 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-07-05 23:18:08 -06:00
c1d89ad749
forward packets by prioritization in desc order ( #25406 )
...
- Forward packets by prioritization in desc order
- Add support of cost-tracking by transaction requested compute units
- Hook up account buckets to forwarder
- Add metrics for forwardable batches count
- Remove redundant invalid packets filtering at end of slot since forwarder will do the same when batch forwardable packets
- Add bench test for forwarding
2022-07-05 23:24:58 -05:00
8eba4d1698
add 2nd pass at hash calc when failure seen ( #26392 )
2022-07-05 18:01:02 -05:00
d3a14f5b30
simplifies packet/shred sanity checks ( #26356 )
2022-07-05 21:41:19 +00:00
ce39c14025
Add end-to-end replay slot metrics ( #25752 )
2022-07-05 13:58:51 -05:00
7e4a5de99c
Refactor ConnectionCache::use_quic ( #26235 )
...
* Remove UseQuic type
Move to storing the UdpSocket on ConnectionCache and accepting a bool
* Remove use_quic from ConnectionCache constructor
Replace with separate with_udp constructor to force callers to choose
2022-07-05 10:49:42 -07:00
61f0a7d9c3
replaces Mutex<PohRecorder> with RwLock<PohRecorder> ( #26370 )
...
Mutex causes superfluous lock contention when a read-only reference suffices.
2022-07-05 14:29:44 +00:00
9ec38a3191
Cleanup snapshot integration tests ( #26390 )
2022-07-05 09:23:23 -05:00
94685e1222
Implement randomized pruning of QUIC connection from staked peers ( #26299 )
2022-06-30 17:56:15 -07:00
88599fd760
skips shreds deserialization before retransmit ( #26230 )
...
Fully deserializing shreds in window-service before sending them to
retransmit stage adds latency to shreds propagation.
This commit instead channels through the payload and relies on only
partial deserialization of a few required fields: slot, shred-index,
shred-type.
2022-06-30 12:13:00 +00:00
4563bf40f6
cleanup feature: tx-wide-compute-cap ( #26326 )
2022-06-29 23:54:45 -07:00
557bf6e656
allow initial hash calc to occur in bg ( #26271 )
...
* allow initial hash calc to occur in bg
* validator_initialized -> startup_verification_complete
* add infos for leader and vote
* rework snapshot for startup verification
* change to assert
2022-06-29 16:48:33 -05:00
f875733a9e
patches bug in retransmit stats where slot stats are erroneously dropped ( #26317 )
...
slot_stats are submitted at a different cadence from the rest of
RetransmitStats. Current code erroneously clears slot_stats before
submitting any metrics.
2022-06-29 21:35:58 +00:00
b3406b5b2a
removes IndexedParallelIterator::with_min_len from retransmit ( #26305 )
...
Testing on mainnet-beta, with_min_len does not seem to have much impact
in the current retransmit code.
2022-06-29 13:27:17 +00:00
348fe9ebe2
verifies shred slot and parent in fetch stage ( #26225 )
...
Shred slot and parent are not verified until window-service where
resources are already wasted to sig-verify and deserialize shreds.
This commit moves above verification to earlier in the pipeline in fetch
stage.
2022-06-28 12:45:50 +00:00
9765034e04
Enable wire compression in Solana CLI and Rust client ( #26236 )
2022-06-27 15:38:07 -07:00
39ca788b95
discards shreds in sigverify if the slot leader is the node itself ( #26229 )
...
Shreds are dropped in window-service if the slot leader is the node
itself:
https://github.com/solana-labs/solana/blob/cd2878acf/core/src/window_service.rs#L181-L185
However this is done after wasting resources verifying signature on
these shreds, and requires a redundant 2nd lookup of the slot leader.
This commit instead discards such shreds in sigverify stage where we
already know the leader for the slot.
2022-06-27 20:12:23 +00:00
67936aaa74
moves Shred::seed to ShredId and adds test coverage ( #26251 )
...
Following commits will skip shreds deserializaton before retransmit, and
so we will only have a ShredId and not a fully deserialized shred to
obtain the shuffling seed from.
2022-06-27 17:58:43 +00:00
662818ef0d
Use `VoteAccount::node_pubkey()` ( #26207 )
2022-06-27 09:09:06 -05:00
d5efbdb19b
Add timing measurement for gossip vote txn processing ( #26163 )
...
* add timing for gossip vote txn processing
* fix build
* fix too many arg error in clippy
* atomic interval
2022-06-27 08:53:34 -05:00
cd2878acf9
Avoid to miss to root for local slots before the hard fork ( #19912 )
...
* Make sure to root local slots even with hard fork
* Address review comments
* Cleanup a bit
* Further clean up
* Further clean up a bit
* Add comment
* Tweak hard fork reconciliation code placement
2022-06-26 15:14:17 +09:00
30d2b112e4
bypasses rayon thread-pool for small retransmit shred batches ( #26222 )
...
In order to preserve current behavior, the threshold is set to the
current value of the argument to IndexedParallelIterator::with_min_len.
Follow up commits will recalibrate this threshold to optimize
performance on mainnet-beta.
2022-06-25 21:15:42 +00:00
7cd7173b71
Refactor: Add get_delegated_stake method to VoteAccounts ( #26221 )
2022-06-25 16:41:35 +00:00
44d1e62007
Refactor: No need to return stake in Bank::get_vote_account ( #26220 )
2022-06-25 16:27:43 +00:00
f1b82ec44d
factors out common retransmit work for shreds of the same slot ( #26218 )
...
Shreds arriving at a node for retransmit tend to belong to the same slot
(or a just a couple of different slots). Slot leader and cluster nodes
are common for the shreds of the same slot, and so the common work to
look up these values can be factored out.
This commit first group-bys shreds by slot to factor out that common
lookup work.
2022-06-25 15:49:05 +00:00
a3395a786a
vote_account uses AccountSharedData to avoid copies ( #23687 )
...
* vote_account uses AccountSharedData to avoid copies
* simpler deserialize
2022-06-24 15:08:01 -05:00
877fedadac
Remove StatusCacheRc type and use StatusCache directly ( #26184 )
2022-06-24 08:38:56 -05:00
23c50a2389
Add StatusCache::root_slot_deltas() and use it ( #26170 )
2022-06-23 15:19:06 -05:00
a6ba5a9a05
Add transaction index in slot to geyser plugin TransactionInfo ( #25688 )
...
* Define shuffle to prep using same shuffle for multiple slices
* Determine transaction indexes and plumb to execute_batch
* Pair transaction_index with transaction in TransactionStatusService
* Add new ReplicaTransactionInfoVersion
* Plumb transaction_indexes through BankingStage
* Prepare BankingStage to receive transaction indexes from PohRecorder
* Determine transaction indexes in PohRecorder; add field to WorkingBank
* Add PohRecorder::record unit test
* Only pass starting_transaction_index around PohRecorder
* Add helper structs to simplify test DashMap
* Pass entry and starting-index into process_entries_with_callback together
* Add tx-index checks to test_rebatch_transactions
* Revert shuffle definition and use zip/unzip
* Only zip/unzip if randomize
* Add confirm_slot_entries test
* Review nits
* Add type alias to make sender docs more clear
2022-06-23 13:37:38 -06:00
f534b8981b
maps number of data shreds to erasure batch size ( #25917 )
...
In prepration of
https://github.com/solana-labs/solana/pull/25807
which reworks erasure batch sizes, this commit:
* adds a helper function mapping the number of data shreds to the
erasure batch size.
* adds ProcessShredsStats to Shredder::entries_to_shreds in order to
replace and remove entries_to_data_shreds from the public interface.
2022-06-23 13:27:54 +00:00
5c2f819f99
Bump Version to 1.11.2 ( #26159 )
2022-06-22 21:16:18 -05:00
55a7e53b9e
chore: bump lru from 0.7.6 to 0.7.7 ( #26140 )
...
* chore: bump lru from 0.7.6 to 0.7.7
Bumps [lru](https://github.com/jeromefroe/lru-rs ) from 0.7.6 to 0.7.7.
- [Release notes](https://github.com/jeromefroe/lru-rs/releases )
- [Changelog](https://github.com/jeromefroe/lru-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/jeromefroe/lru-rs/compare/0.7.6...0.7.7 )
---
updated-dependencies:
- dependency-name: lru
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-06-22 14:39:07 -06:00
bafdb7dd62
Revert handle start_http failure in rpc_service ( #25400 ) ( #26130 )
...
* revert e263be2000
2022-06-22 10:52:27 -07:00
f3639b76ce
Remove some clippy lints
2022-06-22 09:23:22 -07:00
b5d0c7b468
Revert "tvu and tpu timeout on joining its microservices ( #24111 )" ( #26132 )
...
This reverts commit e105547c14
2022-06-22 10:57:46 -05:00
faa6c32162
removes packet modifier from shred_fetch_stage
...
... in favor of just passing packet flags.
2022-06-22 12:17:37 +00:00
1f0f5dc03e
verifies shred-version in fetch stage
...
Shred versions are not verified until window-service where resources are
already wasted to sig-verify and deserialize shreds.
The commit verifies shred-version earlier in the pipeline in fetch stage.
2022-06-22 12:17:37 +00:00
43ff65ece9
Use single send socket in UdpTpuConnection ( #26105 )
2022-06-21 14:56:21 -07:00
75425521b4
moves slot updates notifications after shreds retransmit ( #26094 )
...
RetransmitSlotStats can already be utilized to track when the first
shred for a slot was received; therefore
first_shreds_received: &Mutex<BTreeSet<Slot>>
is redundant. Sending update notifications after shreds retransmit will
also bypass the need for a mutex.
2022-06-21 17:19:40 -04:00
61946a49c3
Weight concurrent streams by stake ( #25993 )
...
Weight concurrent streams by stake for staked nodes
Ported changes from #25056 after address merge conflicts and some refactoring
2022-06-21 12:06:44 -07:00
51f26dc96e
Bump version to 1.11.1 ( #26104 )
2022-06-21 12:07:46 -05:00
d2afa6b418
moves packet-hasher out of the mutex ( #26091 )
...
Packet-hasher is not mutated across threads and does not need to be
wrapped in a mutex.
2022-06-21 16:29:27 +00:00
e344c8476f
Do not use UdpTpuConnection to forward votes ( #26082 )
...
* Do not use UdpTpuConnection to forward votes
* fix tests
2022-06-21 05:56:11 -07:00
611d2ec73c
core: fix double-readlock in validator ( #26053 )
2022-06-20 15:07:00 +00:00
47e62add5b
removes feature gate code adding shred-type to shred seed ( #25963 )
...
The feature is already activated on all clusters, and does not impact
processing of ledger/snapshots.
2022-06-20 14:39:24 +00:00
a5f290a66f
core: disable quic servers on mainnet-beta
2022-06-17 20:04:05 -06:00
b3d1f8d1ac
tracks number of shreds sent and received at different distances from the root ( #25989 )
2022-06-17 21:33:23 +00:00
eacb9183d4
patches bug where the 1st coding shred is not inserted into blockstore ( #25916 )
...
StandardBroadcastRun::insert skips 1st shred with index zero because
the 1st *data* shred is inserted synchronously:
https://github.com/solana-labs/solana/blob/53695ecd2/core/src/broadcast_stage/standard_broadcast_run.rs#L239-L246
https://github.com/solana-labs/solana/blob/53695ecd2/core/src/broadcast_stage/standard_broadcast_run.rs#L334-L339
https://github.com/solana-labs/solana/pull/7481
which added this code was not inserting coding shreds into blockstore.
Starting with
https://github.com/solana-labs/solana/pull/8899
coding shreds are inserted into blockstore as well as data shreds, but
the insert logic erroneously skips first coding shred because it does
not check if shred is code or data.
2022-06-16 13:59:15 +00:00
fe3c1d3d49
removes erroneous uses of &Arc<...> from broadcast-stage ( #25962 )
2022-06-15 13:44:24 +00:00
db9004bd0f
Fix doc warnings ( #25953 )
2022-06-14 21:55:08 -06:00
c96d9d127a
Include forwarding counters in leader slot metrics ( #25874 )
...
* To include forwarding counters in leader slot metrics
* Capture slot_end_detected time when checking leader slots, to be used in reporting later
* Simplify banking stage loop to report leader slot metrics
Co-authored-by: carllin <carl@solana.com>
2022-06-13 17:03:34 -05:00
ace24a7c82
A default tower is no longer considered to contain a stray last vote
2022-06-10 14:17:26 -07:00
29b597cea5
Connection pool support in connection cache and QUIC connection reliability improvement ( #25793 )
...
* Connection pool in connection cache and handle connection errors
1. The connection not has a pool of connections per address, configurable, default 4
2. The connections per address share a lazy initialized endpoint
3. Handle connection issues better, avoid race conditions
4. Various log improvement for help debug connection issues
2022-06-10 09:25:24 -07:00
ee4469c882
Skip compaction in backup_and_clear_blockstore ( #25810 )
...
#### Problem
blockstore clean and compact is quite slow with wait-for-supermajority purge and can take 20-30 minutes
as described in #25710 .
#### Summary of Changes
This PR removes the compaction logic in backup_and_clear_blockstore as the
actual the restoration from a bad fork is handled by `blockstore.purge_slots`
(which is done by issuing rocksdb range-delete that makes the bad fork
unavailable.)
Compaction is irreverent to the shred version, as its main job in this context
is to reclaim disk storage from the deleted slots, which we can let the rocksdb
automatic background compaction to handle it.
Fixes #25710
2022-06-09 17:11:50 +08:00
bf8faa8a30
Report banking stage tracer metrics ( #25620 )
2022-06-09 00:25:37 -05:00
79a8ecd0ac
client: Remove static connection cache, plumb it instead ( #25667 )
...
* client: Remove static connection cache, plumb it instead
* Add TpuClient::new_with_connection_cache to not break downstream
* Refactor get_connection and RwLock into ConnectionCache
* Fix merge conflicts from new async TpuClient
* Remove `ConnectionCache::set_use_quic`
* Move DEFAULT_TPU_USE_QUIC to client, use ConnectionCache::default()
2022-06-08 13:57:12 +02:00
6c9f2eac78
removes fec_set_offset from UnfinishedSlotInfo ( #25815 )
...
If the blockstore has shreds for a slot, it should not recreate the
slot:
https://github.com/solana-labs/solana/blob/ff68bf6c2/ledger/src/leader_schedule_cache.rs#L142-L146
https://github.com/solana-labs/solana/pull/15849/files#r596657314
Therefore in broadcast stage if UnfinishedSlotInfo is None, then
fec_set_offset will be zero:
https://github.com/solana-labs/solana/blob/ff68bf6c2/core/src/broadcast_stage/standard_broadcast_run.rs#L111-L120
As a result fec_set_offset will always be zero, and is so redundant and
can be removed.
2022-06-07 22:17:37 +00:00
ba04063956
Add CPUmetrics ( #25802 )
...
Add in some CPU utilization metrics such as: number of vCPUs, clock frequency, average load across different time intervals, and number of total threads
2022-06-07 11:34:25 -07:00
e6c21a3036
Convert Measure::this to measure! and remove Measure::this ( #25776 )
...
* Remove the args param from Measure::this since we don't ever use it
* banking_stage.rs: convert to measure!
* poh_recorder.rs: convert to measure!
* cost_update_service.rs: convert to measure!
* poh_service.rs: convert to measure!
* bank.rs: convert to measure!
* measure.rs: Remove Measure::this now that all have been converted to measure!
2022-06-06 20:21:05 -05:00
447a3239e7
Add new replay metrics for replay blockstore_into_bank and complete ( #25717 )
2022-06-03 19:45:27 +02:00
5dbf7d8f91
removes raw indexing into packet data ( #25554 )
...
Packets are at the boundary of the system where, vast majority of the
time, they are received from an untrusted source. Raw indexing into the
data buffer can open attack vectors if the offsets are invalid.
Validating offsets beforehand is verbose and error prone.
The commit updates Packet::data() api to take a SliceIndex and always to
return an Option. The call-sites are so forced to explicitly handle the
case where the offsets are invalid.
2022-06-03 01:05:06 +00:00
81231a89b9
adds support for different variants of ShredCode and ShredData
...
The commit implements two new types:
pub enum ShredCode {
Legacy(legacy::ShredCode),
}
pub enum ShredData {
Legacy(legacy::ShredData),
}
Following commits will extend these types by adding merkle variants:
pub enum ShredCode {
Legacy(legacy::ShredCode),
Merkle(merkle::ShredCode),
}
pub enum ShredData {
Legacy(legacy::ShredData),
Merkle(merkle::ShredData),
}
2022-06-02 18:55:50 +00:00
1c2ae470c5
Fix forwarding of transactions over QUIC ( #25674 )
...
* Spawn QUIC server to receive forwarded txs
* Update validator port range
* forward votes using UDP
* no forwarding from unstaked nodes
* forwarding stats in banking stage
* fix test builds
* fix lifetime of forward sender
2022-06-02 11:14:58 -07:00
d3ac4e941b
Bench: preshrink + sigverify ( #25480 )
...
* double shrinking
* add bench
* rename
* aggregate timing
* remove pre/post shrink time
* update api after merge
2022-06-02 09:19:01 -05:00
51ac599915
Add user requested CU (eg. compute_budget.compute_unit_limit) to immutable_deserialized_packet, to be used in cost model and prioritized forwarding ( #25695 )
2022-06-01 22:43:48 +00:00
aedcb05dc8
Record solana-validator ver to metrics at startup ( #25635 )
...
* Record solana-validator ver to metrics at startup
* Update Cargo.lock
2022-06-01 13:37:50 +09:00
02b26ddd82
SigVerify: Fix num_valid_packets metric ( #25643 )
...
It used to report the number of packets with successful signature
validations but was accidentally changed to count packets passed into
the verifier by e4409a87fe
2022-05-31 18:51:20 +10:00
90a3315b69
Detect tracer key in sigverify ( #25579 )
...
* Mark the tracer transaction
* simplify tracer check
2022-05-30 18:41:54 -05:00
e4409a87fe
Add pre shrink pass before sigverify batch ( #25136 )
2022-05-28 01:51:55 +10:00
5b67960c76
(Refactor) Move blocktore options related stuff to blockstore_options.rs ( #25509 )
...
#### Problem
blockstore_db.rs has a mutual dependency between blockstore_metrics.rs.
#### Summary of Changes
This PR removes the mutual dependency by moving the option-related stuff
out from blockstore_db.rs to its new home --- blockstore_options.rs.
By doing this, we address the mutual dependency and also make the code cleaner.
2022-05-26 16:59:26 -07:00
7f4128947b
chore: bump lru from 0.7.5 to 0.7.6 ( #25572 )
...
* chore: bump lru from 0.7.5 to 0.7.6
Bumps [lru](https://github.com/jeromefroe/lru-rs ) from 0.7.5 to 0.7.6.
- [Release notes](https://github.com/jeromefroe/lru-rs/releases )
- [Changelog](https://github.com/jeromefroe/lru-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/jeromefroe/lru-rs/compare/0.7.5...0.7.6 )
---
updated-dependencies:
- dependency-name: lru
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-05-26 19:05:02 +00:00
1ca5c3a7bd
Switch to using enum-dispatch to switch between UDP and Quic ( #24713 )
2022-05-26 11:21:16 -04:00
de612c25b3
removes shred wire layout specs from sigverify ( #25520 )
...
sigverify_shreds relies on wire layout specs of shreds:
https://github.com/solana-labs/solana/blob/0376ab41a/ledger/src/sigverify_shreds.rs#L39-L46
https://github.com/solana-labs/solana/blob/0376ab41a/ledger/src/sigverify_shreds.rs#L298-L305
In preparation of
https://github.com/solana-labs/solana/pull/25237
which adds a new shred variant with different layout and signed message,
this commit removes shred layout specification from sigverify and
instead encapsulate that in shred module.
2022-05-26 13:06:27 +00:00
0efb7478cd
FindPacketSenderStake: Remove parallelism to improve performance ( #25562 )
...
* FindPacketSenderStake: Remove parallelism to improve performance
The work unit sizes were so small that using the thread pool
slowed down this stage significantly.
* fix checks
Co-authored-by: Justin Starry <justin@solana.com>
2022-05-26 21:17:52 +10:00
cafa85bfbb
includes shred-type when computing turbine broadcast seed ( #25556 )
...
Indices for code and data shreds of the same slot overlap; and so they
will have the same random number generator seed when shuffling cluster
nodes for turbine broadcast.
This results in the same propagation path for code and data shreds of
the same index and effectively smaller sample size for re-transmitter
nodes. For example a 32:32 batch (32 code + 32 data shreds), is
retransmitted through _at most_ 32 unique nodes, whereas ideally we want
~64 unique re-transmitters.
This commit adds shred-type to seed function so that code and data
sherds of the same (slot, index) will (most likely) have different
propagation paths.
2022-05-25 20:31:53 +00:00
880684565c
limits read access into Packet data to Packet.meta.size ( #25484 )
...
Bytes past Packet.meta.size are not valid to read from.
The commit makes the buffer field private and instead provides two
methods:
* Packet::data() which returns an immutable reference to the underlying
buffer up to Packet.meta.size. The rest of the buffer is not valid to
read from.
* Packet::buffer_mut() which returns a mutable reference to the entirety
of the underlying buffer to write into. The caller is responsible to
update Packet.meta.size after writing to the buffer.
2022-05-25 16:52:54 +00:00
9651cdad99
Refactor Sigverify trait ( #25359 )
2022-05-24 16:01:41 -05:00
61c5a471e8
preserve optimistic_slot in blockstore ( #25311 )
2022-05-24 12:03:28 -07:00
e66ea7cb6a
Clean up Bank::commit_transactions parameters
2022-05-24 20:24:42 +08:00
cad1c41ce2
Add Packet::deserialize_slice convenience method
2022-05-24 17:31:14 +08:00
514f73f4b1
Remove retain_mut dep ( #25494 )
2022-05-23 21:45:49 +00:00
ec7ca411dd
Make PacketBatch packets vector non-public ( #25413 )
...
Upcoming changes to PacketBatch to support variable sized packets will
modify the internals of PacketBatch. So, this change removes usage of
the internal packet struct and instead uses accessors (which are
currently just wrappers of Vector functions but will change down the
road).
2022-05-23 15:30:15 -05:00
e75569e85a
chore: bump systemstat from 0.1.10 to 0.1.11 ( #25471 )
...
Bumps [systemstat](https://github.com/unrelentingtech/systemstat ) from 0.1.10 to 0.1.11.
- [Release notes](https://github.com/unrelentingtech/systemstat/releases )
- [Commits](https://github.com/unrelentingtech/systemstat/compare/v0.1.10...v0.1.11 )
---
updated-dependencies:
- dependency-name: systemstat
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-23 20:08:47 +00:00
6429aff13b
findpacketsenderstake: add discard after receive ( #25458 )
...
This mimics a similar change in sigverify, see #25388
2022-05-23 21:27:20 +02:00
c248fb3f51
renames Packet Meta::{,set_}addr methods to {,set_}socket_addr ( #25478 )
...
In order to distinguish between Meta.addr field which is an IpAddr and
the methods which refer to a SocketAddr.
2022-05-23 15:48:59 +00:00
b05c7d91ed
Fix derive_partial_eq_without_eq clippy lint
2022-05-22 22:22:21 -07:00
e22be02d3a
sigverify: add discard before dedup ( #25388 )
2022-05-23 03:40:33 +02:00
f8842032c6
clippy: fix "this let-binding has unit value" warnings ( #25429 )
2022-05-22 12:17:59 -04:00
7fb0ef1fa5
Use async send for forwarding transactions ( #25435 )
2022-05-20 21:20:47 -07:00
e263be2000
handle start_http failure in rpc_service ( #25400 )
2022-05-20 17:59:23 -07:00
e025376719
Fix packet accounting after dedup ( #25357 )
...
* Fix packet accounting after dedup
* Rename function to better represent intent
2022-05-20 17:00:13 -07:00
2fdc850176
Use Shared IP to Stake Map ( #25377 )
...
* Find packet sender stake stage use shared IP to stake map
2022-05-20 12:51:07 -07:00
c54e06355f
voteSubscribe pubsub notification now includes the vote transaction signature ( #25291 )
2022-05-19 18:28:46 -07:00
97efbdc303
Defer tower saving until push_vote(), there's no need to do it sooner ( #25374 )
2022-05-19 18:27:58 -07:00
971748b335
fix banking stage starvation ( #25245 )
2022-05-18 22:37:47 +02:00
7402878628
chore: bump raptorq from 1.6.5 to 1.7.0 ( #25330 )
...
Bumps [raptorq](https://github.com/cberner/raptorq ) from 1.6.5 to 1.7.0.
- [Release notes](https://github.com/cberner/raptorq/releases )
- [Commits](https://github.com/cberner/raptorq/compare/v1.6.5...v1.7.0 )
---
updated-dependencies:
- dependency-name: raptorq
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-18 17:56:38 +00:00
542bd0ec3c
chore: bump rayon from 1.5.2 to 1.5.3 ( #25242 )
...
* chore: bump rayon from 1.5.2 to 1.5.3
Bumps [rayon](https://github.com/rayon-rs/rayon ) from 1.5.2 to 1.5.3.
- [Release notes](https://github.com/rayon-rs/rayon/releases )
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md )
- [Commits](https://github.com/rayon-rs/rayon/compare/v1.5.2...v1.5.3 )
---
updated-dependencies:
- dependency-name: rayon
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-05-18 09:39:57 -06:00
5548baf4dd
Don't drop transactions which use a request heap size ix ( #25315 )
2022-05-18 17:47:24 +08:00
b27125815a
Simplify logic around MAX_ORPHAN_REPAIR_RESPONSES constant ( #25032 )
2022-05-17 19:45:45 -06:00
b1b3702e6d
Prioritize transactions in banking stage by their compute unit price ( #25178 )
...
* - get prioritization fee from compute_budget instruction;
- update compute_budget::process_instruction function to take instruction iter to support sanitized versioned message;
- updated runtime.md
* update transaction fee calculation for prioritization fee rate as lamports per 10K CUs
* review changes
* fix test
* fix a bpf test
* fix bpf test
* patch feedback
* fix clippy
* fix bpf test
* feedback
* rename prioritization fee rate to compute unit price
* feedback
Co-authored-by: Justin Starry <justin@solana.com>
2022-05-16 12:06:33 +08:00
52db2e19bc
Lower default batch size to 64 and add 2 banking threads ( #25226 )
2022-05-15 16:52:47 +02:00
3d96a1ab76
Block packets in vote-only mode ( #24906 )
2022-05-14 17:53:37 +02:00
71dd95e842
Tune banking_stage receive loop timing ( #25172 )
2022-05-13 03:42:08 +00:00
896729f25e
keep track of oldest slot used by last hash calculation ( #25152 )
2022-05-12 11:18:08 -05:00
3a4f0d3397
println -> info ( #25163 )
2022-05-12 11:07:13 -05:00
bcf4d54235
Update test_banking_stage_entryfication to be more deterministic ( #25146 )
...
* Update test_banking_stage_entryfication to be more deterministic
* revert to original test with updated checks
2022-05-12 15:36:19 +00:00
41d34d45e0
pass exit by ref ( #25120 )
2022-05-11 09:17:21 -05:00
2fa9bc3e70
Add options to store full and/or incremental snapshots in separate locations ( #24247 )
2022-05-10 16:37:41 -04:00
e3bdc38f0a
Add sanitized types for use in banking stage ( #25067 )
2022-05-11 00:30:48 +08:00
de96663cc4
fix typo ( #25083 )
2022-05-09 12:42:58 -05:00
362b0526cd
Greedy receive in banking stage ( #25060 )
...
* Greedy receive in banking stage
* add upperbound to batch size and batching time
* update test_banking_stage_entryfication test
2022-05-08 10:47:55 -07:00
8e37e364b1
fix typo in measure name ( #25058 )
2022-05-07 10:04:42 -05:00
c920d411f7
Clean up logging and make variables consistent ( #25049 )
2022-05-07 03:52:45 +08:00
082502d4f3
Fail tx sanitization when ix program id uses lookup table ( #25035 )
...
* Fail tx sanitization when ix program id uses lookup table
* feedback
2022-05-07 03:19:50 +08:00
cb6cd5d60f
FindPacketSenderStake: Improve metrics ( #24971 )
...
- separate names for vote and non-vote thread
- time unit postfixes (one is in ns!)
2022-05-06 21:16:13 +02:00
492f89a170
checks account owner when initializing a vote-account ( #25018 )
...
A VoteAccount may only wrap an account if the account owner is
solana_vote_program:id or equivalently this check returns true:
solana_vote_program::check_id(account.owner())
2022-05-06 16:22:49 +00:00
c4aa082efc
Rollback tokio to LTS release v1.14 ( #25028 )
2022-05-06 06:34:49 +00:00
a01291069a
initializes thread-pools with lazy_static instead of thread_local ( #24853 )
...
In addition to thread_local -> lazy_static change, a number of thread-pools are
initialized with get_max_thread_count to achieve parity with the older code in
terms of number of validator threads.
2022-05-05 20:00:50 +00:00
7100f1c94b
Collect stats in streamer receiver and report fetch stage metrics ( #25010 )
2022-05-06 02:56:18 +08:00
870ac80b79
Prioritize BankingStage packets individually in min-max heap ( #24187 )
2022-05-04 21:50:56 -05:00
bece7f32c8
chore: bump log from 0.4.16 to 0.4.17 ( #24987 )
...
* chore: bump log from 0.4.16 to 0.4.17
Bumps [log](https://github.com/rust-lang/log ) from 0.4.16 to 0.4.17.
- [Release notes](https://github.com/rust-lang/log/releases )
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/log/commits/0.4.17 )
---
updated-dependencies:
- dependency-name: log
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-05-05 01:49:18 +00:00
61a9faae17
chore: bump serde_json from 1.0.80 to 1.0.81 ( #24960 )
...
* chore: bump serde_json from 1.0.80 to 1.0.81
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.80 to 1.0.81.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.80...v1.0.81 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2022-05-04 14:57:38 -06:00
9258d81ba3
chore: bump serde from 1.0.136 to 1.0.137 ( #24957 )
...
* chore: bump serde from 1.0.136 to 1.0.137
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.136 to 1.0.137.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.136...v1.0.137 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-05-03 22:14:59 -06:00
2c9d2a2140
chore: bump serde_json from 1.0.79 to 1.0.80 ( #24943 )
...
* chore: bump serde_json from 1.0.79 to 1.0.80
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.79 to 1.0.80.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.79...v1.0.80 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-05-03 13:46:45 -06:00
503d0baf6d
SigVerify: Add total time metrics for dedup/discard/verify ( #24768 )
...
* SigVerify: Add total time metrics for dedup/discard/verify
Previously it was impossible to determine the total time the stage spent
on these activities within a measurement window.
* SigVerify: Add _us postfix to time metrics
2022-05-03 14:59:25 +02:00
eff59193db
enforces that LAST_SHRED_IN_SLOT is also DATA_COMPLETE_SHRED ( #24892 )
...
A data shred cannot be LAST_SHRED_IN_SLOT if not also DATA_COMPLETE_SHRED.
So LAST_SHRED_IN_SLOT should also imply DATA_COMPLETE_SHRED:
https://github.com/solana-labs/solana/blob/74b586ae7/ledger/src/shredder.rs#L116-L117
https://github.com/solana-labs/solana/blob/74b586ae7/core/src/broadcast_stage/standard_broadcast_run.rs#L80-L81
However current shred constructs allow specifying a shred which is
LAST_SHRED_IN_SLOT but not DATA_COMPLETE_SHRED:
https://github.com/solana-labs/solana/blob/74b586ae7/ledger/src/shred.rs#L117-L118
https://github.com/solana-labs/solana/blob/74b586ae7/ledger/src/shred.rs#L272-L273
The commit updates ShredFlags so that if a shred is not
DATA_COMPLETE_SHRED it cannot be LAST_SHRED_IN_SLOT either.
2022-05-02 23:33:53 +00:00
112a0b475a
Revert "Refactor to use EpochSchedule from within RentCollector struct" ( #24893 )
...
* Revert "Ran cargo fmt"
This reverts commit 9052e41b324dfeab3b3887fb78dc56c2f7f2fff836c0bdaa78ed7813e6985472d2e605
2022-05-02 13:46:17 -05:00
e812430e28
defines shred flags using bitflags crate ( #24874 )
...
Shred flags uses raw bit-masking ops which lacks type-safety:
https://github.com/solana-labs/solana/blob/a829ddc92/ledger/src/shred.rs#L112-L114
This commit instead uses bitflags crate to define shred flags.
2022-05-01 19:25:15 +00:00
88c16c0176
Check if quic is enabled before warming up quic connections ( #24821 )
...
* Check if quic is enabled before warming up quic connections
* fix after rebase
* don't start warmup service if quic not enabled
* fix test
2022-05-01 03:52:38 +00:00
a61652104b
Avoid holding lock guards in match expressions ( #24805 )
...
* Avoid holding bank forks read lock for RPC requests
* Avoid using lock guards in temporaries
* revert fetch stage change
2022-04-29 16:32:46 +08:00
4e58b3870c
Update all BankForks methods to return owned values ( #24801 )
2022-04-28 18:51:00 +00:00
b22a14ca68
chore: bump etcd-client from 0.9.0 to 0.9.1 ( #24774 )
...
* chore: bump etcd-client from 0.9.0 to 0.9.1
Bumps [etcd-client](https://github.com/etcdv3/etcd-client ) from 0.9.0 to 0.9.1.
- [Release notes](https://github.com/etcdv3/etcd-client/releases )
- [Commits](https://github.com/etcdv3/etcd-client/compare/v0.9.0...v0.9.1 )
---
updated-dependencies:
- dependency-name: etcd-client
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-04-28 12:30:37 -06:00
5a430c15e2
Separate sigverify metrics for each verifier ( #24744 )
2022-04-28 01:16:17 -07:00
0f60665100
replaces Shred::new_empty_coding with Shred::new_from_parity_shard ( #24749 )
...
Removing implementation details of shreds and payload offsets from
shredder, so that shredder does not need to mutate payload:
https://github.com/solana-labs/solana/blob/71ad12128/ledger/src/shred.rs#L968-L977
Also, Shred::new_from_data can simply obtain a slice as opposed to
Option<&[u8]>:
https://github.com/solana-labs/solana/blob/71ad12128/ledger/src/shred.rs#L268-L278
2022-04-27 18:04:10 +00:00
081c844d6e
removes Shred::new_empty_data_shred ( #24714 )
...
Shred::new_empty_data_shred returns an invalid shred (i.e.
shred.sanitize() returns error). The method is only used in tests and
can be easily replaced with Shred::new_from_data. To keep the shred api
surface small, this commit removes this method.
2022-04-26 23:13:12 +00:00
12ae8d3be5
returns Error when Shred::sanitize fails ( #24653 )
...
Including the error in the output allows to debug when Shred::sanitize
fails.
2022-04-25 23:19:37 +00:00
895f76a93c
hides implementation details of shred from its public interface ( #24563 )
...
Working towards embedding versioning into shreds binary, so that a new
variant of shred struct can include merkle tree hashes of the erasure
set.
2022-04-25 12:43:22 +00:00
8a062273de
Move error counters to be reported by leader only at end of slot ( #24581 )
...
* Add error counters to leader metrics only
* Add dependencies
2022-04-23 18:10:47 -05:00
d0a8a16a57
ReplayStage no longer relies on Validator to reset the poh recorder at start
2022-04-22 21:17:49 -07:00
84e3342612
Process blockstore after starting the TVU
2022-04-22 21:17:49 -07:00
83e041299a
Run real snapshot packager while processing blockstore at validator startup
2022-04-22 21:17:49 -07:00
2d4defa477
fix typo ( #24576 )
2022-04-22 08:43:57 -05:00
0d51596224
sim: Override slot hashes account on simulation bank ( #24543 )
...
* sim: Override slot hashes during simulation
* Add simulation test program
* Address feedback
* Add AccountOverrides explicit type
* Cargo fmt
2022-04-22 12:32:31 +02:00
c544742091
Local cluster test cleanup and refactoring ( #24559 )
...
* remove FixedSchedule.start_epoch
* use duration for timing
* Rename to partition bool to turbine_disabled
* simplify partition config
2022-04-22 12:14:07 +08:00
454ef38e43
caches StakeAccount instead of Delegation in Stakes
...
The commit makes values in stake_delegations map in Stakes struct
generic. Stakes<Delegation> is equivalent to the old code and is used
for backward compatibility in BankFieldsTo{Serialize,Deserialize}.
But banks cache Stakes<StakeAccount> which includes the entire stake
account and StakeState deserialized from account. Doing so, will remove
the need to load stake account from accounts-db when working with
stake-delegations.
2022-04-21 15:28:41 +00:00
02bfb85c16
Refactor transaction processing in banking stage ( #24336 )
...
* Refactor transaction processing in banking stage
* feedback
* more feedback
2022-04-21 21:06:26 +08:00
d5127abf46
Only add hashes for completed blocks to recent blockhashes ( #24389 )
...
* Only add hashes for completed blocks to recent blockhashes
* feedback
2022-04-21 21:05:29 +08:00
a21fc3f303
Apply transaction actual execution units to cost_tracker ( #24311 )
...
* Pass the sum of consumed compute units to cost_tracker
* cost model tracks builtins and bpf programs separately, enabling adjust block cost by actual bpf programs execution costs
* Copied nightly-only experimental `checked_add_(un)signed` implementation to sdk
* Add function to update cost tracker with execution cost adjustment
* Review suggestion - using enum instead of struct for CommitTransactionDetails
Co-authored-by: Justin Starry <justin.m.starry@gmail.com>
* review - rename variable to distinguish accumulated_consumed_units from individual compute_units_consumed
* not to use signed integer operations
* Review - using saturating_add_assign!(), and checked_*().unwrap_or()
* Review - using Ordering enum to cmp
* replace checked_ with saturating_
* review - remove unnecessary Option<>
* Review - add function to report number of non-zero units account to metrics
2022-04-21 07:38:07 +00:00
79923c3b58
Refactor: Rename BlockhashQueue fields and methods for clarity ( #24426 )
2022-04-21 11:57:17 +08:00
d0761d0ca4
demote receive_window_num_slot_shreds to debug logging ( #24505 )
2022-04-20 08:51:46 -05:00
dd15193c69
chore: bump rayon from 1.5.1 to 1.5.2 ( #24470 )
...
* chore: bump rayon from 1.5.1 to 1.5.2
Bumps [rayon](https://github.com/rayon-rs/rayon ) from 1.5.1 to 1.5.2.
- [Release notes](https://github.com/rayon-rs/rayon/releases )
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md )
- [Commits](https://github.com/rayon-rs/rayon/commits )
---
updated-dependencies:
- dependency-name: rayon
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-04-19 23:17:52 -06:00
05f32f287c
solana-validator monitor now reports slot-level progress while loading blockstore
2022-04-19 22:09:48 -07:00
9e4999ef6a
Remove halt_at_slot from RuntimeConfig, it's not a runtime concern
2022-04-19 19:23:58 -07:00
988210908c
Move verify_udp_stats_access out of the way
2022-04-19 19:23:58 -07:00
c6f3da4879
blockstore_processor now accepts an Arc<Rwlock<BankForks>>
2022-04-19 19:23:58 -07:00
0e2e0c8b7d
Extract most storage-related services from the Tvu abstraction
2022-04-19 19:23:58 -07:00
268a2109de
Relocate hard forks info log
2022-04-19 19:23:58 -07:00
dd766042df
Remove LedgerMetricReportService from TVU
2022-04-19 19:23:58 -07:00
705ea53353
moves sign_shred and new_coding_shred_header out of Shredder ( #24487 )
2022-04-19 20:00:05 +00:00
94b0186a96
Cost model tracks builtins and bpf programs separately ( #24468 )
...
* Cost model tracks builtins and bpf programs separatele (enables adjusting block cost by actual bpf programs execution costs)
* Address reviews: expand test; add metrics stat
2022-04-19 13:25:47 -05:00
3bbfaae7b6
moves shred stats to a separate file ( #24484 )
2022-04-19 18:25:09 +00:00
d9d0dad258
report swap mem as bytes like other metrics ( #24455 )
2022-04-19 10:03:25 -05:00
039488b562
drops redundant turbine propagation path ( #24351 )
...
Most nodes in the cluster receive the same shred from two different
nodes: parent, and the first node of their neighborhood:
https://github.com/solana-labs/solana/blob/a8c695ba5/core/src/cluster_nodes.rs#L178-L197
Because of the erasure codings, half of the shreds are already
redundant. So this redundant propagation path will only add extra
overhead.
Additionally the very first node of the broadcast tree has 2x fanout
(i.e. 400 nodes) which adds too much load at one node.
This commit simplifies the broadcast tree by dropping the redundant
propagation path and removing the 2x fanout at root node.
2022-04-19 00:11:29 +00:00
1d50832389
replaces counters with datapoints in gossip metrics ( #24451 )
2022-04-18 23:14:59 +00:00
c2f7f2fff8
Remove redundant epoch_schedule from AccountsPackage
2022-04-18 11:57:40 -05:00
5472d2e605
Removing redundant EpochSchedule param from fns
2022-04-18 11:57:40 -05:00
d2c6c04d3e
banking-bench: Add and rearrange options
...
- Add write-lock-contention option, replacing same_payer
- write-lock-contention also has a same-batch-only value, where
contention happens only inside batches, not between them
- Rename num-threads to batches-per-iteration, which is closer to what
it is actually doing.
- Add num-banking-threads as a new option
- Rename packets-per-chunk to packets-per-batch, because this is closer
to what's happening; and it was previously confusing that num-chunks
had little to do with packets-per-chunk.
Example output for a iterations=100 and a permutation of inputs:
contention,threads,batchsize,batchcount,tps
none, 3,192, 4,65290.30
none, 4,192, 4,77358.06
none, 5,192, 4,86436.65
none, 3, 12,64,43944.57
none, 4, 12,64,65852.15
none, 5, 12,64,70674.37
same-batch-only,3,192, 4,3928.21
same-batch-only,4,192, 4,6460.15
same-batch-only,5,192, 4,7242.85
same-batch-only,3, 12,64,11377.58
same-batch-only,4, 12,64,19582.79
same-batch-only,5, 12,64,24648.45
full, 3,192, 4,3914.26
full, 4,192, 4,2102.99
full, 5,192, 4,3041.87
full, 3, 12,64,11316.17
full, 4, 12,64,2224.99
full, 5, 12,64,5240.32
2022-04-18 09:43:46 -05:00
38f0d60b00
Move repeated logic into common function ( #24373 )
2022-04-18 00:16:06 -05:00
578d59c802
Remove the code that handles cost update for separate pr
2022-04-17 19:26:24 -05:00
e97ffb55cb
nit - renaming variables to concise names
2022-04-17 19:26:24 -05:00
6bc6384f8e
refactor to consolidate info into single return field
2022-04-17 19:26:24 -05:00
9dadfb2e2c
Add checked_add_signed() to apply cost adjustment to cost_tracker
2022-04-17 19:26:24 -05:00
810b1dff40
undo cost of executed-but-not-recorded transactions from cost_tracker
2022-04-17 19:26:24 -05:00
23d365d02f
Address review comment: extract transaction was_executed status to avoid cloning execution_results
2022-04-17 19:26:24 -05:00
094da35b91
Address review comments:
...
1. use was_executed to correctly identify transactions requires cost adjustment;
2. add function to specifically handle executino cost adjustment without have to copy accounts
2022-04-17 19:26:24 -05:00
29ca21ed78
undo transaction cost from cost_tracker if it was not executed successfully
2022-04-17 19:26:24 -05:00
d71986cecf
Separate staked and un-staked on quic tpu port ( #24339 )
2022-04-16 10:54:22 +02:00
1b7d1f78de
Implement QUIC connection warmup service for future leaders ( #24054 )
...
* Increase connection timeouts
* Bump quic connection cache to 1024
* Use constant for quic connection timeout and add warm cache service
* Fixes to QUIC warmup service
* fix check failure
* fixes after rebase
* fix timeout test
Co-authored-by: Pankaj Garg <pankaj@solana.com>
2022-04-15 12:09:24 -07:00
97f2eb8e65
Banking stage: Deserialize packets only once
...
Benchmarks show roughly a 6% improvement. The impact could be more
significant when transactions need to be retried a lot.
after patch:
{'name': 'banking_bench_total', 'median': '72767.43'}
{'name': 'banking_bench_tx_total', 'median': '80240.38'}
{'name': 'banking_bench_success_tx_total', 'median': '72767.43'}
test bench_banking_stage_multi_accounts
... bench: 6,137,264 ns/iter (+/- 1,364,111)
test bench_banking_stage_multi_programs
... bench: 10,086,435 ns/iter (+/- 2,921,440)
before patch:
{'name': 'banking_bench_total', 'median': '68572.26'}
{'name': 'banking_bench_tx_total', 'median': '75704.75'}
{'name': 'banking_bench_success_tx_total', 'median': '68572.26'}
test bench_banking_stage_multi_accounts
... bench: 6,521,007 ns/iter (+/- 1,926,741)
test bench_banking_stage_multi_programs
... bench: 10,526,433 ns/iter (+/- 2,736,530)
2022-04-15 00:57:11 -06:00
7a4a6597c0
Don't enforce ulimit for validator test config ( #24272 )
2022-04-12 22:06:37 +02:00
9b8850f99e
test-validator: Add `--max-compute-units` flag ( #24130 )
...
* test-validator: Add `--max-compute-units` flag
* Add `RuntimeConfig` for tweaking runtime behavior
* Actually add the file
* Move RuntimeConfig to runtime
2022-04-12 02:28:10 +02:00
c1687b0604
Switch to await-aware tokio::sync::Mutex
2022-04-11 18:15:03 -04:00
60b2155bd3
Add accounts-filler-size command line option ( #23896 )
2022-04-11 13:10:09 -05:00
ff3b6d2b8b
Remove duplicate increment ( #24219 )
2022-04-09 15:21:39 -05:00
a058f348a2
Address review comments
2022-04-08 14:37:55 -05:00
2ed29771f2
Unittest for cost tracker after process_and_record_transactions
2022-04-08 14:37:55 -05:00
924b8ea1eb
Adjustments to cost_tracker updates
...
- don't store pending tx signatures and costs in CostTracker
- apply tx costs to global state immediately again
- go from commit_or_cancel to update_or_remove, where the cost tracker
is either updated with the true costs for successful tx, or the costs
of a retryable tx is removed
- move the function into qos_service and hold the cost tracker lock for
the whole loop
2022-04-08 14:37:55 -05:00
9e07272af8
- Only commit successfully executed transactions' cost to cost_tracker;
...
- In-fly transactions are pended in cost_tracker until being committed
or cancelled;
2022-04-08 14:37:55 -05:00
d2702201ca
Bump tonic, tonic-build, prost, and etcd-client ( #24147 )
...
* Bump tonic, prost, and etcd-client
* Restore doc ignores
2022-04-08 10:21:45 -06:00
210f6a6fab
move hash calculation out of acct bg svc ( #23689 )
...
* move hash calculation out of acct bg svc
* pr feedback
2022-04-08 10:42:03 -05:00
1dd63631c0
Add high level overview comments on ledger_cleanup_service ( #24184 )
2022-04-08 00:49:21 -05:00
e105547c14
tvu and tpu timeout on joining its microservices ( #24111 )
...
* panic when test timeout
* nonblocking send when when droping banks
* debug log
* timeout for tvu
* unused varaible
* timeout for tpu
* Revert "debug log"
This reverts commit da780a3301a51d7c496141a85fcd35014fe6dff5.
* add timeout const
* fix typo
* Revert "nonblocking send when when droping banks".
I will create another pull request for this.
This reverts commit 088c98ec0facf825b5eca058fb860deba6d28888.
* Update core/src/tpu.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* Update core/src/tpu.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* Update core/src/tvu.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* Update core/src/tvu.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* Update core/src/validator.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
2022-04-07 20:20:13 -05:00
c27150b1a3
reserialize_bank_fields_with_hash ( #23916 )
...
* reserialize_bank_with_new_accounts_hash
* Update runtime/src/serde_snapshot.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* pr feedback
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-07 14:05:57 -05:00
550ca7bf92
compare contents of serialized banks instead of exact file format ( #24141 )
...
* compare contents of serialized banks instead of exact file format
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* pr feedback
* get rid of clone
* pr feedback
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-06 21:55:44 -05:00
fddd162645
reserialize bank in ahv by first writing to temp file in abs ( #23947 )
2022-04-06 21:39:26 -05:00
fb67ff14de
Remove replica-node crates ( #24152 )
2022-04-06 16:52:19 -06:00
afeb1d3cca
Bump lru crate ( #24150 )
2022-04-06 16:18:42 -06:00
c322842257
Replace channel with Mutex<Option> for AccountsPackage ( #24013 )
2022-04-06 05:47:19 -05:00
302142bb25
fix typo ( #24123 )
2022-04-05 15:55:47 -05:00
db23295e1c
removes legacy weighted_shuffle and weighted_best methods ( #24125 )
...
Older weighted_shuffle is based on a heuristic which results in biased
samples as shown in:
https://github.com/solana-labs/solana/pull/18343
and can be replaced with WeightedShuffle.
Also, as described in:
https://github.com/solana-labs/solana/pull/13919
weighted_best can be replaced with rand::distributions::WeightedIndex,
or WeightdShuffle::first.
2022-04-05 19:19:22 +00:00
4ea59d8cb4
Set drop callback on first root bank ( #23999 )
2022-04-05 13:02:33 -05:00
2282571493
removes outdated and flaky test_skip_repair from retransmit-stage ( #24121 )
...
test_skip_repair in retransmit-stage is no longer relevant because
following: https://github.com/solana-labs/solana/pull/19233
repair packets are filtered out earlier in window-service and so
retransmit stage does not know if a shred is repaired or not.
Also, following turbine peer shuffle changes:
https://github.com/solana-labs/solana/pull/24080
the test has become flaky since it does not take into account how peers
are shuffled for each shred.
2022-04-05 16:02:53 +00:00
2b718d00b0
removes legacy compatibility turbine peers shuffle code
2022-04-05 12:04:12 +00:00
d0b850cdd9
removes turbine peers shuffle patch feature
2022-04-05 12:04:12 +00:00
855801cc95
removes deterministic-shred-seed feature
2022-04-05 12:04:12 +00:00
ee6bb0d5d3
track fec set turbine stats ( #23989 )
2022-04-04 14:44:21 -07:00
6ba4e870c4
Blockstore should drop signals before validator exit ( #24025 )
...
* timeout for validator exits
* clippy
* print backtrace when panic
* add backtrace package
* increase time out to 30s
* debug logging
* make rpc complete service non blocking
* reduce log level
* remove logging
* recv_timeout
* remove backtrace
* remove sleep
* wip
* remove unused variable
* add comments
* Update core/src/validator.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* Update core/src/validator.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* whitespace
* more whitespace
* fix build
* clean up import
* add mutex for signal senders in blockstore
* remove mut
* refactor: extract add signal functions
* make blockstore signal private
* let compiler infer mutex type
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
2022-04-04 11:38:05 -05:00
7cb3b6cbe2
demotes WeightedShuffle failures to error metrics ( #24079 )
...
Since call-sites are calling unwrap anyways, panicking seems too punitive
for our use cases.
2022-04-03 16:20:06 +00:00
ffa4cafe1c
Revert sequential execution of validator_exit and validator_parallel_exit tests ( #24048 )
...
* handle channel disconnect
* revert sequential execution of validator_exit and parallel_validator_exit tests
2022-04-02 10:22:47 -05:00
0b5ed87220
(LedgerStore) Enable performance sampling in column family get() ( #23834 )
...
#### Summary of Changes
This PR enables RocksDB read side performance metrics to report to blockstore_rocksdb_read_perf.
The sampling rate is controlled by an env arg `SOLANA_METRICS_ROCKSDB_PERF_SAMPLES_IN_1K`,
specifies the number of perf samples for every 1000 operations. The default value is set to 10, meaning
we will report 10 out of 1000 (or 1/100) reads.
The metrics are based on the RocksDB [PerfContext](https://github.com/facebook/rocksdb/blob/main/include/rocksdb/perf_context.h ).
It includes many useful metrics including block read time, cache hit rate, and time spent on decompressing the block.
2022-04-01 13:13:32 -07:00
df4d92f9cf
Revert voting service to use UDP instead of QUIC ( #24032 )
2022-04-01 09:34:18 -07:00
51b37f0184
Modify rpc_completed_slot_service to be non-blocking ( #24007 )
...
* timeout for validator exits
* clippy
* print backtrace when panic
* add backtrace package
* increase time out to 30s
* debug logging
* make rpc complete service non blocking
* reduce log level
* remove logging
* recv_timeout
* remove backtrace
* remove sleep
* remove unused variable
* add comments
* Update core/src/validator.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* Update core/src/validator.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* whitespace
* more whitespace
* fix build
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
2022-03-31 16:44:23 -05:00
9c8dad33c7
add epoch_schedule and rent_collector to hash calc ( #24012 )
2022-03-31 10:51:18 -05:00
da001d54e5
calculate_accounts_hash_helper uses config ( #24003 )
2022-03-31 09:29:45 -05:00
125f9634fd
add hash calc config.use_write_cache ( #24005 )
2022-03-30 17:19:34 -05:00
ba770832d0
Poh timing service ( #23736 )
...
* initial work for poh timing report service
* add poh_timing_report_service to validator
* fix comments
* clippy
* imrove test coverage
* delete record when complete
* rename shred full to slot full.
* debug logging
* fix slot full
* remove debug comments
* adding fmt trait
* derive default
* default for poh timing reporter
* better comments
* remove commented code
* fix test
* more test fixes
* delete timestamps for slot that are older than root_slot
* debug log
* record poh start end in bank reset
* report full to start time instead
* fix poh slot offset
* report poh start for normal ticks
* fix typo
* refactor out poh point report fn
* rename
* optimize delete - delete only when last_root changed
* change log level to trace
* convert if to match
* remove redudant check
* fix SlotPohTiming comments
* review feedback on poh timing reporter
* review feedback on poh_recorder
* add test case for out-of-order arrival of timing points and incomplete timing points
* refactor poh_timing_points into its own mod
* remove option for poh_timing_report service
* move poh_timing_point_sender to constructor
* clippy
* better comments
* more clippy
* more clippy
* add slot poh timing point macro
* clippy
* assert in test
* comments and display fmt
* fix check
* assert format
* revise comments
* refactor
* extrac send fn
* revert reporting_poh_timing_point
* align loggin
* small refactor
* move type declaration to the top of the module
* replace macro with constructor
* clippy: remove redundant closure
* review comments
* simplify poh timing point creation
Co-authored-by: Haoran Yi <hyi@Haorans-MacBook-Air.local>
2022-03-30 09:04:49 -05:00
c24de17278
remove index hash calculation as an option ( #23928 )
2022-03-25 15:32:53 -05:00
01af40d6b6
Fix intermittent validator_exit test failure ( #23594 )
...
* run validator_exit_test sequentially
* limit validator exit run to its own serial run subset
add 10ms delay in the validator exit tests
* fix intermittent validator exit failure
* no sleep
* undo the code move
2022-03-25 14:38:19 -05:00
6b85c2104c
Implement forwarding via TpuConnection ( #23817 )
2022-03-25 11:31:40 -04:00
f44c8f296f
fix: thread `enforce_ulimit_nofile` config down when opening blockstore ( #23925 )
2022-03-25 03:13:33 -05:00
51f5524e2f
make verify_accounts_package_hash like other hash calc ( #23906 )
2022-03-24 17:49:48 -05:00
55d61023f7
document 'accounts' hash ( #23907 )
2022-03-24 15:58:52 -05:00
fedf4e984f
typo ( #23910 )
2022-03-24 15:21:59 -05:00
37c36ce3fa
pass stats separately from CalcAccountsHashConfig ( #23892 )
2022-03-24 12:48:47 -05:00
c31db81ac4
Use VoteAccountsHashMap type alias in all applicable spots ( #23904 )
2022-03-24 12:09:48 -05:00
82945ba973
Optimize TpuConnection and its implementations and refactor connection-cache to not use dyn in order to enable those changes ( #23877 )
2022-03-24 11:40:26 -04:00
5b916961b5
HashCalc uses self.accounts_cache ( #23890 )
2022-03-24 10:34:28 -05:00
b22165ad69
hash calc uses self.filler_account_suffix ( #23887 )
2022-03-24 09:58:06 -05:00
9022931689
calc hash uses self.num_hash_scan_passes ( #23883 )
2022-03-24 09:44:42 -05:00
db5d68f01f
HashCalc uses self.accounts_hash_cache_path ( #23882 )
2022-03-24 09:31:55 -05:00
3e22d4b286
calc hash uses self.thread_pool_clean ( #23881 )
2022-03-23 20:52:38 -05:00
9e61fe7583
add AccountsHashConfig to manage parameters ( #23850 )
2022-03-23 13:44:23 -05:00
db49b826f0
seperate blockstore metrics from window service metrics ( #23871 )
2022-03-23 13:38:17 -05:00
7ff8ed869c
typos ( #23870 )
2022-03-23 13:36:55 -05:00
b1280b670a
calculate_accounts_hash_without_index takes &self ( #23846 )
...
* calculate_accounts_hash_without_index takes &self
* Update runtime/src/snapshot_package.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-03-23 11:57:32 -05:00
92462ae031
Manually serialize and use `send_wire_transaction` for votes ( #23826 )
...
* Revert "core: partial versioned transaction support for voting service"
This reverts commit eb3df4c20e
2022-03-23 09:47:55 +08:00
7af48465fa
transaction-status: Add return data to meta ( #23688 )
...
* transaction-status: Add return data to meta
* Add return data to simulation results
* Use pretty-hex for printing return data
* Update arg name, make TransactionRecord struct
* Rename TransactionRecord -> ExecutionRecord
2022-03-22 23:17:05 +01:00
ae75b1a25f
(LedgerStore) Add compression type ( #23578 )
...
This PR adds `--rocksdb-ledger-compression` as a hidden argument to the validator
for specifying the compression algorithm for TransactionStatus. Available compression
algorithms include `lz4`, `snappy`, `zlib`. The default value is `none`.
Experimental results show that with lz4 compression, we can achieve ~37% size-reduction
on the TransactionStatus column family, or ~8% size-reduction of the ledger store size.
2022-03-22 02:27:09 -07:00
eb3df4c20e
core: partial versioned transaction support for voting service
2022-03-21 22:59:05 -06:00
45a7c6edfb
Fix typos and a small refactor ( #23805 )
...
* fix typo
* remove packet_has_more_unprocessed_transactions function
2022-03-21 18:35:31 -05:00
c4ecfa5716
Bump version to v1.11 ( #23807 )
...
* Revert crossbeam_epoch to stable. 0.9.8 only works with nightly
* Remove unneeded unit expression
2022-03-21 17:40:50 -05:00
5d03b188c8
Use QUIC client in voting service ( #23713 )
...
* Use QUIC client in voting service
* guard quic-client usage with a flag
* add measure to time the quic client
* move time measure outside if block
* remove quic vs UDP flag from voting service
2022-03-21 09:10:16 -07:00
71ea05c176
replace nested for_each with flat_map
2022-03-18 16:37:41 -05:00
1c369fb55f
Scan entire UnprocessedPacketBatches buffer to produce stake and locator of each packet
2022-03-18 16:37:41 -05:00
f999eef452
(LedgerStore) Rename BlockstoreAdvancedOptions to LedgerColumnOptions ( #23764 )
...
This PR renames BlockstoreAdvancedOptions to LedgerColumnOptions, as we will
pass-down this struct to LedgerColumn to allow it to perform metric reporting.
2022-03-18 11:13:35 -07:00
56428be629
Not exposing inner cost_table to encapsulating implementation details,
...
making future change easier.
2022-03-18 12:58:43 -05:00
0ed23899e7
directly use compute_budget MAX_UNITS and DEFAULT_UNITS
2022-03-18 08:53:11 -05:00
a4cacf3389
add deterministic default cost
2022-03-18 08:53:11 -05:00
c478fe2047
add timing metrics, some renaming
2022-03-17 19:31:28 -05:00
fd515097d8
leader qos part 2: add stage to find sender stake, set to packet meta
2022-03-17 19:31:28 -05:00
976b138e76
Add tx weighting stage
2022-03-17 19:31:28 -05:00
2f58c9e501
Bump version to 1.10.4 ( #23743 )
2022-03-17 14:02:13 -05:00
3773b753d1
Configure shrink paths during blockstore load
2022-03-15 23:08:07 -07:00
ab373bb1a9
Refactor new_banks_from_ledger() into load and process steps
2022-03-15 23:08:07 -07:00
2da4e3eb6c
Add --no-os-memory-stats-reporting
2022-03-15 17:07:40 -07:00
dbc62f2e28
Use consistent variable naming for DropBankService
2022-03-15 17:07:13 -07:00
d44f3d7216
Remove unhelpful log message
2022-03-15 17:07:13 -07:00
2d3501dff9
make upsert infallible op
2022-03-15 17:05:41 -05:00
62b26f012e
chore: bump reqwest from 0.11.9 to 0.11.10 ( #23671 )
...
* chore: bump reqwest from 0.11.9 to 0.11.10
Bumps [reqwest](https://github.com/seanmonstar/reqwest ) from 0.11.9 to 0.11.10.
- [Release notes](https://github.com/seanmonstar/reqwest/releases )
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.9...v0.11.10 )
---
updated-dependencies:
- dependency-name: reqwest
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-03-15 12:54:23 -06:00
61cead9b9b
Remove injection of exit signal into cost_update_service
2022-03-15 09:58:56 -05:00
eb73dacd58
harden banking tests
2022-03-15 09:58:08 -05:00
8c8f9694e0
Refactor: Sanitized transaction creation ( #23558 )
...
* Refactor: SanitizedTransaction::try_create optionally computes hash
* Refactor: Add SimpleAddressLoader
2022-03-15 12:02:22 +08:00
102dd68a03
Rename AccountsDb plugins to Geyser plugins ( #23604 )
2022-03-14 19:18:46 -06:00
17cc095d28
Slot warping doesn't need to be in new_banks_from_ledger
2022-03-14 15:29:58 -07:00
2e7ee0f177
Tower loading doesn't need to be in new_banks_from_ledger
2022-03-14 15:29:58 -07:00
390dc24608
Create leader schedule before processing blockstore
2022-03-14 15:29:58 -07:00
543d5d4a5d
Reduce new_banks_from_ledger arguments
2022-03-14 15:29:58 -07:00
115f376465
Factor out bank_forks_utils::load_bank_forks()
2022-03-14 15:29:58 -07:00
c2ce152be8
Inline do_process_blockstore_from_root
2022-03-14 15:29:58 -07:00
5ea6a1e500
code review
2022-03-14 13:14:27 -05:00
8590911b0a
Replace type alias with newtype for UnprocesedPacketBatches
2022-03-14 13:14:27 -05:00
63bf0f66af
Bump version to 1.10.3 ( #23648 )
2022-03-14 11:18:45 -05:00
7758c32035
Banking Stage drops transactions that'll exceed the total account data size limit ( #23537 )
2022-03-13 15:58:57 +00:00
1e20bd8f9a
(LedgerStore) Include storage type as a tag in RocksDB metric reporting ( #23523 )
...
#### Summary of Changes
This PR further enables group by operation on storage type in blockstore_rocksdb_cfs metrics.
Such group-by allows us to further compare the performance metrics between rocks-level and
rocks-fifo.
To make things extensible, this PR introduces BlockstoreAdvancedOptions and move shred_storage_type.
All fields in BlockstoreAdvancedOptions will support group-by operation in blockstore_rocksdb_cfs.
Dependency: #23580
2022-03-11 15:17:34 -08:00
b1da7cff66
chore: bump base64 from 0.12.3 to 0.13.0 ( #23616 )
...
* chore: bump base64 from 0.12.3 to 0.13.0
Bumps [base64](https://github.com/marshallpierce/rust-base64 ) from 0.12.3 to 0.13.0.
- [Release notes](https://github.com/marshallpierce/rust-base64/releases )
- [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/marshallpierce/rust-base64/compare/v0.12.3...v0.13.0 )
---
updated-dependencies:
- dependency-name: base64
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Update Cargo.lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2022-03-11 16:01:15 -07:00
d20dd21600
Sort tables in Cargo.toml files ( #23602 )
2022-03-11 11:05:57 -06:00
9b591286d7
Revert "chore: bump dashmap from 4.0.2 to 5.1.0 ( #23372 )" ( #23592 )
...
This reverts commit 3a0271c113
2022-03-11 09:12:27 +01:00
b444836a97
Bump version to 1.10.2 ( #23597 )
2022-03-10 16:41:06 -06:00
35d1235ed0
- move `unprocessed_packet_batches` from `BankingStage` to its own ( #23508 )
...
module
- deserialize packets during receving and buffering
2022-03-10 18:47:46 +00:00
588414a776
Report even if slot begins and ends in process_buffered_packets() ( #23549 )
2022-03-09 23:42:35 -05:00
f68c5a274d
remove persist_cost_table code
2022-03-09 21:05:47 -07:00
9f71958d7d
Patch validator from loading persisted program costs
2022-03-09 21:05:47 -07:00
9bbccbe27c
Use `AsRef<Path>` instead of `PathBuf` for parameters ( #23560 )
2022-03-09 16:08:33 -06:00
a1c45d5acb
typo ( #23564 )
2022-03-09 15:11:21 -06:00
7a9884c831
Quic limit connections ( #23283 )
...
* quic server limit connections
* bump per_ip
* Review comments
* Make the connections per port
2022-03-09 10:52:31 +01:00
5c722519cf
chore: bump etcd-client from 0.8.3 to 0.8.4
...
Bumps [etcd-client](https://github.com/etcdv3/etcd-client ) from 0.8.3 to 0.8.4.
- [Release notes](https://github.com/etcdv3/etcd-client/releases )
- [Commits](https://github.com/etcdv3/etcd-client/compare/0.8.3...v0.8.4 )
---
updated-dependencies:
- dependency-name: etcd-client
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-03-08 17:33:24 -08:00
5a0cd05866
Revert "- estimate a program cost as 2 standard deviation above mean"
...
This reverts commit a25ac1c988
2022-03-08 17:18:44 -08:00
9acbfa5eb1
Revert "use EMA in place of Welford"
...
This reverts commit 6587dbfa47
2022-03-08 17:18:44 -08:00
c878c9e2cb
Revert "1. Persist to blockstore less frequently;"
...
This reverts commit 7aa1fb4e24
2022-03-08 17:18:44 -08:00
0a17edcc1f
Revert "fix tests after merge"
...
This reverts commit ba2d83f580
2022-03-08 17:18:44 -08:00
b719d6a2ad
`solana-validator set-identity` no longer writes a tower file unnecessarily
2022-03-08 15:34:23 -08:00
3114c199bd
Add RPC support for versioned transactions ( #22530 )
...
* Add RPC support for versioned transactions
* fix doc tests
* Add rpc test for versioned txs
* Switch to preflight bank
2022-03-08 15:20:34 +08:00
38d8bbb19c
chore: bump sysctl from 0.4.3 to 0.4.4 ( #23505 )
...
Bumps [sysctl](https://github.com/johalun/sysctl-rs ) from 0.4.3 to 0.4.4.
- [Release notes](https://github.com/johalun/sysctl-rs/releases )
- [Changelog](https://github.com/johalun/sysctl-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/johalun/sysctl-rs/commits )
---
updated-dependencies:
- dependency-name: sysctl
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-07 14:24:19 -07:00
181fffb916
rename status filename to be consistent ( #23501 )
2022-03-07 17:34:35 +00:00
3a0271c113
chore: bump dashmap from 4.0.2 to 5.1.0 ( #23372 )
...
* chore: bump dashmap from 4.0.2 to 5.1.0
Bumps [dashmap](https://github.com/xacrimon/dashmap ) from 4.0.2 to 5.1.0.
- [Release notes](https://github.com/xacrimon/dashmap/releases )
- [Commits](https://github.com/xacrimon/dashmap/commits/v5.1.0 )
---
updated-dependencies:
- dependency-name: dashmap
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-03-07 00:00:26 -05:00
b8b7163b66
(Ledger Store) Report RocksDB Column Family Metrics ( #22503 )
...
This PR enables blockstore to periodically report RocksDB column family properties.
The reported properties are under blockstore_rocksdb_cfs, and the properties also
support group by operation on cf_name.
2022-03-05 16:13:03 -08:00
62d2a4cd88
Make ShredStorageType::RocksLevel public ( #23272 )
...
#### Summary of Changes
This PR adds two hidden arguments to the validator that allow users to use RocksDB's FIFO compaction for storing shreds.
--shred-storage <SHRED_STORAGE>
EXPERIMENTAL: Controls how RocksDB compacts shreds. *WARNING*: You will lose your ledger data
when you switch between options. Possible values are: 'level': stores shreds using RocksDB's default (level)
compaction. 'fifo': stores shreds under RocksDB's FIFO compaction. This option is more efficient on
disk-write-bytes of the ledger store. [default: level] [possible values: level, fifo]
--shred-storage-size <SHRED_STORAGE_SIZE_BYTES>
The shred storage size in bytes. The suggested value is 50% of your ledger storage size in bytes. [default:
268435456000]
2022-03-03 12:43:58 -08:00
f2fa49a771
chore: bump lru from 0.7.2 to 0.7.3 ( #23462 )
...
Bumps [lru](https://github.com/jeromefroe/lru-rs ) from 0.7.2 to 0.7.3.
- [Release notes](https://github.com/jeromefroe/lru-rs/releases )
- [Changelog](https://github.com/jeromefroe/lru-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/jeromefroe/lru-rs/compare/0.7.2...0.7.3 )
---
updated-dependencies:
- dependency-name: lru
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-03 12:01:49 -07:00
61d7bdd66f
chore: bump serde_json from 1.0.78 to 1.0.79 ( #23461 )
...
* chore: bump serde_json from 1.0.78 to 1.0.79
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.78 to 1.0.79.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.78...v1.0.79 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2022-03-02 23:38:06 -07:00
fe7604589d
chore: bump retain_mut from 0.1.5 to 0.1.7 ( #23450 )
...
Bumps [retain_mut](https://github.com/upsuper/retain_mut ) from 0.1.5 to 0.1.7.
- [Release notes](https://github.com/upsuper/retain_mut/releases )
- [Commits](https://github.com/upsuper/retain_mut/commits )
---
updated-dependencies:
- dependency-name: retain_mut
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-02 23:15:33 +00:00
1a99251498
Bump version to 1.10.1 ( #23453 )
2022-03-02 13:47:01 -06:00
26aa18b3f3
fmt ( #23448 )
2022-03-02 11:54:58 -06:00
41f78b9925
small optimization. use shift for pow of 2. ( #22975 )
2022-03-02 09:11:12 -06:00
4f0070a5c6
unittest for bind two consecutive ports ( #23008 )
...
* minor fix of comments in fork-selection tests
* fix doc link
* add unittest for bind_two_consecutive_in_range
2022-03-02 09:10:29 -06:00
8de88d0a55
Refactor packet_threshold adjustment code into its own struct ( #23216 )
...
* refactor packet_threshold adjustment code into own struct and add unittest for it
* fix a typo in error message
* code review feedbacks
* another code review feedback
* Update core/src/ancestor_hashes_service.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* share packet threshold with repair service (credit to carl)
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
2022-03-02 09:09:06 -06:00
86e2f728c3
Fix a batch limits bug in banking ( #23327 )
...
* add thread index in thread name for debugging
* fix batch_limit
* use NUM_VOTE_THREAD instead of hardcoded number (credit to carllin)
2022-03-02 09:08:08 -06:00
d3ebe8d8f5
Remove unneeded jsonrpc dependencies/features; update do-audit ( #23436 )
...
* Update generic-array note
* Remove unneeded jsonrpc deps
* Remove unneeded jsonrpc features
* Rewrite slot-update test without websocket crate
* Rewrite rpc-subscription test without websocket crate, and remove jsonrpc deps
* Update expected balance to accommodate rent-exempt minimum transfer amount
* Remove obsolete audit ignores
2022-03-02 01:42:01 -07:00
c69e3b73ff
bench get_retransmit_peers ( #23292 )
2022-03-01 19:10:29 -08:00
ce4d579499
chore: bump serial_test from 0.5.1 to 0.6.0 ( #23414 )
...
Bumps [serial_test](https://github.com/palfrey/serial_test ) from 0.5.1 to 0.6.0.
- [Release notes](https://github.com/palfrey/serial_test/releases )
- [Commits](https://github.com/palfrey/serial_test/compare/v0.5.1...v0.6.0 )
---
updated-dependencies:
- dependency-name: serial_test
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-01 12:13:40 -07:00
533eca3b4c
Simplify replay_blockstore_into_bank() ( #23282 )
2022-02-25 06:57:04 -06:00
d4292774c5
checks
2022-02-25 08:05:28 +00:00
98f059e89c
chore: bump reqwest from 0.11.6 to 0.11.9 ( #23337 )
...
* chore: bump reqwest from 0.11.6 to 0.11.9
Bumps [reqwest](https://github.com/seanmonstar/reqwest ) from 0.11.6 to 0.11.9.
- [Release notes](https://github.com/seanmonstar/reqwest/releases )
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.6...v0.11.9 )
---
updated-dependencies:
- dependency-name: reqwest
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-02-24 22:40:01 -07:00
d0e85c293f
Fix rustfmt check ( #23296 )
2022-02-23 16:38:53 +08:00
20d031e2b8
Refactor ExecuteTimings w/ enum-indexed array ( #23085 )
2022-02-22 14:46:56 -08:00
7e08ae1d0c
Revert "Add simulation detection countermeasure ( #22880 )" ( #23261 )
...
This reverts commit c42b80f099
2022-02-21 21:15:37 +00:00
8c872e9ce0
(LedgerStore/FIFO) Refactor FIFO options and sanity check. ( #23131 )
2022-02-19 00:58:38 -08:00
70ebab2c82
Add rustfmt.toml and `cargo fmt` ( #23238 )
...
* fmt
* formatted
Co-authored-by: Lucas B <buffalu@jito.network>
2022-02-19 13:32:29 +08:00
1add82aa9e
(Ledger Store Benchmark) Add flags for checking shred storage size. ( #22451 )
2022-02-18 19:35:28 -08:00
619335df1a
Add execute timings ( #23097 )
2022-02-17 01:14:32 -05:00
83d31c9e65
shrink batches when over 80% of the space is wasted ( #23066 )
...
* shrink batches when over 80% of the space is wasted
2022-02-16 08:18:17 -08:00
115d71536b
forward_buffered_packets return packet count in error path ( #23167 )
2022-02-16 07:46:32 -08:00
a6d736572c
`solana-validator set-identity` now supports the `--require-tower` flag
2022-02-15 19:45:00 -08:00
03bf66a51b
flag end-of-slot when poh bank is gone
2022-02-15 15:01:27 -06:00
ab92578b02
Fix the flaky test test_restart_tower_rollback ( #23129 )
...
* Add flag to disable voting until a slot to avoid duplicate voting
* Fix the tower rollback test and remove it from flaky.
2022-02-15 13:19:34 -07:00
c42b80f099
Add simulation detection countermeasure ( #22880 )
...
* Add simulation detection countermeasures
* Add program and test using TestValidator
* Remove incinerator deposit
* Remove incinerator
* Update Cargo.lock
* Add more features to simulation bank
* Update Cargo.lock per rebase
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
2022-02-15 13:09:59 +01:00
824446710b
(Ledger Store Benchmark) Enable FIFO compmaction in the benchmark. ( #22162 )
2022-02-12 00:43:25 -08:00
c04438be4b
Retaining transaction logs when transaction plugin is loaded. ( #22874 )
...
Transaction logs are not being saved to the database through the plugin interface.
Summary of Changes
Retain the transaction logs when transaction notification plugin is loaded.
Fixes #
lijunwangs/solana-accountsdb-plugin-postgres#6
2022-02-11 20:29:07 -08:00
2f9e30a1f7
Introduce slot-specific packet metrics ( #22906 )
2022-02-11 03:07:45 -05:00
59e64d5e99
chore: bump raptorq from 1.6.4 to 1.6.5 ( #23042 )
...
Bumps [raptorq](https://github.com/cberner/raptorq ) from 1.6.4 to 1.6.5.
- [Release notes](https://github.com/cberner/raptorq/releases )
- [Commits](https://github.com/cberner/raptorq/compare/v1.6.4...v1.6.5 )
---
updated-dependencies:
- dependency-name: raptorq
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-10 19:07:07 -07:00
d5dec989b9
Enforce tx metadata upload with static types ( #23028 )
2022-02-10 13:28:18 +08:00
1b287f1b59
(Ledger Cleanup) Add code comments for ledger_cleanup. ( #22807 )
2022-02-08 22:48:56 -08:00
ba2d83f580
fix tests after merge
2022-02-08 16:18:23 -06:00
7aa1fb4e24
1. Persist to blockstore less frequently;
...
2. reduce alpha for EMA to 1 percent to have roughly 200 data points for estimatio
2022-02-08 16:18:23 -06:00
6587dbfa47
use EMA in place of Welford
2022-02-08 16:18:23 -06:00
a25ac1c988
- estimate a program cost as 2 standard deviation above mean
...
- replaced get_average / get_mode with get_default to assign max units to unknown program
2022-02-08 16:18:23 -06:00
e52e48076e
bench should update leader schedule cache ( #22991 )
2022-02-08 02:28:28 +00:00
5acf0f6331
Add feature gate for new vote instruction and plumb through replay ( #21683 )
...
* Add feature gate for new vote instruction and plumb through replay
Add tower versions
* Add check for slot hashes history
* Update is_recent check to exclude voting on hard fork root slot
* Move tower rollback test to flaky and ignore it until #22551 lands
2022-02-07 14:06:19 -08:00
27aaf9df85
removes VoteTracker::new in favor of VoteTracker::default ( #22941 )
...
VoteTracker::new does not need a bank and is so redundant:
https://github.com/solana-labs/solana/blob/5a230f418/core/src/cluster_info_vote_listener.rs#L103-L107
2022-02-04 19:01:59 +00:00
5a230f418d
Add quic port for accepting transactions ( #22753 )
...
using quinn library
streamer: Sign TLS cert with validator identity key
Handle multiple incoming chunks
2022-02-04 15:27:09 +01:00
4bec182b32
Allow buffered packets be consumed if bank is active, regardless leader schedule ( #22913 )
2022-02-03 21:29:41 +00:00
60af1a4cce
Refactor: Add trait for loading addresses ( #22903 )
2022-02-03 11:00:12 +00:00
bd1850df25
Return actual committed transactions from process_transactions() ( #22802 )
2022-02-03 03:56:36 -05:00
c62f9839a2
test-validator-bin: reinstate full rpc method set
2022-02-03 02:43:03 +00:00
58a70d76a3
fix typo in broadcast_duplicates_run.rs ( #22888 )
...
Creat -> Create
2022-02-02 12:29:14 -07:00
dccbddad80
adds reverse lookup index to cluster-nodes ( #22892 )
...
retransmit has to exclude slot leader from set of nodes for each shred;
which currently requires a linear scan:
https://github.com/solana-labs/solana/blob/e3b137066/core/src/cluster_nodes.rs#L238-L242
This commit adds a reverse lookup index to avoid linear scan.
2022-02-02 19:27:50 +00:00
e3b137066d
caches WeightedShuffle struct in ClusterNodes ( #22877 )
...
Instead of reconstructing WeightedShuffle struct for each shred
broadcast or retransmit, we can use the same struct with minimal
mutations.
2022-02-02 15:12:26 +00:00
eac4a6df68
rpc: use minimal mode by default
2022-02-01 19:00:06 -07:00
45e09664b8
removes Rng field from WeightedShuffle struct ( #22850 )
2022-02-01 15:27:23 +00:00
604ca9316c
includes zero weighted entries in WeightedShuffle ( #22829 )
...
Current WeightedShuffle implementation excludes zero weighted entries
from the shuffle:
https://github.com/solana-labs/solana/blob/13e631dcf/gossip/src/weighted_shuffle.rs#L29-L30
Though mathematically this might make more sense, for our use-cases
(turbine specifically), this results in less efficient code:
https://github.com/solana-labs/solana/blob/13e631dcf/core/src/cluster_nodes.rs#L409-L430
This commit changes the implementation so that zero weighted indices are
also included in the shuffle but appear only at the end after non-zero
weighted indices.
2022-01-31 16:23:50 +00:00
cc74693176
chore: bump serde from 1.0.134 to 1.0.136 ( #22817 )
...
* chore: bump serde from 1.0.134 to 1.0.136
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.134 to 1.0.136.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.134...v1.0.136 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-28 12:20:37 -07:00
220aa6ada0
Fix poh recorder initialization on startup ( #22755 )
2022-01-28 14:21:15 +08:00
331b953551
Add vote account address to vote subscription
2022-01-27 08:22:29 -08:00
db481e1799
chore: bump serde_json from 1.0.75 to 1.0.78 ( #22748 )
...
* chore: bump serde_json from 1.0.75 to 1.0.78
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.75 to 1.0.78.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.75...v1.0.78 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-26 18:01:02 -07:00
d9c259a231
Set the correct root in block commitment cache initialization ( #22750 )
...
* Set the correct root in block commitment cache initialization
* clean up test
* bump
2022-01-27 00:48:00 +08:00
2e56c59bcb
Handle already discarded packets in gpu sigverify path ( #22680 )
2022-01-24 14:35:47 +01:00
1240217a73
Refactor: Rename variables and helper method to `PohRecorder` ( #22676 )
...
* Refactor: Rename leader_first_tick_height field
* Refactor: add `PohRecorder::slot_for_tick_height` helper
* Refactor: Add type for poh leader status
2022-01-23 10:28:50 +08:00
d6011ba14d
Dedup bloom filter is too slow ( #22607 )
...
* Faster dedup
* use ahash
* fixup
* single threaded
* use duration type
* remove the count
* fixup
2022-01-21 20:23:48 -07:00
6d5bbca630
Pacify clippy
2022-01-21 19:12:57 -08:00
ce4f7601af
Avoid unstable_name_collisions warning
2022-01-21 19:12:57 -08:00
0cf886302d
chore: bump serde from 1.0.133 to 1.0.134 ( #22650 )
...
* chore: bump serde from 1.0.133 to 1.0.134
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.133 to 1.0.134.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.133...v1.0.134 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-21 17:14:02 -07:00
95bbb70c91
chore: bump serde_json from 1.0.74 to 1.0.75 ( #22541 )
...
* chore: bump serde_json from 1.0.74 to 1.0.75
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.74 to 1.0.75.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.74...v1.0.75 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-21 12:26:51 -07:00
38b02bbcc0
Handle already discarded packets in discard_excess_packets ( #22594 )
2022-01-21 17:22:50 +01:00
e7777281d6
regularly report network limits ( #22563 )
2022-01-20 12:38:42 -08:00
cca3dbc76d
system-monitor-service: support percentages from bigger numbers
2022-01-20 09:51:23 +00:00
f8db314134
(Ledger Store Benchmark) Display storage size of all data shreds ( #22445 )
...
* (Ledger Store) APIs for obtaining physical size of all data and coding shreds
* (Ledger Store Benchmark) Display total data shred storage size.
2022-01-19 19:33:08 -08:00
7f20c6149e
Refactor: move simple vote parsing to runtime ( #22537 )
2022-01-20 10:39:21 +08:00
d343713f61
Optimize packet dedup ( #22571 )
...
* Use bloom filter to dedup packets
* dedup first
* Update bloom/src/bloom.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* Update core/src/sigverify_stage.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* Update core/src/sigverify_stage.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* Update core/src/sigverify_stage.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* fixup
* fixup
* fixup
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
2022-01-19 13:58:20 -08:00
dcf44d2523
improves sigverify discard_excess_packets performance ( #22577 )
...
As shown by the added benchmark, current code does worse if there is a
spam address plus a lot of unique addresses.
on current master:
test bench_packet_discard_many_senders ... bench: 1,997,960 ns/iter (+/- 103,715)
test bench_packet_discard_mixed_senders ... bench: 14,256,116 ns/iter (+/- 534,865)
test bench_packet_discard_single_sender ... bench: 1,306,809 ns/iter (+/- 61,992)
with this commit:
test bench_packet_discard_many_senders ... bench: 1,644,025 ns/iter (+/- 83,715)
test bench_packet_discard_mixed_senders ... bench: 1,089,789 ns/iter (+/- 86,324)
test bench_packet_discard_single_sender ... bench: 955,234 ns/iter (+/- 55,953)
2022-01-19 18:10:02 +00:00
650882217c
Add PacketBatch packet_indexes stat ( #22564 )
...
* collect stats on packet batch indicies
* cleanup
* cleanup
* cleanup
* change name
2022-01-19 08:13:07 +00:00
e616a7ebfc
Track discard time of excess packets in sigverify ( #22554 )
...
* discard time histogram
* closer to the if
* update
2022-01-18 15:09:39 -07:00
8dc6f9f589
Remove unused mut
2022-01-18 12:10:31 -08:00
49443406fd
Use VecDeque instead of Vec in sigverify stage ( #22538 )
...
avoid bad performance of remove(0) for a single sender
2022-01-17 18:37:05 +01:00
2d94e6e5d3
metrics for generate new bank forks ( #22492 )
...
* metrics for generate new bank forks
* fixed
* Apply suggestions from code review
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* --fixup
* fixup!
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
2022-01-17 09:59:47 -07:00
a724fa2347
Add hidden cli option to allow validator reports replayed transaction cost metrics ( #22369 )
...
* add hidden cli option to allow validator reports replayed transaction cost detail metrics
* Update validator/src/main.rs
Co-authored-by: Michael Vines <mvines@gmail.com>
* - rebase master, using unbounded instead of channel; dowgrade to datapoint_trace
* removed cli arg, prefer log at trace
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-01-15 00:31:21 +00:00
1309a9cea0
Add estimated and actual block cost units metrics ( #22326 )
...
* - report cost details for transactions selected to be packed into block;
- report estimated execution units packed into block, and actual units and time after execution
* revert reporting per-transaction details
* rollup transaction cost details (eg signature cost, wirte lock, data cost and execution costs) into block stats
* change naming from units to cu, use struct to replace tuple
2022-01-14 23:44:18 +00:00
9c9f2dd5bd
port counting vote CUs to block cost ( #22477 )
2022-01-14 10:50:29 -06:00
f804ccdece
Store address table lookups in blockstore and bigtable ( #22402 )
2022-01-14 15:24:41 +08:00
4ab7d6c23e
Filter out outdated slots ( #22450 )
...
* Filter out outdated slots
* Fixup error
2022-01-13 19:51:00 -05:00
e291342c4a
Revert "chore: bump dashmap from 4.0.2 to 5.0.0 ( #21824 )" ( #22488 )
...
This reverts commit 8aa3d690b5
2022-01-13 13:06:39 -06:00
6614727be8
downgrade individual per-program-timing to trace to reduce writes to influx
2022-01-12 18:52:13 -06:00
8161cee70f
Remove unnecessary var in banking_stage bench ( #22408 )
2022-01-11 22:25:21 -06:00
637e366b18
Prevent rent-paying account creation ( #22292 )
...
* Fixup typo
* Add new feature
* Add new TransactionError
* Add framework for checking account state before and after transaction processing
* Fail transactions that leave new rent-paying accounts
* Only check rent-state of writable tx accounts
* Review comments: combine process_result success behavior; log and metrics before feature activation
* Fix tests that assume rent-exempt accounts are okay
* Remove test no longer relevant
* Remove native/sysvar special case
* Move metrics submission to report legacy->legacy rent paying transitions as well
2022-01-11 11:32:25 -07:00
8b66625c95
convert std::sync::mpsc to crossbeam_channel ( #22264 )
2022-01-11 02:44:46 -08:00
2cc6f863bf
Bump etcd-client from 0.8.2 to 0.8.3 ( #22415 )
...
Bumps [etcd-client](https://github.com/etcdv3/etcd-client ) from 0.8.2 to 0.8.3.
- [Release notes](https://github.com/etcdv3/etcd-client/releases )
- [Commits](https://github.com/etcdv3/etcd-client/compare/0.8.2...0.8.3 )
---
updated-dependencies:
- dependency-name: etcd-client
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-01-10 13:25:15 -07:00
5f1f4dcbdd
Use struct to pass all Tpu sockets as one argument to Tpu::new() ( #21965 )
...
Tpu::new() now matches Tvu::new() in having struct to reduce argument
list. Additionally, Rust supports partial moves, so there is no need to
clone the Tvu sockets out of Node object.
2022-01-10 11:29:48 -06:00
eeec1ce2ad
Add local cluster test to repro slot hash expiry bug ( #21873 )
2022-01-10 00:58:21 -05:00
28275a33d6
Bump tempfile from 3.2.0 to 3.3.0 ( #22401 )
...
* Bump tempfile from 3.2.0 to 3.3.0
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.2.0 to 3.3.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases )
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS )
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.2.0...v3.3.0 )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-09 12:26:08 -07:00
5771c36d3f
Rename open_with_access_type() to open_with_options() ( #22123 )
2022-01-07 12:11:43 -08:00
52d12cc802
Add runtime support for address table lookups ( #22223 )
...
* Add support for address table lookups in runtime
* feedback
* feedback
2022-01-07 11:59:09 +08:00
390ef0fbcd
Consolidate process instruction execution timings to own struct
2022-01-06 03:56:46 -07:00
848b6dfbdd
Add metrics for executor creation
2022-01-06 03:56:46 -07:00
b25e4a200b
Add execute metrics
2022-01-06 03:56:46 -07:00
7d32909e17
move `ExecuteTimings` from `runtime::bank` to `program_runtime::timings`
2022-01-06 03:56:46 -07:00
47b74e28ec
Add CLEANUP_SERVICE flag to ledger cleanup benchmark ( #22108 )
2022-01-05 23:46:02 -08:00
44d61465f1
(Ledger store benchmark - 3/N) Add comments about the benchmark and its arguments ( #22160 )
...
* Avoid shred generation in ledger_cleanup.rs
* Update comment for test_ledger_cleanup_compaction to include benchmark information.
2022-01-04 23:35:55 -10:00
45458e7139
Refactor: Improve type safety and readability of transaction execution ( #22215 )
...
* Refactor Bank::load_and_execute_transactions
* Refactor: improve type safety of TransactionExecutionResult
* Add enum for extra type safety in execution results
* feedback
2022-01-05 10:15:15 +08:00
e201b41341
Avoid shred generation in ledger_cleanup.rs ( #22091 )
2022-01-04 15:29:43 -10:00
4b24499916
removes total-size from return value of recv_mmsg
2022-01-04 21:06:59 +00:00
01a096adc8
adds bitflags to Packet.Meta
...
Instead of a separate bool type for each flag, all the flags can be
encoded in a type-safe bitflags encoded in a single u8:
https://github.com/solana-labs/solana/blob/d6ec103be/sdk/src/packet.rs#L19-L31
2022-01-04 13:53:40 +00:00
73a7741c49
uses std::net::IpAddr type for Packet.Meta.addr
2022-01-04 13:53:40 +00:00
2486e21ffe
Lower vote-only-mode to 400 ( #22210 )
2022-01-04 12:49:14 +01:00
ca8fef5855
retransmit consecutive leader blocks ( #22157 )
2022-01-04 00:24:16 -08:00
e8b7f96a89
Add struct BlockstoreOptions ( #22121 )
2022-01-03 18:30:45 -10:00
672fed04cb
Bump serde from 1.0.132 to 1.0.133 ( #22233 )
...
* Bump serde from 1.0.132 to 1.0.133
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.132 to 1.0.133.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.132...v1.0.133 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-03 22:19:39 +00:00
005592998d
Fix bug, add error specific timings ( #22225 )
2022-01-03 16:33:54 -05:00
69d71f8f86
removes epoch_authorized_voters from VoteTracker ( #22192 )
...
https://github.com/solana-labs/solana/pull/22169
verifies authorized-voter early on in vote-listener pipeline; and so
VoteTracker no longer needs to maintain and check for epoch authorized
voters.
2022-01-03 21:07:47 +00:00
0e4ede46d1
work around rust 39364 for stats_reporter_sender ( #22227 )
2022-01-03 11:46:02 -08:00
ecbfc70bfa
Bump serde_json from 1.0.73 to 1.0.74 ( #22231 )
...
* Bump serde_json from 1.0.73 to 1.0.74
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.73 to 1.0.74.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.73...v1.0.74 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-03 10:36:59 -07:00
557d35ec79
Remove duplicate code in ledger_cleanup_compaction_test ( #22204 )
2021-12-31 11:19:33 -05:00
f479ab7af2
ledger_cleanup test improvement (1/N) -- make the test lockless and simplify the logic ( #22090 )
2021-12-30 20:18:47 -10:00
d06e6c7425
Count compute units even when transaction errors ( #22182 )
2021-12-30 21:21:42 -05:00
95dfcc546a
bypass retransmission for slots without propagated stats ( #22176 )
2021-12-30 16:07:34 -08:00
c0c6038654
checks for authorized voter early on in the vote-listener pipeline ( #22169 )
...
Before votes are verified that they are signed by the authorized voter,
they might be dropped in verified-vote-packets code. If there are
enough many spam votes from unauthorized voters, this may potentially
drop valid votes but keep the false ones.
https://github.com/solana-labs/solana/blob/57986f982/core/src/verified_vote_packets.rs#L165-L168
2021-12-30 15:03:14 +00:00
33d0b5e011
Revert "Count compute units even when transaction errors ( #22059 )" ( #22174 )
...
This reverts commit eaa8c67bde
2021-12-30 02:42:32 -05:00
f14928a970
Stream additional block metadata via plugin ( #22023 )
...
* Stream additional block metadata through plugin
blockhash, block_height, block_time, rewards are streamed
2021-12-29 15:12:01 -08:00
b1d9a2e60e
Don't forward packets received from TPU forwards port ( #22078 )
...
* Don't forward packets received from TPU forwards port
* Add banking stage test
2021-12-29 19:34:31 +01:00
d20a3774db
Bump lru from 0.7.1 to 0.7.2 ( #22161 )
...
Bumps [lru](https://github.com/jeromefroe/lru-rs ) from 0.7.1 to 0.7.2.
- [Release notes](https://github.com/jeromefroe/lru-rs/releases )
- [Changelog](https://github.com/jeromefroe/lru-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/jeromefroe/lru-rs/compare/0.7.1...0.7.2 )
---
updated-dependencies:
- dependency-name: lru
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-12-29 10:47:37 -07:00
eaa8c67bde
Count compute units even when transaction errors ( #22059 )
2021-12-28 17:05:11 -05:00
f061059e45
Prevent log spam ( #22148 )
2021-12-28 17:04:48 -05:00
3d6ab96587
push live packets straight to buffer, leader only process packets from buffer
2021-12-28 15:21:24 -06:00
b89cd8cd1a
Avoid cloning Vec<Entry> when calling entries_to_test_shreds() ( #22093 )
2021-12-24 12:32:43 -08:00
93c776ce19
Refactor packet deduplication and harden bench test ( #22080 )
2021-12-22 23:05:10 -06:00
dd80a525ef
Leader QoS service metrics ( #21708 )
...
* - qos_service metrics tagged with leader thread ids to separate gossip/tpu votes and transactions;
- qos_service metrics is reported with bank slot;
- replaced timer-based reporting with signal via channel; removed async report test as qos_service now lives within a thread
* - add tpu live packets (eg, not buffered packets) states to qos metrics reporting
2021-12-22 21:39:59 +00:00
f67ecd5c18
removes unused Cargo dependencies ( #22022 )
...
Also moving some to [dev-dependencies] so that they are not propagated
to other packages which depend on the package.
2021-12-22 17:01:33 +00:00
4d62f03297
uses enum instead of trait for VoteTransaction ( #22019 )
...
Box<dyn Trait> involves runtime dispatch, has significant overhead and
is slow. It also requires hacky boilerplate code for implementing Clone
or other basic traits:
https://github.com/solana-labs/solana/blob/e92a81b74/programs/vote/src/vote_state/mod.rs#L70-L102
Only limited known types can be VoteTransaction and they are all defined
in the same crate. So using a trait here only adds overhead.
https://github.com/solana-labs/solana/blob/e92a81b74/programs/vote/src/vote_state/mod.rs#L125-L165
https://github.com/solana-labs/solana/blob/e92a81b74/programs/vote/src/vote_state/mod.rs#L221-L264
2021-12-22 14:25:46 +00:00
9c5d82557a
skip reporting all-zero stats
2021-12-21 16:20:36 -06:00
69d0b08dd8
chore: bump lru from 0.7.0 to 0.7.1 ( #22018 )
...
Bumps [lru](https://github.com/jeromefroe/lru-rs ) from 0.7.0 to 0.7.1.
- [Release notes](https://github.com/jeromefroe/lru-rs/releases )
- [Changelog](https://github.com/jeromefroe/lru-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/jeromefroe/lru-rs/compare/0.7.0...0.7.1 )
---
updated-dependencies:
- dependency-name: lru
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-12-20 21:56:56 -07:00
65d59f4ef0
tracks erasure coding shreds' indices explicitly ( #21822 )
...
The indices for erasure coding shreds are tied to data shreds:
https://github.com/solana-labs/solana/blob/90f41fd9b/ledger/src/shred.rs#L921
However with the upcoming changes to erasure schema, there will be more
erasure coding shreds than data shreds and we can no longer infer coding
shreds indices from data shreds.
The commit adds constructs to track coding shreds indices explicitly.
2021-12-19 22:37:55 +00:00
7476dfeec0
removes Select in favor of recv_timeout/try_iter ( #21981 )
...
crossbeam_channel::Select::ready_timeout might return with success spuriously.
2021-12-18 17:39:07 +00:00
3fe942ab30
new net-stats require a new table ( #21996 )
2021-12-18 00:13:16 -08:00
8f547a6c98
chore: bump serde from 1.0.131 to 1.0.132 ( #21989 )
...
* chore: bump serde from 1.0.131 to 1.0.132
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.131 to 1.0.132.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.131...v1.0.132 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-18 07:45:20 +00:00
7f6fb6937a
Ensure AncestorHashesSerice selects an open port ( #21919 )
2021-12-18 00:44:01 -05:00
97a1fa10a6
streamer send destination metrics for repair, gossip ( #21564 )
2021-12-17 15:21:05 -08:00
76098dd42a
RPC Block Subscription ( #21787 )
...
* add stuff
* compiling
* add notify block
* wip
* feat: add blockSubscribe pubsub method
* address PR comments
Co-authored-by: Lucas B <buffalu@jito.network>
Co-authored-by: Zano <segfaultdoctor@protonmail.com>
2021-12-17 16:03:09 -07:00
89d66c3210
removes next_shred_index from return value of entries to shreds api ( #21961 )
...
next-shred-index is already readily available from returned data shreds.
The commit simplifies the api for upcoming changes to erasure coding
schema which will require explicit tracking of indices for coding shreds
as well as data shreds.
2021-12-17 15:01:55 +00:00
7ec39f5a1e
time based retransmit in replay_stage ( #21498 )
2021-12-17 05:44:40 -08:00
385efae4b3
Remove need to send bank in retransmit request from ReplayStage ( #21943 )
...
* Remove need to send bank in retransmitter
2021-12-16 21:11:01 -05:00
6ff0be6a82
Clean up demote program write lock feature ( #21949 )
...
* Clean up demote program write lock feature
* fix test
2021-12-16 17:27:22 -05:00
e83ca4bb28
Clean up test_ledger_cleanup_compaction prints ( #21875 )
...
- Use info!()/warn!() over println!()/eprintln!()
- Make status prints consistent
- Add default RUST_LOG filter to see test printouts
- Adjust reported data to show shreds and rates we care about
2021-12-16 11:24:29 -06:00
cb395abff7
Fix subtraction overflow ( #21871 )
2021-12-14 14:24:22 -05:00
8d980f07ba
uses Option<Slot> for SlotMeta.parent_slot ( #21808 )
...
SlotMeta.parent_slot for the head of a detached chain of slots is
unknown and that is indicated by u64::MAX which lacks type-safety:
https://github.com/solana-labs/solana/blob/6c108c8fc/ledger/src/blockstore_meta.rs#L203-L205
The commit changes the type to Option<Slot>. Backward compatibility is
maintained by customizing serde serialize/deserialize implementations.
2021-12-14 18:57:11 +00:00
4ceb2689f5
adds ShredId uniquely identifying each shred ( #21820 )
2021-12-14 17:34:02 +00:00
4a9d7318d1
Rework test parameters to be shreds instead of entries ( #21780 )
...
The number of shreds that result from a given number of entries is
variable and in our test case, somewhat unintuitive to think about when
trying to determine how much data we're pushing into the blockstore. So,
this change converts the unit of test parameters from entries to shreds.
This change also cleans up some variable naming for clarity and prints.
2021-12-13 23:34:43 -06:00
018b54dbd7
chore: bump serde_json from 1.0.72 to 1.0.73 ( #21856 )
...
* chore: bump serde_json from 1.0.72 to 1.0.73
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.72 to 1.0.73.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.72...v1.0.73 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-14 04:08:22 +00:00
b81124deec
chore: bump etcd-client from 0.8.1 to 0.8.2 ( #21825 )
...
Bumps [etcd-client](https://github.com/etcdv3/etcd-client ) from 0.8.1 to 0.8.2.
- [Release notes](https://github.com/etcdv3/etcd-client/releases )
- [Commits](https://github.com/etcdv3/etcd-client/compare/0.8.1...0.8.2 )
---
updated-dependencies:
- dependency-name: etcd-client
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-12-13 16:20:22 -07:00
8aa3d690b5
chore: bump dashmap from 4.0.2 to 5.0.0 ( #21824 )
...
* chore: bump dashmap from 4.0.2 to 5.0.0
Bumps [dashmap](https://github.com/xacrimon/dashmap ) from 4.0.2 to 5.0.0.
- [Release notes](https://github.com/xacrimon/dashmap/releases )
- [Commits](https://github.com/xacrimon/dashmap/commits/v5.0.0 )
---
updated-dependencies:
- dependency-name: dashmap
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-13 13:23:58 -07:00
90f41fd9b7
use cost model to limit new account creation ( #21369 )
...
* use cost model to limit new account creation
* handle every system instruction
* remove &
* simplify match
* simplify match
* add datapoint for account data size
* add postgres error handling
* handle accounts:unlock_accounts
2021-12-12 14:57:18 -06:00
e08139f949
uses Option<u64> for SlotMeta.last_index ( #21775 )
...
SlotMeta.last_index may be unknown and current code is using u64::MAX to
indicate that:
https://github.com/solana-labs/solana/blob/6c108c8fc/ledger/src/blockstore_meta.rs#L169-L174
This lacks type-safety and can introduce bugs if not always checked for
Several instances of slot_meta.last_index + 1 are also subject to
overflow.
This commit updates the type to Option<u64>. Backward compatibility is
maintained by customizing serde serialize/deserialize implementations.
2021-12-11 14:47:20 +00:00
254ef3e7b6
Rename Packets to PacketBatch ( #21794 )
2021-12-11 09:44:15 -05:00
65194c7ae8
Add NUM_WRITERS to ledger_cleanup to enable multiple writers. ( #21729 )
...
Summary:
* Add NUM_WRITERS to ledger_cleanup to enable multiple writers.
(Note that our insert_shreds() is still single threaded because
it has a lock that limits only one writer at a time.)
* Make pre-generated slots more performent by directly inserting
into the shared queue. Otherwise, the main-thread which
prepares the slots will be slower than the writers.
* Correct the shred insertion time -- before this diff it did not
wait for joining all writer threads.
2021-12-10 09:42:51 -08:00
d870f566ef
chore: bump serde from 1.0.130 to 1.0.131 ( #21758 )
...
* chore: bump serde from 1.0.130 to 1.0.131
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.130 to 1.0.131.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.130...v1.0.131 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-10 08:47:46 +00:00
8063273d09
adds more sanity checks to shreds ( #21675 )
2021-12-09 16:43:57 +00:00
f0acf7681e
Add vote instructions that directly update on chain vote state ( #21531 )
...
* Add vote state instructions
UpdateVoteState and UpdateVoteStateSwitch
* cargo tree
* extract vote state version conversion to common fn
2021-12-07 16:47:26 -08:00
f6801a4af4
chore: bump itertools from 0.10.1 to 0.10.3 ( #21643 )
...
* chore: bump itertools from 0.10.1 to 0.10.3
Bumps [itertools](https://github.com/rust-itertools/itertools ) from 0.10.1 to 0.10.3.
- [Release notes](https://github.com/rust-itertools/itertools/releases )
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.10.1...v0.10.3 )
---
updated-dependencies:
- dependency-name: itertools
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-06 23:23:26 -07:00
cd17f63d81
adds back position field to coding-shred-header ( #21600 )
...
https://github.com/solana-labs/solana/pull/17004
removed position field from coding-shred-header because as it stands the
field is redundant and unused.
However, with the upcoming changes to erasure coding schema this field
will no longer be redundant and needs to be populated.
2021-12-05 14:42:09 +00:00
d6f22433d0
Bump version to v1.10.0
2021-12-04 20:17:54 +00:00
9c6b95e1e1
fix distance calculation in get_closest_completion ( #21601 )
2021-12-03 22:36:46 -08:00
446089edb4
chore: bump retain_mut from 0.1.4 to 0.1.5 ( #21599 )
...
Bumps [retain_mut](https://github.com/upsuper/retain_mut ) from 0.1.4 to 0.1.5.
- [Release notes](https://github.com/upsuper/retain_mut/releases )
- [Commits](https://github.com/upsuper/retain_mut/compare/v0.1.4...v0.1.5 )
---
updated-dependencies:
- dependency-name: retain_mut
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-12-03 22:11:42 -07:00
1430b58a6d
Remove deprecated slow epoch boundary methods ( #21568 )
2021-12-03 17:59:10 +00:00
b8837c04ec
Reformat imports to a consistent style for imports
...
rustfmt.toml configuration:
imports_granularity = "One"
group_imports = "One"
2021-12-03 09:19:13 -08:00
b78f5b6032
Refactor: Cleanup InstructionProcessor ( #21404 )
...
* Moves create_message(), native_invoke() and process_cross_program_instruction()
from the InstructionProcessor to the InvokeContext so that they can have a useful "self" parameter.
* Moves InstructionProcessor into InvokeContext and Bank.
* Moves ExecuteDetailsTimings into its own file.
* Moves Executor into invoke_context.rs
* Moves PreAccount into its own file.
* impl AbiExample for BuiltinPrograms
2021-12-01 08:54:42 +01:00
dd12d90eac
Upgrade to Rust 2021
2021-11-30 20:43:46 -08:00
de8edad30b
chore: bump etcd-client from 0.7.2 to 0.8.1 ( #21513 )
...
Bumps [etcd-client](https://github.com/etcdv3/etcd-client ) from 0.7.2 to 0.8.1.
- [Release notes](https://github.com/etcdv3/etcd-client/releases )
- [Commits](https://github.com/etcdv3/etcd-client/compare/0.7.2...0.8.1 )
---
updated-dependencies:
- dependency-name: etcd-client
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-11-30 18:16:32 +00:00
a125e7b8c2
chore: bump serde_json from 1.0.71 to 1.0.72 ( #21426 )
...
* chore: bump serde_json from 1.0.71 to 1.0.72
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.71 to 1.0.72.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.71...v1.0.72 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-11-30 18:03:09 +00:00
ba9dfa0d22
Remove frozen account support
2021-11-29 08:38:11 -08:00
9edfc5936d
Refactor accounts.rs with Justin's comments to improve lock accounts ( #21406 )
...
with results code path.
- fix a bug that could unlock accounts that weren't locked
- add test to the refactored function
- skip enumerating transaction accounts if qos results is an error
- add #[must_use] annotation
- avoid clone error in results
- add qos error code to unlock_accounts match statement
- remove unnecessary AbiExample
2021-11-23 21:17:55 +00:00
c29838fce1
Accountsdb plugin transaction part 3: Transaction Notifier ( #21374 )
...
The TransactionNotifierInterface interface for notifying transactions.
Changes to transaction_status_service to notify the notifier of the transaction data.
Interface to query the plugin's interest in transaction data
2021-11-23 09:55:53 -08:00
2602e7c3bc
Fix flaky test ( #21402 )
...
* the async test is flaky on ci
* fix unstable test by increasing stats repoting time
2021-11-23 09:47:17 -06:00
dd338b6c9f
changes Shred::parent return type to Option<Slot> ( #21370 )
...
Shred::parent can return garbage if the struct fields are invalid:
https://github.com/solana-labs/solana/blob/8a50b6302/ledger/src/shred.rs#L446-L453
The commit adds more sanity checks and changes the return type to Option<Slot>.
2021-11-23 14:45:26 +00:00
cd5a39ee43
the async test is flaky on ci ( #21365 )
2021-11-22 18:16:20 -06:00
87831e7f8d
start system monitor earlier in validator so we get memory stats at startup ( #21372 )
2021-11-22 14:37:17 -06:00
f31ca8ba8c
Report cluster slots size ( #21380 )
2021-11-22 17:47:58 +01:00
2ed7e3af89
prioritize slot repairs for unknown last index and close to completion ( #21070 )
2021-11-19 19:17:30 -08:00
0bda0c3e0c
Add bank drop service ( #21322 )
2021-11-19 17:20:18 +01:00
48dfdfb4d5
changes Blockstore::is_shred_duplicate arg type to ShredType
2021-11-19 14:16:39 +00:00
57057f8d39
uses enum for shred type
...
Current code is using u8 which does not have any type-safety and can
contain invalid values:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L167
Checks for invalid shred-types are scattered through the code:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/blockstore.rs#L849-L851
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L346-L348
The commit uses enum for shred type with #[repr(u8)]. Backward
compatibility is maintained by implementing Serialize and Deserialize
compatible with u8, and adding a test to assert that.
2021-11-19 14:16:39 +00:00
b30c94ce55
ClusterInfoVoteListener send only missing votes to BankingStage ( #20873 )
2021-11-18 15:20:41 -08:00
0ca255220e
- Encapsulate QoS Service metrics reporting within QosServioce, so client ( #21191 )
...
code (eg banking_stage) doesn't need to worry about it.
- Remove dead cost_* stats from banking_stage, clean up call path.
2021-11-18 15:35:30 -06:00
89102540b1
chore: bump serde_json from 1.0.70 to 1.0.71 ( #21338 )
...
* chore: bump serde_json from 1.0.70 to 1.0.71
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.70 to 1.0.71.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.70...v1.0.71 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-11-18 02:58:30 +00:00
89c45a57f8
Refactor slot status notification to decouple from accounts notifications ( #21308 )
...
Problem
Slot status can be used of in other scenarios in addition to account information such as transactions, blocks. The current implementation is too tightly coupled.
Summary of Changes
Decouple the slot status notification from accounts notification. Created a new slot status notification module.
2021-11-17 17:11:38 -08:00
66fa062f13
rename process_entries to indicate it's only for tests ( #21321 )
2021-11-17 20:53:40 +01:00
d5de0c8e12
add --no-os-network-stats-reporting option ( #21296 )
2021-11-16 10:26:03 -08:00
398af132a5
More set_root metrics ( #21286 )
2021-11-15 16:28:18 -07:00
55ea672041
chore: bump serde_json from 1.0.69 to 1.0.70 ( #21285 )
...
* chore: bump serde_json from 1.0.69 to 1.0.70
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.69 to 1.0.70.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.69...v1.0.70 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-11-15 15:27:47 -07:00
f2bd9947cc
mem stats: rescale from kb to bytes ( #21282 )
2021-11-15 14:42:41 -06:00
f8dcb2f38b
report mem stats ( #21258 )
2021-11-13 00:59:41 +00:00
b0ca335463
Rename "trusted" to "known" in `validators/` ( #21197 )
...
* Replaced trusted with known validator
* Format Convention
2021-11-12 11:57:55 -07:00
11153e1f87
refactor cost calculation ( #21062 )
...
* - cache calculated transaction cost to allow sharing;
- atomic cost tracking op;
- only lock accounts for transactions eligible for current block;
- moved qos service and stats reporting to its own model;
- add cost_weight default to neutral (as 1), vote has zero weight;
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Update core/src/qos_service.rs
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Update core/src/qos_service.rs
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2021-11-12 01:04:53 -06:00
5fb0ab9d00
removes redundant args from Shredder::try_recovery ( #21226 )
...
Shredder::try_recovery is already taking a Vec<Shred> as an argument. All the
other arguments are embedded in the shreds, and are so redundant.
2021-11-10 21:19:03 +00:00
3c5f505d3e
chore: bump serde_json from 1.0.68 to 1.0.69 ( #21211 )
...
* chore: bump serde_json from 1.0.68 to 1.0.69
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.68 to 1.0.69.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.68...v1.0.69 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-11-09 11:04:40 -07:00
c78f474373
Add validator option to change niceness of snapshot packager thread
2021-11-04 17:16:46 -06:00
7200c5106e
Replaces MockInvokeContext by ThisInvokeContext in tests ( #20881 )
...
* Replaces MockInvokeContext by ThisInvokeContext in BpfLoader, SystemInstructionProcessor, CLIs, ConfigProcessor, StakeProcessor and VoteProcessor.
* Finally, removes MockInvokeContext, MockComputeMeter and MockLogger.
* Adjusts assert_instruction_count test.
* Moves ThisInvokeContext to the program-runtime crate.
2021-11-04 21:47:32 +01:00
140a5f633d
Simplify replay vote tracking by using packet metadata ( #21112 )
2021-11-03 09:02:48 +00:00
e6280fc1fa
Add additional checks for should_retransmit_and_persist() ( #20672 )
...
Add additional checks to should_retransmit_and_persist()
- Check invalid shred index
- Update cases that check if node was leader
- Some comments and variable rename for clarity
2021-11-03 02:01:07 -05:00
a8d78e89d3
Move test-validator to own module to reduce core dependencies ( #20658 )
...
* Move test-validator to own module to reduce core dependencies
* Fix a few TestValidator paths
* Use solana_test_validator crate for solana_test_validator bin
* Move client int tests to separate crate
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-10-29 01:27:07 +00:00
036d7fcc81
Clean up sanitized tx creation for tests ( #21006 )
2021-10-27 18:09:16 +01:00
261dd96ae3
Swap banking stage vote channels ( #20987 )
2021-10-26 21:20:31 +02:00
1297a13586
adds metrics tracking crds writes and votes ( #20953 )
2021-10-26 13:02:30 +00:00
8efba88dde
chore: bump systemstat from 0.1.8 to 0.1.10 ( #20929 )
...
Bumps [systemstat](https://github.com/unrelentingtech/systemstat ) from 0.1.8 to 0.1.10.
- [Release notes](https://github.com/unrelentingtech/systemstat/releases )
- [Commits](https://github.com/unrelentingtech/systemstat/compare/v0.1.8...v0.1.10 )
---
updated-dependencies:
- dependency-name: systemstat
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-10-25 20:32:15 +00:00
43ea579f63
add cli for --accounts-hash-num-passes ( #20827 )
2021-10-25 09:45:46 -05:00
c2bfce90b3
- cost_tracker is data member of a bank, it can report metrics when bank is frozen ( #20802 )
...
- removed cost_tracker_stats and histogram
- move stats reporting outside of bank freeze
2021-10-24 22:19:23 -05:00
5e1cf39c74
adds metrics for number of outgoing shreds in retransmit stage ( #20882 )
2021-10-24 13:12:27 +00:00
350bb561eb
Clippy
2021-10-23 08:21:20 +00:00
bfbbc53dac
Divorce the runtime from FeeCalculator ( #20737 )
2021-10-22 14:32:40 -07:00
735016661b
Report timing info for stakes cache updates from txs ( #20856 )
2021-10-22 12:49:02 -04:00
0ac89841bf
chore: bump reqwest from 0.11.5 to 0.11.6 ( #20844 )
...
* chore: bump reqwest from 0.11.5 to 0.11.6
Bumps [reqwest](https://github.com/seanmonstar/reqwest ) from 0.11.5 to 0.11.6.
- [Release notes](https://github.com/seanmonstar/reqwest/releases )
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.5...v0.11.6 )
---
updated-dependencies:
- dependency-name: reqwest
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
2021-10-21 12:54:18 -06:00
73a7abdadf
chore: bump ahash from 0.7.5 to 0.7.6 ( #20721 )
...
Bumps [ahash](https://github.com/tkaitchuck/ahash ) from 0.7.5 to 0.7.6.
- [Release notes](https://github.com/tkaitchuck/ahash/releases )
- [Commits](https://github.com/tkaitchuck/ahash/commits )
---
updated-dependencies:
- dependency-name: ahash
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-10-21 17:52:21 +00:00
71d0bd4605
Add counter for dropped duplicated packets, fix dropped_packets_count ( #20834 )
2021-10-21 02:56:48 +00:00
fe098b5ddc
rpc-send-tx-svc: add with_config constructor
2021-10-20 13:43:27 -06:00
95e91a4863
disable gossip publish of snapshots when using filler accts ( #20824 )
2021-10-20 18:07:29 +00:00
7496b5784b
- make cost_tracker a member of bank, remove shared instance from TPU; ( #20627 )
...
- decouple cost_model from cost_tracker; allowing one cost_model
instance being shared within a validator;
- update cost_model api to calculate_cost(&self...)->transaction_cost
2021-10-19 14:37:33 -05:00
4cac66244d
report udp stats from validator ( #20587 )
2021-10-15 15:11:11 -07:00
44ff30b65b
Retry `SampleNotDuplicateConfirmed` decisions in AncestorHashesService ( #20240 )
2021-10-15 11:40:03 -07:00
0f03971c3c
adds counters for errors in window-service run_insert ( #20670 )
2021-10-15 14:13:26 +00:00
0c0384ec32
revises turbine peers shuffling order ( #20480 )
...
Turbine randomly shuffles cluster nodes on a broadcast tree for each
shred. This requires knowing the stakes and nodes' contact-infos (from
gossip).
However gossip is subject to partitioning and propogation delays.
Additionally unstaked nodes may join and leave the cluster at any
moment, changing the cluster view from one node to another.
This commit:
* Always arranges the unstaked nodes at the bottom of turbine broadcast
tree.
* Staked nodes are always included regardless of if their contact-info
is available in gossip or not.
* Uses the unbiased WeightedShuffle construct for shuffling nodes.
2021-10-14 15:09:36 +00:00
588168b99d
Add check for shred data header size ( #20668 )
2021-10-14 05:56:14 +02:00
da45be366a
Remove blockhash from fee calculation ( #20641 )
2021-10-13 13:10:58 -07:00
09e7782d76
Refactor code to get block signatures in get_confirmed_signatures_for_address2 ( #20575 )
...
* Refactor get_confirmed_signatures_for_address2
* Move blockstore benches to ledger where they belong
2021-10-13 09:55:19 +02:00
005d6863fd
- move cost tracker into bank, so each bank has its own cost tracker; ( #20527 )
...
- move related modules to runtime
2021-10-12 08:51:33 -05:00
a4c9b0f358
chore: bump ahash from 0.7.4 to 0.7.5 ( #20592 )
...
* chore: bump ahash from 0.7.4 to 0.7.5
Bumps [ahash](https://github.com/tkaitchuck/ahash ) from 0.7.4 to 0.7.5.
- [Release notes](https://github.com/tkaitchuck/ahash/releases )
- [Commits](https://github.com/tkaitchuck/ahash/commits )
---
updated-dependencies:
- dependency-name: ahash
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* Bump ahash patch
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-10-11 14:41:35 -06:00
a8e000a2a6
add filler accounts to bloat validator and predict failure ( #20491 )
...
* add filler accounts to bloat validator and predict failure
* assert no accounts match filler
* cleanup magic numbers
* panic if can't load from snapshot with filler accounts specified
* some renames
* renames
* into_par_iter
* clean filler accts, too
2021-10-11 12:46:27 -05:00
c16510152e
Rework AVX/AVX2 detection again
2021-10-10 12:22:10 -07:00
838ff3b871
Separate out interrupted slots broadcast metrics ( #20537 )
2021-10-09 01:46:06 -07:00
d621994fee
Accountsdb stream plugin improvement ( #20419 )
...
Support using connection pooling and use multiple threads to do Postgres db operations. The performance is improved from 1500 RPS to 40,000 RPS measured during validator start.
Support multiple plugins at the same time.
2021-10-08 20:06:58 -07:00
5440c1d2e1
SnapshotPackagerService pushes incremental snapshot hashes to CRDS ( #20442 )
...
Now that CRDS supports incremental snapshot hashes,
SnapshotPackagerService needs to push 'em!
This commit does two main things:
1. SnapshotPackagerService now knows about incremental snapshot hashes,
and will push SnapshotPackage::IncrementalSnapshot hashes to CRDS.
2. At startup, when loading from a full + incremental snapshot, the
hashes need to be passed all the way to SnapshotPackagerService so it
can push these starting hashes to CRDS. Those values have been piped
through.
Fixes #20441 and #20423
2021-10-08 15:14:56 -05:00
675fa6993b
- update const cost values with data collected by #19627 ( #20314 )
...
- update cost calculation to closely proposed fee schedule #16984
2021-10-08 14:48:50 -05:00
8422a12ac6
chore: bump reqwest from 0.11.4 to 0.11.5 ( #20536 )
...
* chore: bump reqwest from 0.11.4 to 0.11.5
Bumps [reqwest](https://github.com/seanmonstar/reqwest ) from 0.11.4 to 0.11.5.
- [Release notes](https://github.com/seanmonstar/reqwest/releases )
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.4...v0.11.5 )
---
updated-dependencies:
- dependency-name: reqwest
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
2021-10-08 13:25:31 -06:00
Will Hickey
apfitzge
Jeff Washington (jwash)
behzad nouri
Christian Kamm
Michael Vines
apfitzge
Maximilian Schneider
dependabot[bot]
Xiang Zhu
Brooks Prumo
Lijun Wang
Jeff Biseda
Brennan Watt
Tyera Eulberg
Tao Zhu
Tyera Eulberg
Haoran Yi
Trent Nelson
Jeff Washington (jwash)
Ashwin Sekar
Justin Starry
janlegner
Yueh-Hsuan Chiang
Richard Patel
steviez
Boqin Qin(秦 伯钦)
github-actions[bot]
Pankaj Garg
carllin
Jack May
sakridge
Nicholas Clarke
Nick Rempel
Steven Luscher
Ryo Onodera
Brian Anderson
Jon Cinque
ryleung-solana
buffalu
DimAn
Jason Davis
Tao Zhu
Giorgio Gambino
Gavin Chan
anatoly yakovenko
Ikko Ashimine
Eric Warehime
carllin
segfaultdoctor
Alexander Meißner
Jeff Washington (jwash)
Michael Keleti
Ivan Mironov