5625959f7e
(LedgerStore) Change perf_samples_counter from Arc<AtomicUsize> to AtomicUsize ( #25043 )
...
#### Problem
After #25042 , each LedgerColumn has its own BlockstoreRocksDbWritePerfMetrics
and BlockstoreRocksDbReadPerfMetrics instances. As it has total ownership,
its member field does not need to use Arc.
#### Summary of Changes
Change perf_samples_counter from Arc<AtomicUsize> to AtomicUsize
under BlockstoreRocksDbWritePerfMetrics and BlockstoreRocksDbReadPerfMetrics.
2022-05-16 11:31:07 -07:00
bc005e3408
Add configurable limit to number of blocks to check before Bigtable upload ( #24716 )
...
* Add ConfirmedBlockUploadConfig, no behavior changes
* Add comment
* A little DRY cleanup
* Add configurable limit to number of blocks to check in Blockstore and Bigtable before uploading
* Limit blockstore and bigtable look-ahead
* Exit iterator early when reach ending_slot
* Use rooted_slot_iterator instead of slot_meta_iterator
* Only check blocks in the ledger
2022-05-13 07:34:02 +00:00
08da486c05
additional costs in block capacity calc ( #25059 )
...
* Added additional costs to block capacity computation, and pushed alloc of CostModel all the way to the top of the call chain, instead of reallocing
* Fix two compiler errors
* Update block processing to propagate computed costs, rather than re-computing deeper in the call stack
* Clippy fix
* Reformatting fix after merge
* Add CostModel::sum_without_bpf
2022-05-12 13:52:20 -05:00
b2dcda8980
(LedgerStore) Move metric sample counters out from LedgerColumnOptions ( #25042 )
...
#### Problem
LedgerColumnOptions contain two fields, perf_read_counter and perf_write_counter,
that are not really options but internal counters.
#### Summary of Changes
This PR introduces BlockstoreRocksDbPerfSamplingStatus, a struct that holds internal
status for RocksDB perf sampling and moves perf_read_counter and perf_write_counter
out from LedgerColumnOptions.
2022-05-10 16:13:19 -07:00
c838e15234
Unset needs_unlock for rebatched transactions batches ( #25095 )
...
* Unset needs_unlock for rebatched transactions batches
* address review comments
2022-05-10 13:39:08 -07:00
2fa9bc3e70
Add options to store full and/or incremental snapshots in separate locations ( #24247 )
2022-05-10 16:37:41 -04:00
85f4a3e4c4
chore: bump reed-solomon-erasure from 5.0.1 to 5.0.2 ( #25081 )
...
* chore: bump reed-solomon-erasure from 5.0.1 to 5.0.2
Bumps [reed-solomon-erasure](https://github.com/darrenldl/reed-solomon-erasure ) from 5.0.1 to 5.0.2.
- [Release notes](https://github.com/darrenldl/reed-solomon-erasure/releases )
- [Changelog](https://github.com/rust-rse/reed-solomon-erasure/blob/master/CHANGELOG.md )
- [Commits](https://github.com/darrenldl/reed-solomon-erasure/compare/v5.0.1...v5.0.2 )
---
updated-dependencies:
- dependency-name: reed-solomon-erasure
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2022-05-10 10:28:44 -06:00
082502d4f3
Fail tx sanitization when ix program id uses lookup table ( #25035 )
...
* Fail tx sanitization when ix program id uses lookup table
* feedback
2022-05-07 03:19:50 +08:00
8a4f975416
chore: bump prost from 0.10.1 to 0.10.3 ( #25038 )
...
* chore: bump prost from 0.10.1 to 0.10.3
Bumps [prost](https://github.com/tokio-rs/prost ) from 0.10.1 to 0.10.3.
- [Release notes](https://github.com/tokio-rs/prost/releases )
- [Commits](https://github.com/tokio-rs/prost/compare/v0.10.1...v0.10.3 )
---
updated-dependencies:
- dependency-name: prost
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-05-06 11:14:58 -06:00
492f89a170
checks account owner when initializing a vote-account ( #25018 )
...
A VoteAccount may only wrap an account if the account owner is
solana_vote_program:id or equivalently this check returns true:
solana_vote_program::check_id(account.owner())
2022-05-06 16:22:49 +00:00
c4aa082efc
Rollback tokio to LTS release v1.14 ( #25028 )
2022-05-06 06:34:49 +00:00
a01291069a
initializes thread-pools with lazy_static instead of thread_local ( #24853 )
...
In addition to thread_local -> lazy_static change, a number of thread-pools are
initialized with get_max_thread_count to achieve parity with the older code in
terms of number of validator threads.
2022-05-05 20:00:50 +00:00
bece7f32c8
chore: bump log from 0.4.16 to 0.4.17 ( #24987 )
...
* chore: bump log from 0.4.16 to 0.4.17
Bumps [log](https://github.com/rust-lang/log ) from 0.4.16 to 0.4.17.
- [Release notes](https://github.com/rust-lang/log/releases )
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/log/commits/0.4.17 )
---
updated-dependencies:
- dependency-name: log
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-05-05 01:49:18 +00:00
e8bf812a8a
allow ledger-tool halt at slot to calc hash using write cache ( #24974 )
2022-05-04 16:27:49 -05:00
9258d81ba3
chore: bump serde from 1.0.136 to 1.0.137 ( #24957 )
...
* chore: bump serde from 1.0.136 to 1.0.137
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.136 to 1.0.137.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.136...v1.0.137 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-05-03 22:14:59 -06:00
63bd0cdd5d
(LedgerStore) Move BlockstoreRocksDbColumnFamilyMetrics to blockstore_metric.rs ( #24856 )
...
#### Problem
blockstore_db.rs becomes bigger.
#### Summary of Changes
Move BlockstoreRocksDbColumnFamilyMetrics to blockstore_metric.rs out from blockstore_db.rs.
2022-05-03 14:46:59 -07:00
8fa2913d08
chore: bump libc from 0.2.124 to 0.2.125 ( #24914 )
...
* chore: bump libc from 0.2.124 to 0.2.125
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.124 to 0.2.125.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.124...0.2.125 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-05-03 10:10:55 -06:00
9587c8537f
limits pre-allocation size when deserializing shreds ( #24921 )
...
Though current Shred struct is not vulnerable to this, adding with_limit
causes pre-allocation size to be limited to prevent against memory
exhaustion attacks:
https://github.com/bincode-org/bincode/blob/2d3f42034/readme.md
2022-05-03 12:13:45 +00:00
0b9d04808f
(LedgerStore) Move trait ColumnMetrics and metric-macros to blockstore_metric.rs ( #24855 )
...
#### Problem
blockstore_db.rs becomes bigger.
#### Summary of Changes
Move trait ColumnMetrics and metric-macros to blockstore_metric.rs out from blockstore_db.rs.
2022-05-02 22:58:31 -07:00
eca0eb9585
(LedgerStore) Move metric-related functions to blockstore_metric.rs ( #24854 )
...
#### Problem
blockstore_db.rs becomes bigger.
#### Summary of Changes
This PR creates blockstore_metric.rs and moves metric-related functions out from blockstore_db.rs.
2022-05-02 20:53:25 -07:00
eff59193db
enforces that LAST_SHRED_IN_SLOT is also DATA_COMPLETE_SHRED ( #24892 )
...
A data shred cannot be LAST_SHRED_IN_SLOT if not also DATA_COMPLETE_SHRED.
So LAST_SHRED_IN_SLOT should also imply DATA_COMPLETE_SHRED:
https://github.com/solana-labs/solana/blob/74b586ae7/ledger/src/shredder.rs#L116-L117
https://github.com/solana-labs/solana/blob/74b586ae7/core/src/broadcast_stage/standard_broadcast_run.rs#L80-L81
However current shred constructs allow specifying a shred which is
LAST_SHRED_IN_SLOT but not DATA_COMPLETE_SHRED:
https://github.com/solana-labs/solana/blob/74b586ae7/ledger/src/shred.rs#L117-L118
https://github.com/solana-labs/solana/blob/74b586ae7/ledger/src/shred.rs#L272-L273
The commit updates ShredFlags so that if a shred is not
DATA_COMPLETE_SHRED it cannot be LAST_SHRED_IN_SLOT either.
2022-05-02 23:33:53 +00:00
cb96edcaa7
Pass ProcessOptions struct around by reference ( #24900 )
2022-05-02 16:58:00 -05:00
68c7fb941d
chore: bump serde_bytes from 0.11.5 to 0.11.6 ( #24882 )
...
* chore: bump serde_bytes from 0.11.5 to 0.11.6
Bumps [serde_bytes](https://github.com/serde-rs/bytes ) from 0.11.5 to 0.11.6.
- [Release notes](https://github.com/serde-rs/bytes/releases )
- [Commits](https://github.com/serde-rs/bytes/compare/0.11.5...0.11.6 )
---
updated-dependencies:
- dependency-name: serde_bytes
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-05-02 14:33:01 -06:00
4f947a0db3
(de-)serializes shred headers through a Cursor ( #24876 )
...
Current serialize/de-serialize code for shreds manually tracks offsets
into the payload; This can instead be done with std::io::Cursor.
https://github.com/solana-labs/solana/blob/e812430e2/ledger/src/shred.rs#L232-L258
2022-05-02 15:02:06 +00:00
e812430e28
defines shred flags using bitflags crate ( #24874 )
...
Shred flags uses raw bit-masking ops which lacks type-safety:
https://github.com/solana-labs/solana/blob/a829ddc92/ledger/src/shred.rs#L112-L114
This commit instead uses bitflags crate to define shred flags.
2022-05-01 19:25:15 +00:00
a829ddc922
adds hard-coded tests for backward compatibility of shreds serialization ( #24866 )
...
In preparation of the upcoming changes to shred struct, the added
hard-coded tests in this commit ensure that shreds are backward
compatible when serialized and de-serialized.
2022-05-01 13:11:45 +00:00
428cf54c91
Change BlockStore TryPrimaryThenSecondary to just Secondary ( #23391 )
2022-04-29 20:05:39 -05:00
5245eb4229
(LedgerStore) Hidden validator argument for RocksDB perf samples ( #24684 )
...
#### Summary of Changes
This PR replaces the use of thread_rng in RocksDB perf metric samples by
AtomicU32 with Ordering::Relaxed to improve the performance of determining
whether to sample the current RocksDB's read/write perf metric.
2022-04-29 17:55:34 -07:00
d7aec2a7e6
moves ErasureConfig to blockstore_meta next to ErasureMeta
2022-04-29 23:02:29 +00:00
4b62c93f87
removes erasure::Session, using Reed-Solomon constructs directly
...
The extra wrapping and indirection by the Session struct is not used in
any form. The commit removes Session and instead uses Reed-Solomon
constructs directly.
2022-04-29 23:02:29 +00:00
b56c091b37
(LedgerStore) Hidden validator argument for RocksDB perf samples ( #24682 )
...
#### Problem
Currently, the number of RocksDB perf samples is controlled by an env arg
which is later handled using a lazy_static variable. However, there is a known
performance overhead of using lazy_static as mentioned in
https://github.com/solana-labs/solana/pull/6472 .
#### Summary of Changes
Instead, this PR uses a hidden validator argument, --rocksdb-perf-sample-interval,
for controlling how often RocksDB read/write performance sample is collected.
2022-04-29 15:28:50 -07:00
08e1727926
minor edits in shred and shredder ( #24841 )
...
Removed Default implementation for ShredType. ShredType should always be
explicitly specified, and not rely on default values.
Simplified single-arg Shred Error variants to use shorter syntax.
Renamed erasure blocks to shards, to be consistent with reed_solomon
crate and not to confuse with FEC blocks.
2022-04-29 19:42:15 +00:00
27efcae16c
(LedgerStore) Convert Rocks from tuple to struct with named fields ( #24761 )
...
#### Problem
The RocksDB wrapper,`Rocks`, under blockstore_db is currently implemented
as a tuple with unnamed fields. Accessing its fields requires syntax like `self.0`
which limits readability.
#### Summary of Changes
This PR converts Rocks from tuple to struct so that it has more human-readable
fields.
2022-04-28 21:32:48 -07:00
008860bb36
removes SHRED_PAYLOAD_SIZE from shred public interface ( #24806 )
2022-04-28 23:42:37 +00:00
4e58b3870c
Update all BankForks methods to return owned values ( #24801 )
2022-04-28 18:51:00 +00:00
b6a18e0e87
Fix get_first_available_block for genesis; also make get_blocks(_with_limit) consistent ( #24760 )
...
* Handle genesis more appropriately in get_first_available_block
* Add unit test
* Make get_blocks starting-slots consistent with other methods
2022-04-28 00:36:19 -06:00
1e86d3ab59
moves shredder out of the shred module ( #24755 )
...
Working towards embedding versioning into shreds binary; so Shredder can
no longer depend on internals of Shred.
2022-04-27 22:00:04 +00:00
0f60665100
replaces Shred::new_empty_coding with Shred::new_from_parity_shard ( #24749 )
...
Removing implementation details of shreds and payload offsets from
shredder, so that shredder does not need to mutate payload:
https://github.com/solana-labs/solana/blob/71ad12128/ledger/src/shred.rs#L968-L977
Also, Shred::new_from_data can simply obtain a slice as opposed to
Option<&[u8]>:
https://github.com/solana-labs/solana/blob/71ad12128/ledger/src/shred.rs#L268-L278
2022-04-27 18:04:10 +00:00
081c844d6e
removes Shred::new_empty_data_shred ( #24714 )
...
Shred::new_empty_data_shred returns an invalid shred (i.e.
shred.sanitize() returns error). The method is only used in tests and
can be easily replaced with Shred::new_from_data. To keep the shred api
surface small, this commit removes this method.
2022-04-26 23:13:12 +00:00
913ad79cd9
patches bug in data-shred index sanitize ( #24707 )
...
https://github.com/solana-labs/solana/pull/24653
introduced an off-by-one error in data-shred index sanitize.
2022-04-26 21:10:27 +00:00
b6bc5d2fa1
Aggregate executor update timing metrics
2022-04-26 23:15:22 +08:00
edf3f05ab5
Make test_next_leader_slot_next_epoch() aware of stake minimum delegation ( #24660 )
2022-04-26 08:31:06 -04:00
12ae8d3be5
returns Error when Shred::sanitize fails ( #24653 )
...
Including the error in the output allows to debug when Shred::sanitize
fails.
2022-04-25 23:19:37 +00:00
74b05eebd8
chore: bump num_enum from 0.5.4 to 0.5.7 ( #24634 )
...
Bumps [num_enum](https://github.com/illicitonion/num_enum ) from 0.5.4 to 0.5.7.
- [Release notes](https://github.com/illicitonion/num_enum/releases )
- [Commits](https://github.com/illicitonion/num_enum/commits )
---
updated-dependencies:
- dependency-name: num_enum
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-25 16:05:59 -06:00
895f76a93c
hides implementation details of shred from its public interface ( #24563 )
...
Working towards embedding versioning into shreds binary, so that a new
variant of shred struct can include merkle tree hashes of the erasure
set.
2022-04-25 12:43:22 +00:00
92ad767738
simplifies Serialize/Deserialize implementations for ShredType ( #24601 )
2022-04-23 13:33:59 +00:00
83e041299a
Run real snapshot packager while processing blockstore at validator startup
2022-04-22 21:17:49 -07:00
c544742091
Local cluster test cleanup and refactoring ( #24559 )
...
* remove FixedSchedule.start_epoch
* use duration for timing
* Rename to partition bool to turbine_disabled
* simplify partition config
2022-04-22 12:14:07 +08:00
d5127abf46
Only add hashes for completed blocks to recent blockhashes ( #24389 )
...
* Only add hashes for completed blocks to recent blockhashes
* feedback
2022-04-21 21:05:29 +08:00
96d977fd05
test_shred_fec_set_index fix ( #24540 )
2022-04-20 16:26:44 -07:00
Yueh-Hsuan Chiang
Tyera Eulberg
Jason
Pankaj Garg
DimAn
dependabot[bot]
Justin Starry
behzad nouri
Jeff Washington (jwash)
steviez
Brooks Prumo
Michael Vines
Jeff Biseda