blockworks-foundation
/
solana
mirror of https://github.com/blockworks-foundation/solana.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
18,774 Commits 168 Branches 497 Tags 539 MiB
Commit Graph

901 Commits

Author SHA1 Message Date
Jeff Washington (jwash) e8bf812a8a
allow ledger-tool halt at slot to calc hash using write cache (#24974) 2022-05-04 16:27:49 -05:00
dependabot[bot] 9258d81ba3
chore: bump serde from 1.0.136 to 1.0.137 (#24957) 
* chore: bump serde from 1.0.136 to 1.0.137

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.136 to 1.0.137.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.136...v1.0.137)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-05-03 22:14:59 -06:00
Yueh-Hsuan Chiang 63bd0cdd5d
(LedgerStore) Move BlockstoreRocksDbColumnFamilyMetrics to blockstore_metric.rs (#24856) 
#### Problem
blockstore_db.rs becomes bigger.

#### Summary of Changes
Move BlockstoreRocksDbColumnFamilyMetrics to blockstore_metric.rs out from blockstore_db.rs.
 2022-05-03 14:46:59 -07:00
dependabot[bot] 8fa2913d08
chore: bump libc from 0.2.124 to 0.2.125 (#24914) 
* chore: bump libc from 0.2.124 to 0.2.125

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.124 to 0.2.125.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.124...0.2.125)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-05-03 10:10:55 -06:00
behzad nouri 9587c8537f
limits pre-allocation size when deserializing shreds (#24921) 
Though current Shred struct is not vulnerable to this, adding with_limit
causes pre-allocation size to be limited to prevent against memory
exhaustion attacks:
https://github.com/bincode-org/bincode/blob/2d3f42034/readme.md
 2022-05-03 12:13:45 +00:00
Yueh-Hsuan Chiang 0b9d04808f
(LedgerStore) Move trait ColumnMetrics and metric-macros to blockstore_metric.rs (#24855) 
#### Problem
blockstore_db.rs becomes bigger.

#### Summary of Changes
Move trait ColumnMetrics and metric-macros to blockstore_metric.rs out from blockstore_db.rs.
 2022-05-02 22:58:31 -07:00
Yueh-Hsuan Chiang eca0eb9585
(LedgerStore) Move metric-related functions to blockstore_metric.rs (#24854) 
#### Problem
blockstore_db.rs becomes bigger.

#### Summary of Changes
This PR creates blockstore_metric.rs and moves metric-related functions out from blockstore_db.rs.
 2022-05-02 20:53:25 -07:00
behzad nouri eff59193db
enforces that LAST_SHRED_IN_SLOT is also DATA_COMPLETE_SHRED (#24892) 
A data shred cannot be LAST_SHRED_IN_SLOT if not also DATA_COMPLETE_SHRED.
So LAST_SHRED_IN_SLOT should also imply DATA_COMPLETE_SHRED:
https://github.com/solana-labs/solana/blob/74b586ae7/ledger/src/shredder.rs#L116-L117
https://github.com/solana-labs/solana/blob/74b586ae7/core/src/broadcast_stage/standard_broadcast_run.rs#L80-L81

However current shred constructs allow specifying a shred which is
LAST_SHRED_IN_SLOT but not DATA_COMPLETE_SHRED:
https://github.com/solana-labs/solana/blob/74b586ae7/ledger/src/shred.rs#L117-L118
https://github.com/solana-labs/solana/blob/74b586ae7/ledger/src/shred.rs#L272-L273

The commit updates ShredFlags so that if a shred is not
DATA_COMPLETE_SHRED it cannot be LAST_SHRED_IN_SLOT either.
 2022-05-02 23:33:53 +00:00
steviez cb96edcaa7
Pass ProcessOptions struct around by reference (#24900) 2022-05-02 16:58:00 -05:00
dependabot[bot] 68c7fb941d
chore: bump serde_bytes from 0.11.5 to 0.11.6 (#24882) 
* chore: bump serde_bytes from 0.11.5 to 0.11.6

Bumps [serde_bytes](https://github.com/serde-rs/bytes) from 0.11.5 to 0.11.6.
- [Release notes](https://github.com/serde-rs/bytes/releases)
- [Commits](https://github.com/serde-rs/bytes/compare/0.11.5...0.11.6)

---
updated-dependencies:
- dependency-name: serde_bytes
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-05-02 14:33:01 -06:00
behzad nouri 4f947a0db3
(de-)serializes shred headers through a Cursor (#24876) 
Current serialize/de-serialize code for shreds manually tracks offsets
into the payload; This can instead be done with std::io::Cursor.
https://github.com/solana-labs/solana/blob/e812430e2/ledger/src/shred.rs#L232-L258
 2022-05-02 15:02:06 +00:00
behzad nouri e812430e28
defines shred flags using bitflags crate (#24874) 
Shred flags uses raw bit-masking ops which lacks type-safety:
https://github.com/solana-labs/solana/blob/a829ddc92/ledger/src/shred.rs#L112-L114

This commit instead uses bitflags crate to define shred flags.
 2022-05-01 19:25:15 +00:00
behzad nouri a829ddc922
adds hard-coded tests for backward compatibility of shreds serialization (#24866) 
In preparation of the upcoming changes to shred struct, the added
hard-coded tests in this commit ensure that shreds are backward
compatible when serialized and de-serialized.
 2022-05-01 13:11:45 +00:00
steviez 428cf54c91
Change BlockStore TryPrimaryThenSecondary to just Secondary (#23391) 2022-04-29 20:05:39 -05:00
Yueh-Hsuan Chiang 5245eb4229
(LedgerStore) Hidden validator argument for RocksDB perf samples (#24684) 
#### Summary of Changes
This PR replaces the use of thread_rng in RocksDB perf metric samples by
AtomicU32 with Ordering::Relaxed to improve the performance of determining
whether to sample the current RocksDB's read/write perf metric.
 2022-04-29 17:55:34 -07:00
behzad nouri d7aec2a7e6 moves ErasureConfig to blockstore_meta next to ErasureMeta 2022-04-29 23:02:29 +00:00
behzad nouri 4b62c93f87 removes erasure::Session, using Reed-Solomon constructs directly 
The extra wrapping and indirection by the Session struct is not used in
any form. The commit removes Session and instead uses Reed-Solomon
constructs directly.
 2022-04-29 23:02:29 +00:00
Yueh-Hsuan Chiang b56c091b37
(LedgerStore) Hidden validator argument for RocksDB perf samples (#24682) 
#### Problem
Currently, the number of RocksDB perf samples is controlled by an env arg
which is later handled using a lazy_static variable.  However, there is a known
performance overhead of using lazy_static as mentioned in
https://github.com/solana-labs/solana/pull/6472.

#### Summary of Changes
Instead, this PR uses a hidden validator argument, --rocksdb-perf-sample-interval,
for controlling how often RocksDB read/write performance sample is collected.
 2022-04-29 15:28:50 -07:00
behzad nouri 08e1727926
minor edits in shred and shredder (#24841) 
Removed Default implementation for ShredType. ShredType should always be
explicitly specified, and not rely on default values.

Simplified single-arg Shred Error variants to use shorter syntax.

Renamed erasure blocks to shards, to be consistent with reed_solomon
crate and not to confuse with FEC blocks.
 2022-04-29 19:42:15 +00:00
Yueh-Hsuan Chiang 27efcae16c
(LedgerStore) Convert Rocks from tuple to struct with named fields (#24761) 
#### Problem
The RocksDB wrapper,`Rocks`, under blockstore_db is currently implemented
as a tuple with unnamed fields.  Accessing its fields requires syntax like `self.0`
which limits readability. 

#### Summary of Changes
This PR converts Rocks from tuple to struct so that it has more human-readable
fields.
 2022-04-28 21:32:48 -07:00
behzad nouri 008860bb36
removes SHRED_PAYLOAD_SIZE from shred public interface (#24806) 2022-04-28 23:42:37 +00:00
Justin Starry 4e58b3870c
Update all BankForks methods to return owned values (#24801) 2022-04-28 18:51:00 +00:00
Tyera Eulberg b6a18e0e87
Fix get_first_available_block for genesis; also make get_blocks(_with_limit) consistent (#24760) 
* Handle genesis more appropriately in get_first_available_block

* Add unit test

* Make get_blocks starting-slots consistent with other methods
 2022-04-28 00:36:19 -06:00
behzad nouri 1e86d3ab59
moves shredder out of the shred module (#24755) 
Working towards embedding versioning into shreds binary; so Shredder can
no longer depend on internals of Shred.
 2022-04-27 22:00:04 +00:00
behzad nouri 0f60665100
replaces Shred::new_empty_coding with Shred::new_from_parity_shard (#24749) 
Removing implementation details of shreds and payload offsets from
shredder, so that shredder does not need to mutate payload:
https://github.com/solana-labs/solana/blob/71ad12128/ledger/src/shred.rs#L968-L977

Also, Shred::new_from_data can simply obtain a slice as opposed to
Option<&[u8]>:
https://github.com/solana-labs/solana/blob/71ad12128/ledger/src/shred.rs#L268-L278
 2022-04-27 18:04:10 +00:00
behzad nouri 081c844d6e
removes Shred::new_empty_data_shred (#24714) 
Shred::new_empty_data_shred returns an invalid shred (i.e.
shred.sanitize() returns error). The method is only used in tests and
can be easily replaced with Shred::new_from_data. To keep the shred api
surface small, this commit removes this method.
 2022-04-26 23:13:12 +00:00
behzad nouri 913ad79cd9
patches bug in data-shred index sanitize (#24707) 
https://github.com/solana-labs/solana/pull/24653
introduced an off-by-one error in data-shred index sanitize.
 2022-04-26 21:10:27 +00:00
Justin Starry b6bc5d2fa1 Aggregate executor update timing metrics 2022-04-26 23:15:22 +08:00
Brooks Prumo edf3f05ab5
Make test_next_leader_slot_next_epoch() aware of stake minimum delegation (#24660) 2022-04-26 08:31:06 -04:00
behzad nouri 12ae8d3be5
returns Error when Shred::sanitize fails (#24653) 
Including the error in the output allows to debug when Shred::sanitize
fails.
 2022-04-25 23:19:37 +00:00
dependabot[bot] 74b05eebd8
chore: bump num_enum from 0.5.4 to 0.5.7 (#24634) 
Bumps [num_enum](https://github.com/illicitonion/num_enum) from 0.5.4 to 0.5.7.
- [Release notes](https://github.com/illicitonion/num_enum/releases)
- [Commits](https://github.com/illicitonion/num_enum/commits)

---
updated-dependencies:
- dependency-name: num_enum
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-25 16:05:59 -06:00
behzad nouri 895f76a93c
hides implementation details of shred from its public interface (#24563) 
Working towards embedding versioning into shreds binary, so that a new
variant of shred struct can include merkle tree hashes of the erasure
set.
 2022-04-25 12:43:22 +00:00
behzad nouri 92ad767738
simplifies Serialize/Deserialize implementations for ShredType (#24601) 2022-04-23 13:33:59 +00:00
Michael Vines 83e041299a Run real snapshot packager while processing blockstore at validator startup 2022-04-22 21:17:49 -07:00
Justin Starry c544742091
Local cluster test cleanup and refactoring (#24559) 
* remove FixedSchedule.start_epoch

* use duration for timing

* Rename to partition bool to turbine_disabled

* simplify partition config
 2022-04-22 12:14:07 +08:00
Justin Starry d5127abf46
Only add hashes for completed blocks to recent blockhashes (#24389) 
* Only add hashes for completed blocks to recent blockhashes

* feedback
 2022-04-21 21:05:29 +08:00
Jeff Biseda 96d977fd05
test_shred_fec_set_index fix (#24540) 2022-04-20 16:26:44 -07:00
dependabot[bot] 40b48f807a
chore: bump libc from 0.2.120 to 0.2.124 (#24520) 
* chore: bump libc from 0.2.120 to 0.2.124

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.120 to 0.2.124.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.120...0.2.124)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-04-20 15:45:13 -06:00
dependabot[bot] dd15193c69
chore: bump rayon from 1.5.1 to 1.5.2 (#24470) 
* chore: bump rayon from 1.5.1 to 1.5.2

Bumps [rayon](https://github.com/rayon-rs/rayon) from 1.5.1 to 1.5.2.
- [Release notes](https://github.com/rayon-rs/rayon/releases)
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md)
- [Commits](https://github.com/rayon-rs/rayon/commits)

---
updated-dependencies:
- dependency-name: rayon
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-04-19 23:17:52 -06:00
Tyera Eulberg 77f6ec5625
Add program_id to TokenBalances structs (#24513) 
* Add program_id to TokenBalances structs

* Update docs
 2022-04-19 22:01:22 -06:00
Michael Vines 9e4999ef6a Remove halt_at_slot from RuntimeConfig, it's not a runtime concern 2022-04-19 19:23:58 -07:00
Michael Vines c6f3da4879 blockstore_processor now accepts an Arc<Rwlock<BankForks>> 2022-04-19 19:23:58 -07:00
behzad nouri 705ea53353
moves sign_shred and new_coding_shred_header out of Shredder (#24487) 2022-04-19 20:00:05 +00:00
behzad nouri 3bbfaae7b6
moves shred stats to a separate file (#24484) 2022-04-19 18:25:09 +00:00
HaoranYi c6a751d658
Detect and report bank drop signal queue full and disconnect events (#24112) 
* nonblocking send when when droping banks

* detect and report drop signal queue full/disconnect events

* comments

* use counter for reporting bank_drop_queue events

* reduce log

* use datapoint to report stats

* logging instead of reporting bank drop signal full

* fix a corner case for reporting

* fix build
 2022-04-18 11:19:11 -05:00
Justin Starry 4ed647d8ec
Test that tick slot hashes update the recent blockhash queue (#24242) 2022-04-16 00:30:20 +08:00
HaoranYi e3ef0741be
simplify bank drop calls (#24142) 
* simplify bank drop calls

* clippy: import

* Update ledger/src/blockstore_processor.rs

Co-authored-by: Brooks Prumo <brooks@prumo.org>

* Update runtime/src/accounts_background_service.rs

Co-authored-by: Brooks Prumo <brooks@prumo.org>

* Update runtime/src/bank.rs

Co-authored-by: Brooks Prumo <brooks@prumo.org>

* cleanup

* format

* use msb of bank_id to indicates that we are dropping

* clippy

* restore bank id

* clippy

* revert is-serialized_with_abs flag

* assert

* clippy

* whitespace

* fix bank drop callback check

* more fix

* remove msb dropping implementation

* fix

Co-authored-by: Brooks Prumo <brooks@prumo.org>
 2022-04-14 08:43:54 -05:00
Yueh-Hsuan Chiang 077bc4f407
(LedgerStore) Change the default RocksDB perf sample rate to 1 / 1000. (#24234) 2022-04-12 04:12:47 +00:00
Yueh-Hsuan Chiang 5a48ef72fd
(LedgerStore) Skip sampling check when ROCKSDB_PERF_CONTEXT_SAMPLES_IN_1K_DEFAULT = 0 (#24221) 
#### Problem
Currently, even if SOLANA_METRICS_ROCKSDB_PERF_SAMPLES_IN_1K == 0, we are still doing
the sampling check for every RocksDB read.

```
thread_rng().gen_range(0, METRIC_SAMPLES_1K) > *ROCKSDB_PERF_CONTEXT_SAMPLES_IN_1K
```

#### Summary of Changes
This PR skips the sampling check when SOLANA_METRICS_ROCKSDB_PERF_SAMPLES_IN_1K
is set to 0.
 2022-04-11 20:39:46 -07:00
Jon Cinque 9b8850f99e
test-validator: Add `--max-compute-units` flag (#24130) 
* test-validator: Add `--max-compute-units` flag

* Add `RuntimeConfig` for tweaking runtime behavior

* Actually add the file

* Move RuntimeConfig to runtime
 2022-04-12 02:28:10 +02:00