Commit Graph

855 Commits

Author SHA1 Message Date
HaoranYi 83f5f8bfc3
Move test_purge_huge test (#23587)
* ignore test_purge_huge tests it is expensive.

* move test_purge to integration tests
2022-03-10 15:31:43 -06:00
Brooks Prumo 3c6840050c
Ensure blocks do not exceed the max accounts data size during Replay Stage (#23422) 2022-03-10 10:24:31 -06:00
steviez 58c0db9704
Cleanup several blockstore functions (#23390)
* Rename excludes_from_compaction to should_exclude_from_compaction
* Make subfunction to create all cf descriptors
* Condense logic for when to disable compactions
2022-03-10 02:08:38 -06:00
dependabot[bot] 46ec5d563b
chore: bump libc from 0.2.117 to 0.2.119 (#23527)
* chore: bump libc from 0.2.117 to 0.2.119

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.117 to 0.2.119.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.117...0.2.119)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-03-08 14:25:38 -07:00
Michael Vines 5599bd9442 Remove BankFromArchiveTimings from ledger/ 2022-03-08 08:11:50 -08:00
Michael Vines 9cfa21f7d1 Inline load_from_genesis 2022-03-08 08:11:50 -08:00
Yueh-Hsuan Chiang b8b7163b66
(Ledger Store) Report RocksDB Column Family Metrics (#22503)
This PR enables blockstore to periodically report RocksDB column family properties.
The reported properties are under blockstore_rocksdb_cfs, and the properties also
support group by operation on cf_name.
2022-03-05 16:13:03 -08:00
Michael Vines 36ad59673c drop mut 2022-03-04 09:52:46 +01:00
Michael Vines 0d33b54d74 Rework do_process_blockstore_from_root to use BankForks 2022-03-04 09:52:46 +01:00
Michael Vines 93c8e04d51 Simplify do_process_blockstore_from_root slightly 2022-03-04 09:52:46 +01:00
dependabot[bot] 360f6466a3
chore: bump futures from 0.3.19 to 0.3.21 (#23476)
* chore: bump futures from 0.3.19 to 0.3.21

Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.19 to 0.3.21.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.19...0.3.21)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-03-03 16:44:43 -07:00
Yueh-Hsuan Chiang 62d2a4cd88
Make ShredStorageType::RocksLevel public (#23272)
#### Summary of Changes
This PR adds two hidden arguments to the validator that allow users to use RocksDB's FIFO compaction for storing shreds.

        --shred-storage <SHRED_STORAGE>
            EXPERIMENTAL: Controls how RocksDB compacts shreds.  *WARNING*: You will lose your ledger data
            when you switch between options. Possible values are: 'level': stores shreds using RocksDB's default (level)
            compaction. 'fifo': stores shreds under RocksDB's FIFO compaction. This option is more efficient on
            disk-write-bytes of the ledger store. [default: level]  [possible values: level, fifo]

        --shred-storage-size <SHRED_STORAGE_SIZE_BYTES>
            The shred storage size in bytes. The suggested value is 50% of your ledger storage size in bytes. [default:
            268435456000]
2022-03-03 12:43:58 -08:00
Yueh-Hsuan Chiang 634f4eb37d
(LedgerStore) Use different path for different blockstore storage type. (#23236)
#### Summary of Changes
To avoid mixing the use of different shred storage types, each shred storage type
will have its blockstore in a different directory.

This PR still keeps the RocksFifo setting hidden.  The default ShredStorageType and
blockstore directory are still RocksLevel and `rocksdb`.

Will follow-up with PRs on making FIFO option public in ledger-tool and validator.

#### Test Plan
* Added a new test to verify the existence of `rocksdb-fifo` directory when FIFO compaction is used.
* Updated existing test to verify the current setting still store ledger under `rocksdb` directory.
* Manually ran ledger_cleanup_test with both level and fifo compaction and verified the resulting ledger.
* Ran a validator with this PR.
2022-03-02 18:30:22 -08:00
Will Hickey 1a99251498
Bump version to 1.10.1 (#23453) 2022-03-02 13:47:01 -06:00
Jeff Washington (jwash) ef8b7d9c62
ledger tool halt at slot verify hash (#23424) 2022-03-02 11:11:18 -06:00
Tyera Eulberg 3b5b71ce44
Improve UX querying rpc for blocks at or before the snapshot from boot (#23403)
* Bump first-available block to first complete block

* Remove obsolete purges in tests (PrimaryIndex toggling no longer in use

* Check first-available block in Rpc check_slot_cleaned_up
2022-02-28 23:57:41 -07:00
dependabot[bot] 985af71241
chore: bump sha2 from 0.10.1 to 0.10.2 (#23343)
* chore: bump sha2 from 0.10.1 to 0.10.2

Bumps [sha2](https://github.com/RustCrypto/hashes) from 0.10.1 to 0.10.2.
- [Release notes](https://github.com/RustCrypto/hashes/releases)
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.10.1...sha2-v0.10.2)

---
updated-dependencies:
- dependency-name: sha2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-02-24 23:57:42 -07:00
Jeff Washington (jwash) 39c86c6c44
fmt (#23329) 2022-02-24 12:12:29 -06:00
steviez 12bddbc4ec
Use rocksdb property constant instead of local constant (#23147) 2022-02-23 23:02:11 -06:00
Justin Starry d0e85c293f
Fix rustfmt check (#23296) 2022-02-23 16:38:53 +08:00
Gavin Chan 20d031e2b8
Refactor ExecuteTimings w/ enum-indexed array (#23085) 2022-02-22 14:46:56 -08:00
Yueh-Hsuan Chiang 8c872e9ce0
(LedgerStore/FIFO) Refactor FIFO options and sanity check. (#23131) 2022-02-19 00:58:38 -08:00
Justin Starry 1719d2349f
Skip adding builtins if they will be removed (#23233)
* Add failing test for precompile transition

* Skip adding builtins if they will be removed

* cargo clean

* nits

* fix abi check

* remove workaround

Co-authored-by: Jack May <jack@solana.com>
2022-02-18 18:36:59 -08:00
Yueh-Hsuan Chiang 7939fdc3e5
Minor refactor on Shreds column family descriptor construction. (#23103) 2022-02-16 00:29:11 -08:00
Lijun Wang c04438be4b
Retaining transaction logs when transaction plugin is loaded. (#22874)
Transaction logs are not being saved to the database through the plugin interface.

Summary of Changes

Retain the transaction logs when transaction notification plugin is loaded.

Fixes #
lijunwangs/solana-accountsdb-plugin-postgres#6
2022-02-11 20:29:07 -08:00
Yueh-Hsuan Chiang 6399647e64
(Ledger Store) Add code comments for SlotMetaWorkingSetEntry. (#22662) 2022-02-11 10:04:26 -08:00
HaoranYi e4a1799334
fix typos in ledge code comments (#23068) 2022-02-10 20:43:44 -07:00
Yueh-Hsuan Chiang 9213fcb11b
Introduce experimental FIFO-compaction option for shreds in blockstore (#22140) 2022-02-10 11:34:03 -08:00
Justin Starry d5dec989b9
Enforce tx metadata upload with static types (#23028) 2022-02-10 13:28:18 +08:00
steviez 2ca7ae8e30
Update rocksdb to 0.18.0 release (#23031) 2022-02-09 21:51:56 -06:00
dependabot[bot] 9d036be61b
chore: bump libc from 0.2.115 to 0.2.117 (#23034)
* chore: bump libc from 0.2.115 to 0.2.117

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.115 to 0.2.117.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.115...0.2.117)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-02-09 16:49:44 -07:00
Pankaj Garg a146f2d853 suppress clippy 2022-02-07 08:02:58 -08:00
Pankaj Garg c5d8560cdb updates to address review feedback 2022-02-07 08:02:58 -08:00
Pankaj Garg dfef68f985 fix build 2022-02-07 08:02:58 -08:00
Pankaj Garg 4de14e530b Optimize batching of transactions during replay for parallel processing 2022-02-07 08:02:58 -08:00
Yueh-Hsuan Chiang 37afdd1a65
(Ledger Store) Improve comments for blockstore_purge (#22808) 2022-02-06 21:56:23 -08:00
steviez c7ca2f41f5
Add some comments for RocksDB option configuration (#22655) 2022-02-06 23:28:24 -06:00
Justin Starry ba215e94f6
Refactor: Add AccountKeys struct for static and dynamic message keys (#22960) 2022-02-05 20:00:31 +08:00
Jeff Washington (jwash) fb95fa68a2
cleanup api (#22845) 2022-01-31 11:39:23 -06:00
Yueh-Hsuan Chiang aaae5b3ba6
(Ledger Store) Add comment blocks for six pub functions in blockstore.rs (#22476) 2022-01-28 22:25:07 -08:00
Yueh-Hsuan Chiang 40a49081b9
(Ledger Store) Add comments for find slot functions. (#22666) 2022-01-28 12:07:09 -08:00
dependabot[bot] cc74693176
chore: bump serde from 1.0.134 to 1.0.136 (#22817)
* chore: bump serde from 1.0.134 to 1.0.136

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.134 to 1.0.136.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.134...v1.0.136)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-28 12:20:37 -07:00
dependabot[bot] 7376efe8ea
chore: bump libc from 0.2.112 to 0.2.115 (#22796)
* chore: bump libc from 0.2.112 to 0.2.115

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.112 to 0.2.115.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.112...0.2.115)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-27 21:54:13 +00:00
Michael Vines 6d5bbca630 Pacify clippy 2022-01-21 19:12:57 -08:00
dependabot[bot] 0cf886302d
chore: bump serde from 1.0.133 to 1.0.134 (#22650)
* chore: bump serde from 1.0.133 to 1.0.134

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.133 to 1.0.134.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.133...v1.0.134)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-21 17:14:02 -07:00
Yueh-Hsuan Chiang fe7543c31a
(Ledger Store) APIs for obtaining physical size of all data and coding shreds (#22443) 2022-01-19 19:31:19 -08:00
Tao Zhu a724fa2347
Add hidden cli option to allow validator reports replayed transaction cost metrics (#22369)
* add hidden cli option to allow validator reports replayed transaction cost detail metrics

* Update validator/src/main.rs

Co-authored-by: Michael Vines <mvines@gmail.com>

* - rebase master, using unbounded instead of channel; dowgrade to datapoint_trace

* removed cli arg, prefer log at trace

Co-authored-by: Michael Vines <mvines@gmail.com>
2022-01-15 00:31:21 +00:00
Justin Starry f804ccdece
Store address table lookups in blockstore and bigtable (#22402) 2022-01-14 15:24:41 +08:00
carllin eca8d21249
log internals (#22493) 2022-01-13 19:50:46 -05:00
Jeff Biseda 8b66625c95
convert std::sync::mpsc to crossbeam_channel (#22264) 2022-01-11 02:44:46 -08:00
dependabot[bot] 28275a33d6
Bump tempfile from 3.2.0 to 3.3.0 (#22401)
* Bump tempfile from 3.2.0 to 3.3.0

Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.2.0 to 3.3.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases)
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.2.0...v3.3.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-09 12:26:08 -07:00
dependabot[bot] 2759cb860b
Bump sha2 from 0.10.0 to 0.10.1 (#22359)
* Bump sha2 from 0.10.0 to 0.10.1

Bumps [sha2](https://github.com/RustCrypto/hashes) from 0.10.0 to 0.10.1.
- [Release notes](https://github.com/RustCrypto/hashes/releases)
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.10.0...sha2-v0.10.1)

---
updated-dependencies:
- dependency-name: sha2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-08 00:17:32 -07:00
carllin 813006b33b
remove per program timings from blockstore processor ledger replay (#22370) 2022-01-08 01:52:08 -05:00
Yueh-Hsuan Chiang 5771c36d3f
Rename open_with_access_type() to open_with_options() (#22123) 2022-01-07 12:11:43 -08:00
steviez 207825d30b
Minimize boilerplate code around Rocks column families (#22345) 2022-01-06 23:39:09 -06:00
Justin Starry 52d12cc802
Add runtime support for address table lookups (#22223)
* Add support for address table lookups in runtime

* feedback

* feedback
2022-01-07 11:59:09 +08:00
Trent Nelson 7d32909e17 move `ExecuteTimings` from `runtime::bank` to `program_runtime::timings` 2022-01-06 03:56:46 -07:00
Justin Starry 45458e7139
Refactor: Improve type safety and readability of transaction execution (#22215)
* Refactor Bank::load_and_execute_transactions

* Refactor: improve type safety of TransactionExecutionResult

* Add enum for extra type safety in execution results

* feedback
2022-01-05 10:15:15 +08:00
behzad nouri 01a096adc8 adds bitflags to Packet.Meta
Instead of a separate bool type for each flag, all the flags can be
encoded in a type-safe bitflags encoded in a single u8:
https://github.com/solana-labs/solana/blob/d6ec103be/sdk/src/packet.rs#L19-L31
2022-01-04 13:53:40 +00:00
Jeff Biseda ca8fef5855
retransmit consecutive leader blocks (#22157) 2022-01-04 00:24:16 -08:00
Yueh-Hsuan Chiang e8b7f96a89
Add struct BlockstoreOptions (#22121) 2022-01-03 18:30:45 -10:00
dependabot[bot] 672fed04cb
Bump serde from 1.0.132 to 1.0.133 (#22233)
* Bump serde from 1.0.132 to 1.0.133

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.132 to 1.0.133.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.132...v1.0.133)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-03 22:19:39 +00:00
Justin Starry 2a00382d71
Refactor: cleanup solana_transaction_status crate (#22230) 2022-01-03 15:45:18 +00:00
carllin 33d0b5e011
Revert "Count compute units even when transaction errors (#22059)" (#22174)
This reverts commit eaa8c67bde.
2021-12-30 02:42:32 -05:00
Omar Kilani bac6821e19
get_signatures_for_address does not correctly account for result sets that span local and Bigtable sources (#22115)
* get_signatures_for_address does not correctly account for result sets that span Blockstore and Bigtable.

This causes Bigtable to return `RowNotFound` until the new tx is uploaded.

Check that `before` exists in Bigtable, and if not, set it to `None` to return the full data set.

References #21442
Closes #22110

* Differentiate between before sig not found and no newer signatures

* Dedupe bigtable results to account for potential upload race

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-12-29 10:25:10 -07:00
carllin eaa8c67bde
Count compute units even when transaction errors (#22059) 2021-12-28 17:05:11 -05:00
Yueh-Hsuan Chiang b89cd8cd1a
Avoid cloning Vec<Entry> when calling entries_to_test_shreds() (#22093) 2021-12-24 12:32:43 -08:00
behzad nouri f67ecd5c18
removes unused Cargo dependencies (#22022)
Also moving some to [dev-dependencies] so that they are not propagated
to other packages which depend on the package.
2021-12-22 17:01:33 +00:00
Yueh-Hsuan Chiang bf8fbf8383
Add code comment for handle_chaining in blockstore.rs (#21876) 2021-12-21 22:36:24 -08:00
Yueh-Hsuan Chiang 3155a04189
Add comment block for insert_shreds_handle_duplicate in blockstore.rs (#21877) 2021-12-21 22:36:13 -08:00
dependabot[bot] 41ec7c8be9
chore: bump num_cpus from 1.13.0 to 1.13.1 (#22044)
* chore: bump num_cpus from 1.13.0 to 1.13.1

Bumps [num_cpus](https://github.com/seanmonstar/num_cpus) from 1.13.0 to 1.13.1.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases)
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.13.0...v1.13.1)

---
updated-dependencies:
- dependency-name: num_cpus
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-21 00:06:14 -07:00
dependabot[bot] e810400716
chore: bump futures from 0.3.18 to 0.3.19 (#22020)
* chore: bump futures from 0.3.18 to 0.3.19

Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.18 to 0.3.19.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.18...0.3.19)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-20 20:45:19 +00:00
behzad nouri 65d59f4ef0
tracks erasure coding shreds' indices explicitly (#21822)
The indices for erasure coding shreds are tied to data shreds:
https://github.com/solana-labs/solana/blob/90f41fd9b/ledger/src/shred.rs#L921

However with the upcoming changes to erasure schema, there will be more
erasure coding shreds than data shreds and we can no longer infer coding
shreds indices from data shreds.

The commit adds constructs to track coding shreds indices explicitly.
2021-12-19 22:37:55 +00:00
dependabot[bot] 8f547a6c98
chore: bump serde from 1.0.131 to 1.0.132 (#21989)
* chore: bump serde from 1.0.131 to 1.0.132

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.131 to 1.0.132.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.131...v1.0.132)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-18 07:45:20 +00:00
Michael Vines 5f054cd51b Update to reed-solomon-erasure 5.0.1, to get simd-accel on M1 macs 2021-12-17 14:19:39 -08:00
behzad nouri 89d66c3210
removes next_shred_index from return value of entries to shreds api (#21961)
next-shred-index is already readily available from returned data shreds.
The commit simplifies the api for upcoming changes to erasure coding
schema which will require explicit tracking of indices for coding shreds
as well as data shreds.
2021-12-17 15:01:55 +00:00
behzad nouri efd64a3862
simplifies ShredIndex api (#21932) 2021-12-16 19:17:32 +00:00
behzad nouri 8183f28636
adds ErasureSetId identifying erasure coding sets of shreds (#21928) 2021-12-16 14:18:55 +00:00
Yueh-Hsuan Chiang 882f886450
Add comment block for commit_slot_meta_working_set in blockstore.rs (#21852) 2021-12-15 13:12:50 -08:00
Yueh-Hsuan Chiang e374fb1d60
Add code comment for get_slot_meta_entry in blockstore.rs (#21844) 2021-12-15 13:12:38 -08:00
Yueh-Hsuan Chiang 71b12b1f56
Add comment for clear_unconfirmed_slot() in blockstore.rs (#21837) 2021-12-15 00:37:09 -08:00
Yueh-Hsuan Chiang e476e17abf
Add code comment for check_insert_data_shred in blockstore.rs (#21845) 2021-12-15 00:36:11 -08:00
Michael Vines 2a6dcb2ffd Futures 0.3.18 has been yanked, back off to .17 2021-12-14 14:14:43 -08:00
behzad nouri 8d980f07ba
uses Option<Slot> for SlotMeta.parent_slot (#21808)
SlotMeta.parent_slot for the head of a detached chain of slots is
unknown and that is indicated by u64::MAX which lacks type-safety:
https://github.com/solana-labs/solana/blob/6c108c8fc/ledger/src/blockstore_meta.rs#L203-L205

The commit changes the type to Option<Slot>. Backward compatibility is
maintained by customizing serde serialize/deserialize implementations.
2021-12-14 18:57:11 +00:00
behzad nouri 4ceb2689f5
adds ShredId uniquely identifying each shred (#21820) 2021-12-14 17:34:02 +00:00
Ikko Ashimine e19c7923c3
chore: fix typo in shred.rs (#21890)
begining -> beginning
2021-12-14 11:19:27 -05:00
dependabot[bot] 8a63812c4e
chore: bump libc from 0.2.109 to 0.2.112 (#21870)
* chore: bump libc from 0.2.109 to 0.2.112

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.109 to 0.2.112.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.109...0.2.112)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-14 00:27:57 -07:00
steviez 4a9d7318d1
Rework test parameters to be shreds instead of entries (#21780)
The number of shreds that result from a given number of entries is
variable and in our test case, somewhat unintuitive to think about when
trying to determine how much data we're pushing into the blockstore. So,
this change converts the unit of test parameters from entries to shreds.

This change also cleans up some variable naming for clarity and prints.
2021-12-13 23:34:43 -06:00
behzad nouri e08139f949
uses Option<u64> for SlotMeta.last_index (#21775)
SlotMeta.last_index may be unknown and current code is using u64::MAX to
indicate that:
https://github.com/solana-labs/solana/blob/6c108c8fc/ledger/src/blockstore_meta.rs#L169-L174

This lacks type-safety and can introduce bugs if not always checked for
Several instances of slot_meta.last_index + 1 are also subject to
overflow.

This commit updates the type to Option<u64>. Backward compatibility is
maintained by customizing serde serialize/deserialize implementations.
2021-12-11 14:47:20 +00:00
Justin Starry 254ef3e7b6
Rename Packets to PacketBatch (#21794) 2021-12-11 09:44:15 -05:00
behzad nouri 49ba09b333
adds back ErasureMeta::first_coding_index field (#21623)
https://github.com/solana-labs/solana/pull/16646
removed first_coding_index since the field is currently redundant and
always equal to fec_set_index.
However, with upcoming changes to erasure coding schema, this will no
longer be the same as fec_set_index and so requires a separate field to
represent.
2021-12-10 20:08:04 +00:00
dependabot[bot] d870f566ef
chore: bump serde from 1.0.130 to 1.0.131 (#21758)
* chore: bump serde from 1.0.130 to 1.0.131

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.130 to 1.0.131.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.130...v1.0.131)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-10 08:47:46 +00:00
behzad nouri 8063273d09
adds more sanity checks to shreds (#21675) 2021-12-09 16:43:57 +00:00
dependabot[bot] b61b7189a5
chore: bump hmac from 0.11.0 to 0.12.0 (#21681)
* chore: bump hmac from 0.11.0 to 0.12.0

Bumps [hmac](https://github.com/RustCrypto/MACs) from 0.11.0 to 0.12.0.
- [Release notes](https://github.com/RustCrypto/MACs/releases)
- [Commits](https://github.com/RustCrypto/MACs/compare/hmac-v0.11.0...hmac-v0.12.0)

---
updated-dependencies:
- dependency-name: hmac
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

* Update dependabot-pr.sh

* Bump sha2 and pbkdf2 as well

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: axleiro <83293196+axleiro@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-12-08 18:12:35 -07:00
dependabot[bot] f6801a4af4
chore: bump itertools from 0.10.1 to 0.10.3 (#21643)
* chore: bump itertools from 0.10.1 to 0.10.3

Bumps [itertools](https://github.com/rust-itertools/itertools) from 0.10.1 to 0.10.3.
- [Release notes](https://github.com/rust-itertools/itertools/releases)
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.10.1...v0.10.3)

---
updated-dependencies:
- dependency-name: itertools
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-06 23:23:26 -07:00
dependabot[bot] b5353e2130
chore: bump libc from 0.2.108 to 0.2.109 (#21627)
* chore: bump libc from 0.2.108 to 0.2.109

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.108 to 0.2.109.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.108...0.2.109)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-12-06 09:37:26 -07:00
behzad nouri cd17f63d81
adds back position field to coding-shred-header (#21600)
https://github.com/solana-labs/solana/pull/17004
removed position field from coding-shred-header because as it stands the
field is redundant and unused.
However, with the upcoming changes to erasure coding schema this field
will no longer be redundant and needs to be populated.
2021-12-05 14:42:09 +00:00
Trent Nelson d6f22433d0 Bump version to v1.10.0 2021-12-04 20:17:54 +00:00
Michael Vines b8837c04ec Reformat imports to a consistent style for imports
rustfmt.toml configuration:
  imports_granularity = "One"
  group_imports = "One"
2021-12-03 09:19:13 -08:00
Michael Vines dd12d90eac Upgrade to Rust 2021 2021-11-30 20:43:46 -08:00