behzad nouri
705ea53353
moves sign_shred and new_coding_shred_header out of Shredder ( #24487 )
2022-04-19 20:00:05 +00:00
behzad nouri
3bbfaae7b6
moves shred stats to a separate file ( #24484 )
2022-04-19 18:25:09 +00:00
HaoranYi
c6a751d658
Detect and report bank drop signal queue full and disconnect events ( #24112 )
...
* nonblocking send when when droping banks
* detect and report drop signal queue full/disconnect events
* comments
* use counter for reporting bank_drop_queue events
* reduce log
* use datapoint to report stats
* logging instead of reporting bank drop signal full
* fix a corner case for reporting
* fix build
2022-04-18 11:19:11 -05:00
Justin Starry
4ed647d8ec
Test that tick slot hashes update the recent blockhash queue ( #24242 )
2022-04-16 00:30:20 +08:00
HaoranYi
e3ef0741be
simplify bank drop calls ( #24142 )
...
* simplify bank drop calls
* clippy: import
* Update ledger/src/blockstore_processor.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/accounts_background_service.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/bank.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* cleanup
* format
* use msb of bank_id to indicates that we are dropping
* clippy
* restore bank id
* clippy
* revert is-serialized_with_abs flag
* assert
* clippy
* whitespace
* fix bank drop callback check
* more fix
* remove msb dropping implementation
* fix
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-14 08:43:54 -05:00
Yueh-Hsuan Chiang
077bc4f407
(LedgerStore) Change the default RocksDB perf sample rate to 1 / 1000. ( #24234 )
2022-04-12 04:12:47 +00:00
Yueh-Hsuan Chiang
5a48ef72fd
(LedgerStore) Skip sampling check when ROCKSDB_PERF_CONTEXT_SAMPLES_IN_1K_DEFAULT = 0 ( #24221 )
...
#### Problem
Currently, even if SOLANA_METRICS_ROCKSDB_PERF_SAMPLES_IN_1K == 0, we are still doing
the sampling check for every RocksDB read.
```
thread_rng().gen_range(0, METRIC_SAMPLES_1K) > *ROCKSDB_PERF_CONTEXT_SAMPLES_IN_1K
```
#### Summary of Changes
This PR skips the sampling check when SOLANA_METRICS_ROCKSDB_PERF_SAMPLES_IN_1K
is set to 0.
2022-04-11 20:39:46 -07:00
Jon Cinque
9b8850f99e
test-validator: Add `--max-compute-units` flag ( #24130 )
...
* test-validator: Add `--max-compute-units` flag
* Add `RuntimeConfig` for tweaking runtime behavior
* Actually add the file
* Move RuntimeConfig to runtime
2022-04-12 02:28:10 +02:00
Giorgio Gambino
60b2155bd3
Add accounts-filler-size command line option ( #23896 )
2022-04-11 13:10:09 -05:00
steviez
6ca84f8a40
Move PurgeType enum to blockstore_purge.rs ( #24185 )
2022-04-08 11:46:12 -05:00
Tyera Eulberg
d2702201ca
Bump tonic, tonic-build, prost, and etcd-client ( #24147 )
...
* Bump tonic, prost, and etcd-client
* Restore doc ignores
2022-04-08 10:21:45 -06:00
Yueh-Hsuan Chiang
1f136de294
(LedgerStore) Report perf metrics for RocksDB deletes ( #24138 )
...
#### Summary of Changes
This PR enables perf metrics reporting for RocksDB deletes.
Samples are reported under "blockstore_rocksdb_write_perf" with op=delete
The sampling rate is still controlled by env arg SOLANA_METRICS_ROCKSDB_PERF_SAMPLES_IN_1K
and its default to 10 (meaning we report 10 in 1000 perf samples).
2022-04-08 00:18:05 -07:00
Yueh-Hsuan Chiang
b84521d47d
(LedgerStore) Report perf metrics for RocksDB write batch ( #24061 )
...
#### Summary of Changes
This PR enables perf metrics reporting for RocksDB write-batches.
Samples are reported under "blockstore_rocksdb_write_perf" with op=write_batch
Its cf_name tag is set to "write_batch" as well as each write-batch could include multiple column families.
The sampling rate is still controlled by env arg SOLANA_METRICS_ROCKSDB_PERF_SAMPLES_IN_1K
and its default to 10 (meaning we report 10 in 1000 perf samples).
2022-04-08 00:17:51 -07:00
Yueh-Hsuan Chiang
206c3dd402
(LedgerStore) Enable RocksDB Perf metrics reporting for get_bytes and put_bytes ( #24066 )
...
#### Summary of Changes
Enable RocksDB Perf metrics reporting for get_bytes and put_bytes.
2022-04-07 00:24:10 -07:00
Yueh-Hsuan Chiang
4f0e887702
(LedgerStore) Report RocksDB perf metrics for Protobuf Columns ( #24065 )
...
This PR enables the reporting of both RocksDB read and write perf metrics for ProtobufColumns,
including TransactionStatus and Rewards.
2022-04-07 00:15:00 -07:00
Tyera Eulberg
afeb1d3cca
Bump lru crate ( #24150 )
2022-04-06 16:18:42 -06:00
Yueh-Hsuan Chiang
2d1f27ed8e
(LedgerStore) Perf Metric for RocksDB Writes ( #23951 )
...
#### Summary of Changes
This PR implements the reporting of RocksDB write perf metrics to blockstore_rocksdb_write_perf
based on RocksDB's PerfContext. The default sample rate is 10 in 1000, and the env arg SOLANA_METRICS_ROCKSDB_PERF_SAMPLES_IN_1K can control the sample rate.
2022-04-06 12:12:38 -07:00
Brooks Prumo
c322842257
Replace channel with Mutex<Option> for AccountsPackage ( #24013 )
2022-04-06 05:47:19 -05:00
Yueh-Hsuan Chiang
24cc6c33de
(LedgerStore)(Refactor) Move metric reporting functions to a dedicate mod ( #24060 )
...
Previously, the metric reporting functions are implemented under LedgerColumnMetric.
However, there're operations like write batch which is issued by the function inside Rocks.
This PR moves reporting functions to its own dedicate mod so that both LedgerColumn and
Rocks can report column perf metrics.
2022-04-05 15:06:17 -07:00
carllin
4ea59d8cb4
Set drop callback on first root bank ( #23999 )
2022-04-05 13:02:33 -05:00
HaoranYi
eb65ffb779
Optimize replay waking up ( #24051 )
...
* timeout for validator exits
* clippy
* print backtrace when panic
* add backtrace package
* increase time out to 30s
* debug logging
* make rpc complete service non blocking
* reduce log level
* remove logging
* recv_timeout
* remove backtrace
* remove sleep
* wip
* remove unused variable
* add comments
* Update core/src/validator.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* Update core/src/validator.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* whitespace
* more whitespace
* fix build
* clean up import
* add mutex for signal senders in blockstore
* remove mut
* refactor: extract add signal functions
* make blockstore signal private
* increase replay wake up channel bounds
* reduce replay wakeup signal bound to 1
* reduce log level
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
2022-04-05 08:57:12 -05:00
behzad nouri
855801cc95
removes deterministic-shred-seed feature
2022-04-05 12:04:12 +00:00
Jeff Biseda
ee6bb0d5d3
track fec set turbine stats ( #23989 )
2022-04-04 14:44:21 -07:00
HaoranYi
6ba4e870c4
Blockstore should drop signals before validator exit ( #24025 )
...
* timeout for validator exits
* clippy
* print backtrace when panic
* add backtrace package
* increase time out to 30s
* debug logging
* make rpc complete service non blocking
* reduce log level
* remove logging
* recv_timeout
* remove backtrace
* remove sleep
* wip
* remove unused variable
* add comments
* Update core/src/validator.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* Update core/src/validator.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* whitespace
* more whitespace
* fix build
* clean up import
* add mutex for signal senders in blockstore
* remove mut
* refactor: extract add signal functions
* make blockstore signal private
* let compiler infer mutex type
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
2022-04-04 11:38:05 -05:00
Yueh-Hsuan Chiang
0b5ed87220
(LedgerStore) Enable performance sampling in column family get() ( #23834 )
...
#### Summary of Changes
This PR enables RocksDB read side performance metrics to report to blockstore_rocksdb_read_perf.
The sampling rate is controlled by an env arg `SOLANA_METRICS_ROCKSDB_PERF_SAMPLES_IN_1K`,
specifies the number of perf samples for every 1000 operations. The default value is set to 10, meaning
we will report 10 out of 1000 (or 1/100) reads.
The metrics are based on the RocksDB [PerfContext](https://github.com/facebook/rocksdb/blob/main/include/rocksdb/perf_context.h ).
It includes many useful metrics including block read time, cache hit rate, and time spent on decompressing the block.
2022-04-01 13:13:32 -07:00
HaoranYi
ba770832d0
Poh timing service ( #23736 )
...
* initial work for poh timing report service
* add poh_timing_report_service to validator
* fix comments
* clippy
* imrove test coverage
* delete record when complete
* rename shred full to slot full.
* debug logging
* fix slot full
* remove debug comments
* adding fmt trait
* derive default
* default for poh timing reporter
* better comments
* remove commented code
* fix test
* more test fixes
* delete timestamps for slot that are older than root_slot
* debug log
* record poh start end in bank reset
* report full to start time instead
* fix poh slot offset
* report poh start for normal ticks
* fix typo
* refactor out poh point report fn
* rename
* optimize delete - delete only when last_root changed
* change log level to trace
* convert if to match
* remove redudant check
* fix SlotPohTiming comments
* review feedback on poh timing reporter
* review feedback on poh_recorder
* add test case for out-of-order arrival of timing points and incomplete timing points
* refactor poh_timing_points into its own mod
* remove option for poh_timing_report service
* move poh_timing_point_sender to constructor
* clippy
* better comments
* more clippy
* more clippy
* add slot poh timing point macro
* clippy
* assert in test
* comments and display fmt
* fix check
* assert format
* revise comments
* refactor
* extrac send fn
* revert reporting_poh_timing_point
* align loggin
* small refactor
* move type declaration to the top of the module
* replace macro with constructor
* clippy: remove redundant closure
* review comments
* simplify poh timing point creation
Co-authored-by: Haoran Yi <hyi@Haorans-MacBook-Air.local>
2022-03-30 09:04:49 -05:00
behzad nouri
cda3d66b21
uses first_coding_index for erasure meta obtained from coding shreds ( #23974 )
...
Now that nodes correctly populate position field in coding shreds, and
first_coding_index in erasure meta, the old code to maintain backward
compatibility can be removed.
The commit is working towards changing erasure coding schema to 32:64.
2022-03-30 13:55:11 +00:00
Jeff Washington (jwash)
c24de17278
remove index hash calculation as an option ( #23928 )
2022-03-25 15:32:53 -05:00
behzad nouri
1f9c89c1e8
expands lifetime of SlotStats ( #23872 )
...
Current slot stats are removed when the slot is full or every 30 seconds
if the slot is before root:
https://github.com/solana-labs/solana/blob/493a8e234/ledger/src/blockstore.rs#L2017-L2027
In order to track if the slot is ultimately marked as dead or rooted and
emit more metrics, this commit expands lifetime of SlotStats while
bounding total size of cache using an LRU eviction policy.
2022-03-25 19:32:22 +00:00
steviez
c31db81ac4
Use VoteAccountsHashMap type alias in all applicable spots ( #23904 )
2022-03-24 12:09:48 -05:00
Yueh-Hsuan Chiang
c83c95b56b
(LedgerStore) Create ColumnMetrics trait for CF metric reporting ( #23763 )
...
This PR does a refactoring on column family-related metrics reporting.
As the metric reporting is per column family basis, the PR creates
ColumnMetrics trait and move the metric reporting logic into it.
This refactoring will make future column metric reporting (such as
read PerfContext) much cleaner.
2022-03-23 20:51:49 -07:00
Jon Cinque
7af48465fa
transaction-status: Add return data to meta ( #23688 )
...
* transaction-status: Add return data to meta
* Add return data to simulation results
* Use pretty-hex for printing return data
* Update arg name, make TransactionRecord struct
* Rename TransactionRecord -> ExecutionRecord
2022-03-22 23:17:05 +01:00
Yueh-Hsuan Chiang
ae75b1a25f
(LedgerStore) Add compression type ( #23578 )
...
This PR adds `--rocksdb-ledger-compression` as a hidden argument to the validator
for specifying the compression algorithm for TransactionStatus. Available compression
algorithms include `lz4`, `snappy`, `zlib`. The default value is `none`.
Experimental results show that with lz4 compression, we can achieve ~37% size-reduction
on the TransactionStatus column family, or ~8% size-reduction of the ledger store size.
2022-03-22 02:27:09 -07:00
Will Hickey
c4ecfa5716
Bump version to v1.11 ( #23807 )
...
* Revert crossbeam_epoch to stable. 0.9.8 only works with nightly
* Remove unneeded unit expression
2022-03-21 17:40:50 -05:00
Yueh-Hsuan Chiang
f999eef452
(LedgerStore) Rename BlockstoreAdvancedOptions to LedgerColumnOptions ( #23764 )
...
This PR renames BlockstoreAdvancedOptions to LedgerColumnOptions, as we will
pass-down this struct to LedgerColumn to allow it to perform metric reporting.
2022-03-18 11:13:35 -07:00
behzad nouri
6b0d34d70d
removes redundant Arcs from Blockstore ( #23735 )
2022-03-17 19:43:57 +00:00
Will Hickey
2f58c9e501
Bump version to 1.10.4 ( #23743 )
2022-03-17 14:02:13 -05:00
Yueh-Hsuan Chiang
86c695268e
(LedgerStore) Improve the function API of new_cf_descriptor ( #23696 )
...
As we start adding more options into BlockstoreOptions, it's better to allow
new_cf_descriptor to take the reference to BlockstoreOptions so that
we can avoid future function API changes on new_cf_descriptor.
2022-03-16 11:47:49 -07:00
dependabot[bot]
44ab660172
chore: bump libc from 0.2.119 to 0.2.120 ( #23694 )
...
* chore: bump libc from 0.2.119 to 0.2.120
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.119 to 0.2.120.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.119...0.2.120 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-03-16 00:34:40 -06:00
Michael Vines
390c5667f7
Setup bank hard_forks in load_bank_forks()
2022-03-15 23:08:07 -07:00
steviez
e7cf84a593
Remove AccessType arg from create_new_ledger ( #23591 )
...
Creating a new ledger implicitly means that no other process could have
previously held access to it. Additionally, creating a new ledger
implicitly requires writing, so it follows that Primary access is
required and we can drop access type as an argument.
2022-03-15 00:07:16 -05:00
Michael Vines
390dc24608
Create leader schedule before processing blockstore
2022-03-14 15:29:58 -07:00
Michael Vines
115f376465
Factor out bank_forks_utils::load_bank_forks()
2022-03-14 15:29:58 -07:00
Michael Vines
63324be5b3
Remove last_full_snapshot_slot return value when it can be derived by the caller
2022-03-14 15:29:58 -07:00
Michael Vines
cf5048faaa
Eliminate to_loadresult()
2022-03-14 15:29:58 -07:00
Michael Vines
3d4bf1d00a
Refactor bank_forks_utils::load() to invoke a common process_blockstore_from_root()
2022-03-14 15:29:58 -07:00
Michael Vines
c2ce152be8
Inline do_process_blockstore_from_root
2022-03-14 15:29:58 -07:00
Michael Vines
07d5ee062d
Push recyclers down the stack
2022-03-14 15:29:58 -07:00
HaoranYi
fc2d1a61f3
signing_coding_time is not used ( #23655 )
2022-03-14 17:16:30 -05:00
Will Hickey
63bf0f66af
Bump version to 1.10.3 ( #23648 )
2022-03-14 11:18:45 -05:00
Yueh-Hsuan Chiang
1e20bd8f9a
(LedgerStore) Include storage type as a tag in RocksDB metric reporting ( #23523 )
...
#### Summary of Changes
This PR further enables group by operation on storage type in blockstore_rocksdb_cfs metrics.
Such group-by allows us to further compare the performance metrics between rocks-level and
rocks-fifo.
To make things extensible, this PR introduces BlockstoreAdvancedOptions and move shred_storage_type.
All fields in BlockstoreAdvancedOptions will support group-by operation in blockstore_rocksdb_cfs.
Dependency: #23580
2022-03-11 15:17:34 -08:00
Brooks Prumo
d20dd21600
Sort tables in Cargo.toml files ( #23602 )
2022-03-11 11:05:57 -06:00
Yueh-Hsuan Chiang
99f1a22b9d
(LedgerStore) Generalize RocksDB metric reporting macro ( #23580 )
...
#### Summary of Changes
This PR generalizes RocksDB metric reporting macro so that future RocksDB metrics can
reuse the same macro.
2022-03-10 23:13:59 -08:00
Will Hickey
b444836a97
Bump version to 1.10.2 ( #23597 )
2022-03-10 16:41:06 -06:00
HaoranYi
83f5f8bfc3
Move test_purge_huge test ( #23587 )
...
* ignore test_purge_huge tests it is expensive.
* move test_purge to integration tests
2022-03-10 15:31:43 -06:00
Brooks Prumo
3c6840050c
Ensure blocks do not exceed the max accounts data size during Replay Stage ( #23422 )
2022-03-10 10:24:31 -06:00
steviez
58c0db9704
Cleanup several blockstore functions ( #23390 )
...
* Rename excludes_from_compaction to should_exclude_from_compaction
* Make subfunction to create all cf descriptors
* Condense logic for when to disable compactions
2022-03-10 02:08:38 -06:00
dependabot[bot]
46ec5d563b
chore: bump libc from 0.2.117 to 0.2.119 ( #23527 )
...
* chore: bump libc from 0.2.117 to 0.2.119
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.117 to 0.2.119.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.117...0.2.119 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-03-08 14:25:38 -07:00
Michael Vines
5599bd9442
Remove BankFromArchiveTimings from ledger/
2022-03-08 08:11:50 -08:00
Michael Vines
9cfa21f7d1
Inline load_from_genesis
2022-03-08 08:11:50 -08:00
Yueh-Hsuan Chiang
b8b7163b66
(Ledger Store) Report RocksDB Column Family Metrics ( #22503 )
...
This PR enables blockstore to periodically report RocksDB column family properties.
The reported properties are under blockstore_rocksdb_cfs, and the properties also
support group by operation on cf_name.
2022-03-05 16:13:03 -08:00
Michael Vines
36ad59673c
drop mut
2022-03-04 09:52:46 +01:00
Michael Vines
0d33b54d74
Rework do_process_blockstore_from_root to use BankForks
2022-03-04 09:52:46 +01:00
Michael Vines
93c8e04d51
Simplify do_process_blockstore_from_root slightly
2022-03-04 09:52:46 +01:00
dependabot[bot]
360f6466a3
chore: bump futures from 0.3.19 to 0.3.21 ( #23476 )
...
* chore: bump futures from 0.3.19 to 0.3.21
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.19 to 0.3.21.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.19...0.3.21 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-03-03 16:44:43 -07:00
Yueh-Hsuan Chiang
62d2a4cd88
Make ShredStorageType::RocksLevel public ( #23272 )
...
#### Summary of Changes
This PR adds two hidden arguments to the validator that allow users to use RocksDB's FIFO compaction for storing shreds.
--shred-storage <SHRED_STORAGE>
EXPERIMENTAL: Controls how RocksDB compacts shreds. *WARNING*: You will lose your ledger data
when you switch between options. Possible values are: 'level': stores shreds using RocksDB's default (level)
compaction. 'fifo': stores shreds under RocksDB's FIFO compaction. This option is more efficient on
disk-write-bytes of the ledger store. [default: level] [possible values: level, fifo]
--shred-storage-size <SHRED_STORAGE_SIZE_BYTES>
The shred storage size in bytes. The suggested value is 50% of your ledger storage size in bytes. [default:
268435456000]
2022-03-03 12:43:58 -08:00
Yueh-Hsuan Chiang
634f4eb37d
(LedgerStore) Use different path for different blockstore storage type. ( #23236 )
...
#### Summary of Changes
To avoid mixing the use of different shred storage types, each shred storage type
will have its blockstore in a different directory.
This PR still keeps the RocksFifo setting hidden. The default ShredStorageType and
blockstore directory are still RocksLevel and `rocksdb`.
Will follow-up with PRs on making FIFO option public in ledger-tool and validator.
#### Test Plan
* Added a new test to verify the existence of `rocksdb-fifo` directory when FIFO compaction is used.
* Updated existing test to verify the current setting still store ledger under `rocksdb` directory.
* Manually ran ledger_cleanup_test with both level and fifo compaction and verified the resulting ledger.
* Ran a validator with this PR.
2022-03-02 18:30:22 -08:00
Will Hickey
1a99251498
Bump version to 1.10.1 ( #23453 )
2022-03-02 13:47:01 -06:00
Jeff Washington (jwash)
ef8b7d9c62
ledger tool halt at slot verify hash ( #23424 )
2022-03-02 11:11:18 -06:00
Tyera Eulberg
3b5b71ce44
Improve UX querying rpc for blocks at or before the snapshot from boot ( #23403 )
...
* Bump first-available block to first complete block
* Remove obsolete purges in tests (PrimaryIndex toggling no longer in use
* Check first-available block in Rpc check_slot_cleaned_up
2022-02-28 23:57:41 -07:00
dependabot[bot]
985af71241
chore: bump sha2 from 0.10.1 to 0.10.2 ( #23343 )
...
* chore: bump sha2 from 0.10.1 to 0.10.2
Bumps [sha2](https://github.com/RustCrypto/hashes ) from 0.10.1 to 0.10.2.
- [Release notes](https://github.com/RustCrypto/hashes/releases )
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.10.1...sha2-v0.10.2 )
---
updated-dependencies:
- dependency-name: sha2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-02-24 23:57:42 -07:00
Jeff Washington (jwash)
39c86c6c44
fmt ( #23329 )
2022-02-24 12:12:29 -06:00
steviez
12bddbc4ec
Use rocksdb property constant instead of local constant ( #23147 )
2022-02-23 23:02:11 -06:00
Justin Starry
d0e85c293f
Fix rustfmt check ( #23296 )
2022-02-23 16:38:53 +08:00
Gavin Chan
20d031e2b8
Refactor ExecuteTimings w/ enum-indexed array ( #23085 )
2022-02-22 14:46:56 -08:00
Yueh-Hsuan Chiang
8c872e9ce0
(LedgerStore/FIFO) Refactor FIFO options and sanity check. ( #23131 )
2022-02-19 00:58:38 -08:00
Justin Starry
1719d2349f
Skip adding builtins if they will be removed ( #23233 )
...
* Add failing test for precompile transition
* Skip adding builtins if they will be removed
* cargo clean
* nits
* fix abi check
* remove workaround
Co-authored-by: Jack May <jack@solana.com>
2022-02-18 18:36:59 -08:00
Yueh-Hsuan Chiang
7939fdc3e5
Minor refactor on Shreds column family descriptor construction. ( #23103 )
2022-02-16 00:29:11 -08:00
Lijun Wang
c04438be4b
Retaining transaction logs when transaction plugin is loaded. ( #22874 )
...
Transaction logs are not being saved to the database through the plugin interface.
Summary of Changes
Retain the transaction logs when transaction notification plugin is loaded.
Fixes #
lijunwangs/solana-accountsdb-plugin-postgres#6
2022-02-11 20:29:07 -08:00
Yueh-Hsuan Chiang
6399647e64
(Ledger Store) Add code comments for SlotMetaWorkingSetEntry. ( #22662 )
2022-02-11 10:04:26 -08:00
HaoranYi
e4a1799334
fix typos in ledge code comments ( #23068 )
2022-02-10 20:43:44 -07:00
Yueh-Hsuan Chiang
9213fcb11b
Introduce experimental FIFO-compaction option for shreds in blockstore ( #22140 )
2022-02-10 11:34:03 -08:00
Justin Starry
d5dec989b9
Enforce tx metadata upload with static types ( #23028 )
2022-02-10 13:28:18 +08:00
steviez
2ca7ae8e30
Update rocksdb to 0.18.0 release ( #23031 )
2022-02-09 21:51:56 -06:00
dependabot[bot]
9d036be61b
chore: bump libc from 0.2.115 to 0.2.117 ( #23034 )
...
* chore: bump libc from 0.2.115 to 0.2.117
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.115 to 0.2.117.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.115...0.2.117 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-02-09 16:49:44 -07:00
Pankaj Garg
a146f2d853
suppress clippy
2022-02-07 08:02:58 -08:00
Pankaj Garg
c5d8560cdb
updates to address review feedback
2022-02-07 08:02:58 -08:00
Pankaj Garg
dfef68f985
fix build
2022-02-07 08:02:58 -08:00
Pankaj Garg
4de14e530b
Optimize batching of transactions during replay for parallel processing
2022-02-07 08:02:58 -08:00
Yueh-Hsuan Chiang
37afdd1a65
(Ledger Store) Improve comments for blockstore_purge ( #22808 )
2022-02-06 21:56:23 -08:00
steviez
c7ca2f41f5
Add some comments for RocksDB option configuration ( #22655 )
2022-02-06 23:28:24 -06:00
Justin Starry
ba215e94f6
Refactor: Add AccountKeys struct for static and dynamic message keys ( #22960 )
2022-02-05 20:00:31 +08:00
Jeff Washington (jwash)
fb95fa68a2
cleanup api ( #22845 )
2022-01-31 11:39:23 -06:00
Yueh-Hsuan Chiang
aaae5b3ba6
(Ledger Store) Add comment blocks for six pub functions in blockstore.rs ( #22476 )
2022-01-28 22:25:07 -08:00
Yueh-Hsuan Chiang
40a49081b9
(Ledger Store) Add comments for find slot functions. ( #22666 )
2022-01-28 12:07:09 -08:00
dependabot[bot]
cc74693176
chore: bump serde from 1.0.134 to 1.0.136 ( #22817 )
...
* chore: bump serde from 1.0.134 to 1.0.136
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.134 to 1.0.136.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.134...v1.0.136 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-28 12:20:37 -07:00
dependabot[bot]
7376efe8ea
chore: bump libc from 0.2.112 to 0.2.115 ( #22796 )
...
* chore: bump libc from 0.2.112 to 0.2.115
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.112 to 0.2.115.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.112...0.2.115 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-27 21:54:13 +00:00
Michael Vines
6d5bbca630
Pacify clippy
2022-01-21 19:12:57 -08:00
dependabot[bot]
0cf886302d
chore: bump serde from 1.0.133 to 1.0.134 ( #22650 )
...
* chore: bump serde from 1.0.133 to 1.0.134
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.133 to 1.0.134.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.133...v1.0.134 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-21 17:14:02 -07:00
Yueh-Hsuan Chiang
fe7543c31a
(Ledger Store) APIs for obtaining physical size of all data and coding shreds ( #22443 )
2022-01-19 19:31:19 -08:00
Tao Zhu
a724fa2347
Add hidden cli option to allow validator reports replayed transaction cost metrics ( #22369 )
...
* add hidden cli option to allow validator reports replayed transaction cost detail metrics
* Update validator/src/main.rs
Co-authored-by: Michael Vines <mvines@gmail.com>
* - rebase master, using unbounded instead of channel; dowgrade to datapoint_trace
* removed cli arg, prefer log at trace
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-01-15 00:31:21 +00:00
Justin Starry
f804ccdece
Store address table lookups in blockstore and bigtable ( #22402 )
2022-01-14 15:24:41 +08:00
carllin
eca8d21249
log internals ( #22493 )
2022-01-13 19:50:46 -05:00
Jeff Biseda
8b66625c95
convert std::sync::mpsc to crossbeam_channel ( #22264 )
2022-01-11 02:44:46 -08:00
dependabot[bot]
28275a33d6
Bump tempfile from 3.2.0 to 3.3.0 ( #22401 )
...
* Bump tempfile from 3.2.0 to 3.3.0
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.2.0 to 3.3.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases )
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS )
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.2.0...v3.3.0 )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-09 12:26:08 -07:00
dependabot[bot]
2759cb860b
Bump sha2 from 0.10.0 to 0.10.1 ( #22359 )
...
* Bump sha2 from 0.10.0 to 0.10.1
Bumps [sha2](https://github.com/RustCrypto/hashes ) from 0.10.0 to 0.10.1.
- [Release notes](https://github.com/RustCrypto/hashes/releases )
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.10.0...sha2-v0.10.1 )
---
updated-dependencies:
- dependency-name: sha2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-08 00:17:32 -07:00
carllin
813006b33b
remove per program timings from blockstore processor ledger replay ( #22370 )
2022-01-08 01:52:08 -05:00
Yueh-Hsuan Chiang
5771c36d3f
Rename open_with_access_type() to open_with_options() ( #22123 )
2022-01-07 12:11:43 -08:00
steviez
207825d30b
Minimize boilerplate code around Rocks column families ( #22345 )
2022-01-06 23:39:09 -06:00
Justin Starry
52d12cc802
Add runtime support for address table lookups ( #22223 )
...
* Add support for address table lookups in runtime
* feedback
* feedback
2022-01-07 11:59:09 +08:00
Trent Nelson
7d32909e17
move `ExecuteTimings` from `runtime::bank` to `program_runtime::timings`
2022-01-06 03:56:46 -07:00
Justin Starry
45458e7139
Refactor: Improve type safety and readability of transaction execution ( #22215 )
...
* Refactor Bank::load_and_execute_transactions
* Refactor: improve type safety of TransactionExecutionResult
* Add enum for extra type safety in execution results
* feedback
2022-01-05 10:15:15 +08:00
behzad nouri
01a096adc8
adds bitflags to Packet.Meta
...
Instead of a separate bool type for each flag, all the flags can be
encoded in a type-safe bitflags encoded in a single u8:
https://github.com/solana-labs/solana/blob/d6ec103be/sdk/src/packet.rs#L19-L31
2022-01-04 13:53:40 +00:00
Jeff Biseda
ca8fef5855
retransmit consecutive leader blocks ( #22157 )
2022-01-04 00:24:16 -08:00
Yueh-Hsuan Chiang
e8b7f96a89
Add struct BlockstoreOptions ( #22121 )
2022-01-03 18:30:45 -10:00
dependabot[bot]
672fed04cb
Bump serde from 1.0.132 to 1.0.133 ( #22233 )
...
* Bump serde from 1.0.132 to 1.0.133
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.132 to 1.0.133.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.132...v1.0.133 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-03 22:19:39 +00:00
Justin Starry
2a00382d71
Refactor: cleanup solana_transaction_status crate ( #22230 )
2022-01-03 15:45:18 +00:00
carllin
33d0b5e011
Revert "Count compute units even when transaction errors ( #22059 )" ( #22174 )
...
This reverts commit eaa8c67bde
.
2021-12-30 02:42:32 -05:00
Omar Kilani
bac6821e19
get_signatures_for_address does not correctly account for result sets that span local and Bigtable sources ( #22115 )
...
* get_signatures_for_address does not correctly account for result sets that span Blockstore and Bigtable.
This causes Bigtable to return `RowNotFound` until the new tx is uploaded.
Check that `before` exists in Bigtable, and if not, set it to `None` to return the full data set.
References #21442
Closes #22110
* Differentiate between before sig not found and no newer signatures
* Dedupe bigtable results to account for potential upload race
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-12-29 10:25:10 -07:00
carllin
eaa8c67bde
Count compute units even when transaction errors ( #22059 )
2021-12-28 17:05:11 -05:00
Yueh-Hsuan Chiang
b89cd8cd1a
Avoid cloning Vec<Entry> when calling entries_to_test_shreds() ( #22093 )
2021-12-24 12:32:43 -08:00
behzad nouri
f67ecd5c18
removes unused Cargo dependencies ( #22022 )
...
Also moving some to [dev-dependencies] so that they are not propagated
to other packages which depend on the package.
2021-12-22 17:01:33 +00:00
Yueh-Hsuan Chiang
bf8fbf8383
Add code comment for handle_chaining in blockstore.rs ( #21876 )
2021-12-21 22:36:24 -08:00
Yueh-Hsuan Chiang
3155a04189
Add comment block for insert_shreds_handle_duplicate in blockstore.rs ( #21877 )
2021-12-21 22:36:13 -08:00
dependabot[bot]
41ec7c8be9
chore: bump num_cpus from 1.13.0 to 1.13.1 ( #22044 )
...
* chore: bump num_cpus from 1.13.0 to 1.13.1
Bumps [num_cpus](https://github.com/seanmonstar/num_cpus ) from 1.13.0 to 1.13.1.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases )
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.13.0...v1.13.1 )
---
updated-dependencies:
- dependency-name: num_cpus
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-21 00:06:14 -07:00
dependabot[bot]
e810400716
chore: bump futures from 0.3.18 to 0.3.19 ( #22020 )
...
* chore: bump futures from 0.3.18 to 0.3.19
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.18 to 0.3.19.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.18...0.3.19 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-20 20:45:19 +00:00
behzad nouri
65d59f4ef0
tracks erasure coding shreds' indices explicitly ( #21822 )
...
The indices for erasure coding shreds are tied to data shreds:
https://github.com/solana-labs/solana/blob/90f41fd9b/ledger/src/shred.rs#L921
However with the upcoming changes to erasure schema, there will be more
erasure coding shreds than data shreds and we can no longer infer coding
shreds indices from data shreds.
The commit adds constructs to track coding shreds indices explicitly.
2021-12-19 22:37:55 +00:00
dependabot[bot]
8f547a6c98
chore: bump serde from 1.0.131 to 1.0.132 ( #21989 )
...
* chore: bump serde from 1.0.131 to 1.0.132
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.131 to 1.0.132.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.131...v1.0.132 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-18 07:45:20 +00:00
Michael Vines
5f054cd51b
Update to reed-solomon-erasure 5.0.1, to get simd-accel on M1 macs
2021-12-17 14:19:39 -08:00
behzad nouri
89d66c3210
removes next_shred_index from return value of entries to shreds api ( #21961 )
...
next-shred-index is already readily available from returned data shreds.
The commit simplifies the api for upcoming changes to erasure coding
schema which will require explicit tracking of indices for coding shreds
as well as data shreds.
2021-12-17 15:01:55 +00:00
behzad nouri
efd64a3862
simplifies ShredIndex api ( #21932 )
2021-12-16 19:17:32 +00:00
behzad nouri
8183f28636
adds ErasureSetId identifying erasure coding sets of shreds ( #21928 )
2021-12-16 14:18:55 +00:00
Yueh-Hsuan Chiang
882f886450
Add comment block for commit_slot_meta_working_set in blockstore.rs ( #21852 )
2021-12-15 13:12:50 -08:00
Yueh-Hsuan Chiang
e374fb1d60
Add code comment for get_slot_meta_entry in blockstore.rs ( #21844 )
2021-12-15 13:12:38 -08:00
Yueh-Hsuan Chiang
71b12b1f56
Add comment for clear_unconfirmed_slot() in blockstore.rs ( #21837 )
2021-12-15 00:37:09 -08:00
Yueh-Hsuan Chiang
e476e17abf
Add code comment for check_insert_data_shred in blockstore.rs ( #21845 )
2021-12-15 00:36:11 -08:00
Michael Vines
2a6dcb2ffd
Futures 0.3.18 has been yanked, back off to .17
2021-12-14 14:14:43 -08:00
behzad nouri
8d980f07ba
uses Option<Slot> for SlotMeta.parent_slot ( #21808 )
...
SlotMeta.parent_slot for the head of a detached chain of slots is
unknown and that is indicated by u64::MAX which lacks type-safety:
https://github.com/solana-labs/solana/blob/6c108c8fc/ledger/src/blockstore_meta.rs#L203-L205
The commit changes the type to Option<Slot>. Backward compatibility is
maintained by customizing serde serialize/deserialize implementations.
2021-12-14 18:57:11 +00:00
behzad nouri
4ceb2689f5
adds ShredId uniquely identifying each shred ( #21820 )
2021-12-14 17:34:02 +00:00
Ikko Ashimine
e19c7923c3
chore: fix typo in shred.rs ( #21890 )
...
begining -> beginning
2021-12-14 11:19:27 -05:00
dependabot[bot]
8a63812c4e
chore: bump libc from 0.2.109 to 0.2.112 ( #21870 )
...
* chore: bump libc from 0.2.109 to 0.2.112
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.109 to 0.2.112.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.109...0.2.112 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-14 00:27:57 -07:00
steviez
4a9d7318d1
Rework test parameters to be shreds instead of entries ( #21780 )
...
The number of shreds that result from a given number of entries is
variable and in our test case, somewhat unintuitive to think about when
trying to determine how much data we're pushing into the blockstore. So,
this change converts the unit of test parameters from entries to shreds.
This change also cleans up some variable naming for clarity and prints.
2021-12-13 23:34:43 -06:00
behzad nouri
e08139f949
uses Option<u64> for SlotMeta.last_index ( #21775 )
...
SlotMeta.last_index may be unknown and current code is using u64::MAX to
indicate that:
https://github.com/solana-labs/solana/blob/6c108c8fc/ledger/src/blockstore_meta.rs#L169-L174
This lacks type-safety and can introduce bugs if not always checked for
Several instances of slot_meta.last_index + 1 are also subject to
overflow.
This commit updates the type to Option<u64>. Backward compatibility is
maintained by customizing serde serialize/deserialize implementations.
2021-12-11 14:47:20 +00:00
Justin Starry
254ef3e7b6
Rename Packets to PacketBatch ( #21794 )
2021-12-11 09:44:15 -05:00
behzad nouri
49ba09b333
adds back ErasureMeta::first_coding_index field ( #21623 )
...
https://github.com/solana-labs/solana/pull/16646
removed first_coding_index since the field is currently redundant and
always equal to fec_set_index.
However, with upcoming changes to erasure coding schema, this will no
longer be the same as fec_set_index and so requires a separate field to
represent.
2021-12-10 20:08:04 +00:00
dependabot[bot]
d870f566ef
chore: bump serde from 1.0.130 to 1.0.131 ( #21758 )
...
* chore: bump serde from 1.0.130 to 1.0.131
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.130 to 1.0.131.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.130...v1.0.131 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-10 08:47:46 +00:00
behzad nouri
8063273d09
adds more sanity checks to shreds ( #21675 )
2021-12-09 16:43:57 +00:00
dependabot[bot]
b61b7189a5
chore: bump hmac from 0.11.0 to 0.12.0 ( #21681 )
...
* chore: bump hmac from 0.11.0 to 0.12.0
Bumps [hmac](https://github.com/RustCrypto/MACs ) from 0.11.0 to 0.12.0.
- [Release notes](https://github.com/RustCrypto/MACs/releases )
- [Commits](https://github.com/RustCrypto/MACs/compare/hmac-v0.11.0...hmac-v0.12.0 )
---
updated-dependencies:
- dependency-name: hmac
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* Update dependabot-pr.sh
* Bump sha2 and pbkdf2 as well
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: axleiro <83293196+axleiro@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-12-08 18:12:35 -07:00
dependabot[bot]
f6801a4af4
chore: bump itertools from 0.10.1 to 0.10.3 ( #21643 )
...
* chore: bump itertools from 0.10.1 to 0.10.3
Bumps [itertools](https://github.com/rust-itertools/itertools ) from 0.10.1 to 0.10.3.
- [Release notes](https://github.com/rust-itertools/itertools/releases )
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.10.1...v0.10.3 )
---
updated-dependencies:
- dependency-name: itertools
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-06 23:23:26 -07:00
dependabot[bot]
b5353e2130
chore: bump libc from 0.2.108 to 0.2.109 ( #21627 )
...
* chore: bump libc from 0.2.108 to 0.2.109
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.108 to 0.2.109.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.108...0.2.109 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-12-06 09:37:26 -07:00
behzad nouri
cd17f63d81
adds back position field to coding-shred-header ( #21600 )
...
https://github.com/solana-labs/solana/pull/17004
removed position field from coding-shred-header because as it stands the
field is redundant and unused.
However, with the upcoming changes to erasure coding schema this field
will no longer be redundant and needs to be populated.
2021-12-05 14:42:09 +00:00
Trent Nelson
d6f22433d0
Bump version to v1.10.0
2021-12-04 20:17:54 +00:00
Michael Vines
b8837c04ec
Reformat imports to a consistent style for imports
...
rustfmt.toml configuration:
imports_granularity = "One"
group_imports = "One"
2021-12-03 09:19:13 -08:00
Michael Vines
dd12d90eac
Upgrade to Rust 2021
2021-11-30 20:43:46 -08:00
ryleung-solana
8cf36e5cb0
Add GPU sigverify for verify path ( #20851 )
...
Allows the use of GPU acceleration in verifying the signatures in Entry's after deserialization in the replay stage
Co-authored-by: Stephen Akridge <sakridge@gmail.com>
Co-authored-by: Ryan Leung <ryan.leung@solana.com>
2021-11-30 21:16:13 -05:00
Yueh-Hsuan Chiang
38fcfb7542
Persist coding shreds ( #21214 )
2021-11-30 09:18:36 -08:00
Michael Vines
e31b469f56
Quash M1 build warning
2021-11-29 22:42:09 -08:00
Michael Vines
ba9dfa0d22
Remove frozen account support
2021-11-29 08:38:11 -08:00
dependabot[bot]
3c2276fd09
chore: bump futures from 0.3.17 to 0.3.18 ( #21412 )
...
* chore: bump futures from 0.3.17 to 0.3.18
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.17 to 0.3.18.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.17...0.3.18 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-11-23 21:55:17 -07:00
Tyera Eulberg
9e043d2055
Use max_complete_transaction_status_slot in BigTableUploadService ( #21401 )
...
* Use max_complete_transaction_status_slot in BigTableUploadService
* Use node root to limit BigTableUploadService
2021-11-23 17:36:13 -07:00
behzad nouri
dd338b6c9f
changes Shred::parent return type to Option<Slot> ( #21370 )
...
Shred::parent can return garbage if the struct fields are invalid:
https://github.com/solana-labs/solana/blob/8a50b6302/ledger/src/shred.rs#L446-L453
The commit adds more sanity checks and changes the return type to Option<Slot>.
2021-11-23 14:45:26 +00:00
dependabot[bot]
191cb62c37
chore: bump libc from 0.2.107 to 0.2.108 ( #21377 )
...
* chore: bump libc from 0.2.107 to 0.2.108
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.107 to 0.2.108.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.107...0.2.108 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-11-22 17:02:34 -07:00
Jeff Biseda
2ed7e3af89
prioritize slot repairs for unknown last index and close to completion ( #21070 )
2021-11-19 19:17:30 -08:00
behzad nouri
7da2df7d8c
adds methods to obtain shreds' erasure coded block and index ( #21325 )
2021-11-19 20:01:15 +00:00
behzad nouri
48dfdfb4d5
changes Blockstore::is_shred_duplicate arg type to ShredType
2021-11-19 14:16:39 +00:00
behzad nouri
57057f8d39
uses enum for shred type
...
Current code is using u8 which does not have any type-safety and can
contain invalid values:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L167
Checks for invalid shred-types are scattered through the code:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/blockstore.rs#L849-L851
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L346-L348
The commit uses enum for shred type with #[repr(u8)]. Backward
compatibility is maintained by implementing Serialize and Deserialize
compatible with u8, and adding a test to assert that.
2021-11-19 14:16:39 +00:00
Justin Starry
66fa062f13
rename process_entries to indicate it's only for tests ( #21321 )
2021-11-17 20:53:40 +01:00
Yueh-Hsuan Chiang
3aa49e2c69
Keep all erasure coding shreds even on successful recovery ( #20968 ) ( #21052 )
...
Problem
Blockstore currently removes erasure shreds if the data shreds are
successfully recovered on insert, which is an issue if we want to
serve coding shreds over repair.
Summary of Changes
This diff keeps all coding shreds even on successful recovery and
changes change the signature of prev_inserted_codes to immutable
reference to ensure its immunity.
Fixes #20968
2021-11-16 11:26:34 -08:00
behzad nouri
3fc858eb60
adds methods to obtain data/coding shreds indices from ErasureMeta
2021-11-13 17:08:05 +00:00
behzad nouri
d25d9be555
removes Default implementaion of ErasureConfig
...
The Default implementation of ErasureConfig is off and never useful:
https://github.com/solana-labs/solana/blob/150951333/ledger/src/erasure.rs#L49-L68
2021-11-13 17:08:05 +00:00
behzad nouri
7e98b77b33
marks ErasureMeta.size as unused
...
ErasureMeta.size is unused and redundant:
https://github.com/solana-labs/solana/blob/150951333/ledger/src/blockstore_meta.rs#L57-L58
2021-11-13 17:08:05 +00:00
behzad nouri
5fb0ab9d00
removes redundant args from Shredder::try_recovery ( #21226 )
...
Shredder::try_recovery is already taking a Vec<Shred> as an argument. All the
other arguments are embedded in the shreds, and are so redundant.
2021-11-10 21:19:03 +00:00
Kirill Fomichev
b4d09388aa
Disable JIT compiler on unsupported platforms ( #21215 )
2021-11-09 11:31:21 +00:00
dependabot[bot]
c2754b8a51
chore: bump libc from 0.2.106 to 0.2.107 ( #21202 )
...
* chore: bump libc from 0.2.106 to 0.2.107
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.106 to 0.2.107.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.106...0.2.107 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-11-08 12:14:12 -07:00
behzad nouri
11a53de0e3
rewrites Shredder::try_recovery ( #21082 )
...
For every missing data-shred, Shredder::try_recovery calls into
new_empty_data_shred which does a redundant serialization into payload
buffer which is then immediately overwritten by the erasure recovery:
https://github.com/solana-labs/solana/blob/696501500/ledger/src/shred.rs#L372-L417
Additionally, the implementation is unnecessary complex hindering
upcoming changes to erasure coding generation
https://github.com/solana-labs/solana/blob/696501500/ledger/src/shred.rs#L814-L938
The commit simplifies the Shredder::try_recovery implementation.
2021-11-04 16:15:51 +00:00
dependabot[bot]
ca51ea3e04
chore: bump libc from 0.2.105 to 0.2.106 ( #21093 )
...
* chore: bump libc from 0.2.105 to 0.2.106
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.105 to 0.2.106.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.105...0.2.106 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-11-01 14:17:45 -06:00
behzad nouri
484ead01ed
optimizes and simplifies SlotMeta::completed_data_indexes ops ( #21059 )
...
SlotMeta::completed_data_indexes is defined as a Vec<u32>:
https://github.com/solana-labs/solana/blob/a8d78e89d/ledger/src/blockstore_meta.rs#L31-L32
which results in inefficient updates:
https://github.com/solana-labs/solana/blob/a8d78e89d/ledger/src/blockstore.rs#L3245-L3326
This commit changes the type to BTreeSet<u32> for efficient and simpler
updates and lookups.
The change should be backward compatible because Vec<T> and BTreeSet<T>
are both serialized as seq:
https://github.com/serde-rs/serde/blob/ce0844b9e/serde/src/ser/impls.rs#L207-L208
https://github.com/serde-rs/serde/blob/ce0844b9e/serde/src/ser/impls.rs#L216-L217
2021-10-31 12:56:25 +00:00
sakridge
4642a2c856
Optimize test_recovery_with_expanded_coding_shreds ( #20849 )
2021-10-28 11:38:08 +02:00
Tyera Eulberg
f0de3e9bf0
chore: bump tonic from 0.5.2 to 0.6.1 ( #21035 )
...
* Bump tonic and prost crates
* connect_lazy no longer infallible
2021-10-28 06:59:36 +00:00
Justin Starry
036d7fcc81
Clean up sanitized tx creation for tests ( #21006 )
2021-10-27 18:09:16 +01:00
dependabot[bot]
cb1947bea2
chore: bump libc from 0.2.104 to 0.2.105 ( #20972 )
...
* chore: bump libc from 0.2.104 to 0.2.105
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.104 to 0.2.105.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.104...0.2.105 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-10-26 00:14:58 -06:00
Michael Vines
350bb561eb
Clippy
2021-10-23 08:21:20 +00:00
dependabot[bot]
66e431dad5
chore: bump libc from 0.2.103 to 0.2.104 ( #20764 )
...
* chore: bump libc from 0.2.103 to 0.2.104
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.103 to 0.2.104.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.103...0.2.104 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-10-19 22:38:02 -06:00
carllin
b5f21d5e34
Add ledger tool to remove dead slot ( #20810 )
2021-10-19 20:23:16 -07:00
behzad nouri
231b58b5f1
adds more metrics to blockstore insert shreds stats ( #20701 )
2021-10-17 11:02:34 -04:00
Jeff Washington (jwash)
70b2e5fef2
hash calc uses all threads during startup ( #20735 )
2021-10-15 17:41:23 -05:00
Tyera Eulberg
e806fa6904
Include token owners in TransactionTokenBalances ( #20642 )
...
* Cache owners in TransactionTokenBalances
* Light cleanup
* Use return struct, and remove pub
* Single-use statements
* Why not, just do the whole crate
* Add metrics
* Make datapoint_debug to prevent spam unless troubleshooting
2021-10-13 21:46:52 -06:00
sakridge
09e7782d76
Refactor code to get block signatures in get_confirmed_signatures_for_address2 ( #20575 )
...
* Refactor get_confirmed_signatures_for_address2
* Move blockstore benches to ledger where they belong
2021-10-13 09:55:19 +02:00
Tao Zhu
005d6863fd
- move cost tracker into bank, so each bank has its own cost tracker; ( #20527 )
...
- move related modules to runtime
2021-10-12 08:51:33 -05:00
Jeff Washington (jwash)
a8e000a2a6
add filler accounts to bloat validator and predict failure ( #20491 )
...
* add filler accounts to bloat validator and predict failure
* assert no accounts match filler
* cleanup magic numbers
* panic if can't load from snapshot with filler accounts specified
* some renames
* renames
* into_par_iter
* clean filler accts, too
2021-10-11 12:46:27 -05:00
Brooks Prumo
5440c1d2e1
SnapshotPackagerService pushes incremental snapshot hashes to CRDS ( #20442 )
...
Now that CRDS supports incremental snapshot hashes,
SnapshotPackagerService needs to push 'em!
This commit does two main things:
1. SnapshotPackagerService now knows about incremental snapshot hashes,
and will push SnapshotPackage::IncrementalSnapshot hashes to CRDS.
2. At startup, when loading from a full + incremental snapshot, the
hashes need to be passed all the way to SnapshotPackagerService so it
can push these starting hashes to CRDS. Those values have been piped
through.
Fixes #20441 and #20423
2021-10-08 15:14:56 -05:00
Tao Zhu
675fa6993b
- update const cost values with data collected by #19627 ( #20314 )
...
- update cost calculation to closely proposed fee schedule #16984
2021-10-08 14:48:50 -05:00
Trent Nelson
767f740305
Bump version to 1.9.0
2021-10-06 17:57:41 -07:00
Michael Vines
7027d56064
Resolve nightly-2021-10-05 clippy complaints
2021-10-06 10:37:58 -07:00
steviez
d8e4840b49
Follow up to blockstore test cleanup ( #19656 ) ( #20143 )
2021-10-04 10:35:03 -05:00
Lijun Wang
fe97cb2ddf
AccountsDb plugin framework ( #20047 )
...
Summary of Changes
Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows
Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.
The code comprises 4 major parts:
accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
2021-09-30 14:26:17 -07:00
dependabot[bot]
9393654af0
chore: bump libc from 0.2.102 to 0.2.103 ( #20280 )
...
* chore: bump libc from 0.2.102 to 0.2.103
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.102 to 0.2.103.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.102...0.2.103 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
2021-09-28 07:50:46 +00:00
Justin Starry
c71fab6cb3
Add `delete` subcommand to `ledger-tool bigtable` ( #19931 )
...
* Add `delete` subcommand to `ledger-tool bigtable` command
* feedback
2021-09-16 23:37:45 +00:00
dependabot[bot]
ea15c0b40b
chore: bump libc from 0.2.101 to 0.2.102 ( #19924 )
...
* chore: bump libc from 0.2.101 to 0.2.102
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.101 to 0.2.102.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.101...0.2.102 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-16 09:45:01 -06:00
steviez
4fdd70c931
Properly clean temp files in Blockstore::create_new_ledger() ( #19589 )
...
Use TempDir::path() instead of TempDir::into_path(); into_path() returns
the underlying PathBuf and voids the promise to delete the directory
when TempDir goes out of scope.
2021-09-13 23:13:56 -04:00
Brooks Prumo
97fc64dd0c
Call exhaustively_free_unused_resource() instead ( #19847 )
2021-09-13 18:08:32 -05:00
Brooks Prumo
ef749a2506
Flush accounts cache before calling clean_accounts() ( #19839 )
2021-09-13 14:56:13 -05:00
Brooks Prumo
3f6eb96d6e
Call `Bank::update_accounts_hash()` before taking bank snapshot ( #19765 )
...
This is a bugfix.
When `load_frozen_forks()` called `snapshot_bank()`, the accounts hash
was not computed. So then when a snapshot archive was eventually
created, its hash would be all 1s, which is clearly wrong. The fix is
to update the bank's accounts hash before taking the bank snapshot.
2021-09-13 08:27:50 -05:00
Brooks Prumo
62c8bcf565
Add default() to SnapshotConfig ( #19776 )
2021-09-12 13:44:27 -05:00
dependabot[bot]
857fce13f9
chore: bump sha2 from 0.9.6 to 0.9.8 ( #19760 )
...
* chore: bump sha2 from 0.9.6 to 0.9.8
Bumps [sha2](https://github.com/RustCrypto/hashes ) from 0.9.6 to 0.9.8.
- [Release notes](https://github.com/RustCrypto/hashes/releases )
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.9.6...sha2-v0.9.8 )
---
updated-dependencies:
- dependency-name: sha2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-11 16:22:40 -06:00
Giles Cope
c3a610374a
unused deps
2021-09-11 14:33:53 -07:00
steviez
2dee098b91
Blockstore test cleanup ( #19656 )
...
* Use auto-deleting macro for ledger path
2021-09-10 07:33:08 -05:00
Jeff Washington (jwash)
456bf15012
AccountsIndexConfig -> AccountsDbConfig ( #19687 )
2021-09-08 04:30:38 +00:00
Jeff Washington (jwash)
d3f938f0cf
Remove Copy from AccountsIndexConfig. Not all types will support it ( #19686 )
2021-09-07 20:09:40 -05:00
Brooks Prumo
4a5f83d3a7
Add fn to check when to take snapshots ( #19682 )
2021-09-07 23:26:35 +00:00
Brooks Prumo
a0552e5b46
Make startup aware of Incremental Snapshots ( #19600 )
2021-09-07 20:43:43 +00:00
Tyera Eulberg
5fa3e5744c
Populate memo in blockstore signatures-for-address ( #19515 )
...
* Add TransactionMemos column family
* Traitify extract_memos
* Write TransactionMemos in TransactionStatusService
* Populate memos from column
* Dedupe and add unit test
2021-09-03 00:28:52 -06:00
Jack May
170927b7c4
nit: Move builtins types to builtins file ( #19597 )
2021-09-02 21:29:11 -07:00
Brooks Prumo
e9374d32a3
Revert "Make startup aware of Incremental Snapshots ( #19550 )" ( #19599 )
...
This reverts commit d45ced0a5d
.
2021-09-02 19:14:41 -05:00
Brooks Prumo
d45ced0a5d
Make startup aware of Incremental Snapshots ( #19550 )
2021-09-02 19:05:15 -05:00
Brooks Prumo
1d2f0f6641
Simplify process_blockstore tests ( #19553 )
...
Some of the blockstore_processor tests weren't using
`test_process_blockstore()`, but could. I updated those tests, and also
changed to using `..` for ignoring, instead of `_`, since I'll be adding
another return value here shortly (thus reducing the need to change
these again).
2021-09-01 20:13:52 +00:00
behzad nouri
6d9818b8e4
skips retransmit for shreds with unknown slot leader ( #19472 )
...
Shreds' signatures should be verified before they reach retransmit
stage, and if the leader is unknown they should fail signature check.
Therefore retransmit-stage can as well expect to know who the slot
leader is and otherwise just skip the shred.
Blockstore checking signature of recovered shreds before sending them to
retransmit stage:
https://github.com/solana-labs/solana/blob/4305d4b7b/ledger/src/blockstore.rs#L884-L930
Shred signature verifier:
https://github.com/solana-labs/solana/blob/4305d4b7b/core/src/sigverify_shreds.rs#L41-L57
https://github.com/solana-labs/solana/blob/4305d4b7b/ledger/src/sigverify_shreds.rs#L105
2021-09-01 15:44:26 +00:00
dependabot[bot]
8e4360cf9c
chore: bump futures from 0.3.16 to 0.3.17 ( #19513 )
...
* chore: bump futures from 0.3.16 to 0.3.17
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.16 to 0.3.17.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.16...0.3.17 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-08-30 22:12:38 -06:00
dependabot[bot]
dc3d7ad2bd
chore: bump sha2 from 0.9.5 to 0.9.6 ( #19511 )
...
* chore: bump sha2 from 0.9.5 to 0.9.6
Bumps [sha2](https://github.com/RustCrypto/hashes ) from 0.9.5 to 0.9.6.
- [Release notes](https://github.com/RustCrypto/hashes/releases )
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.9.5...sha2-v0.9.6 )
---
updated-dependencies:
- dependency-name: sha2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-08-30 19:58:26 -06:00
dependabot[bot]
6a53ec28e0
chore: bump serde from 1.0.129 to 1.0.130 ( #19497 )
...
* chore: bump serde from 1.0.129 to 1.0.130
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.129 to 1.0.130.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.129...v1.0.130 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-08-30 18:19:34 +00:00
dependabot[bot]
fbffdf9f3d
chore: bump futures-util from 0.3.16 to 0.3.17 ( #19498 )
...
* chore: bump futures-util from 0.3.16 to 0.3.17
Bumps [futures-util](https://github.com/rust-lang/futures-rs ) from 0.3.16 to 0.3.17.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.16...0.3.17 )
---
updated-dependencies:
- dependency-name: futures-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-08-30 17:45:47 +00:00
behzad nouri
8ad52fa095
implements copy-on-write for vote-accounts ( #19362 )
...
Bank::vote_accounts redundantly clones vote-accounts HashMap even though
an immutable reference will suffice:
https://github.com/solana-labs/solana/blob/95c998a19/runtime/src/bank.rs#L5174-L5186
This commit implements copy-on-write semantics for vote-accounts by
wrapping the underlying HashMap in Arc<...>.
2021-08-30 15:54:01 +00:00
dependabot[bot]
e492638b7d
chore: bump libc from 0.2.100 to 0.2.101 ( #19442 )
...
* chore: bump libc from 0.2.100 to 0.2.101
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.100 to 0.2.101.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.100...0.2.101 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
2021-08-26 12:38:34 -06:00
Jeff Washington (jwash)
f1d9c014f8
create test function test_process_blockstore ( #19432 )
2021-08-26 12:48:52 -05:00
dependabot[bot]
46890ac197
chore: bump serde from 1.0.128 to 1.0.129 ( #19395 )
...
* chore: bump serde from 1.0.128 to 1.0.129
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.128 to 1.0.129.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.128...v1.0.129 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-08-24 12:32:30 -06:00
Brooks Prumo
f3ce5fa5e8
clippy: branches-sharing-code
...
```text
error: all if blocks contain the same code at the end
--> ledger/src/blockstore.rs:3473:5
|
3473 | / Ok(insert_map.get(&slot).unwrap().clone())
3474 | | }
| |_____^
|
= note: `-D clippy::branches-sharing-code` implied by `-D warnings`
= note: The end suggestion probably needs some adjustments to use the expression result correctly
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#branches_sharing_code
help: consider moving the end statements out like this
|
3473 ~ }
3474 + Ok(insert_map.get(&slot).unwrap().clone())
|
error: could not compile `solana-ledger` due to previous error
```
2021-08-23 11:17:21 -05:00
dependabot[bot]
c0c95e88d8
chore: bump libc from 0.2.99 to 0.2.100 ( #19371 )
...
* chore: bump libc from 0.2.99 to 0.2.100
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.99 to 0.2.100.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.99...0.2.100 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-08-23 09:34:32 -06:00
dependabot[bot]
aea9960c75
chore: bump serde from 1.0.127 to 1.0.128 ( #19370 )
...
* chore: bump serde from 1.0.127 to 1.0.128
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.127 to 1.0.128.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.127...v1.0.128 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
2021-08-23 09:33:34 -06:00
Brooks Prumo
6d939811e9
Name snapshots consistently ( #19346 )
...
#### Problem
Snapshot names are overloaded, and there are multiple terms that mean the same thing. This is confusing. Here's a list of ones in the codebase that I've found:
```
- snapshot_dir
- snapshots_dir
- snapshot_path
- snapshot_output_dir
- snapshot_package_output_path
- snapshot_archives_dir
```
#### Summary of Changes
For all the ones that are about the directory where snapshot archives are stored, ensure they are `snapshot_archives_dir`. For the ones about the (bank) snapshots directory, set to `bank_snapshots_dir`.
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-08-21 15:41:03 -05:00
Tao Zhu
4982dc20f9
replace function with const var for better readability ( #19285 )
2021-08-19 14:59:53 -05:00
Tao Zhu
d7ba15cde8
Update block max cost limit to fix performance regession ( #19276 )
...
* increase block_cost_max by 10 times to accommodate updated account read and write costs; Otherwise during performace test, banking_stage will start reject and retry transactions in next block, result regressed performance.
* fix typo
2021-08-18 16:38:44 +00:00
Justin Starry
c50b01cb60
Store versioned transactions in the ledger, disabled by default ( #19139 )
...
* Add support for versioned transactions, but disable by default
* merge conflicts
* trent's feedback
* bump Cargo.lock
* Fix transaction error encoding
* Rename legacy_transaction method
* cargo clippy
* Clean up casts, int arithmetic, and unused methods
* Check for duplicates in sanitized message conversion
* fix clippy
* fix new test
* Fix bpf conditional compilation for message module
2021-08-17 15:17:56 -07:00
Jeff Washington (jwash)
7c70f2158b
accounts_index_bins to AccountsIndexConfig ( #19257 )
...
* accounts_index_bins to AccountsIndexConfig
* rename param bins -> config
* rename BINS_FOR* to ACCOUNTS_INDEX_CONFIG_FOR*
2021-08-17 14:50:01 -05:00
behzad nouri
7a8807b8bb
retransmits shreds recovered from erasure codes
...
Shreds recovered from erasure codes have not been received from turbine
and have not been retransmitted to other nodes downstream. This results
in more repairs across the cluster which is slower.
This commit channels through recovered shreds to retransmit stage in
order to further broadcast the shreds to downstream nodes in the tree.
2021-08-17 13:44:10 +00:00
behzad nouri
3c71670bd9
returns completed-data-set-info from insert_data_shred
...
instead of opaque (u32, u32) which are then converted to
CompletedDataSetInfo at the call-site.
2021-08-17 13:44:10 +00:00
behzad nouri
b64eeb7729
removes erroneous uses of &Arc<...> from window-service
2021-08-13 17:26:31 +00:00
dependabot[bot]
5b9671d01a
chore: bump matches from 0.1.8 to 0.1.9 ( #19220 )
...
* chore: bump matches from 0.1.8 to 0.1.9
Bumps [matches](https://github.com/SimonSapin/rust-std-candidates ) from 0.1.8 to 0.1.9.
- [Release notes](https://github.com/SimonSapin/rust-std-candidates/releases )
- [Commits](https://github.com/SimonSapin/rust-std-candidates/commits )
---
updated-dependencies:
- dependency-name: matches
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-08-13 16:37:45 +00:00
behzad nouri
d192d78ac2
removes unused code from staking_utils ( #19189 )
2021-08-13 02:42:41 +00:00
Brooks Prumo
5fb6b341c4
Handle cleaning of zero-lamport accounts w.r.t. Incremental Snapshots ( #18870 )
...
* Handle cleaning zero-lamport accounts
Handle cleaning zero-lamport accounts in slots higher than the last full
snapshot slot. This is part of the Incremental Snapshot work.
Fixes #18825
2021-08-12 15:56:08 -05:00
Tao Zhu
414d904959
Reject blocks for costs above the max block cost ( #18994 )
...
* added realtime cost checking logic to reject block that would exceed max limit:
- defines max limits at block_cost_limits.rs
- right after each bath's execution, accumulate its cost and check again
limit, return error if limit is exceeded
* update abi that changed due to adding additional TransactionError
* To avoid counting stats mltiple times, only accumulate execute-timing when a bank is completed
* gate it by a feature
* move cost const def into block_cost_limits.rs
* redefine the cost for signature and account access, removed signer part as it is not well defined for now
* check if per_program_timings of execute_timings before sending
2021-08-12 10:48:47 -05:00
Jeff Washington (jwash)
e91988c977
cli for num account index bins ( #19085 )
2021-08-11 11:45:25 -05:00
behzad nouri
00e5e12906
renames solana_runtime::vote_account::VoteAccount
...
Rename:
VoteAccount -> VoteAccountInner # the private type
ArcVoteAccount -> VoteAccount # the public type
2021-08-10 22:54:17 +00:00
Alexander Meißner
85befbc8cd
Removes bpf_loader_upgradeable_program feature gate. ( #19102 )
2021-08-11 00:08:57 +02:00
dependabot[bot]
536b763751
chore: bump libc from 0.2.98 to 0.2.99 ( #19142 )
...
* chore: bump libc from 0.2.98 to 0.2.99
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.98 to 0.2.99.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.98...0.2.99 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-08-10 10:54:54 -06:00
Jeff Washington (jwash)
651343688d
Use option for account index bucket parameter ( #19150 )
2021-08-10 11:32:25 -05:00
behzad nouri
f302774cf7
implements copy-on-write for staked-nodes ( #19090 )
...
Bank::staked_nodes and Bank::epoch_staked_nodes redundantly clone
staked-nodes HashMap even though an immutable reference will suffice:
https://github.com/solana-labs/solana/blob/a9014cece/runtime/src/vote_account.rs#L77
This commit implements copy-on-write semantics for staked-nodes by
wrapping the underlying HashMap in Arc<...>.
2021-08-10 12:59:12 +00:00
Jeff Washington (jwash)
47e0d9aa95
plumb more accounts_index bins ( #19123 )
2021-08-10 05:45:46 -05:00
Jeff Washington (jwash)
5874f75a26
restore name: Bank::new_with_paths ( #19122 )
2021-08-09 17:30:37 -05:00
Brooks Prumo
00890957ee
Add snapshot_utils::bank_from_latest_snapshot_archives() ( #18983 )
...
While reviewing PR #18565 , as issue was brought up to refactor some code
around verifying the bank after rebuilding from snapshots. A new
top-level function has been added to get the latest snapshot archives
and load the bank then verify. Additionally, new tests have been
written and existing tests have been updated to use this new function.
Fixes #18973
While resolving the issue, it became clear there was some additional
low-hanging fruit this change enabled. Specifically, the functions
`bank_to_xxx_snapshot_archive()` now return their respective
`SnapshotArchiveInfo`. And on the flip side,
`bank_from_snapshot_archives()` now takes `SnapshotArchiveInfo`s instead
of separate paths and archive formats. This bundling simplifies bank
rebuilding.
2021-08-06 20:16:06 -05:00
Jeff Washington (jwash)
ca37873e16
rework bank::new_with_paths ( #19087 )
...
* rework bank::new_with_paths
* missing 1 bench
2021-08-06 09:30:40 -05:00
Jeff Washington (jwash)
14361906ca
for all tests, bank::new -> bank::new_for_tests ( #19064 )
2021-08-05 08:42:38 -05:00
Jeff Washington (jwash)
3280ae3e9f
add validator option --accounts-db-skip-shrink ( #19028 )
...
* add validator option --accounts-db-skip-shrink
* typo
2021-08-04 17:28:33 -05:00
carllin
03353d500f
Actively manage dead slots in AncestorHashesService ( #18912 )
2021-08-02 14:33:28 -07:00
dependabot[bot]
c16bf02448
chore: bump serde from 1.0.126 to 1.0.127 ( #19010 )
...
* chore: bump serde from 1.0.126 to 1.0.127
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.126 to 1.0.127.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.126...v1.0.127 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-08-02 21:16:34 +00:00
dependabot[bot]
4e41c05acd
chore: bump futures from 0.3.15 to 0.3.16 ( #18914 )
...
* chore: bump futures from 0.3.15 to 0.3.16
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.15 to 0.3.16.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.15...0.3.16 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-07-27 07:59:34 -06:00
dependabot[bot]
a4024168de
chore: bump rocksdb from 0.16.0 to 0.17.0 ( #18904 )
...
Bumps [rocksdb](https://github.com/rust-rocksdb/rust-rocksdb ) from 0.16.0 to 0.17.0.
- [Release notes](https://github.com/rust-rocksdb/rust-rocksdb/releases )
- [Changelog](https://github.com/rust-rocksdb/rust-rocksdb/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-rocksdb/rust-rocksdb/compare/v0.16.0...v0.17.0 )
---
updated-dependencies:
- dependency-name: rocksdb
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-07-27 12:36:42 +09:00
dependabot[bot]
b97113408b
chore: bump futures-util from 0.3.15 to 0.3.16 ( #18878 )
...
* chore: bump futures-util from 0.3.15 to 0.3.16
Bumps [futures-util](https://github.com/rust-lang/futures-rs ) from 0.3.15 to 0.3.16.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.15...0.3.16 )
---
updated-dependencies:
- dependency-name: futures-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* Nudge Cargo.lock
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-07-26 18:19:53 +00:00
Brooks Prumo
d1debcd971
Add incremental snapshot utils ( #18504 )
...
This commit adds high-level functions for creating and loading-from
incremental snapshots, plus all low-level functions required to perform
those tasks. This commit **does not** add taking incremental snapshots
as part of a running validator, nor starting up a node with an
incremental snapshot; just laying ground work.
Additionally, `snapshot_utils` and `serde_snapshot` have been
refactored to use a common code paths for the different snapshots.
Also of note, some renaming has happened:
1. Snapshots are now either `full_` or `incremental_` throughout the
codebase. If not specified, the code applies to both.
2. Bank snapshots now are called "bank snapshots"
(before they were called "slot snapshots", "bank snapshots", or
just "snapshots"). The one exception is within `Bank`, where they
are still just "snapshots", because they are already "bank
snapshots".
3. Snapshot archives now have `_archive` in the code. This
should clear up an ambiguity between bank snapshots and snapshot
archives.
2021-07-22 14:40:37 -05:00
carllin
588c0464b8
Add sampling logic and DuplicateSlotRepairStatus module ( #18721 )
2021-07-21 11:15:08 -07:00
Justin Starry
d166b9856a
Move transaction sanitization earlier in the pipeline ( #18655 )
...
* Move transaction sanitization earlier in the pipeline
* Renamed HashedTransaction to SanitizedTransaction
* Implement deref for sanitized transaction
* bring back process_transactions test method
* Use sanitized transactions for cost model calculation
2021-07-15 22:51:27 -05:00
Brian Anderson
37ee0b5599
Eliminate doc warnings and fix some markdown ( #18566 )
...
* Fix link target in doc comment
* Fix formatting of log examples in process_instruction
* Fix doc markdown in solana-gossip
* Fix doc markdown in solana-runtime
* Escape square braces in doc comments to avoid warnings
* Surround 'account references' doc items in code spans to avoid warnings
* Fix code block in loader_upgradeable_instruction
* Fix doctest for loader_upgradable_instruction
2021-07-16 00:40:07 +00:00
Trent Nelson
3a85b77bb5
hijack secp256k1 enablement feature plumbing for libsecp256k1 upgrade
2021-07-15 18:43:55 +00:00
Trent Nelson
568660b402
Revert "Remove feature switch for secp256k1 program ( #18467 )"
...
This reverts commit fd574dcb3b
.
2021-07-15 18:43:55 +00:00
sakridge
7f2254225e
Move entry/poh to own crate to speed up poh bench build ( #18225 )
2021-07-14 14:16:29 +02:00
Jeff Washington (jwash)
d092fa1f03
add ledger-tool verify verify-accounts-index option ( #18375 )
...
* add ledger-tool verify verify-accounts-index option
* comment, merge, respond to feedback, cleanup
2021-07-13 11:06:18 -05:00
carllin
a1c0f144f4
Add blockstore column for frozen hashes and duplicate confirmed ( #18533 )
2021-07-12 20:59:16 -07:00
Michael Vines
4098af3b5b
Record vote account commission with voting/staking rewards and surface in RPC
2021-07-12 15:09:44 -07:00
Tyera Eulberg
761de8b1a3
Bump prost, prost-types, and tonic ( #18537 )
...
* Bump prost+tonic and accommodate generated service changes
* Unignore advisory
* Fixup .proto error list
2021-07-09 17:31:26 +00:00
Justin Starry
fd574dcb3b
Remove feature switch for secp256k1 program ( #18467 )
...
* Remove feature switch for secp256k1 program
* fix tests
2021-07-09 10:08:03 -05:00
dependabot[bot]
c853da7424
chore: bump libc from 0.2.97 to 0.2.98 ( #18516 )
...
* chore: bump libc from 0.2.97 to 0.2.98
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.97 to 0.2.98.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.97...0.2.98 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-07-08 17:31:16 +00:00
jbiseda
a86ced0bac
generate deterministic seeds for shreds ( #17950 )
...
* generate shred seed from leader pubkey
* clippy
* clippy
* review
* review 2
* fmt
* review
* check
* review
* cleanup
* fmt
2021-07-07 08:21:12 -07:00
Justin Starry
5dd399dafa
Fix transaction logs and inner ixs for leader nodes ( #18395 )
...
* Fix transaction logs and inner ixs for leader nodes
* Fix cpi log storage flag
2021-07-06 19:35:24 +00:00
carllin
0eca92de18
Make set roots an iterator ( #18357 )
2021-07-01 20:02:40 -07:00
Justin Starry
d5961e9d9f
Reject transactions with extra signatures ( #18306 )
...
* Reject transactions with extra signatures
* fix tests
* fix check
* fix check
* tx method
* fix checks
2021-07-01 18:06:59 +00:00
Brooks Prumo
45d54b1fc6
Add SnapshotArchiveInfo and refactor functions in snapshot_utils ( #18232 )
2021-07-01 12:20:56 -05:00
Tao Zhu
5e424826ba
Persist cost table to blockstore ( #18123 )
...
* Add `ProgramCosts` Column Family to blockstore, implement LedgerColumn; add `delete_cf` to Rocks
* Add ProgramCosts to compaction excluding list alone side with TransactionStatusIndex in one place: `excludes_from_compaction()`
* Write cost table to blockstore after `replay_stage` replayed active banks; add stats to measure persist time
* Deletes program from `ProgramCosts` in blockstore when they are removed from cost_table in memory
* Only try to persist to blockstore when cost_table is changed.
* Restore cost table during validator startup
* Offload `cost_model` related operations from replay main thread to dedicated service thread, add channel to send execute_timings between these threads;
* Move `cost_update_service` to its own module; replay_stage is now decoupled from cost_model.
2021-07-01 11:32:41 -05:00
dependabot[bot]
644632fc8a
chore: bump assert_matches from 1.3.0 to 1.5.0 ( #18358 )
...
* chore: bump assert_matches from 1.3.0 to 1.5.0
Bumps [assert_matches](https://github.com/murarth/assert_matches ) from 1.3.0 to 1.5.0.
- [Release notes](https://github.com/murarth/assert_matches/releases )
- [Commits](https://github.com/murarth/assert_matches/commits )
---
updated-dependencies:
- dependency-name: assert_matches
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-07-01 09:59:01 -06:00
Brooks Prumo
89a3e4f91e
Move SnapshotConfig into its own module ( #18331 )
...
Also move ArchiveFormat to snapshot_utils, and do not
reexport SnapshotVersion.
2021-07-01 08:55:26 -05:00
Lijun Wang
a67d26a1e8
Fixed an issue doing the set_roots repeatedly for the same set. Instead doing the per chunks. ( #18314 )
2021-06-30 13:29:16 -07:00
dependabot[bot]
78968d132f
chore: bump log from 0.4.11 to 0.4.14 ( #18323 )
...
* chore: bump log from 0.4.11 to 0.4.14
Bumps [log](https://github.com/rust-lang/log ) from 0.4.11 to 0.4.14.
- [Release notes](https://github.com/rust-lang/log/releases )
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/log/compare/0.4.11...0.4.14 )
---
updated-dependencies:
- dependency-name: log
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* Make version consistent
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-06-30 18:32:01 +00:00
sakridge
8d9a6deda4
Add repair number per slot ( #18082 )
2021-06-30 18:20:07 +02:00
dependabot[bot]
52fd10ce03
chore: bump byteorder from 1.3.4 to 1.4.3 ( #18300 )
...
* chore: bump byteorder from 1.3.4 to 1.4.3
Bumps [byteorder](https://github.com/BurntSushi/byteorder ) from 1.3.4 to 1.4.3.
- [Release notes](https://github.com/BurntSushi/byteorder/releases )
- [Changelog](https://github.com/BurntSushi/byteorder/blob/master/CHANGELOG.md )
- [Commits](https://github.com/BurntSushi/byteorder/compare/1.3.4...1.4.3 )
---
updated-dependencies:
- dependency-name: byteorder
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* Bump expected instruction count
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-06-29 15:08:04 -06:00
sakridge
e808f34b0b
Add batch stats ( #18096 )
2021-06-22 15:23:26 +02:00
Michael Vines
84b9de8c18
Shredder no longer holds a keypair
2021-06-21 21:29:52 -07:00
dependabot[bot]
d458fac2ff
chore: bump bincode from 1.3.1 to 1.3.3 ( #18087 )
...
* chore: bump bincode from 1.3.1 to 1.3.3
Bumps [bincode](https://github.com/servo/bincode ) from 1.3.1 to 1.3.3.
- [Release notes](https://github.com/servo/bincode/releases )
- [Commits](https://github.com/servo/bincode/compare/v1.3.1...v1.3.3 )
---
updated-dependencies:
- dependency-name: bincode
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-06-21 16:06:31 +00:00
Jeff Washington (jwash)
ec2f930475
user process.accounts_db_test_hash_calculation for debug_verify hash ( #18053 )
2021-06-21 10:20:27 -05:00
dependabot[bot]
491e4450cc
chore: bump libc from 0.2.96 to 0.2.97 ( #18060 )
...
* chore: bump libc from 0.2.96 to 0.2.97
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.96 to 0.2.97.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.96...0.2.97 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-06-18 11:51:06 -06:00
Alexander Meißner
6514096a67
chore: cargo +nightly clippy --fix -Z unstable-options
2021-06-18 10:42:46 -07:00
Tyera Eulberg
d0511de9a6
chore: bump trees from 0.2.1 to 0.4.2 ( #18052 )
...
* chore: bump trees from 0.2.1 to 0.4.2 (#18041 )
Bumps [trees](https://github.com/oooutlk/trees ) from 0.2.1 to 0.4.2.
- [Release notes](https://github.com/oooutlk/trees/releases )
- [Commits](https://github.com/oooutlk/trees/commits )
---
updated-dependencies:
- dependency-name: trees
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Accommodate field & type changes
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-17 22:45:09 +00:00
Jeff Washington (jwash)
bab96e878e
Revert "chore: bump trees from 0.2.1 to 0.4.2 ( #18041 )" ( #18045 )
...
This reverts commit f2a7152c0c
.
2021-06-17 14:04:59 -05:00
dependabot[bot]
f2a7152c0c
chore: bump trees from 0.2.1 to 0.4.2 ( #18041 )
...
Bumps [trees](https://github.com/oooutlk/trees ) from 0.2.1 to 0.4.2.
- [Release notes](https://github.com/oooutlk/trees/releases )
- [Commits](https://github.com/oooutlk/trees/commits )
---
updated-dependencies:
- dependency-name: trees
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-17 08:07:16 -06:00
Jeff Washington (jwash)
dbd4dc04b0
ledger tool limit_load_slot_count_from_snapshot avoids assert failures ( #17974 )
2021-06-15 15:39:22 -05:00
Jeff Washington (jwash)
f558b9b6bf
verify bank hash on startup with ledger tool option ( #17939 )
2021-06-15 11:52:12 -05:00
Jon Cinque
1b1d34da59
Refactor stake program into solana_program ( #17906 )
...
* Move stake state / instructions into solana_program
* Update account-decoder
* Update cli and runtime
* Update all other parts
* Commit Cargo.lock changes in programs/bpf
* Update cli stake instruction import
* Allow integer arithmetic
* Update ABI digest
* Bump rust mem instruction count
* Remove useless structs
* Move stake::id() -> stake::program::id()
* Re-export from solana_sdk and mark deprecated
* Address feedback
* Run cargo fmt
2021-06-15 18:04:00 +02:00
Jeff Washington (jwash)
471b34132e
add metrics for startup ( #17913 )
...
* add metrics for startup
* roll timings up higher
* fix test
* fix duplicate
2021-06-14 17:46:49 -05:00
dependabot[bot]
a0872232d3
chore: bump itertools from 0.9.0 to 0.10.1 ( #17929 )
...
* chore: bump itertools from 0.9.0 to 0.10.1
Bumps [itertools](https://github.com/rust-itertools/itertools ) from 0.9.0 to 0.10.1.
- [Release notes](https://github.com/rust-itertools/itertools/releases )
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.9.0...v0.10.1 )
---
updated-dependencies:
- dependency-name: itertools
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Fix versions
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-06-14 18:32:20 +00:00
steviez
d52632f895
Add extra checks to verify_index_integrity ( #17851 )
2021-06-14 13:23:31 -05:00
Jeff Washington (jwash)
d4cc975fe9
calculate_capitalization uses hash calculation ( #17443 )
...
* calculate_capitalization uses hash calculation
* feedback
* remove debugging code, clean up slot math
2021-06-14 08:53:07 -05:00
dependabot[bot]
f08ed1eb2d
chore: bump rayon from 1.5.0 to 1.5.1 ( #17869 )
...
* chore: bump rayon from 1.5.0 to 1.5.1
Bumps [rayon](https://github.com/rayon-rs/rayon ) from 1.5.0 to 1.5.1.
- [Release notes](https://github.com/rayon-rs/rayon/releases )
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md )
- [Commits](https://github.com/rayon-rs/rayon/compare/rayon-core-v1.5.0...v1.5.1 )
---
updated-dependencies:
- dependency-name: rayon
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-06-10 13:18:47 -06:00
Lijun Wang
269d995832
Make account shrink configurable #17544 ( #17778 )
...
1. Added both options for measuring space usage using total accounts usage and for individual store shrink ratio using an enum. Validator CLI options: --accounts-shrink-optimize-total-space and --accounts-shrink-ratio
2. Added code for selecting candidates based on total usage in a separate function select_candidates_by_total_usage
3. Added unit tests for the new functions added
4. The default implementations is kept at 0.8 shrink ratio with --accounts-shrink-optimize-total-space set to true
Fixes #17544
2021-06-09 21:21:32 -07:00
dependabot[bot]
9a2ca8dd2f
chore: bump rustc_version from 0.2.3 to 0.4.0 ( #17854 )
...
* chore: bump rustc_version from 0.2.3 to 0.4.0
Bumps [rustc_version](https://github.com/Kimundi/rustc-version-rs ) from 0.2.3 to 0.4.0.
- [Release notes](https://github.com/Kimundi/rustc-version-rs/releases )
- [Commits](https://github.com/Kimundi/rustc-version-rs/compare/v0.2.3...v0.4.0 )
---
updated-dependencies:
- dependency-name: rustc_version
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Make versions consistent
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-06-09 16:57:39 +00:00
Alfonso Subiotto Marques
8a9b7f5ef2
Disable reed-solomon-erasure/simd-accel feature on aarch64 (Apple M1)
...
This feature requires clang to support -march=native on M1. Before this change,
cargo build would output a build error when building for aarch64.
2021-06-08 12:09:34 -07:00
Tyera Eulberg
a66566e75b
Remove budget program ( #17816 )
2021-06-08 01:20:17 +00:00
steviez
4677e6c132
Combine repeated logic for checking if slot has been cleaned up ( #17713 )
2021-06-04 17:40:27 -05:00
Tyera Eulberg
544b3c0d17
Create solana-poh and move remaining rpc modules to solana-rpc ( #17698 )
...
* Create solana-poh crate
* Move BigTableUploadService to solana-ledger
* Add solana-rpc to workspace
* Move dependencies to solana-rpc
* Move remaining rpc modules to solana-rpc
* Single use statement solana-poh
* Single use statement solana-rpc
2021-06-04 09:23:06 -06:00
carllin
96ba2edfeb
Switch EpochSlots to be frozen slots, not completed slots ( #17168 )
2021-06-03 00:20:00 +00:00
dependabot[bot]
19c49e2e25
chore: bump crossbeam-channel from 0.4.4 to 0.5.0 ( #17665 )
...
* chore: bump crossbeam-channel from 0.4.4 to 0.5.0
Bumps [crossbeam-channel](https://github.com/crossbeam-rs/crossbeam ) from 0.4.4 to 0.5.0.
- [Release notes](https://github.com/crossbeam-rs/crossbeam/releases )
- [Changelog](https://github.com/crossbeam-rs/crossbeam/blob/master/CHANGELOG.md )
- [Commits](https://github.com/crossbeam-rs/crossbeam/compare/crossbeam-channel-0.4.4...crossbeam-channel-0.5.0 )
---
updated-dependencies:
- dependency-name: crossbeam-channel
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-06-02 19:36:28 +00:00
Ryo Onodera
96cdbfdcc0
Purge expired BlockHeight data from blockstore ( #17634 )
...
* Purge expired BlockHeight data from blockstore
* Also call compact_storage and add comment....
2021-06-01 20:21:14 +00:00
dependabot[bot]
a0d721c6bd
Bump sha2 from 0.9.2 to 0.9.5 ( #17627 )
...
* Bump sha2 from 0.9.2 to 0.9.5
Bumps [sha2](https://github.com/RustCrypto/hashes ) from 0.9.2 to 0.9.5.
- [Release notes](https://github.com/RustCrypto/hashes/releases )
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.9.2...sha2-v0.9.5 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-06-01 08:58:38 -06:00
dependabot[bot]
ab0f4ff835
Bump serde from 1.0.122 to 1.0.126 ( #17618 )
...
* Bump serde from 1.0.122 to 1.0.126
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.122 to 1.0.126.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.122...v1.0.126 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
2021-05-31 22:41:25 +00:00
dependabot[bot]
ac452f9bb8
Bump tempfile from 3.1.0 to 3.2.0 ( #17587 )
...
* Bump tempfile from 3.1.0 to 3.2.0
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.1.0 to 3.2.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases )
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS )
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.1.0...v3.2.0 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-05-31 18:05:07 +00:00
dependabot[bot]
cf5ad7cb7d
Bump futures from 0.3.8 to 0.3.15 ( #17574 )
...
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.8 to 0.3.15.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.8...0.3.15 )
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-05-28 20:04:07 +00:00
Ryo Onodera
1f97b2365f
Avoid full-range compactions with periodic filtered b.g. ones ( #16697 )
...
* Update rocksdb to v0.16.0
* Promote the infrequent and important log to info!
* Force background compaction by ttl without manual compaction
* Fix test
* Support no compaction mode in test_ledger_cleanup_compaction
* Fix comment
* Make compaction_interval customizable
* Avoid major compaction with periodic filtering...
* Adress lazy_static, special cfs and range check
* Clean up a bit and add comment
* Add comment
* More comments...
* Config code cleanup
* Add comment
* Use .conflicts_with()
* Nullify unneeded delete_range ops for special CFs
* Some clean ups
* Clarify the locking intention
* Ensure special CFs' consistency with PurgeType::CompactionFilter
* Fix comment
* Fix bad copy paste
* Fix various types...
* Don't use tuples
* Add a unit test for compaction_filter
* Fix typo...
* Remove flag and just use new behavior always
* Fix wrong condition negation...
* Doc. about no set_last_purged_slot in purge_slots
* Write a test and fix off-by-one bug....
* Apply suggestions from code review
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Follow up to github review suggestions
* Fix line-wrapping
* Fix conflict
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2021-05-28 16:42:56 +09:00
dependabot[bot]
4f74c77146
chore: bump chrono-humanize from 0.1.1 to 0.2.1 ( #16895 )
...
* chore: bump chrono-humanize from 0.1.1 to 0.2.1
Bumps [chrono-humanize](https://gitlab.com/imp/chrono-humanize-rs ) from 0.1.1 to 0.2.1.
- [Release notes](https://gitlab.com/imp/chrono-humanize-rs/tags )
- [Commits](https://gitlab.com/imp/chrono-humanize-rs/compare/0.1.1...0.2.1 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
2021-05-27 18:03:15 +00:00
dependabot[bot]
6e2ae68643
Bump serde_bytes from 0.11.4 to 0.11.5 ( #16299 )
...
* Bump serde_bytes from 0.11.4 to 0.11.5
Bumps [serde_bytes](https://github.com/serde-rs/bytes ) from 0.11.4 to 0.11.5.
- [Release notes](https://github.com/serde-rs/bytes/releases )
- [Commits](https://github.com/serde-rs/bytes/compare/0.11.4...0.11.5 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-05-27 11:48:44 -06:00
Brooks Prumo
1953543274
Refactor bank_forks_utils::load() and some snapshot logic ( #17492 )
...
Refactor a few functions that are on the load-from-snapshot path, to facilitate
adding in incremental snapshots more easily.
Additionally, add some tests and doc comments.
2021-05-27 12:00:27 -05:00
steviez
983828a2a9
Remove redundant copy from RocksDB get_cf() wrapper ( #17529 )
2021-05-27 10:53:51 -05:00
Michael Vines
8eab0e8602
Bump version to v1.8.0 ( #17541 )
2021-05-27 08:51:53 -07:00
Tyera Eulberg
ab581dafc2
Add block height to ConfirmedBlock structs ( #17523 )
...
* Add BlockHeight CF to blockstore
* Rename CacheBlockTimeService to be more general
* Cache block-height using service
* Fixup previous proto mishandling
* Add block_height to block structs
* Add block-height to solana block
* Fallback to BankForks if block time or block height are not yet written to Blockstore
* Add docs
* Review comments
2021-05-26 22:16:16 -06:00
Michael Vines
9541411c15
Plumb transaction-level rewards (aka "rent debits") into the `getTransaction` RPC method
2021-05-27 03:05:05 +00:00
carllin
52dccc656a
Purge slots greater than new last index ( #16071 )
2021-05-26 16:12:57 -07:00
Jeff Washington (jwash)
6b9d8d41a3
add --limit_load_slot_count_from_snapshot to ledger-tool ( #17417 )
2021-05-26 10:36:12 -05:00
carllin
3dfe87973b
Propagate dead slots up to replay ( #17227 )
2021-05-25 13:43:47 -07:00
Jeff Washington (jwash)
99313a4ba2
add ledger-tool verify option to verify hash calculation ( #17412 )
2021-05-24 18:15:57 -05:00
Tyera Eulberg
41ec1c8d50
Add blockstore-root-scan for api nodes on boot ( #17402 )
...
* Add blockstore-root-scan for api nodes on boot
* Ensure cluster-confirmed root and parents are set as root in blockstore in load_frozen_forks()
* Plumb rpc-scan-and-fix-roots validator flag
2021-05-24 13:24:47 -06:00
sakridge
a8dca3976b
Refactor genesis download/load/check functions ( #17276 )
...
* Refactor genesis ingest functions
* Consolidate genesis.bin/genesis.tar.bz2 references
2021-05-24 16:45:36 +02:00
Tao Zhu
0781fe1b4f
Upgrade Rust to 1.52.0 ( #17096 )
...
* Upgrade Rust to 1.52.0
update nightly_version to newly pushed docker image
fix clippy lint errors
1.52 comes with grcov 0.8.0, include this version to script
* upgrade to Rust 1.52.1
* disabling Serum from downstream projects until it is upgraded to Rust 1.52.1
2021-05-19 09:31:47 -05:00
Jeff Washington (jwash)
0c07f1ffe4
add data point for ledger processing ( #17210 )
2021-05-14 15:58:31 -05:00
Brooks Prumo
b074e86ad2
Fix clippy ( #17214 )
...
Newer clippy from +nightly have new errors about where annotations can and
cannot go. This commit fixes two of them.
2021-05-13 17:51:36 -05:00
Tyera Eulberg
6e9deaf1bd
Move block-time caching earlier ( #17109 )
...
* Require that blockstore block-time only be recognized slot, instead of root
* Move cache_block_time to after Bank freeze
* Single use statement
* Pass transaction_status_sender by reference
* Remove unnecessary slot-existence check before caching block time altogether
* Move block-time existence check into Blockstore::cache_block_time, Blockstore no longer needed in blockstore_processor helper
2021-05-10 13:14:56 -06:00
Jeff Washington (jwash)
f39dda00e0
type AccountSecondaryIndexes = HashSet ( #17108 )
2021-05-10 14:22:48 +00:00
behzad nouri
81ad795d46
removes position field in coding-shred-header
...
CodingShredHeader.position is equal to
ShredCommonHeader.index - ShredCommonHeader.fec_set_index
and is so redundant. The extra position field can add bugs if not
consistent with index and fec_set_index.
2021-05-10 13:20:56 +00:00
carllin
bc7e741514
Integrate gossip votes into switching threshold ( #16973 )
2021-05-04 00:51:42 -07:00
steviez
475b00c42f
Add test to ensure data shreds with empty data would be inserted ( #16955 )
...
* Add test to ensure data shreds with empty data would be inserted
* Add error log statements for malformed shreds
2021-04-30 10:38:15 -05:00
steviez
bc31378797
Trim extra shred bytes in blockstore ( #16602 )
...
Strip the zero-padding off of data shreds before insertion into blockstore
Co-authored-by: Stephen Akridge <sakridge@gmail.com>
Co-authored-by: Nathan Hawkins <utsl@utsl.org>
2021-04-27 17:40:41 -05:00
behzad nouri
0f3ac51cf1
limits to data_header.size when combining shreds' payloads ( #16708 )
...
Shredder::deshred is ignoring data_header.size when combining shreds' payloads:
https://github.com/solana-labs/solana/blob/37b8587d4/ledger/src/shred.rs#L940-L961
Also adding more sanity checks on the alignment of data shreds indices.
2021-04-27 12:04:44 +00:00
behzad nouri
03194145c0
removes first_coding_index from erasure recovery code ( #16646 )
...
first_coding_index is the same as the set_index and is so redundant:
https://github.com/solana-labs/solana/blob/37b8587d4/ledger/src/blockstore_meta.rs#L49-L60
2021-04-23 12:00:37 +00:00
behzad nouri
37b8587d4e
expands number of erasure coding shreds in the last batch in slots ( #16484 )
...
Number of parity coding shreds is always less than the number of data
shreds in FEC blocks:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L719
Data shreds are batched in chunks of 32 shreds each:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L714
However the very last batch of data shreds in a slot can be small, in
which case the loss rate can be exacerbated.
This commit expands the number of coding shreds in the last FEC block in
slots to: 64 - number of data shreds; so that FEC blocks are always 64
data and parity coding shreds each.
As a consequence of this, the last FEC block has more parity coding
shreds than data shreds. So for some shred indices we will have a coding
shred but no data shreds. This should not cause any kind of overlapping
FEC blocks as in:
https://github.com/solana-labs/solana/pull/10095
since this is done only for the very last batch in a slot, and the next
slot will reset the shred index.
2021-04-21 12:47:50 +00:00
Michael Vines
a911ae00ba
clippy
2021-04-18 20:55:02 -07:00
behzad nouri
e405747409
Revert "Add limit and shrink policy for recycler ( #15320 )"
...
This reverts commit c2e8814dce
.
2021-04-18 19:29:24 +00:00
steviez
bb24318ef0
Document shreds ( #16514 )
...
No functionality changes from this commit
2021-04-16 14:04:46 -05:00
Justin Starry
85eb37fab0
Merge pull request from GHSA-8v47-8c53-wwrc
...
* Track transaction check time separately from account loads
* banking packet process metrics
* Remove signature clone in status cache lookup
* Reduce allocations when converting packets to transactions
* Add blake3 hash of transaction messages in status cache
* Bug fixes
* fix tests and run fmt
* Address feedback
* fix simd tx entry verification
* Fix rebase
* Feedback
* clean up
* Add tests
* Remove feature switch and fall back to signature check
* Bump programs/bpf Cargo.lock
* clippy
* nudge benches
* Bump `BankSlotDelta` frozen ABI hash`
* Add blake3 to sdk/programs/Cargo.lock
* nudge bpf tests
* short circuit status cache checks
Co-authored-by: Trent Nelson <trent@solana.com>
2021-04-13 00:28:08 -06:00
Christian Drappi
54a04bac3d
Apple M1 compatibility ( #16346 )
...
Co-authored-by: Christian Drappi <christiandrappi@Christians-MacBook-Pro.local>
2021-04-09 17:21:01 -07:00
behzad nouri
22a18a68e3
stops consuming pinned vectors with a recycler ( #16441 )
...
If the vector is pinned and has a recycler, From<PinnedVec>
implementation of Vec should clone (instead of consuming) the underlying
vector so that the next allocation of a PinnedVec will recycle an
already pinned one.
2021-04-09 16:55:24 +00:00
François Garillot
b08cff9e77
Simplify some pattern-matches ( #16402 )
...
When those match an exact combinator on Option / Result.
Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust ).
2021-04-08 12:40:37 -06:00
Michael Vines
43feef7362
Adjust tokio version to just "1"
2021-04-05 17:13:20 +00:00
Tyera Eulberg
b8b6777262
Only get Blockstore::last_root once ( #16362 )
2021-04-05 04:14:02 +00:00
Tyera Eulberg
1a13d22984
Fixup iterator method ( #16357 )
2021-04-04 23:32:51 +00:00
carllin
4e5ef6bce2
Add cluster state verifier logging ( #16330 )
...
* Add cluster state verifier logging
* Add duplicate-slots iterator to ledger tool
2021-04-02 21:48:44 -07:00
Tyera Eulberg
da27acabcc
Rpc: enable getConfirmedSignaturesForAddress2 to return confirmed (not yet finalized) data ( #16281 )
...
* Update blockstore method to allow return of unfinalized signature
* Support confirmed sigs in getConfirmedSignaturesForAddress2
* Add deprecated comments
* Update docs
* Enable confirmed transaction-history in cli
* Return real confirmation_status; fill in not-yet-finalized block time if possible
2021-04-01 04:35:57 +00:00
Tyera Eulberg
18bd47dbe1
Rpc: fix getConfirmedTransaction slot ( #16288 )
...
* Fix transaction blockstore apis
* Update blockstore apis in rpc
2021-03-31 21:04:00 -06:00
behzad nouri
3f63ed9a72
removes OrderedIterator and transaction batch iteration order ( #16153 )
...
In TransactionBatch,
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/transaction_batch.rs#L4-L11
lock_results[i] is aligned with transactions[iteration_order[i]]:
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2414-L2424
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/accounts.rs#L788-L817
However load_and_execute_transactions is iterating over
lock_results[iteration_order[i]]
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2878-L2889
and then returning i as for the index of the retryable transaction.
If iteratorion_order is [1, 2, 0], and i is 0, then:
lock_results[iteration_order[i]] = lock_results[1]
which corresponds to
transactions[iteration_order[1]] = transactions[2]
so neither i = 0, nor iteration_order[i] = 1 gives the correct index for the
corresponding transaction (which is 2).
This commit removes OrderedIterator and transaction batch iteration order
entirely. There is only one place in blockstore processor which the
iteration order is not ordinal:
https://github.com/solana-labs/solana/blob/e50f59844/ledger/src/blockstore_processor.rs#L269-L271
It seems like, instead of using an iteration order, that can shuffle entry
transactions in-place.
2021-03-31 23:59:19 +00:00
Tyera Eulberg
433f1ead1c
Rpc: enable getConfirmedBlock and getConfirmedTransaction to return confirmed (not yet finalized) data ( #16142 )
...
* Add Blockstore block and tx apis that allow unrooted responses
* Add TransactionStatusMessage, and send on bank freeze; also refactor TransactionStatusSender
* Track highest slot with tx-status writes complete
* Rename and unpub fn
* Add commitment to GetConfirmed input configs
* Support confirmed blocks in getConfirmedBlock
* Support confirmed txs in getConfirmedTransaction
* Update sigs-for-addr2 comment
* Enable confirmed block in cli
* Enable confirmed transaction in cli
* Review comments
* Rename blockstore method
2021-03-26 16:47:35 -06:00
Jeff Washington (jwash)
4f4cffbd03
Throttle PoH ticks by cumulative slot time ( #16139 )
...
* Throttle PoH ticks by cumulative slot time
* respond to pr feedback
* saturating sub
* updated comment
2021-03-26 18:54:16 +00:00
Justin Starry
e7fd7d46cf
rpc: add getSlotLeaders method ( #16057 )
2021-03-23 17:48:54 +00:00
behzad nouri
4f82b897bc
buffers data shreds to make larger erasure coded sets ( #15849 )
...
Broadcast stage batches up to 8 entries:
https://github.com/solana-labs/solana/blob/79280b304/core/src/broadcast_stage/broadcast_utils.rs#L26-L29
which will be serialized into some number of shreds and chunked into FEC
sets of at most 32 shreds each:
https://github.com/solana-labs/solana/blob/79280b304/ledger/src/shred.rs#L576-L597
So depending on the size of entries, FEC sets can be small, which may
aggravate loss rate.
For example 16 FEC sets of 2:2 data/code shreds each have higher loss
rate than one 32:32 set.
This commit broadcasts data shreds immediately, but also buffers them
until it has a batch of 32 data shreds, at which point 32 coding shreds
are generated and broadcasted.
2021-03-23 14:52:38 +00:00
carllin
d76ad33597
Handle blockstore insert dup checks ( #16051 )
2021-03-22 16:18:22 -07:00
Tyera Eulberg
654449ce91
Bump tokio to 1.1 ( #15926 )
2021-03-16 21:32:50 +00:00
behzad nouri
3b85cbc504
renames is_last_in_fec_set back to is_last_data ( #15848 )
...
https://github.com/solana-labs/solana/pull/10095
renamed is_last_data to is_last_in_fec_set. However, the code shows that
this is actually meant to indicate where the serialized data is
complete:
https://github.com/solana-labs/solana/blob/420174d3d/ledger/src/shred.rs#L599-L600
https://github.com/solana-labs/solana/blob/420174d3d/ledger/src/shred.rs#L229-L231
There are multiple FEC sets for each `&[Entry]` serialized and this flag
does not represent shreds last in FEC sets (only the very last one by
overlap). So the name is wrong and confusing
2021-03-16 10:09:16 +00:00
Michael Vines
a2eb655322
=1.7.0
2021-03-16 07:51:07 +00:00
Michael Vines
0c9ca5522c
Bump version to v1.7.0
2021-03-13 09:01:21 +00:00
Justin Starry
918d04e3f0
Add more slot update notifications ( #15734 )
...
* Add more slot update notifications
* fix merge
* Address feedback and add integration test
* switch to datapoint
* remove unused shred method
* fix clippy
* new thread for rpc completed slots
* remove extra constant
* fixes
* rely on channel closing
* fix check
2021-03-12 21:44:06 +08:00
Trent Nelson
24d18b3cf2
docs: add docs links for crates published to crates.io
2021-03-11 08:38:18 +00:00
Jeff Washington (jwash)
1135ffd595
mut data refs as slice ( #15782 )
2021-03-10 15:28:03 -06:00
Jeff Washington (jwash)
8a3135d17b
Account->AccountSharedData ( #15691 )
2021-03-09 15:06:07 -06:00
Jeff Washington (jwash)
9448540ea6
log timing after ledger process ( #15716 )
2021-03-05 11:01:52 -06:00
Tyera Eulberg
7e65289729
Convert blockstore TransactionStatus column family to protobufs ( #15733 )
...
* Prevent panic if TransactionStatus can't be deserialized
* Convert Blockstore TransactionStatus column to protobuf
* Add compatability test
2021-03-05 09:05:35 -07:00
Jeff Washington (jwash)
be35c1c1b7
add execute detail timings ( #15638 )
2021-03-03 17:07:45 -06:00
sakridge
f1223fb783
Lower blockstore processor error severity ( #15578 )
2021-03-01 14:57:37 -08:00
Trent Nelson
ef63ffce7d
Bump ed25519-dalek to 1.0.1
2021-03-01 21:10:46 +00:00
Michael Vines
33eaa2b238
Sort forks in "ledger processed..." log message
2021-02-28 17:32:50 -08:00
Michael Vines
e92cff3efa
create-snapshot subcommad now accepts the ROOT keyword
2021-02-26 16:40:10 -08:00
carllin
c2e8814dce
Add limit and shrink policy for recycler ( #15320 )
2021-02-24 00:15:58 -08:00
Michael Vines
5df36aec7d
Pacify clippy
2021-02-19 20:08:41 -08:00
Tyera Eulberg
170cb792eb
Return blockstore error if previous_blockhash cannot be determined ( #15382 )
...
* Return blockstore error if previous_blockhash cannot be determined
* Add require_previous_blockshash flag
2021-02-18 01:04:52 +00:00
Trent Nelson
7f7370c306
Re-allow clippy::integer_arithmetic at crate-level
2021-02-17 13:55:08 -07:00
Michael Vines
9ba69a7381
Adapt to fs_extra 1.2.0
2021-02-17 12:44:58 -08:00
Tyera Eulberg
98e3e570d2
Add --force arg for bigtable upload
2021-02-16 17:24:07 -08:00
behzad nouri
e403aeaf05
adds an inverted index to leader schedule ( #15249 )
...
next_leader_slot is doing a linear search for slots in which a pubkey is
the leader:
https://github.com/solana-labs/solana/blob/e59a24d9f/ledger/src/leader_schedule_cache.rs#L123-L157
This can be done more efficiently by adding an inverted index to leader
schedule.
2021-02-15 00:52:52 +00:00
Tyera Eulberg
0812931c38
Log if unsanitary transactions are read from blockstore ( #15319 )
2021-02-14 06:32:43 +00:00
Tyera Eulberg
da6753b8c0
Warp timestamp and extend max-allowable-drift for accommodate slow blocks ( #15204 )
...
* Remove timestamp_correction feature gating
* Remove timestamp_bounding feature gating
* Remove unused deprecated ledger code
* Remove unused deprecated unbounded-timestamp code
* Enable independent adjustment of fast/slow timestamp bounding
* Update timestamp bounds to 25% fast, 80% slow; warp timestamp
* Update bank hash test
* Add PR number to feature
Co-authored-by: Michael Vines <mvines@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-02-09 15:49:00 -07:00
sakridge
bbae23358c
ledger-tool cleanup and additions ( #15179 )
...
* Plumb allow-dead-slots to ledger-tool verify
* ledger-tool cleanup and add some useful missing args
Print root slots and how many unrooted past last root.
2021-02-06 17:26:42 -08:00
sakridge
774416a546
Ledger purge printing ( #15176 )
2021-02-06 15:45:08 -08:00
Ryo Onodera
c69027e5aa
chore: bump serde from 1.0.118 to 1.0.122 ( #15126 )
...
* chore: bump serde from 1.0.118 to 1.0.122
* Update programs/bpf/Cargo.lock
2021-02-05 19:23:24 +09:00
Tyera Eulberg
d1563f0ccd
Bump tonic, prost, tarpc, tokio ( #15013 )
...
* Update tonic & prost, and regenerate proto
* Reignore doc code
* Revert pull #14367 , but pin tokio to v0.2 for jsonrpc
* Bump backoff and goauth -> and therefore tokio
* Bump tokio in faucet, net-utils
* Bump remaining tokio, plus tarpc
2021-02-05 00:21:53 -07:00
Michael Vines
42bf6dc2ab
Revert hard nofile limit back to 500000
2021-02-03 19:43:55 +00:00
Ryo Onodera
8376781ec8
Don't squash caught errors, please ( #15046 )
...
* Don't squash caught errors, please
* Update blockstore.rs
* Update blockstore.rs
2021-02-03 14:42:34 +00:00
Trent Nelson
894b412aef
docs: bump nofiles recommendations to match maps
2021-02-02 21:59:21 -07:00
dependabot[bot]
1df93fa2be
chore: bump serde from 1.0.112 to 1.0.118 ( #14828 )
...
* chore: bump serde from 1.0.112 to 1.0.122
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.112 to 1.0.122.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.122 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* Update frozen_abi digest following serde update
* Revert "chore: bump serde from 1.0.112 to 1.0.122"
This reverts commit a3ef4442a4c985144ae2bd7ceaf8899a7ab8d7c0.
* Revert "[auto-commit] Update all Cargo lock files"
This reverts commit c41c3b005fb1ccade55155302c52cd5736c4b55f.
* chore: bump serde from 1.0.112 to 1.0.118
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.112 to 1.0.118.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.118 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* Remove serum-dex pinning
* blind commit!
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2021-02-02 23:28:16 +09:00
Tyera Eulberg
cbb8b79a60
Add validator flag to opt in to cpi and logs storage ( #14922 )
...
* Add validator flag to opt in to cpi and logs storage
* Default TestValidator to opt-in; allow using in multinode-demo
* No clone
Co-authored-by: Carl Lin <carl@solana.com>
2021-02-01 14:00:51 -07:00
Michael Vines
cbffab7850
Upgrade to Rust v1.49.0
2021-01-23 19:16:36 -08:00
sakridge
191193289f
Upgrade sha2 to 0.9.3 ( #14746 )
2021-01-22 22:25:22 -08:00
Josh
1de6d28eaf
Add block_time to getConfirmedSignaturesForAddress2 and getConfirmedTransaction ( #14572 )
...
* add block_time to get_confirmed_signatures_for_address2 and protobuf implementation for tx_by_addr
* add tests for convert
* update cargo lock
* run cargo format after rebase
* introduce legacy TransactionByAddrInfo
* move LegacyTransactionByAddrInfo back to storage-bigtable
2021-01-20 22:10:35 -08:00
dependabot[bot]
4ecc0f4fa6
chore: bump rayon from 1.4.1 to 1.5.0 ( #14177 )
...
* chore: bump rayon from 1.4.1 to 1.5.0
Bumps [rayon](https://github.com/rayon-rs/rayon ) from 1.4.1 to 1.5.0.
- [Release notes](https://github.com/rayon-rs/rayon/releases )
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md )
- [Commits](https://github.com/rayon-rs/rayon/compare/rayon-core-v1.4.1...rayon-core-v1.5.0 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-01-15 18:07:35 +09:00
sakridge
907f518f6d
Add load/execute/store timings ( #14561 )
2021-01-14 14:14:16 -08:00
Ryo Onodera
e95ebcf864
Don't stop to find newer cluster-confirmed roots ( #14557 )
...
* Don't stop to find newer cluster-confirmed roots
* Fix and add new tests
* nits
2021-01-13 15:47:06 +00:00
carllin
6dfad0652f
Cache account stores, flush from AccountsBackgroundService ( #13140 )
2021-01-11 17:00:23 -08:00
sakridge
d8105bb7d7
Add rocskdb high priority threads ( #14515 )
...
Without them, memtable writes can stall on compactions.
2021-01-11 13:32:02 -08:00
Michael Vines
86c81a0ba2
Humanize the 'ledger processed...' time
2021-01-08 22:56:21 -08:00
carllin
5affd8aa72
Add secondary indexes ( #14212 )
2020-12-31 18:06:03 -08:00