Commit Graph

1127 Commits

Author SHA1 Message Date
behzad nouri db3d926633
implements shred::layout::get_merkle_root (#29437)
In preparation of removing merkle roots form shreds binary, the commit
adds api to recover the root from the merkle proof embedded in shreds
payload.
2022-12-29 23:32:58 +00:00
behzad nouri 1de161c13a
simplifies shreds sigverify (#29436)
Simplifying this code in preparation of removing merkle roots from
shreds binary.
2022-12-29 20:51:57 +00:00
behzad nouri fba4a57060
expands test coverage for sign_shreds_gpu (#29429) 2022-12-29 00:33:51 +00:00
behzad nouri d172ab342f
expands test coverage for merkle/legacy shreds sigverify (#29424) 2022-12-28 17:42:24 +00:00
steviez 58d66e1113
Minor cleanup on bigtable_upload (#29379)
Adjust some logs, and remove an unnecessary cloned().
2022-12-23 20:19:15 +00:00
Tyera 81394cf92c
Fix BigTable upload early return (#29378)
* Change Err when slot range is empty to Ok and log; add method docs

* Update var and log to be more correct

* Promote log level to warn
2022-12-23 18:11:27 +00:00
behzad nouri 92ca725c6b
patches shred::merkle::make_shreds_from_data when data is empty (#29295)
If data is empty, make_shreds_from_data will now return one data shred
with empty data. This preserves invariants verified in tests regardless
of data size.
2022-12-16 18:56:21 +00:00
steviez ff8bb5362c
Remove repetitive logic in SlotMeta first insert detection logic (#29153) 2022-12-15 17:38:27 -06:00
behzad nouri 73c42dde6e
sanitizes shreds recovered from erasure shards (#29286)
Instead of sanitizing shreds late in recover code:
https://github.com/solana-labs/solana/blob/50ad0390f/ledger/src/shred/merkle.rs#L786

Shred{Code,Data}::from_recovered_shard should sanitize shreds before
returning:
https://github.com/solana-labs/solana/blob/50ad0390f/ledger/src/shred/merkle.rs#L192-L216
https://github.com/solana-labs/solana/blob/50ad0390f/ledger/src/shred/merkle.rs#L324-L350
2022-12-15 20:09:15 +00:00
Ikko Ashimine 50c1de5597
fix typo in shred.rs (#29149)
calcuation -> calculation
2022-12-15 01:14:39 -06:00
dependabot[bot] dca5d7f9b4
chore: bump test-case from 2.1.0 to 2.2.2 (#28184)
Bumps [test-case](https://github.com/frondeus/test-case) from 2.1.0 to 2.2.2.
- [Release notes](https://github.com/frondeus/test-case/releases)
- [Changelog](https://github.com/frondeus/test-case/blob/master/CHANGELOG.md)
- [Commits](https://github.com/frondeus/test-case/compare/v2.1.0...v2.2.2)

---
updated-dependencies:
- dependency-name: test-case
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-13 16:07:50 +00:00
Jeff Washington (jwash) bb0bfc4214
remove bank_from_latest_snapshot_archives(caching_enabled) (#29238) 2022-12-13 07:16:24 -08:00
Jeff Washington (jwash) fec8f61566
remove ProcessOptions::accounts_db_caching_enabled (#29217) 2022-12-12 20:25:00 -08:00
Jeff Washington (jwash) 2c2324f4ea
remove caching_enabled from Bank::new_with_paths_for_tests (#29214) 2022-12-12 15:30:46 -08:00
Jeff Washington (jwash) dd4393e46d
remove accounts_caching from Bank::new_with_paths (#29213) 2022-12-12 15:18:12 -08:00
behzad nouri 9524c9dbff patches errors from clippy::uninlined_format_args
https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
2022-12-06 19:32:15 +00:00
Jon Cinque 504091f9be
perf: Change underlying Elems to *const u8 (#29089)
* perf: Change underlying Elems to *const u8

* Remove clippy allows
2022-12-06 18:06:27 +01:00
Jon Cinque b1340d77a2
sdk: Make Packet::meta private, use accessor functions (#29092)
sdk: Make packet meta private
2022-12-06 12:54:49 +01:00
steviez 34e5bf1b13
Add SlotMeta unit test for connected at slot zero (#29025) 2022-12-02 15:37:27 -06:00
steviez 01cd55a27a
Change SlotMeta is_connected bool to bitflags (#29001)
We currently use the is_connected field to be able to signal to
ReplayStage that a slot has replayable updates. It was discovered that
this functionality is effectively broken, and that is_connected is never
true. In order to convey this information to ReplayStage more
effectively, we need extra state information so this PR changes the
existing bool to bitflags with two bits.

From a compatibility standpoint, the is_connected bool was already
occupying one byte in the serialized SlotMeta in blockstore. Thus, the
change from a bool to bitflags still "fits" in that one byte allotment.

In consideration of a case where a client may wish to downgrade software
and use the same ledger, deserializing the bitflags into a bool could
fail if the new bit is set. As such, this PR introduces the second bit
field, but does not set it anywhere. Once clusters have mass adopted a
software version with this PR, a subsequent change to actually set and
use the new field can be introduced.
2022-12-01 14:42:35 -06:00
steviez 3c42c87098
Remove obsoleted return value from Blockstore insert shred method (#28992) 2022-12-01 11:17:46 -06:00
steviez b6dce6cf3b
Move BlockstoreInsertionMetrics field update to blockstore.rs (#28991)
The num_repair field is only blockstore insertion metric being updated
outside of Blockstore::insert() call chain; move the update to insert()
with the rest of the fields in BlockstoreInsertionMetrics struct.
2022-11-30 11:46:35 -06:00
Brennan Watt 9a6ab5e7fe
Distinguish turbine vs repair insertion metrics (#28980) 2022-11-30 09:03:53 -08:00
hanako mumei 015784ad07 update to spl-token-2022 0.5.0
* support CpiGuard and PermanentDelegate extensions in transaction-status and account-decoder
* update transaction-status and account-decoder to new ConfidentialTransfer interfaces
2022-11-28 11:30:41 -08:00
Maximilian Schneider c8b0c3ede9
Update cost model to use requested_cu instead of estimated cu #27608 (#28281)
* Update cost model to use requested_cu instead of estimated cu #27608

* remove CostUpdate and CostModel from replay/tvu

* revive cost update service to send cost tracker stats

* CostModel is now static

* remove unused package

Co-authored-by: Tao Zhu <tao@solana.com>
2022-11-22 11:55:56 -06:00
Brooks Prumo 2bafb0cb12
Requires EAH state cannot be Invalid (#28817) 2022-11-17 11:01:01 -05:00
Brooks Prumo d1ba42180d
clippy for rust 1.65.0 (#28765) 2022-11-09 19:39:38 +00:00
Brooks Prumo 59bf1809fe
Uses SnapshotHash type in snapshot archive fields (#28681) 2022-10-31 14:28:35 -04:00
Dmitri Makarov 34865d032c chore: update Solana docs and code comments that specify "BPF" to "SBF" 2022-10-31 14:14:25 -04:00
steviez 2272fd807e
Remove Blockstore manual compaction code (#28409)
The manual Blockstore compaction that was being initiated from
LedgerCleanupService has been disabled for quite some time in favor of
several optimizations.

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2022-10-28 10:39:00 +02:00
Justin Starry 2d8665d307
Record inner instruction stack height (#28430)
* Record inner instruction stack height

* fix sbf tests

* feedback
2022-10-26 10:37:44 +08:00
Yueh-Hsuan Chiang f26a457cf5
Add comment block for DeadSlots LedgerColumn (#28341)
Add comment block for DeadSlots LedgerColumn
2022-10-20 09:47:19 -07:00
Yueh-Hsuan Chiang bf37228768
Add comment block for Root ledger column (#28358)
Add comment block for Root ledger column.
2022-10-19 17:06:21 -07:00
Yueh-Hsuan Chiang 9be7ada68d
Improve SlotColumn's code comment. (#28447)
Improve SlotColumn's code comment to include how
LedgerCleanupService manages the clean-up of a SlotColumn.
2022-10-18 23:50:22 -07:00
Yueh-Hsuan Chiang f910f2d7eb
Add comment block for Orphans column family. (#28340)
Add comment block for Orphans column family.
2022-10-16 23:47:02 -07:00
Yueh-Hsuan Chiang 6a96a4c2ee
Add comment block for ErasureMeta ledger column (#28356)
Add comment block for ErasureMeta ledger column.
2022-10-14 12:59:34 -07:00
Yueh-Hsuan Chiang 4539cb75fb
Add comment block for SlotMeta column family (#28339)
Add comment block for SlotMeta column family.
2022-10-14 12:56:56 -07:00
Yueh-Hsuan Chiang 5df10173dd
Add comment block for BankHash ledger column (#28357)
Add comment block for BankHash ledger column.
2022-10-13 11:41:58 -07:00
steviez b8acb1b350
Fix blockstore_processor::load_frozen_forks() halt_at_slot behavior II (#28367)
PR #28317 previously attempted to fix a case where blockstore processing
would create children banks for slots past the halt_at_slot.

However, the previous fix didn't handle the case where a slot could be
strictly less than the halt_at_slot, but have children that were greater
than the halt_at_slot. For example, this could happen if a child of slot
S is S+n where n > 1.

Thus, this change covers our processing logic to cover this second case
as well.
2022-10-12 17:24:27 -05:00
steviez 2929c8f7a2
Fix blockstore_processor::load_frozen_forks() halt_at_slot behavior (#28317)
load_frozen_forks() finds new slots to process by creating new Banks for
the children of the current slot in process_next_slots(). Prior to this
change, we would then immediately check if we had reached the
halt_at_slot and correctly halt processing when appropriate. As such, it
would be possible for Banks to be created for slots beyond the
halt_at_slot.

While a potential child slot that is past halt_at_slot wouldn't be
replayed, the Bank being created still alters some universal state in
AccountsDb. So, this change moves the halt_at_slot check before we
create children Banks in process_next_slots().
2022-10-10 14:37:23 -05:00
steviez 60f6e24b76
Make Blockstore::get_entries_in_data_block() use multi_get() (#28245) 2022-10-09 15:34:03 -04:00
steviez 624f5cfcd5
Add rocksdb multi_get_bytes() method (#28244) 2022-10-07 18:05:13 -04:00
behzad nouri d9ef04772d
moves merkle proof size sanity check to Shred{Code,Data}::merkle_branch (#28266) 2022-10-06 18:54:24 +00:00
steviez 9e8f21ec26
Add metric to track number of slots processed in load_frozen_forks() (#28247) 2022-10-06 12:17:49 -05:00
behzad nouri a6512016a7
uses references for MerkleBranch root and proof fields (#28243) 2022-10-06 15:41:55 +00:00
Brooks Prumo a8c6a9e5fc
Bank::freeze() waits for EAH calculation to complete (#28170) 2022-10-05 17:44:35 -04:00
Jeff Washington (jwash) 7ae7a3ff0e
after replay at validator startup don't verify hash calc (#28222) 2022-10-04 14:04:04 -07:00
Jeff Washington (jwash) 990fb68c34
add sub timings in load_frozen_forks (#28223) 2022-10-04 13:46:02 -07:00
behzad nouri 9e7a0e7420
rolls out merkle shreds to ~5% of testnet (#28199) 2022-10-04 19:36:16 +00:00
steviez 7fef7d569a
Make RocksFifo storage size API expose an Option<u64> (#28192)
A fifo rocksdb instance must be opened with max size parameter on the
fifo columns. To support this, we previously plumbed a constant up to
callers that provided a default if unbounded growth desired.

This change attempts to be more rusty by exposing an option for this
value, and converting the option to a constant at the lowest level
possible.
2022-10-04 12:38:40 -05:00