Commit Graph

1450 Commits

Author SHA1 Message Date
mergify[bot] 2088c5870c
v2.0: replay: extend last fec set check for 32+ retransmitter signed shreds (backport of #2101) (#2192)
replay: extend last fec set check for 32+ retransmitter signed shreds (#2101)

* replay: extend last fec set check for 32+ retransmitter signed shreds

* pr feedback: use separate feature flag

* pr feedback: is_retransmitter_signed -> is_retransmitter_signed_variant, false for legacy

* pr feedback: update doc comment fail -> error

* pr feedback: hash -> bank_hash for report metrics

* refactor metrics inside blockstore fn, return block_id for future use

* pr feedback: gate metrics reporting

* pr feedback: do not distinguish impossible combos, simplify check code

* pr feedback: remove report_metrics helper fn

* pr feedback: remove metric

* pr feedback: block_id -> last_fec_set_merkle_root

Co-authored-by: Ashwin Sekar <ashwin@anza.xyz>
2024-08-28 00:24:08 -04:00
mergify[bot] 590a23c7d8
v2.0: gossip: ignore retransmitter signatures when comparing duplicate shreds (backport of #2673) (#2699)
* gossip: ignore retransmitter signatures when comparing duplicate shreds (#2673)

* gossip: ignore retransmitter signatures when comparing duplicate shreds

* pr feedback: compare rest of payload instead of setting sig

* pr feedback: remove dcou, pub(super)

(cherry picked from commit ff87ed9187)

# Conflicts:
#	ledger/src/shred.rs

* fix conflicts

---------

Co-authored-by: Ashwin Sekar <ashwin@anza.xyz>
Co-authored-by: Ashwin Sekar <ashwin@solana.com>
2024-08-23 15:59:02 -04:00
mergify[bot] fd3b4f324f
v2.0: replay: do not start leader for a block we already have shreds for (backport of #2416) (#2484)
* replay: do not start leader for a block we already have shreds for (#2416)

* replay: do not start leader for a block we already have shreds for

* pr feedback: comment, move existing check to blockstore fn

* move blockstore read after tick height check

* pr feedback: resuse blockstore fn in next_leader_slot

(cherry picked from commit 15dbe7fb0f)

# Conflicts:
#	poh/src/poh_recorder.rs

* fix conflicts

---------

Co-authored-by: Ashwin Sekar <ashwin@anza.xyz>
Co-authored-by: Ashwin Sekar <ashwin@solana.com>
2024-08-15 13:40:38 -04:00
mergify[bot] 0b1143e149
v2.0: verifies retransmitter signature on chained Merkle shreds (backport of #1735) (#2201)
verifies retransmitter signature on chained Merkle shreds (#1735)

(cherry picked from commit 6f94686181)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2024-08-02 18:01:46 +00:00
mergify[bot] b57b883355
v2.0: Fix BankForks::new_rw_arc memory leak (backport of #1893) (#2066)
* Fix BankForks::new_rw_arc memory leak (#1893)

(cherry picked from commit d441c0f577)

* Fix flaky test test_banking_stage_entries_only_central_scheduler (#2082)

---------

Co-authored-by: Andrei Silviu Dragnea <andreisilviudragnea@gmail.com>
Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>
2024-07-17 16:11:40 -07:00
mergify[bot] 5d680fd1a2
v2.0: blockstore: only consume duplicate proofs from root_slot + 1 on startup (backport of #1971) (#2114)
blockstore: only consume duplicate proofs from root_slot + 1 on startup (#1971)

* blockstore: only consume duplicate proofs from root_slot + 1 on startup

* pr feedback: update test comments

* pr feedback: add pub behind dcou for test fns

(cherry picked from commit 2a48564b59)

Co-authored-by: Ashwin Sekar <ashwin@anza.xyz>
2024-07-16 21:02:23 -04:00
mergify[bot] 0001afe8ed
v2.0: Refactor cost tracking (backport of #1954) (#1975)
* Refactor cost tracking (#1954)

* Refactor and additional metrics for cost tracking (#1888)

* Refactor and add metrics:
- Combine remove_* and update_* functions to reduce locking on cost-tracker and iteration.
- Add method to calculate executed transaction cost by directly using actual execution cost and loaded accounts size;
- Wireup histogram to report loaded accounts size;
- Report time of block limits checking;
- Move account counters from ExecuteDetailsTimings to ExecuteAccountsDetails;

* Move committed transactions adjustment into its own function

* remove histogram for loaded accounts size due to performance impact

(cherry picked from commit f8630a3522)

* rename cost_tracker.account_data_size to better describe its purpose is to tracker per-block new account allocation

---------

Co-authored-by: Tao Zhu <82401714+tao-stones@users.noreply.github.com>
Co-authored-by: Tao Zhu <tao@solana.com>
2024-07-03 14:59:02 -05:00
Tao Zhu f8ae688668
Revert "v2.0: Refactor and additional metrics for cost tracking (backport of #1888) (#1900) (#1937)
Revert "v2.0: Refactor and additional metrics for cost tracking (backport of #1888) (#1900)"

This reverts commit 0aef62eac7.
2024-07-01 10:33:44 -06:00
mergify[bot] 0aef62eac7
v2.0: Refactor and additional metrics for cost tracking (backport of #1888) (#1900)
* Refactor and additional metrics for cost tracking (#1888)

* Refactor and add metrics:
- Combine remove_* and update_* functions to reduce locking on cost-tracker and iteration.
- Add method to calculate executed transaction cost by directly using actual execution cost and loaded accounts size;
- Wireup histogram to report loaded accounts size;
- Report time of block limits checking;
- Move account counters from ExecuteDetailsTimings to ExecuteAccountsDetails;

* Move committed transactions adjustment into its own function

(cherry picked from commit c3fadacf69)

* rename cost_tracker.account_data_size to better describe its purpose is to tracker per-block new account allocation

---------

Co-authored-by: Tao Zhu <82401714+tao-stones@users.noreply.github.com>
Co-authored-by: Tao Zhu <tao@solana.com>
2024-06-28 16:09:36 -05:00
Ryo Onodera 3e53b644a8
Adjust replay-related metrics for unified scheduler (#1741)
* Adjust replay-related metrics for unified schduler

* Fix grammar

* Don't compute slowest for unified scheduler

* Rename to is_unified_scheduler_enabled

* Hoist uses to top of file

* Conditionally disable replay-slot-end-to-end-stats

* Remove the misleading fairly balanced text
2024-06-20 09:12:04 +09:00
Tyera 034cd7396a
Add num_partitions to Blockstore rewards (#1601)
* Add num_partitions field to Rewards proto definition

* Add type to hold rewards plus num_partitions

* Add Bank method to get rewards plus num_partitions for recording

* Update Blockstore::write_rewards to use num_partitions

* Update RewardsRecorderService to handle num_partitions

* Populate num_partitions in ReplayStage::record_rewards

* Write num_partitions to Bigtable

* Reword KeyedRewardsAndNumPartitions method

* Clone immediately

* Determine epoch boundary by checking parent epoch

* Rename UiConfirmedBlock field

* nit: fix comment typo

* Add test_get_rewards_and_partitions

* Add pre-activation test

* Add should_record unit test
2024-06-17 16:42:33 -06:00
Kevin Heavey 7e0688341e
remove superfluous usage of allow(dead_code) (#1669) 2024-06-13 22:52:49 +08:00
Jon C e0dc5dc707
rpc: Include interest-bearing configuration for UI amount calculation (#1549)
* Plumb new SplTokenAdditionalData everywhere

* account-decoder: Calculate ui amount with interest

* rpc / ledger: Populate interest bearing information

* rpc: Test interest-bearing config

* Deprecate `parse_token` for `parse_token_v2`

* Deprecate parse_account_data and AccountAdditionalData for v2

* Deprecate token_amount_to_ui_amount -> v2

* Make get_mint_owner_and_additional_data pub(crate)

* ledger: Revert changes, always use raw amount
2024-06-06 18:17:51 +02:00
Joe C 31334c44eb
Vote: hoist vote sender types up to runtime (#1623)
vote: hoist vote sender types up to runtime
2024-06-06 10:59:48 -05:00
Ryo Onodera 74dcd3b0ad
Make unified schedler abort on tx execution errors (#1211)
* Make unified schedler abort on tx execution errors

* Fix typo and improve wording

* Indicate panic _should_ be enforced by trait impls

* Fix typo

* Avoid closure of closure...

* Rename: is_threads_joined => are_threads_joined

* Add some comments for trashed_scheduler_inners

* Ensure 100% coverage of end_session by more tests

* Document relation of aborted thread and is_trashed

* Replace sleep()s with sleepless_testing

* Fix lints from newer rust
2024-05-31 16:13:41 +09:00
Ashwin Sekar ecb491c392
blockstore: always send a coding shred for chained merkle root conflicts (#1353)
* blockstore: always send a coding shred for chained merkle root conflicts

* pr feedback: return ErasureMeta

* fix format string indentation
2024-05-28 11:24:07 -04:00
behzad nouri 63bc98c23c
dedups code across Merkle Shred{Code,Data} implementations (#1389) 2024-05-24 13:52:55 +00:00
steviez da029625d1
Format strings in solana-ledger crate (#1392)
The long lines are known to sometimes prevent clippy from properly
formatting a file
2024-05-20 00:59:19 -05:00
Jeff Washington (jwash) 3aec34a65b
use StorageAccess from config (#1412) 2024-05-17 15:47:51 -05:00
Ashwin Sekar 8c67696346
replay: only vote on blocks with >= 32 data shreds in last fec set (#1002)
* replay: only vote on blocks with >= 32 data shreds in last fec set

* pr feedback: pub(crate), inspect_err

* pr feedback: error variants, collapse function, dedup

* pr feedback: remove set_last_in_slot, rework test

* pr feedback: add metric, perform check regardless of ff

* pr feedback: mark block as dead rather than duplicate

* pr feedback: self.meta, const_assert, no collect

* pr feedback: cfg(test) assertion, remove expect and collect, error fmt

* Keep the collect to preserve error

* pr feedback: do not hold bank_forks lock for mark_dead_slot
2024-05-17 16:42:59 +00:00
Kevin Heavey cadba689cb
Make solana-frozen-abi optional in all remaining crates (#1278)
* put most AbiExample derivations behind a cfg_attr

* feature gate all `extern crate solana_frozen_abi_macro;`

* use cfg_attr wherever we were deriving both AbiExample and AbiEnumVisitor

* fix cases where AbiEnumVisitor was still being derived unconditionally

* fix a case where AbiExample was derived unconditionally

* fix more cases where both AbiEnumVisitor and AbiExample were derived unconditionally

* two more cases where AbiExample and AbiEnumVisitor were unconditionally derived

* fix remaining unconditional derivations of AbiEnumVisitor

* fix cases where AbiExample is the first thing derived

* fix most remaining unconditional derivations of AbiExample

* move all `frozen_abi(digest =` behind cfg_attr

* replace incorrect cfg with cfg_attr

* fix one more unconditionally derived AbiExample

* feature gate AbiExample impls

* add frozen-abi feature to required Cargo.toml files

* make frozen-abi features activate recursively

* fmt

* add missing feature gating

* fix accidentally changed digest

* activate frozen-abi in relevant test scripts

* don't activate solana-program's frozen-abi in sdk dev-dependencies

* update to handle AbiExample derivation on new AppendVecFileBacking enum

* revert toml formatting

* remove unused frozen-abi entries from address-lookup-table Cargo.toml

* remove toml references to solana-address-lookup-table-program/frozen-abi

* update lock file

* remove no-longer-used generic param
2024-05-17 14:42:58 +02:00
Yihau Chen 4d5ad77e23
clippy: readonly_write_lock (#1336)
* clippy: readonly_write_lock

* add some comment

* update comments
2024-05-16 16:50:09 +00:00
behzad nouri d6cb2d6244
attaches retransmitters signature after erasure recovery (#1321)
Recovered shreds need to be resigned before being retransmitted to the
other nodes. Because shreds are resigned immediately after signature
verification, shreds belonging to the same erasure batch will have the
same signature and we can attach that signature to recovered shreds.
2024-05-16 16:26:17 +00:00
Illia Bobyr 8fcb8eca0a
ledger::shred: Document the `dispatch!` macro (#1360) 2024-05-15 20:31:34 -07:00
steviez bd09ba69b1
ledger-tool: Remove duplicate slot output code (#1255)
This makes the slot subcommand use the same output method as bigtable
block to output blocks (-vv). Doing so creates consistency between the
two commands as well as removing a duplicate implementation in the
ledger-tool code. The shared type also supports json output which the
ledger-tool implementation did not.
2024-05-15 17:31:19 -05:00
behzad nouri 517e358d78
resigns chained Merkle shreds after signature verification (#1207)
Chained Merkle shreds with retransmitter's signature should be resigned
before broadcast to other nodes.
2024-05-15 16:39:19 +00:00
Yihau Chen ec9bd79849
clippy: fix legacy_numeric_constants (#1314)
clippy: legacy_numeric_constants
2024-05-15 11:29:19 +08:00
Yihau Chen b652b6b57d
clippy: unused_enumerate_index (#1316) 2024-05-14 13:49:09 +08:00
Kevin Heavey 973b6efd61
deactivate serde feature of bitflags where not needed (#1288) 2024-05-13 10:30:01 -05:00
behzad nouri 6fa6ee293c
patches Blockstore::is_shred_duplicate for resigned chained Merkle shreds (#1271)
With chained Merkle shreds which are signed by the retransmitter,
different retransmitter signatures does not make a shred duplicate.
2024-05-13 12:42:03 +00:00
Jeff Washington (jwash) 835eec02ba
add StorageAccess to specify how to access storages (#1270)
* add StorageAccess to specify how to access storages

* pr feedback
2024-05-10 12:55:22 -05:00
Ashwin Sekar b5c5bd3160
blockstore: account for blockstore cleanup during shred insertion (#1259) 2024-05-09 18:12:43 -07:00
Ashwin Sekar 6cd0128da1
blockstore: use erasure meta index field to find conflicting shreds (#1151)
* blockstore: use erasure meta index field to find conflicting shreds

* pr feedback: error msg, let Some in case of cleanup

* pr feedback: add error if conflicting shred is not found
2024-05-09 18:12:20 -07:00
steviez c84c478e32
Remove explicit assert in Blockstore::get_slot_entries_in_block() (#1258)
This is a private function and the callers should NOT be calling with an
empty completed range vector. Regardless, it is safer to just return an
empty Entry vector should an empty CompletedRanges be provided as input.
2024-05-09 15:56:08 -05:00
Yihau Chen f050705f98
clippy: unnecessary_get_then_check (#1242) 2024-05-09 00:17:18 +08:00
behzad nouri 7bf02b71a1
renames ErasureMeta.set_index to fec_set_index (#1194)
"set_index" is ambiguous and inconsistent with shred code.
2024-05-06 12:37:49 +00:00
Ashwin Sekar a645d07e06
blockstore: relax backwards chained merkle root check for upgrades (#1163)
* blockstore: relax backwards chained merkle root check for upgrades

* s/v1.18.12/v1.18.13

Co-authored-by: Trent Nelson <490004+t-nelson@users.noreply.github.com>

---------

Co-authored-by: Trent Nelson <490004+t-nelson@users.noreply.github.com>
2024-05-05 19:20:32 -07:00
behzad nouri 93749446cb
removes redundant collect::<Vec<_>> in Rocks::multi_get_cf (#1184)
Rocks::multi_get_cf collects an iterator into a vector:
https://github.com/anza-xyz/agave/blob/6b45dc95d/ledger/src/blockstore_db.rs#L661-L666

which is then immediately converted back into an iterator:
https://github.com/anza-xyz/agave/blob/6b45dc95d/ledger/src/blockstore_db.rs#L1589-L1590
https://github.com/anza-xyz/agave/blob/6b45dc95d/ledger/src/blockstore_db.rs#L1705-L1706

The redundant collect::<Vec<_>> can be avoided by returning an iterator
from multi_get_cf.
2024-05-05 17:46:11 +00:00
Brooks b7bbe36918
clippy: multiple bound locations (#1179) 2024-05-03 11:33:09 -04:00
Brooks fbbae8a59a
clippy: clone_from() (#1177)
```
error: assigning the result of `Clone::clone()` may be inefficient
   --> bucket_map/src/bucket.rs:979:17
    |
979 |                 hashed = hashed_raw.clone();
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: use `clone_from()`: `hashed.clone_from(&hashed_raw)`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#assigning_clones
    = note: `-D clippy::assigning-clones` implied by `-D warnings`
    = help: to override `-D warnings` add `#[allow(clippy::assigning_clones)]`
```
2024-05-03 15:21:10 +00:00
steviez e628360e04
Remove intermediate Vec<_>'s in rocksdb multi_get functions (#1144)
Currently, several intermediate Vec<_>'s are created to represent
the keys passed down to multi_get(). These intermediate Vec<_>'s are
not strictly necessary and incur extra allocations. We can instead pass
iterators at several levels to avoid several allocations
2024-05-02 16:44:47 -05:00
dependabot[bot] be80b92259
build(deps): bump rocksdb from 0.21.0 to 0.22.0 (#535)
* build(deps): bump rocksdb from 0.21.0 to 0.22.0

Bumps [rocksdb](https://github.com/rust-rocksdb/rust-rocksdb) from 0.21.0 to 0.22.0.
- [Release notes](https://github.com/rust-rocksdb/rust-rocksdb/releases)
- [Changelog](https://github.com/rust-rocksdb/rust-rocksdb/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-rocksdb/rust-rocksdb/compare/v0.21.0...v0.22.0)

---
updated-dependencies:
- dependency-name: rocksdb
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2024-05-01 15:34:46 -05:00
Jon C b36eec5ab8
spl: Upgrade to newest library versions (#994)
* spl: Upgrade to newest versions

* Patch more crates
2024-04-24 18:48:58 +02:00
Ashwin Sekar 175e152714
blockstore: send duplicate proofs for chained merkle root conflicts (#835)
* blockstore: send duplicate proofs for chained merkle root conflicts

* pr feedback: avoid deserialization, inline warn logs

* remove allow[dead_code] after rebase

* pr feedback: cow to simplify, avoid extra indentation
2024-04-23 21:36:23 +00:00
Ashwin Sekar f1336970d0
blockstore: store first received coding shred index in ErasureMeta (#961)
* blockstore: store first received coding shred index in ErasureMeta

* pr feedback: use u64 instead of usize
2024-04-22 16:03:18 -07:00
behzad nouri a94e221d52
caches Merkle shreds signature verifications (#230)
Merkle shreds sign the Merkle root of the erasure batch, so all shreds
within the same erasure batch have the same signature. The commit
improves shreds signature verification by adding an LRU cache.
2024-04-19 17:43:44 +00:00
behzad nouri aa0922d684
reworks shred::merkle::Shred{Code,Data}::get_chained_merkle_root_offset (#802)
Shred{Code,Data}::get_chained_merkle_root_offset is only applicable to
"chained" Merkle shreds however the code or API does not enforce this.

The commit explicitly checks for "chained" variant in
Shred{Code,Data}::get_chained_merkle_root_offset.
2024-04-17 16:32:41 +00:00
behzad nouri f2612ef7b5
enables chained Merkle shreds for ClusterType::Development (#54)
The commit enables chained Merkle shreds for ClusterType::Development so that
we get ample coverage for further development.
2024-04-16 18:14:39 +00:00
Justin Starry f27f91bed7
Deprecate `solana_program::sdk_ids` module (#790)
Deprecate solana_program::sdk_ids module
2024-04-16 06:21:20 +08:00
Jeff Washington (jwash) 6bd0bb611c
introduce AccountForStorage for StorableAccounts (#795)
introduce `AccountForStorage` for `StorableAccounts`
2024-04-15 13:21:07 -05:00