During a cluster upgrade when only half of the cluster can ingest the new shred
variant, sending shreds of the new variant can cause nodes to diverge.
The commit adds a feature to enable chained Merkle shreds explicitly.
#### Problem
In HotStorageReader, the account_matches_owners takes
&[&Pubkey] as the address candidates. However, it should
be &[Pubkey] as defined in the accounts_file API.
#### Summary of Changes
Correct HotStorageReader::account_matches_owners() to
take &[Pubkey] instead.
#### Test Plan
Existing unit-tests
#### Problem
Using non-reference type of AccountHash in
AccountMetaOptionalFields causes an unnecessary copy
as mentioned in #34948.
#### Summary of Changes
Uses &AccountHash in AccountMetaOptionalFields to
avoid copying.
#### Test Plan
Existing unit tests.
Fixes#34948
* gossip: notify state machine of duplicate proofs
* Add feature flag for ingesting duplicate proofs from Gossip.
* Use the Epoch the shred is in instead of the root bank epoch.
* Fix unittest by activating the feature.
* Add a test for feature disabled case.
* EpochSchedule is now not copyable, clone it explicitly.
* pr feedback: read epoch schedule on startup, add guard for ff recache
* pr feedback: bank_forks lock, -cached_slots_in_epoch, init ff
* pr feedback: bank.forks_try_read() -> read()
* pr feedback: fix local-cluster setup
* local-cluster: do not expose gossip internals, use retry mechanism instead
* local-cluster: split out case 4b into separate test and ignore
* pr feedback: avoid taking lock if ff is already found
* pr feedback: do not cache ff epoch
* pr feedback: bank_forks lock, revert to cached_slots_in_epoch
* pr feedback: move local variable into helper function
* pr feedback: use let else, remove epoch 0 hack
---------
Co-authored-by: Wen <crocoxu@gmail.com>
#### Problem
So far the current HotStorageWriter::write_accounts() only writes
accounts blocks and index block.
#### Summary of Changes
The PR further writes owners block in HotStorageWriter::write_accounts().
#### Test Plan
Extended existing test for HotStorageWriter to cover the owners block.
The accounts command currently dumps every single account in the
AccountsDb. This is obviously a lot of output, so a previous change
streamed the accounts instead of collecting and dumping at the end.
The streaming approach is much more performant, but the implementation
is non-trivial. This change
- Moves the accounts output code to output.rs
- Refactor the logic to several objects that implment the functionality
- Adjust the json output to also include the summary
This change lays the groundwork for cleanly adding several more flags
that will allow for querying different subsets of accounts.
This argument is a flag and doesn't take a value; however, it had the
.value_name() modifier set with "FORMAT". This could be confusing so
remove .value_name() and add .takes_value(false)
#### Problem
In HotStorageWriter::write_accounts, it skips storing rent-epoch when
the rent-epoch equals Epoch::MAX. While the value is correct, it is
more suitable to use RENT_EXEMPT_RENT_EPOCH instead as the
goal here is to save bytes for rent-exempt accounts.
#### Summary of Changes
Replace Epoch::MAX by RENT_EXEMPT_RENT_EPOCH when checking
whether to skip storing rent-epoch in HotStorageWriter.
* Check feature_set for enable_partitioned_epoch_reward
* Keep common variable outside if case
* Keep common early return out of if case, since the first_slot_in_epoch must exist for partiion PDA to exist
* Get and parse epoch partition data PDA
* Find partition index for all addresses
* Pull relevant blocks and get rewards
* Reuse ordering and reformatting
* Remove feature deactivation from TestValidator
* Restore rewards iteration in first block in epoch for feature case to catch Voting rewards
* Add fn get_reward_map helper to dedupe code
* No need to start 2nd get_block_with_limit call with first block again
* Replace filter_map to parameterize RewardType filter expression
* Weird thing to make clippy and compiler agree (https://github.com/rust-lang/rust-clippy/issues/8098)
* Use activated_slot to ensure the right approach for past rewards epochs
* Improve help text
* Add max_accounts cli arg and halt loop accordingly
* Log totals before max_account exit
* Add test of early end on max_accounts
* Add logging when account-creation loop breaks
#### Problem
The implementation of write_accounts() for HotAccountStorage is missing.
It consists of the writing of account blocks, index block, and owners block.
#### Summary of Changes
This PR completes part of the HotStorageWriter::write_accounts().
Specifically, it finishes the writing of account blocks and index block.
#### Test Plan
A new unit-test is added to verify the correctness of the work-in-progress
HotStorageWriter::write_accounts().
* implement a custom parser for `VoteState` which is usuable in a bpf context
* derive or impl `Arbitrary` for `VoteStateVersions` and its component types, for test builds only
* Move EpochRewardsHasher to solana-sdk
* Cargo.lock
* Apparently we're allowing arithmetic_side_effects in all of runtime
* Move allow stmt to block instead of module
* Also allow in test mod
* add `_from_source` function variants for signer, keypair, and pubkey
* make `parse_signer_source` an associated function of `SignerSource`
* refactor `SignerSource` into `input_parsers::signer`
* make `_from_source` functions public
* remove unnecessary import
* Refactor bank load_and_execute_transactions
* Remove redundant clippy annotation
* Report check time where it is done in load_and_execute_transactions
* add `try_split_u64`
* add `try_combine_lo_hi_u64`
* add `try` variants of ciphertext arithmetic functions
* use try functions in proof generaiton and verification logic
* deprecate non-`try` functions
* use try functions in proof generaiton and verification logic
* Apply suggestions from code review
Co-authored-by: Jon C <me@jonc.dev>
* cargo fmt
---------
Co-authored-by: Jon C <me@jonc.dev>
* Introduce primitive threading in unified scheduler
* Make the internal struct ExecutedTask not pub
* Improve wording a bit
* Explain scheduler main loop's overhead sensitivity
* Improve wording a bit
* Define ChainedChannel{Sender, Receiver} wrappers
* Clean up a bit
* Use derivative to avoid manual Clone impl
* Clarify comment
* Remove extra whitespace in comment
* Remove unneeded dyn trait for ChainedChannel
* Remove the accumulator thread for now
* Fix typo
* Use unimplemented!() to convey intention better
#### Problem
To write the owners-block, it requires an in-memory struct that maintains
a set of unique owner addresses while providing a look-up function to
obtain the OwnerOffset with the specified owner address.
#### Summary of Changes
This PR adds OwnersTable, the in-memory struct that maintains
a set of unique owner addresses while providing a look-up function to
obtain the OwnerOffset with the specified owner address.
#### Test Plan
A new unit-test is added.
hana
Pankaj Garg
behzad nouri
Brooks
HaoranYi
Tao Zhu
Yueh-Hsuan Chiang
Ashwin Sekar
dependabot[bot]
steviez
ellttBen
Tyera
Andrew Fitzgerald
Alexander Meißner
Dmitri Makarov
samkim-crypto
Wen
Ryo Onodera
Will Hickey