* allow pedantic invalid cast lint
* allow lint with false-positive triggered by `test-case` crate
* nightly `fmt` correction
* adapt to rust layout changes
* remove dubious test
* Use transmute instead of pointer cast and de/ref when check_aligned is false.
* Renames clippy::integer_arithmetic to clippy::arithmetic_side_effects.
* bump rust nightly to 2023-08-25
* Upgrades Rust to 1.72.0
---------
Co-authored-by: Trent Nelson <trent@solana.com>
* remove unnecessary hashes around raw string literals
* remove unncessary literal `unwrap()`s
* remove panicking `unwrap()`
* remove unnecessary `unwrap()`
* use `[]` instead of `vec![]` where applicable
* remove (more) unnecessary explicit `into_iter()` calls
* remove redundant pattern matching
* don't cast to same type and constness
* do not `cfg(any(...` a single item
* remove needless pass by `&mut`
* prefer `or_default()` to `or_insert_with(T::default())`
* `filter_map()` better written as `filter()`
* incorrect `PartialOrd` impl on `Ord` type
* replace "slow zero-filled `Vec` initializations"
* remove redundant local bindings
* add required lifetime to associated constant
* sdk: Add concurrent support for rand 0.7 and 0.8
* Update rand, rand_chacha, and getrandom versions
* Run command to replace `gen_range`
Run `git grep -l gen_range | xargs sed -i'' -e 's/gen_range(\(\S*\), /gen_range(\1../'
* sdk: Fix users of older `gen_range`
* Replace `hash::new_rand` with `hash::new_with_thread_rng`
Run:
```
git grep -l hash::new_rand | xargs sed -i'' -e 's/hash::new_rand([^)]*/hash::new_with_thread_rng(/'
```
* perf: Use `Keypair::new()` instead of `generate`
* Use older rand version in zk-token-sdk
* program-runtime: Inline random key generation
* bloom: Fix clippy warnings in tests
* streamer: Scope rng usage correctly
* perf: Fix clippy warning
* accounts-db: Map to char to generate a random string
* Remove `from_secret_key_bytes`, it's just `keypair_from_seed`
* ledger: Generate keypairs by hand
* ed25519-tests: Use new rand
* runtime: Use new rand in all tests
* gossip: Clean up clippy and inline keypair generators
* core: Inline keypair generation for tests
* Push sbf lockfile change
* sdk: Sort dependencies correctly
* Remove `hash::new_with_thread_rng`, use `Hash::new_unique()`
* Use Keypair::new where chacha isn't used
* sdk: Fix build by marking rand 0.7 optional
* Hardcode secret key length, add static assertion
* Unify `getrandom` crate usage to fix linking errors
* bloom: Fix tests that require a random hash
* Remove some dependencies, try to unify others
* Remove unnecessary uses of rand and rand_core
* Update lockfiles
* Add back some dependencies to reduce rebuilds
* Increase max rebuilds from 14 to 15
* frozen-abi: Remove `getrandom`
* Bump rebuilds to 17
* Remove getrandom from zk-token-proof
In most cases, either a &Bank or an Arc<Bank> is more proper.
- &Bank is used if the function only needs a momentary reference
- Arc<Bank> is used if the function needs its' own copy
This PR leaves several instances of &Arc<Bank> around; these instances
are situations where a clone may only happen conditionally.
* Integrate program loader-v4 with bank
* fix tests
* new struct for ProgramRuntimeEnvironments
* remove environment from program_runtime_environment_v
* move find_program_in_cache() to invoke_context
* cleanup
When a consensus divergance occurs, the current workflow involves a
handful of manual steps to hone in on the offending slot and
transaction. This process isn't overly difficult to execute; however, it
is tedious and currently involves creating and parsing logs.
This change introduces functionality to output a debug file that
contains the components go into the bank hash. The file can be generated
in two ways:
- Via solana-validator when the node realizes it has diverged
- Via solana-ledger-tool verify by passing a flag
When a divergance occurs now, the steps to debug would be:
- Grab the file from the node that diverged
- Generate a file for the same slot with ledger-tool with a known good
version
- Diff the files, they are pretty-printed json
* remove some stats that are no longer kept
* Update runtime/src/accounts_hash.rs
Co-authored-by: Brooks <brooks@prumo.org>
* reorder to remove drop
---------
Co-authored-by: Brooks <brooks@prumo.org>
* stake: deprecate on chain warmup/cooldown rate and config
* Pr feedback: Deprecate since 1.16.7
Co-authored-by: Jon Cinque <me@jonc.dev>
---------
Co-authored-by: Jon Cinque <me@jonc.dev>
Using atomics for entry indices allows load function to use self.cache.get
instead of get_mut which reduces lock contention on the respective
dash-map shard.
#### Summary of Changes
This PR implements TieredStorage::new_readonly() and introduces
TieredStorageReader and HotStorageReader.
#### Test Plan
Updated the existing unit test.
#### Summary of Changes
This PR initiates the implementation of TieredAccountsFile::write_accounts,
which will later support AccountsFile::append_accounts.
This PR also introduces TieredAccountsFileWriter -- the main writer struct for a
TieredAccountsFile.
#### Test Plan
Extended existing unit tests.
#### Summary of Changes
This PR initiates the implementation of the main struct for the
tiered accounts storage --- TieredStorage. Specifically,
it defines the TieredStorage struct, TieredStorageFormat,
and skeleton implementation of new_writable().
#### Test Plan
Unit tests are included in this PR.
#### Summary of Changes
This PR implements AccountIndexFormat::AddressAndOffset, the index format
that will be used by the hot account storage.
#### Test Plan
Unit tests are included in this PR.
Tested via the prototype implementation of tiered-storage.
#### Problem
`TieredAccountMeta` has default impls on many of its methods. But I don't think it should, because these defaults are not actually useful defaults (i.e. they are `unimplemented!()`).
We're only going to have a small number of structs that'll implement this trait, and each struct should be responsible for picking the correct impl for each method.
Currently, `HotAccountMeta` is the only one, and it implements every method.
#### Summary of Changes
Remove default impls on TieredAccountMeta
* update max reward block factor
* review comments
* more tests
* reduce the number of stakes per block to speed up tests
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
#### Problem
While TieredReadableAccount implements ReadableAccount that
already includes `fn data(&self) -> &'a [u8]`, a `data()` function that
directly under TieredReadableAccount is still needed in order to
correctly link the lifetime of the returned value and its member
account_block. Otherwise, cargo will complain lifetime may not
live long enough.
#### Summary of Changes
This PR adds TieredReadableAccount::data() that directly links the lifetime
of its account_block to the returned value.
* Test rewards compuation and partitioned rewards distribution at the epoch boundary
* reviews
* match to if
* clippy
* peeling of bank1 from the loop
* collapse if_same
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
* Move CostModel and CostTracker to its own crate
* compile new crate and update imports
* update sbf Cargo.lock
* fix AbiExample
* fix cargo sort
* Fix AbiExample
#### Problem
Both AppendVec::cloned_account() and ReadableAccount::to_account_shard_data()
returns an AccountSharedData instance by cloning itself as pointed out under
a comment in PR #32380.
#### Summary of Changes
This PR removes AccountsFile::cloned_account() and replaces its call-sites by
ReadableAccount::to_account_shared_data().
#### Summary of Changes
This PR introduces TieredReadableAccount, a struct that takes a generic
TieredAccountMeta that implements ReadableAccount and support
StoredAccountMeta.
#### Test Plan
More tests will be added once more reader and writer functions for hot
accounts have been added.
#### Problem
In TieredAccountMeta, it uses `data` and `account_data` to refer to
the data associated with the account, and this could be confusing.
#### Summary of Changes
This PR makes TieredAccountMeta only use `account_data` to refer
to the data associated with the account.
#### Problem
Function data_size_for_shared_block() is designed to provide
the data_size when its account shares its account block with
other accounts. However, TieredAccountMeta already has another
function for returning data size.
#### Summary of Changes
This PR removes data_size_for_shared_block() from TieredAccountMeta.
Will follow-up with another PR that includes a better API.
#### Summary of Changes
This PR implements HotAccountMeta's getter functions for accessing
optional fields (e.g. rent_epoch, account_hash, etc) and account data.
#### Test Plan
A new unit test is included in this PR.
#### Summary of Changes
This PR adds several functions to AccountMetaOptionalFields
for obtaining the offset of each optional field.
#### Test Plan
New unit test cases are included in this PR.
#### Summary of Changes
This PR adds byte_block::read_type(), an util function that
reads the raw part of the input byte block at the specified offset
as type T. This function will be later used by the hot storage.
#### Test Plan
Modified existing unit tests to use byte_block::read_type()
when the input byte block is properly aligned.
* When there are too many pubkeys in one slot, kick the one with lowest
stake out.
* Cache last_root to reduce read locks we need.
* Use slots_in_epoch to limit number of slots in the map.
* Fix lint errors.
* Only cache stake and slots per epoch once per epoch.
* Revert "Only cache stake and slots per epoch once per epoch."
This reverts commit 8658aad0083456794b4c4403adaf9c74d1a71d09.
* Vote at the tip of current fork if last vote is outside SlotHash
of the tip and last vote expired.
* Add unittest when last vote is outside slothash, we should vote at the tip
of the current fork.
* Revert "Use slots_in_epoch to limit number of slots in the map."
This reverts commit 93574f57a48d2a70fbbc0f62fa8810d3b6bee0af.
* Revert "Cache last_root to reduce read locks we need."
This reverts commit bb114ec2b62cb9c0207328b19c415f6116be0f1c.
* Revert "When there are too many pubkeys in one slot, kick the one with lowest"
This reverts commit 711e29a6a025fd4f11fbc97dcbbe90e4832be04c.
* Move new vote generation when last vote is outside slothash into the
main path, this actually makes more sense since we don't select where
to vote in two different places, and all the vote generation logic
is seamlessly inherited.
* - Move vote refresh to be behind select vote and do not refresh vote if a new
vote is selected.
- Check whether last vote is inside slothash inside select_vote_and_reset_forks
- rename slot_within_slothash to is_in_slothashes_history
- remove one unittest for now, more tests will be added in a separate CL
* Remove new test, it will be in another file.
* Add is_in_slot_hashes_history test in the new file.
* Add unittest for the case when last vote is outside slot hashes.
* Small improvements and more unittests.
* Fix bad merge.
* Update docs/src/terminology.md
Co-authored-by: mvines <mvines@gmail.com>
* Put SwitchForkDecision::FailedSwitchThreshold logic into separate function.
* Make linter happy.
---------
Co-authored-by: mvines <mvines@gmail.com>
#### Summary of Changes
This PR adds getter functions to the TieredAccountMeta for accessing
optional fields and account data.
#### Test Plan
This PR only adds methods to a trait. More tests will be included in the
follow-up PRs.
Slot::MAX was used to specify that a type of snapshots should not be
created; define a constant to be that value and reference the constant
to have a single point of edit.
Tao Zhu
Pankaj Garg
Brooks
behzad nouri
steviez
Alexander Meißner
Ryo Onodera
Trent Nelson
Jon Cinque
HaoranYi
Jeff Washington (jwash)
Yueh-Hsuan Chiang
Ashwin Sekar
Justin Starry
Illia Bobyr
Wen