Commit Graph

4404 Commits

Author SHA1 Message Date
Christian Kamm 525e59f01a
sdk, programs/bpf_loader: add sol_remaining_compute_units syscall (#31640)
bpf_loader: add sol_remaining_compute_units syscall

Co-authored-by: jonch <9093549+jon-chuang@users.noreply.github.com>
2023-09-13 08:57:08 -06:00
Pankaj Garg 5562f79cc5
Do not unnecessarily re-verify unloaded program (#32722)
* Do not unnecessarily re-verify unloaded program

* clippy fixes

* new unit test for extract

* fixes after rebase

* fixes after rebase

* address review comments

* fix clippy
2023-09-13 06:25:56 -07:00
Alexander Meißner bbb57be0a5
Refactor - Move interfaces of address-lookup-table into the program SDK (#33165)
* Adds a module `address_lookup_table` to the SDK.

* Adds a module `address_lookup_table::instruction` to the SDK.

* Adds a module `address_lookup_table::error` to the SDK.

* Adds a module `address_lookup_table::state` to the SDK.

* Moves AddressLookupTable into SDK as well.

* Moves AddressLookupTableAccount into address_lookup_table.

* Adds deprecation messages.

* Disentangles dependencies across cargo files.
2023-09-11 21:10:40 +02:00
Tao Zhu 4f4ce69f5f
purge duplicated bank prioritization fee from cache (#33062)
* purge duplicated bank prioritization fee from cache

* add test for purge dup bank

* Added metrics counts to monitor anomalies

* fix a flaky test
2023-09-11 10:08:55 -05:00
Pankaj Garg 377b6003a8
Check loader-v4::id() as a valid program owner (#33167) 2023-09-06 11:54:18 -07:00
Brooks a80819b029
PrunedBanksRequestHandler::handle_request() is only pub with dcou (#33155) 2023-09-06 17:02:52 +00:00
behzad nouri 88ee8f5820
replaces once_cell::sync::OnceCell with std::sync::OnceLock (#33140)
std::sync::OnceLock has become stable since rust 1.70.0 and there is no
longer a need for an external crate dependency.
2023-09-06 16:46:51 +00:00
Brooks 9e156f88f4
Removes invariant `is_serialized_with_abs` param (#33154) 2023-09-06 12:00:18 -04:00
Brooks d921b9a44e
Adds metrics when purging banks with the same slot (#33153) 2023-09-06 11:38:30 -04:00
steviez 224eea42d5
Add unit test for Bank::register_hard_fork() (#32902) 2023-09-06 09:07:09 -05:00
Brooks 2cf53b6293
Purge banks, with the same slot, sequentially (#33149) 2023-09-05 17:06:35 -04:00
Alexander Meißner e9542200e8
Fix - Superfluous `if` condition for overwriting `environments` in feature activation code (#33138)
Removes the `if` block around the overwriting of the `environments`.
2023-09-05 15:01:20 +02:00
Alexander Meißner 9e703f85de
Upgrades Rust to 1.72.0 & nightly-2023-08-25 (#32961)
* allow pedantic invalid cast lint

* allow lint with false-positive triggered by `test-case` crate

* nightly `fmt` correction

* adapt to rust layout changes

* remove dubious test

* Use transmute instead of pointer cast and de/ref when check_aligned is false.

* Renames clippy::integer_arithmetic to clippy::arithmetic_side_effects.

* bump rust nightly to 2023-08-25

* Upgrades Rust to 1.72.0

---------

Co-authored-by: Trent Nelson <trent@solana.com>
2023-09-01 07:26:13 +00:00
Ryo Onodera 4fe0812f3c
Make should_panic tests compatible with 1.73+ panic format (#33099)
* Adjust should_panic tests for 1.73+ panic format

* Update more panic format change affected tests...
2023-09-01 02:04:22 +00:00
behzad nouri 39615bd075
removes manual let...else (#33089)
https://rust-lang.github.io/rust-clippy/master/index.html#/manual_let_else
2023-08-31 22:35:47 +00:00
behzad nouri 4ec5ea6f7b
replaces assert!(matches!(...)) with assert_matches!(...) (#33068)
assert_matches!(...) provides more informative error message when it
fails and it is part of nightly rust:
https://doc.rust-lang.org/std/assert_matches/macro.assert_matches.html
2023-08-30 13:48:27 -04:00
Trent Nelson b8dc5daedb
preliminaries for bumping nightly to 2023-08-25 (#33047)
* remove unnecessary hashes around raw string literals

* remove unncessary literal `unwrap()`s

* remove panicking `unwrap()`

* remove unnecessary `unwrap()`

* use `[]` instead of `vec![]` where applicable

* remove (more) unnecessary explicit `into_iter()` calls

* remove redundant pattern matching

* don't cast to same type and constness

* do not `cfg(any(...` a single item

* remove needless pass by `&mut`

* prefer `or_default()` to `or_insert_with(T::default())`

* `filter_map()` better written as `filter()`

* incorrect `PartialOrd` impl on `Ord` type

* replace "slow zero-filled `Vec` initializations"

* remove redundant local bindings

* add required lifetime to associated constant
2023-08-29 23:05:35 +00:00
Alexander Meißner 1238e53c9f
Refactor - Loader v4 deployment status (#33024)
Refactors "is_deployed: bool" and "authority_address: Option<Pubkey>".
Replaces them with "status: LoaderV4Status" and "authority_address: Pubkey".
2023-08-28 20:14:01 +02:00
Ryo Onodera d3fb54e0b2
Remove From<LogCollector> for Vec<String> (#33012) 2023-08-27 13:10:02 +09:00
Alexander Meißner 29bbda0c11
General cleanup (Part 2) (#32998)
* bucket-map: remove unused `mut`

* remove unnecessary explicit `into_iter()` calls

* don't default-construct unit-structs

* prefer `or_default()` to `or_insert_with(T::default())`

* replace "slow zero-filled vec initialization"

---------

Co-authored-by: Trent Nelson <trent@solana.com>
2023-08-25 19:07:38 +02:00
Alexander Meißner a8be70fa7a
General cleanup (#32980)
* Fixes `cargo::` in build.rs

* Fixes overshadowing reexports.

* Removes unnecessary `as *const u8`.

* Removes unnecessary `.into_iter()`.

* cargo clippy
2023-08-24 21:44:19 +02:00
Alexander Meißner 025b952598
Fix - missing delay visibility feature triggering cache flush (#32974)
Adds delay_visibility_of_program_deployment to FEATURES_AFFECTING_RBPF.
2023-08-24 16:41:23 +02:00
steviez cd96d5b6c3
Only update last restart slot sysvar if value has changed (#32925)
Read account first to check for change in value
2023-08-22 22:33:03 +02:00
Jon Cinque 0fe902ced7
Bump rand to 0.8, rand_chacha to 0.3, getrandom to 0.2 (#32871)
* sdk: Add concurrent support for rand 0.7 and 0.8

* Update rand, rand_chacha, and getrandom versions

* Run command to replace `gen_range`

Run `git grep -l gen_range | xargs sed -i'' -e 's/gen_range(\(\S*\), /gen_range(\1../'

* sdk: Fix users of older `gen_range`

* Replace `hash::new_rand` with `hash::new_with_thread_rng`

Run:
```
git grep -l hash::new_rand | xargs sed -i'' -e 's/hash::new_rand([^)]*/hash::new_with_thread_rng(/'
```

* perf: Use `Keypair::new()` instead of `generate`

* Use older rand version in zk-token-sdk

* program-runtime: Inline random key generation

* bloom: Fix clippy warnings in tests

* streamer: Scope rng usage correctly

* perf: Fix clippy warning

* accounts-db: Map to char to generate a random string

* Remove `from_secret_key_bytes`, it's just `keypair_from_seed`

* ledger: Generate keypairs by hand

* ed25519-tests: Use new rand

* runtime: Use new rand in all tests

* gossip: Clean up clippy and inline keypair generators

* core: Inline keypair generation for tests

* Push sbf lockfile change

* sdk: Sort dependencies correctly

* Remove `hash::new_with_thread_rng`, use `Hash::new_unique()`

* Use Keypair::new where chacha isn't used

* sdk: Fix build by marking rand 0.7 optional

* Hardcode secret key length, add static assertion

* Unify `getrandom` crate usage to fix linking errors

* bloom: Fix tests that require a random hash

* Remove some dependencies, try to unify others

* Remove unnecessary uses of rand and rand_core

* Update lockfiles

* Add back some dependencies to reduce rebuilds

* Increase max rebuilds from 14 to 15

* frozen-abi: Remove `getrandom`

* Bump rebuilds to 17

* Remove getrandom from zk-token-proof
2023-08-21 19:11:21 +02:00
Brooks a8c1dabf6a
Renames AccountsPackageType to AccountsPackageKind (#32908) 2023-08-21 13:00:00 -04:00
HaoranYi 910b0f5d12
fix sbf sysvar test (#32803)
* fix sbf sysvar test

* typo

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-08-21 09:49:48 -06:00
Brooks a563b4cccb
Renames AccountsHashEnum to AccountsHashKind (#32904) 2023-08-21 10:34:56 -04:00
Brooks 9a6cf3f528
Renames SnapshotRequestType to SnapshotRequestKind (#32907) 2023-08-21 07:12:20 -04:00
Brooks f9bc3cec24
Renames SnapshotType to SnapshotKind (#32906) 2023-08-19 19:55:29 -04:00
steviez a4c8cc3ce0
Remove improper uses of &Arc<Bank> (#32802)
In most cases, either a &Bank or an Arc<Bank> is more proper.
- &Bank is used if the function only needs a momentary reference
- Arc<Bank> is used if the function needs its' own copy

This PR leaves several instances of &Arc<Bank> around; these instances
are situations where a clone may only happen conditionally.
2023-08-18 16:46:34 -05:00
Brooks 3cda810358
Adds docs to get_all_accounts_modified_since_parent() (#32895) 2023-08-18 16:01:50 -04:00
steviez 480851b865
Reduce repeated logic in BankForks unit tests (#32852)
* Move repeated BankForks logic into a loop
* Do away with redundant Vec<Arc<Bank>>
* Condense repeated logic into helper function
2023-08-18 11:58:04 -05:00
Brooks 280bb53802
Adds docs to get_all_accounts() (#32887) 2023-08-18 16:09:50 +00:00
Brooks 004577d94a
Adds docs to scan_all_accounts() (#32888) 2023-08-18 16:04:07 +00:00
Brooks 1e8f7b5d1e
Adds docs to non_vote_transaction_count_since_restart() (#32886) 2023-08-18 15:37:44 +00:00
Alexander Meißner c5a251e8c3
Refactor - `Bank::compute_active_feature_set()` (#32872)
Returns the feature set instead of overwriting it inside Bank::compute_active_feature_set().
2023-08-17 19:14:10 +02:00
Pankaj Garg c17b938204
Integrate program loader-v4 with bank (#32832)
* Integrate program loader-v4 with bank

* fix tests

* new struct for ProgramRuntimeEnvironments

* remove environment from program_runtime_environment_v

* move find_program_in_cache() to invoke_context

* cleanup
2023-08-16 10:50:23 -07:00
Brooks e316db2ab4
dcou: set_accounts_hash() (#32829) 2023-08-15 07:18:59 -04:00
steviez 6bbf514e78
Add ability to output components that go into Bank hash (#32632)
When a consensus divergance occurs, the current workflow involves a
handful of manual steps to hone in on the offending slot and
transaction. This process isn't overly difficult to execute; however, it
is tedious and currently involves creating and parsing logs.

This change introduces functionality to output a debug file that
contains the components go into the bank hash. The file can be generated
in two ways:
- Via solana-validator when the node realizes it has diverged
- Via solana-ledger-tool verify by passing a flag

When a divergance occurs now, the steps to debug would be:
- Grab the file from the node that diverged
- Generate a file for the same slot with ledger-tool with a known good
  version
- Diff the files, they are pretty-printed json
2023-08-15 00:12:05 -05:00
Brooks ce57cac370
Use update_accounts_hash() in AccountsHashVerifier (#32830) 2023-08-14 13:11:23 -04:00
Ryo Onodera ce63640d6d
dcou: make internal function `pub` conditionally (#32822)
dcou: make internal function pub conditionally
2023-08-13 14:12:20 +09:00
Pankaj Garg a1a0829a8b
Initialize and store PRv2 environment in cache (#32812)
* Initialize and store PRv2 environment in cache

* address review comments
2023-08-11 18:58:22 +00:00
Jon Cinque c73a56faf8
stake: Rename `StakeStateWithFlags` -> `StakeStateV2` (#32795) 2023-08-11 01:07:21 +02:00
Pankaj Garg f4287d70bb
Move accounts-db code to its own crate (#32766) 2023-08-09 13:03:36 -07:00
Jon Cinque 8e4a9a94ed
sdk: Add new version of `StakeState` to avoid breaking downstream users (#32736)
* sdk: Rename `StakeState` -> `StakeStateWithFlags`

* Add back `StakeFlags` with a deprecation warning
2023-08-09 00:05:40 +02:00
Jeff Washington (jwash) 7c1cf298aa
add active stats for pieces of hash calc (#32750) 2023-08-08 07:16:10 -07:00
Jeff Washington (jwash) 32cb381f69
remove some stats that are no longer kept (#32751)
* remove some stats that are no longer kept

* Update runtime/src/accounts_hash.rs

Co-authored-by: Brooks <brooks@prumo.org>

* reorder to remove drop

---------

Co-authored-by: Brooks <brooks@prumo.org>
2023-08-08 07:15:37 -07:00
Jeff Washington (jwash) 21b8b70bee
add metrics around purge_exact (#32752) 2023-08-08 07:15:21 -07:00
Yueh-Hsuan Chiang bf6d0aba4c
TieredStorage struct (4/N) -- remove_on_drop (#32603)
#### Summary of Changes
This PR makes TieredStorage implement Drop which does the remove-on-drop
behavior that follows the AccountsFile convention.
2023-08-07 15:39:46 -07:00
Brooks df31bc13db
Removes `set_no_remove_on_drop()` from AppendVec/AccountsFile (#32742) 2023-08-07 17:48:31 -04:00
Brooks 4894eb0333
Moves some accounts-db test-only code into a dev-context-only-utils feature (#32748) 2023-08-07 17:28:15 -04:00
Ashwin Sekar fa3506631a
stake: deprecate on chain warmup/cooldown rate and config (#32723)
* stake: deprecate on chain warmup/cooldown rate and config

* Pr feedback: Deprecate since 1.16.7

Co-authored-by: Jon Cinque <me@jonc.dev>

---------

Co-authored-by: Jon Cinque <me@jonc.dev>
2023-08-07 13:23:24 -07:00
behzad nouri b9a2030537
uses atomics for read-only accounts cache entry index (#32518)
Using atomics for entry indices allows load function to use self.cache.get
instead of get_mut which reduces lock contention on the respective
dash-map shard.
2023-08-07 17:47:09 +00:00
Brooks c2dec254c8
Removes `remove_on_drop` field from AppendVec (#32741) 2023-08-07 13:25:07 -04:00
Pankaj Garg 511cf28be8
Move `serde-snapshot dependent accounts-db tests (#32671)
Move serde-snapshot dependent accounts-db tests
2023-08-07 09:35:41 -07:00
Brooks 6ce647a3e0
Do not drop AppendVec in store-tool (#32739) 2023-08-07 12:15:01 -04:00
Yueh-Hsuan Chiang a5bde0a79c
TieredStorage struct (3/N) -- new_readonly and reader structs (#32579)
#### Summary of Changes
This PR implements TieredStorage::new_readonly() and introduces
TieredStorageReader and HotStorageReader.


#### Test Plan
Updated the existing unit test.
2023-08-04 12:18:38 -07:00
HaoranYi 533f42dae6
remove type cast in scan metric report (#32704)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-08-04 13:03:32 -05:00
Brooks 04e25ff3ce
Adds doc comments for Bank fns that get accounts/snapshot hashes (#32716) 2023-08-04 13:56:15 -04:00
Jeff Washington (jwash) ca7a7ad2f3
add some doc comments identified in audit (#32709)
* add some doc comments identified in audit

* Update runtime/src/bank.rs

Co-authored-by: Brooks <brooks@prumo.org>

* Update runtime/src/accounts_db.rs

Co-authored-by: Brooks <brooks@prumo.org>

* Update runtime/src/accounts_db.rs

Co-authored-by: Brooks <brooks@prumo.org>

---------

Co-authored-by: Brooks <brooks@prumo.org>
2023-08-04 10:52:01 -07:00
Alexander Meißner a310dd776c
Fix - Uses `std::mem::transmute` and `std::ptr::write` in unsafe code in append_vec.rs (#32711)
Uses std::mem::transmute and std::ptr::write in unsafe code in append_vec.rs
2023-08-04 17:18:13 +02:00
steviez 5edd032c79
Remove unnecessary clone (#32707) 2023-08-03 13:54:43 -06:00
Jeff Washington (jwash) f5c8bb789a
remove unused bank creation/freezing progress (#32699) 2023-08-03 10:12:44 -07:00
Alexander Meißner 33c33f5f79
Fix - Adds missing feature pubkeys to the `FEATURES_AFFECTING_RBPF` list (#32703)
Adds missing feature pubkeys to the FEATURES_AFFECTING_RBPF list.
2023-08-03 17:45:30 +02:00
Jeff Washington (jwash) 3dcb382731
avoid giant memory allocation in hash calc (#32646)
* avoid giant memory allocation in hash calc

* update comment

* reorder to avoid clone

* simplify references

* update comment on get_item
2023-07-31 13:13:19 -07:00
Brooks f4504d055a
Uses Mutex::into_inner() at end of de_dup_accounts() (#32667) 2023-07-31 17:02:45 +00:00
Jeff Washington (jwash) 91fd2a3a93
abs purge slots in parallel (#32536)
* abs purge slots in parallel

* remove arc clone
2023-07-31 09:22:53 -07:00
Jeff Washington (jwash) bb407d2b62
add metric read_only_cache_load_us (#32559) 2023-07-31 07:07:13 -07:00
Brooks 58e796dc64
Removes unnecessary clippy-allow for needless-collect when generating index (#32656) 2023-07-29 22:30:11 -04:00
Brooks fc8cfb63ae
Unstable sort slots when generating index (#32653) 2023-07-29 22:28:33 -04:00
behzad nouri ad4ddd3cb0
stores stake-accounts in parallel after epoch rewards calculations (#32633) 2023-07-29 20:47:28 +00:00
Brooks 16ec9a68c9
Unzips Option when deserializing snapshot data files (#32652) 2023-07-28 15:21:22 -04:00
Brooks 965ed1ab1f
Simplifies if-let when rebuilding bank (#32654)
Simplifies let-Some when rebuilding bank
2023-07-28 17:55:58 +00:00
Brooks e1d0785f4f
Unzips Option when rebuilding bank (#32651) 2023-07-28 12:28:49 -04:00
Brooks de02601d73
Cleanups metrics in add_bank_snapshot() (#32645) 2023-07-27 20:24:25 +00:00
Brooks 4417f49728
Adds more timing metrics to add_bank_snapshot() (#32639) 2023-07-27 14:52:34 -04:00
Yueh-Hsuan Chiang 2448486106
TieredStorage struct (2/N) -- write_accounts part 1 (#32541)
#### Summary of Changes
This PR initiates the implementation of TieredAccountsFile::write_accounts,
which will later support AccountsFile::append_accounts.

This PR also introduces TieredAccountsFileWriter -- the main writer struct for a
TieredAccountsFile.

#### Test Plan
Extended existing unit tests.
2023-07-27 14:55:48 +08:00
Brooks 89c1f94cc4
Do not add bank snapshots for AccountsHashVerifier requests (#32589) 2023-07-25 18:30:06 -04:00
Pankaj Garg ef8d3206d7
Move bank specific code out of epoch_accounts_hash/utils.rs (#32623)
* Move bank specific code out of epoch_accounts_hash/utils.rs

* cleanup
2023-07-25 14:12:20 -07:00
Yueh-Hsuan Chiang 28d7e59cac
TieredStorage struct (1/N) -- new_writable() (#32538)
#### Summary of Changes
This PR initiates the implementation of the main struct for the
tiered accounts storage --- TieredStorage.  Specifically,
it defines the TieredStorage struct, TieredStorageFormat,
and skeleton implementation of new_writable().

#### Test Plan
Unit tests are included in this PR.
2023-07-26 01:51:56 +08:00
Pankaj Garg aba637d5d9
Split snapshot_utils.rs into snapshot_bank_utils.rs (#32612) 2023-07-24 16:31:03 -07:00
Brooks de4ff348b4
Removes manual call to `.0.display()` with thiserror (#32605) 2023-07-24 12:02:55 -04:00
HaoranYi 17af3ab10a
Add stake_flags to stake state (#32524)
add stake_flags to stake state

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-07-24 09:09:40 -05:00
Tao Zhu 62b9fcfa76
cleanup feature: Use default units per instruction in fee calculation (#32570)
* cleanup feature: Use default units per instruction in fee calculation
2023-07-21 18:37:58 -05:00
Pankaj Garg fc35b13365
Cleanup load_program() in bank.rs (#32146) 2023-07-21 13:43:46 -07:00
behzad nouri 952d8861c6
removes feature-gate code for updating rewards from cached accounts (#32514) 2023-07-21 19:52:44 +00:00
HaoranYi 9e8639f7ae
Add distributed rewards for the block in partitioned reward metrics (#32580)
report distributed rewards for the block in metrics

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-07-21 14:42:52 -05:00
Pankaj Garg faff34403c
Move bank dependent tests out of serde_snapshot/tests.rs (#32571) 2023-07-21 12:08:32 -07:00
Tao Zhu 19f5b49abb
Priority queue supports deprecated compute budget ix (#32567)
Supports deprecated compute budget instruction when getting transaction's priority details
2023-07-20 19:17:56 -05:00
Brooks faa002c2b6
Adds error types for all of `add_bank_snapshot()` (#32564) 2023-07-20 18:30:19 -04:00
Brooks 979913e1d5
Returns Self instead of Result<Self> for AccountsPackage::new_for_snapshot() (#32545) 2023-07-20 12:52:31 -04:00
Brooks 3fbfac41da
AccountsBackgroundService gracefully exits when snapshot request handling errors (#32543) 2023-07-20 11:53:06 -04:00
Jeff Washington (jwash) 0f70098410
allow shrinking to be in progress during `scan_account_storage` (#32553) 2023-07-20 08:01:10 -07:00
Brooks 6d30429dd1
Replaces `unwrap` with `?` when writing snapshot version file (#32542) 2023-07-19 16:42:46 -04:00
behzad nouri a8b0c92873
simplifies update_stakes_cache implementation (#32534) 2023-07-19 18:13:30 +00:00
HaoranYi 5408872476
test force reward interval end fn (#32513)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-07-17 15:37:10 -05:00
Yueh-Hsuan Chiang a2d9870678
[TieredStorage] Implementation of AccountIndexFormat for hot accounts (#32497)
#### Summary of Changes
This PR implements AccountIndexFormat::AddressAndOffset, the index format
that will be used by the hot account storage.

#### Test Plan
Unit tests are included in this PR.
Tested via the prototype implementation of tiered-storage.
2023-07-16 07:23:08 +08:00
Pankaj Garg 2a86420777
Move calculate_fee() out of bank.rs (#32409)
* Move calculate_fee() to fee.rs

* fixup code for compilation
2023-07-14 20:17:38 -07:00
behzad nouri cfb028819a
deprecates Signature::new in favor of Signature::{try_,}from (#32481) 2023-07-14 22:51:12 +00:00
Brooks 78230050fb
Fixes measure in log when calculating accounts hash (#32502) 2023-07-14 17:02:14 -04:00