Commit Graph

329 Commits

Author SHA1 Message Date
Alexander Meißner 8033be333e
Refactor - `LoadedPrograms` (#33482)
* Adds type ProgramRuntimeEnvironment.

* Moves LoadedPrograms::remove_expired_entries() into LoadedPrograms::prune().

* Adds Stats::prunes_environment and renames Stats::prunes_orphan and Stats::prunes_expired.

* Adds LoadedPrograms::latest_root_epoch.

* Typo fix, authored-by: Dmitri Makarov <dmakarov@users.noreply.github.com>
2023-10-02 19:01:23 +02:00
Tao Zhu 57e78a16dc
heap_size type to be consistent with request instruction (#33354)
* heap_size type to be consistent with request instruction

* update tests
2023-09-25 13:11:26 -05:00
ananas-block 997aa0a3f8
Feat(syscall): add altbn128 g1 & g2 compression (#32870)
* solana-program - altbn128: add g1 & g2 compression

still fixing tests for point of infinity

feat: proof compression syscall working

add rust test to ci

remove prints

added c test

added sycall pricing

* fixed ci checks

* refactored altbn128 and compression
2023-09-25 05:43:34 -07:00
Tao Zhu a2ad820309
compute_budget heap_size does not have to be optional (#33313) 2023-09-21 09:24:47 -05:00
Tao Zhu 8b8a21a52f
cleanup feature: enable request heap frame instruction #30076 (#33243)
* cleanup feature: enable request heap frame instruction #30076

* update sbf tests

* removed out dated comments and test
2023-09-18 16:06:24 -05:00
Christian Kamm 525e59f01a
sdk, programs/bpf_loader: add sol_remaining_compute_units syscall (#31640)
bpf_loader: add sol_remaining_compute_units syscall

Co-authored-by: jonch <9093549+jon-chuang@users.noreply.github.com>
2023-09-13 08:57:08 -06:00
Pankaj Garg 5562f79cc5
Do not unnecessarily re-verify unloaded program (#32722)
* Do not unnecessarily re-verify unloaded program

* clippy fixes

* new unit test for extract

* fixes after rebase

* fixes after rebase

* address review comments

* fix clippy
2023-09-13 06:25:56 -07:00
Alexander Meißner 424666e341
Bump solana_rbpf to v0.7.0 (#33104)
* Upgrades RBPF in Cargo.toml

* Adjustments to updated interfaces.
2023-09-06 10:54:15 +02:00
Alexander Meißner 9e703f85de
Upgrades Rust to 1.72.0 & nightly-2023-08-25 (#32961)
* allow pedantic invalid cast lint

* allow lint with false-positive triggered by `test-case` crate

* nightly `fmt` correction

* adapt to rust layout changes

* remove dubious test

* Use transmute instead of pointer cast and de/ref when check_aligned is false.

* Renames clippy::integer_arithmetic to clippy::arithmetic_side_effects.

* bump rust nightly to 2023-08-25

* Upgrades Rust to 1.72.0

---------

Co-authored-by: Trent Nelson <trent@solana.com>
2023-09-01 07:26:13 +00:00
behzad nouri 4ec5ea6f7b
replaces assert!(matches!(...)) with assert_matches!(...) (#33068)
assert_matches!(...) provides more informative error message when it
fails and it is part of nightly rust:
https://doc.rust-lang.org/std/assert_matches/macro.assert_matches.html
2023-08-30 13:48:27 -04:00
vadorovsky e6dc3dac45
feat(sbf): Add Poseidon syscall (#32680)
Computing Poseidon[0] hashes is too expensive to be done in a Solana
program in one transaction. Poseidon is a zero-knowlege proof friendly
hash function, used by the majority of ZK-based projects, including the
ones built on top of Solana.

This change introduces the `sol_poseidon` syscall which takes 2D byte
slice as an input and then calculates a Poseidon hash using a BN254
curve and the following Poseidon parameters:

* x^5 S-boxes
* width - 2 ≤ t ≤ 13
* inputs - 1 ≤ n ≤ 12
* 8 full rounds and partial rounds depending on t: [56, 57, 56, 60, 60,
  63, 64, 63, 60, 66, 60, 65]

Computation of Poseidon hashes is done with the light-poseidon[1]
crate, which is audited[2] and compatible with Circom[3] (BN254 curve,
the same parameters and constants).

Proposed compute costs depend on number of inputs and are based on
light-poseidon benchmarks[4].

[0] https://www.poseidon-hash.info/
[1] https://crates.io/crates/light-poseidon
[2] https://github.com/Lightprotocol/light-poseidon/blob/main/assets/audit.pdf
[3] https://docs.circom.io/
[4] https://github.com/Lightprotocol/light-poseidon/tree/main#performance
2023-08-30 09:43:31 -07:00
Alessandro Decina 0f41719918
direct mapping: misc fixes (#32649)
* transaction_context: update make_data_mut comment

* bpf_loader: cpi: pass SerializeAccountMetadata to CallerAccount::from*

We now have a way to provide CallerAccount with trusted values coming
from our internal serialization code and not from untrusted vm space

* bpf_loader: direct_mapping: enforce account info pointers to be immutable

When direct mapping is enabled, we might need to update account data
memory regions across CPI calls. Since the only way we have to retrieve
the regions is based on their vm addresses, we enforce vm addresses to
be stable.  Accounts can still be mutated and resized of course, but it
must be done in place.

This also locks all other AccountInfo pointers, since there's no legitimate
reason to make them point to anything else.

* bpf_loader: cpi: access ref_to_len_in_vm through VmValue

Direct mapping needs to translate vm values at each access since
permissions of the underlying memory might have changed.

* direct mapping: improve memory permission tracking across CPI calls

Ensure that the data and realloc regions of an account always track the
account's permissions. In order to do this, we also need to split
realloc regions in their own self contained regions, where before we
had:

[account fields][account data][account realloc + more account fields + next account fields][next account data][...]

we now have:

[account fields][account data][account realloc][more account fields + next account fields][next account data][...]

Tested in TEST_[FORBID|ALLOW]_WRITE_AFTER_OWNERSHIP_CHANGE*

Additionally when direct mapping is on, we must update all perms at once before
doing account data updates. Otherwise, updating an account might write into
another account whose perms we haven't updated yet. Tested in
TEST_FORBID_LEN_UPDATE_AFTER_OWNERSHIP_CHANGE.

* bpf_loader: serialization: address review comment don't return vm_addr from push_account_region

* bpf_loader: rename push_account_region to push_account_data_region

* cpi: fix slow edge case zeroing extra account capacity after shrinking an account

When returning from CPI we need to zero all the account memory up to the
original length only if we know we're potentially dealing with uninitialized
memory.

When we know that the spare capacity has deterministic content, we only need to
zero new_len..prev_len.

This fixes a slow edge case that was triggerable by the following scenario:

- load a large account (say 10MB) into the vm
- shrink to 10 bytes - would memset 10..10MB
- shrink to 9 bytes - would memset 9..10MB
- shrink to 8 bytes - would memset 8..10MB
- ...

Now instead in the scenario above the following will happen:

- load a large account (say 10MB) into the vm
- shrink to 10 bytes - memsets 10..10MB
- shrink to 9 bytes - memsets 9..10
- shrink to 8 bytes - memset 8..9
- ...

* bpf_loader: add account_data_region_memory_state helper

Shared between serialization and CPI to figure out the MemoryState of an
account.

* cpi: direct_mapping: error out if ref_to_len_in_vm points to account memory

If ref_to_len_in_vm is allowed to be in account memory, calles could mutate it,
essentially letting callees directly mutate callers memory.

* bpf_loader: direct_mapping: map AccessViolation -> InstructionError

Return the proper ReadonlyDataModified / ExecutableDataModified /
ExternalAccountDataModified depending on where the violation occurs

* bpf_loader: cpi: remove unnecessary infallible slice::get call
2023-08-30 16:57:24 +07:00
Alexander Meißner 150a798d32
Fix - Upcoming `arithmetic_side_effects` lints (#33000)
* dereplicode address alignment check

* Uses `checked_div` and `checked_rem` in built-in loaders.

* Uses `checked_div` and `checked_rem`.

* sdk: replace sub() with saturating_sub()

* eliminate `String` "arithmetic"

* allow arithmetic side-effects in tests and benches and on types we don't control

---------

Co-authored-by: Trent Nelson <trent@solana.com>
2023-08-29 20:58:53 +02:00
Pankaj Garg e036a0375d
Handle closing of the program in the same slot as deployment (#33025) 2023-08-28 09:05:49 -07:00
Ryo Onodera d3fb54e0b2
Remove From<LogCollector> for Vec<String> (#33012) 2023-08-27 13:10:02 +09:00
Pankaj Garg dbe4017143
Prune programs deployed in duplicate unconfirmed slot (#32999)
* Prune programs deployed in duplicate unconfirmed slot

* unit test
2023-08-25 11:02:20 -07:00
Alexander Meißner 29bbda0c11
General cleanup (Part 2) (#32998)
* bucket-map: remove unused `mut`

* remove unnecessary explicit `into_iter()` calls

* don't default-construct unit-structs

* prefer `or_default()` to `or_insert_with(T::default())`

* replace "slow zero-filled vec initialization"

---------

Co-authored-by: Trent Nelson <trent@solana.com>
2023-08-25 19:07:38 +02:00
Alexander Meißner 3bd14cc1e1
Fix - Reordering in second level index of LoadedPrograms (#32973)
Replaces "swap_remove" => "remove".
2023-08-24 16:41:04 +02:00
Jon Cinque 0fe902ced7
Bump rand to 0.8, rand_chacha to 0.3, getrandom to 0.2 (#32871)
* sdk: Add concurrent support for rand 0.7 and 0.8

* Update rand, rand_chacha, and getrandom versions

* Run command to replace `gen_range`

Run `git grep -l gen_range | xargs sed -i'' -e 's/gen_range(\(\S*\), /gen_range(\1../'

* sdk: Fix users of older `gen_range`

* Replace `hash::new_rand` with `hash::new_with_thread_rng`

Run:
```
git grep -l hash::new_rand | xargs sed -i'' -e 's/hash::new_rand([^)]*/hash::new_with_thread_rng(/'
```

* perf: Use `Keypair::new()` instead of `generate`

* Use older rand version in zk-token-sdk

* program-runtime: Inline random key generation

* bloom: Fix clippy warnings in tests

* streamer: Scope rng usage correctly

* perf: Fix clippy warning

* accounts-db: Map to char to generate a random string

* Remove `from_secret_key_bytes`, it's just `keypair_from_seed`

* ledger: Generate keypairs by hand

* ed25519-tests: Use new rand

* runtime: Use new rand in all tests

* gossip: Clean up clippy and inline keypair generators

* core: Inline keypair generation for tests

* Push sbf lockfile change

* sdk: Sort dependencies correctly

* Remove `hash::new_with_thread_rng`, use `Hash::new_unique()`

* Use Keypair::new where chacha isn't used

* sdk: Fix build by marking rand 0.7 optional

* Hardcode secret key length, add static assertion

* Unify `getrandom` crate usage to fix linking errors

* bloom: Fix tests that require a random hash

* Remove some dependencies, try to unify others

* Remove unnecessary uses of rand and rand_core

* Update lockfiles

* Add back some dependencies to reduce rebuilds

* Increase max rebuilds from 14 to 15

* frozen-abi: Remove `getrandom`

* Bump rebuilds to 17

* Remove getrandom from zk-token-proof
2023-08-21 19:11:21 +02:00
Alexander Meißner be6d6fa5ae
Refactor - `LoadedPrograms::get_entries_sorted_by_tx_usage()` (#32874)
Factor get_entries_sorted_by_tx_usage() out of sort_and_unload().
2023-08-17 19:14:32 +02:00
Pankaj Garg c17b938204
Integrate program loader-v4 with bank (#32832)
* Integrate program loader-v4 with bank

* fix tests

* new struct for ProgramRuntimeEnvironments

* remove environment from program_runtime_environment_v

* move find_program_in_cache() to invoke_context

* cleanup
2023-08-16 10:50:23 -07:00
Pankaj Garg a1a0829a8b
Initialize and store PRv2 environment in cache (#32812)
* Initialize and store PRv2 environment in cache

* address review comments
2023-08-11 18:58:22 +00:00
Jon Cinque 8c14886891
sdk: Limited Borsh 0.9 support (Pubkey and helpers) (#32511)
* sdk: Implement Borsh 0.9 traits on Pubkey

* Alphabetize cargo.toml

* Add backwards-compatible borsh file

* Add borsh0_10.rs for more clarity

* Deprecate `borsh` utils, use borsh0_10 everywhere

* Mark borsh 0.9 helpers as deprecated

* Add macros for deriving helper impls

* Add borsh 0.9 tests

* Refactor tests into macro
2023-08-02 23:15:24 +02:00
Alessandro Decina e3f253d559
introduce SerializedAccountMetadata (#32644)
* bpf_loader: move computing original account lengths inside serialize_paramters_(aligned|unaligned)

This is in preparation of returning more than just the original length

* bpf_loader: deserialize*: take original lens as an iterator instead of a slice

This is in preparation of extracting account lenghts from a larger
context

* bpf_loader: introduce SerializedAccountMetadata

Instead of passing original_account_lengths around as Vec<usize>,
introduce an explicit type that includes the length and soon more.
2023-07-28 18:34:27 +07:00
Tao Zhu 62b9fcfa76
cleanup feature: Use default units per instruction in fee calculation (#32570)
* cleanup feature: Use default units per instruction in fee calculation
2023-07-21 18:37:58 -05:00
Andrew Fitzgerald a7eda70ddb
Preliminary Changes for Updating Cargo (#32547) 2023-07-21 13:43:00 -07:00
Will Hickey 83ac15dd97
Remove inappropriate use of usize. Refactor upcast. (#32539)
* Remove inappropriate use of usize. Refactor upcast.
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
2023-07-20 14:53:08 -05:00
Tao Zhu c69bc00f69
cost model could double count builtin instruction cost (#32422)
1. add tests to demo builtin cost could be double counted;
2. quick fix for now
2023-07-17 15:50:13 -05:00
Pankaj Garg 7177dd9809
Remove calculate_fee()'s dependency on ComputeBudget (#32487) 2023-07-14 12:25:04 -07:00
Ryo Onodera d80745cfaf
dcou: BorrowedAccount::set_data() (#32424)
* dcou: BorrowedAccount::set_data()

* Properly depend on dev-context-only-utils
2023-07-13 22:44:08 +09:00
behzad nouri d54b6204be
removes instances of clippy::manual_let_else (#32417) 2023-07-09 21:41:36 +00:00
Alexander Meißner 9731331be4
Bump solana_rbpf to v0.6.0 (#32313) 2023-07-05 19:46:21 +02:00
Pankaj Garg 089fb464a9
Move message_processor to program-runtime (#32268) 2023-06-24 15:55:39 -07:00
HaoranYi 203544293b
Add epoch_rewards to sysvar cache (#32155)
add epoch_rewards to sysvar cache

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-06-20 15:01:34 -05:00
Tyera 748223c928
Use workspace serde dependency in program-runtime (#32196)
Use workspace serde dependency
2023-06-19 19:36:05 +00:00
galactus 2ceabd9368
Simd 47 syscall sysvar last restart slot (#31957)
* add sysvar and logic for last restart slot

* cleanup

* add test for getting last restart slot from account

* format code

* add some basic rustdoc

* copy+paste error

* feature flag for last_restart_slot

* add to sysvars.md

* updated wording in sysvars.md

* rename sol_get_last_restart_slot_sysvar > sol_get_last_restart_slot

* create sbf C header for sol_get_last_restart_slot

* cleanup imports

* reverted hardened_unpack workaround

* cleanup imports

* cleanup logs + blank lines

* Implementing ui changes for last restart slot, nit

* Some more nit change and implementing the UI for sysvar

* fixing the CI

* Minor clippy fix

* format changes

* changes suggested by mvines and lichtso

* increase timeout in local_cluster test

* fix code format

* use keypair for feature flag from mvines

* delete test.json file

* Revert "increase timeout in local_cluster test"

This reverts commit a67465ae22.

* last restart slot should be always less than or equal to current slot

* fixing bug

* changes after  steviez comments

* format issue fixed

* fixing the comment on premature application of future hardfork

* nit change in test

Co-authored-by: steviez <steven@solana.com>

* reverting sysvar_cache.rs because change was not necessary

---------

Co-authored-by: steve-gg <grooviegermanikus@gmail.com>
Co-authored-by: steviez <steven@solana.com>
2023-06-16 20:14:02 +00:00
Pankaj Garg c86e160438
Add runtime environment to FailedVerification tombstones (#31991)
* Add runtime environment to FailedVerification tombstones

* modify default variant

* prune update

* add DelayVisibility in prune_feature_set_transition
2023-06-06 22:24:39 +00:00
Alexander Meißner 89207a3fe5
Feature - Prune on feature set transition (#31945)
* Call create_program_runtime_environment() in Bank::apply_builtin_program_feature_transitions().

* Adds LoadedPrograms::prune_feature_set_transition().
2023-06-06 17:49:02 +02:00
Pankaj Garg 1b30de4f32
Retain runtime environment config for unloaded programs (#31953)
* Retain runtime environment config for unloaded programs
2023-06-03 09:38:11 +02:00
Pankaj Garg 37ebb709e6
Evict program cache using transaction usage counter (#31927)
* Evict program cache using tx usage counter

* address review feedback

* address review feedback
2023-06-02 11:26:56 -07:00
Alexander Meißner 3792204598
Bump solana_rbpf to v0.5.0 (#31943)
* Renames RBPF BuiltIn to Builtin.

* Bump solana_rbpf to v0.5.0
2023-06-02 16:34:01 +02:00
Pankaj Garg 29d7a57cce
Reset program cache stats after submit (#31929) 2023-06-02 01:16:49 -07:00
Pankaj Garg 29904bb06b
More metrics for LoadedPrograms cache (#31735) 2023-05-22 07:53:54 -07:00
Alexander Meißner 83f692ce67
Refactor - Share RBPF config and syscall registry globally (#31727)
* Adds LoadedPrograms::program_runtime_environment_v1.

* No need to explicitly remove precompiles from program_accounts_map.
2023-05-20 20:28:00 +02:00
Pankaj Garg 8fbe471cc8
Remove special handling of builtin while pruning (#31719) 2023-05-19 09:35:48 -07:00
Pankaj Garg e6d490708a
Do not overwrite builtin program in the same slot (#31718) 2023-05-19 05:26:31 -07:00
Trent Nelson a0850a7501
runtime: rename executor cache datapoint (#31716) 2023-05-18 20:37:01 +00:00
Alexander Meißner 520c647918
Refactor - Merges `BuiltinPrograms` into `LoadedPrograms` (#31654)
* Merges BuiltinPrograms into LoadedPrograms.

* Prevents built-ins from being pruned.
2023-05-18 22:18:28 +02:00
Pankaj Garg b03422968b
Remove Rc and RefCell to carry program cache in invoke_context (#31684) 2023-05-17 06:10:50 -07:00
Alexander Meißner 5c8b5a2a68
Bump solana_rbpf to v0.4.0 (#31594)
* Moves "disable_deploy_of_alloc_free_syscall" parameter inside create_loader().

* Removes the "is_abi_v2" flag.

* Bumps solana_rbpf to v0.4.0
2023-05-12 18:07:13 +02:00