iter_memory_pair_chunks was iterating regions in reverse, but not memory
_within_ regions in reverse.
This commit fixes the issue and simplifies the implementation by removing
nested loops which made control flow hard to reason about.
If the vector holding an account is reallocated during execution of a callee,
we must zero the spare capacity regardless of whether the account size changed,
because the underlying vector might contain uninitialized memory in the spare
capacity.
reserve(additional) reserves additional bytes on top of the current _length_
not capacity. Before this fix we could potentially reserve less capacity than
required.
* updated inner_call for hashing fn to follow generic approach
* different hash compute budget values for all digests
* fixed conflicts
* reverted changes to compute_budget.rs and added 3method to trait to get compute budget values
* updated type for result fn for HasherImpl
* using Hash directly in result fn, got rid of HASH_BYTES and removed comment form compute_budget
* updated import statement
* cargo fmt -all
* removed unused import and reference related warning
* oops forgot semicolon
* removed trailing white space
* solana-program - altbn128: add g1 & g2 compression
still fixing tests for point of infinity
feat: proof compression syscall working
add rust test to ci
remove prints
added c test
added sycall pricing
* fixed ci checks
* refactored altbn128 and compression
* Do not unnecessarily re-verify unloaded program
* clippy fixes
* new unit test for extract
* fixes after rebase
* fixes after rebase
* address review comments
* fix clippy
* allow pedantic invalid cast lint
* allow lint with false-positive triggered by `test-case` crate
* nightly `fmt` correction
* adapt to rust layout changes
* remove dubious test
* Use transmute instead of pointer cast and de/ref when check_aligned is false.
* Renames clippy::integer_arithmetic to clippy::arithmetic_side_effects.
* bump rust nightly to 2023-08-25
* Upgrades Rust to 1.72.0
---------
Co-authored-by: Trent Nelson <trent@solana.com>
Computing Poseidon[0] hashes is too expensive to be done in a Solana
program in one transaction. Poseidon is a zero-knowlege proof friendly
hash function, used by the majority of ZK-based projects, including the
ones built on top of Solana.
This change introduces the `sol_poseidon` syscall which takes 2D byte
slice as an input and then calculates a Poseidon hash using a BN254
curve and the following Poseidon parameters:
* x^5 S-boxes
* width - 2 ≤ t ≤ 13
* inputs - 1 ≤ n ≤ 12
* 8 full rounds and partial rounds depending on t: [56, 57, 56, 60, 60,
63, 64, 63, 60, 66, 60, 65]
Computation of Poseidon hashes is done with the light-poseidon[1]
crate, which is audited[2] and compatible with Circom[3] (BN254 curve,
the same parameters and constants).
Proposed compute costs depend on number of inputs and are based on
light-poseidon benchmarks[4].
[0] https://www.poseidon-hash.info/
[1] https://crates.io/crates/light-poseidon
[2] https://github.com/Lightprotocol/light-poseidon/blob/main/assets/audit.pdf
[3] https://docs.circom.io/
[4] https://github.com/Lightprotocol/light-poseidon/tree/main#performance
* transaction_context: update make_data_mut comment
* bpf_loader: cpi: pass SerializeAccountMetadata to CallerAccount::from*
We now have a way to provide CallerAccount with trusted values coming
from our internal serialization code and not from untrusted vm space
* bpf_loader: direct_mapping: enforce account info pointers to be immutable
When direct mapping is enabled, we might need to update account data
memory regions across CPI calls. Since the only way we have to retrieve
the regions is based on their vm addresses, we enforce vm addresses to
be stable. Accounts can still be mutated and resized of course, but it
must be done in place.
This also locks all other AccountInfo pointers, since there's no legitimate
reason to make them point to anything else.
* bpf_loader: cpi: access ref_to_len_in_vm through VmValue
Direct mapping needs to translate vm values at each access since
permissions of the underlying memory might have changed.
* direct mapping: improve memory permission tracking across CPI calls
Ensure that the data and realloc regions of an account always track the
account's permissions. In order to do this, we also need to split
realloc regions in their own self contained regions, where before we
had:
[account fields][account data][account realloc + more account fields + next account fields][next account data][...]
we now have:
[account fields][account data][account realloc][more account fields + next account fields][next account data][...]
Tested in TEST_[FORBID|ALLOW]_WRITE_AFTER_OWNERSHIP_CHANGE*
Additionally when direct mapping is on, we must update all perms at once before
doing account data updates. Otherwise, updating an account might write into
another account whose perms we haven't updated yet. Tested in
TEST_FORBID_LEN_UPDATE_AFTER_OWNERSHIP_CHANGE.
* bpf_loader: serialization: address review comment don't return vm_addr from push_account_region
* bpf_loader: rename push_account_region to push_account_data_region
* cpi: fix slow edge case zeroing extra account capacity after shrinking an account
When returning from CPI we need to zero all the account memory up to the
original length only if we know we're potentially dealing with uninitialized
memory.
When we know that the spare capacity has deterministic content, we only need to
zero new_len..prev_len.
This fixes a slow edge case that was triggerable by the following scenario:
- load a large account (say 10MB) into the vm
- shrink to 10 bytes - would memset 10..10MB
- shrink to 9 bytes - would memset 9..10MB
- shrink to 8 bytes - would memset 8..10MB
- ...
Now instead in the scenario above the following will happen:
- load a large account (say 10MB) into the vm
- shrink to 10 bytes - memsets 10..10MB
- shrink to 9 bytes - memsets 9..10
- shrink to 8 bytes - memset 8..9
- ...
* bpf_loader: add account_data_region_memory_state helper
Shared between serialization and CPI to figure out the MemoryState of an
account.
* cpi: direct_mapping: error out if ref_to_len_in_vm points to account memory
If ref_to_len_in_vm is allowed to be in account memory, calles could mutate it,
essentially letting callees directly mutate callers memory.
* bpf_loader: direct_mapping: map AccessViolation -> InstructionError
Return the proper ReadonlyDataModified / ExecutableDataModified /
ExternalAccountDataModified depending on where the violation occurs
* bpf_loader: cpi: remove unnecessary infallible slice::get call
* dereplicode address alignment check
* Uses `checked_div` and `checked_rem` in built-in loaders.
* Uses `checked_div` and `checked_rem`.
* sdk: replace sub() with saturating_sub()
* eliminate `String` "arithmetic"
* allow arithmetic side-effects in tests and benches and on types we don't control
---------
Co-authored-by: Trent Nelson <trent@solana.com>
* sdk: Add concurrent support for rand 0.7 and 0.8
* Update rand, rand_chacha, and getrandom versions
* Run command to replace `gen_range`
Run `git grep -l gen_range | xargs sed -i'' -e 's/gen_range(\(\S*\), /gen_range(\1../'
* sdk: Fix users of older `gen_range`
* Replace `hash::new_rand` with `hash::new_with_thread_rng`
Run:
```
git grep -l hash::new_rand | xargs sed -i'' -e 's/hash::new_rand([^)]*/hash::new_with_thread_rng(/'
```
* perf: Use `Keypair::new()` instead of `generate`
* Use older rand version in zk-token-sdk
* program-runtime: Inline random key generation
* bloom: Fix clippy warnings in tests
* streamer: Scope rng usage correctly
* perf: Fix clippy warning
* accounts-db: Map to char to generate a random string
* Remove `from_secret_key_bytes`, it's just `keypair_from_seed`
* ledger: Generate keypairs by hand
* ed25519-tests: Use new rand
* runtime: Use new rand in all tests
* gossip: Clean up clippy and inline keypair generators
* core: Inline keypair generation for tests
* Push sbf lockfile change
* sdk: Sort dependencies correctly
* Remove `hash::new_with_thread_rng`, use `Hash::new_unique()`
* Use Keypair::new where chacha isn't used
* sdk: Fix build by marking rand 0.7 optional
* Hardcode secret key length, add static assertion
* Unify `getrandom` crate usage to fix linking errors
* bloom: Fix tests that require a random hash
* Remove some dependencies, try to unify others
* Remove unnecessary uses of rand and rand_core
* Update lockfiles
* Add back some dependencies to reduce rebuilds
* Increase max rebuilds from 14 to 15
* frozen-abi: Remove `getrandom`
* Bump rebuilds to 17
* Remove getrandom from zk-token-proof
* Integrate program loader-v4 with bank
* fix tests
* new struct for ProgramRuntimeEnvironments
* remove environment from program_runtime_environment_v
* move find_program_in_cache() to invoke_context
* cleanup
* bpf_loader: move computing original account lengths inside serialize_paramters_(aligned|unaligned)
This is in preparation of returning more than just the original length
* bpf_loader: deserialize*: take original lens as an iterator instead of a slice
This is in preparation of extracting account lenghts from a larger
context
* bpf_loader: introduce SerializedAccountMetadata
Instead of passing original_account_lengths around as Vec<usize>,
introduce an explicit type that includes the length and soon more.
* add sysvar and logic for last restart slot
* cleanup
* add test for getting last restart slot from account
* format code
* add some basic rustdoc
* copy+paste error
* feature flag for last_restart_slot
* add to sysvars.md
* updated wording in sysvars.md
* rename sol_get_last_restart_slot_sysvar > sol_get_last_restart_slot
* create sbf C header for sol_get_last_restart_slot
* cleanup imports
* reverted hardened_unpack workaround
* cleanup imports
* cleanup logs + blank lines
* Implementing ui changes for last restart slot, nit
* Some more nit change and implementing the UI for sysvar
* fixing the CI
* Minor clippy fix
* format changes
* changes suggested by mvines and lichtso
* increase timeout in local_cluster test
* fix code format
* use keypair for feature flag from mvines
* delete test.json file
* Revert "increase timeout in local_cluster test"
This reverts commit a67465ae22.
* last restart slot should be always less than or equal to current slot
* fixing bug
* changes after steviez comments
* format issue fixed
* fixing the comment on premature application of future hardfork
* nit change in test
Co-authored-by: steviez <steven@solana.com>
* reverting sysvar_cache.rs because change was not necessary
---------
Co-authored-by: steve-gg <grooviegermanikus@gmail.com>
Co-authored-by: steviez <steven@solana.com>
Andrew Fitzgerald
Alessandro Decina
Alexander Meißner
Justin Starry
Brooks
Justin Starry
Sanjay Singh
Tao Zhu
ananas-block
Christian Kamm
Pankaj Garg
behzad nouri
vadorovsky
Jon Cinque
HaoranYi
ripatel-fd
samkim-crypto
galactus