solana/sdk/src/transaction_context.rs

1239 lines
46 KiB
Rust
Raw Normal View History

//! Data shared between program runtime and built-in programs as well as SBF programs.
#![deny(clippy::indexing_slicing)]
#[cfg(all(not(target_os = "solana"), debug_assertions))]
use crate::signature::Signature;
#[cfg(not(target_os = "solana"))]
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
use {
crate::{
account::WritableAccount,
rent::Rent,
system_instruction::{
MAX_PERMITTED_ACCOUNTS_DATA_ALLOCATIONS_PER_TRANSACTION, MAX_PERMITTED_DATA_LENGTH,
},
},
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
solana_program::entrypoint::MAX_PERMITTED_DATA_INCREASE,
std::mem::MaybeUninit,
};
use {
crate::{
account::{AccountSharedData, ReadableAccount},
instruction::InstructionError,
pubkey::Pubkey,
},
std::{
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
cell::{Ref, RefCell, RefMut},
collections::HashSet,
pin::Pin,
rc::Rc,
},
};
/// Index of an account inside of the TransactionContext or an InstructionContext.
pub type IndexOfAccount = u16;
/// Contains account meta data which varies between instruction.
///
/// It also contains indices to other structures for faster lookup.
#[derive(Clone, Debug, Eq, PartialEq)]
pub struct InstructionAccount {
/// Points to the account and its key in the `TransactionContext`
pub index_in_transaction: IndexOfAccount,
/// Points to the first occurrence in the parent `InstructionContext`
///
/// This excludes the program accounts.
pub index_in_caller: IndexOfAccount,
/// Points to the first occurrence in the current `InstructionContext`
///
/// This excludes the program accounts.
pub index_in_callee: IndexOfAccount,
/// Is this account supposed to sign
pub is_signer: bool,
/// Is this account allowed to become writable
pub is_writable: bool,
}
/// An account key and the matching account
pub type TransactionAccount = (Pubkey, AccountSharedData);
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
#[derive(Clone, Debug, PartialEq)]
pub struct TransactionAccounts {
accounts: Vec<RefCell<AccountSharedData>>,
touched_flags: RefCell<Box<[bool]>>,
is_early_verification_of_account_modifications_enabled: bool,
}
impl TransactionAccounts {
#[cfg(not(target_os = "solana"))]
fn new(
accounts: Vec<RefCell<AccountSharedData>>,
is_early_verification_of_account_modifications_enabled: bool,
) -> TransactionAccounts {
TransactionAccounts {
touched_flags: RefCell::new(vec![false; accounts.len()].into_boxed_slice()),
accounts,
is_early_verification_of_account_modifications_enabled,
}
}
fn len(&self) -> usize {
self.accounts.len()
}
pub fn get(&self, index: IndexOfAccount) -> Option<&RefCell<AccountSharedData>> {
self.accounts.get(index as usize)
}
#[cfg(not(target_os = "solana"))]
pub fn touch(&self, index: IndexOfAccount) -> Result<(), InstructionError> {
if self.is_early_verification_of_account_modifications_enabled {
*self
.touched_flags
.borrow_mut()
.get_mut(index as usize)
.ok_or(InstructionError::NotEnoughAccountKeys)? = true;
}
Ok(())
}
#[cfg(not(target_os = "solana"))]
pub fn touched_count(&self) -> usize {
self.touched_flags
.borrow()
.iter()
.fold(0usize, |accumulator, was_touched| {
accumulator.saturating_add(*was_touched as usize)
})
}
pub fn try_borrow(
&self,
index: IndexOfAccount,
) -> Result<Ref<'_, AccountSharedData>, InstructionError> {
self.accounts
.get(index as usize)
.ok_or(InstructionError::MissingAccount)?
.try_borrow()
.map_err(|_| InstructionError::AccountBorrowFailed)
}
pub fn try_borrow_mut(
&self,
index: IndexOfAccount,
) -> Result<RefMut<'_, AccountSharedData>, InstructionError> {
self.accounts
.get(index as usize)
.ok_or(InstructionError::MissingAccount)?
.try_borrow_mut()
.map_err(|_| InstructionError::AccountBorrowFailed)
}
pub fn into_accounts(self) -> Vec<AccountSharedData> {
self.accounts
.into_iter()
.map(|account| account.into_inner())
.collect()
}
}
/// Loaded transaction shared between runtime and programs.
///
/// This context is valid for the entire duration of a transaction being processed.
#[derive(Debug, Clone, PartialEq)]
pub struct TransactionContext {
account_keys: Pin<Box<[Pubkey]>>,
accounts: Rc<TransactionAccounts>,
instruction_stack_capacity: usize,
instruction_trace_capacity: usize,
instruction_stack: Vec<usize>,
instruction_trace: Vec<InstructionContext>,
return_data: TransactionReturnData,
accounts_resize_delta: RefCell<i64>,
#[cfg(not(target_os = "solana"))]
rent: Option<Rent>,
#[cfg(not(target_os = "solana"))]
is_cap_accounts_data_allocations_per_transaction_enabled: bool,
/// Useful for debugging to filter by or to look it up on the explorer
#[cfg(all(not(target_os = "solana"), debug_assertions))]
signature: Signature,
}
impl TransactionContext {
/// Constructs a new TransactionContext
#[cfg(not(target_os = "solana"))]
pub fn new(
transaction_accounts: Vec<TransactionAccount>,
rent: Option<Rent>,
instruction_stack_capacity: usize,
instruction_trace_capacity: usize,
) -> Self {
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
let (account_keys, accounts): (Vec<_>, Vec<_>) = transaction_accounts
.into_iter()
.map(|(key, account)| (key, RefCell::new(account)))
.unzip();
Self {
account_keys: Pin::new(account_keys.into_boxed_slice()),
accounts: Rc::new(TransactionAccounts::new(accounts, rent.is_some())),
instruction_stack_capacity,
instruction_trace_capacity,
instruction_stack: Vec::with_capacity(instruction_stack_capacity),
instruction_trace: vec![InstructionContext::default()],
return_data: TransactionReturnData::default(),
accounts_resize_delta: RefCell::new(0),
rent,
is_cap_accounts_data_allocations_per_transaction_enabled: false,
#[cfg(all(not(target_os = "solana"), debug_assertions))]
signature: Signature::default(),
}
}
/// Used in mock_process_instruction
#[cfg(not(target_os = "solana"))]
pub fn deconstruct_without_keys(self) -> Result<Vec<AccountSharedData>, InstructionError> {
if !self.instruction_stack.is_empty() {
return Err(InstructionError::CallDepth);
}
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
Ok(Rc::try_unwrap(self.accounts)
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
.expect("transaction_context.accounts has unexpected outstanding refs")
.into_accounts())
}
#[cfg(not(target_os = "solana"))]
pub fn accounts(&self) -> &Rc<TransactionAccounts> {
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
&self.accounts
}
/// Returns true if `enable_early_verification_of_account_modifications` is active
#[cfg(not(target_os = "solana"))]
pub fn is_early_verification_of_account_modifications_enabled(&self) -> bool {
self.rent.is_some()
}
/// Stores the signature of the current transaction
#[cfg(all(not(target_os = "solana"), debug_assertions))]
pub fn set_signature(&mut self, signature: &Signature) {
self.signature = *signature;
}
/// Returns the signature of the current transaction
#[cfg(all(not(target_os = "solana"), debug_assertions))]
pub fn get_signature(&self) -> &Signature {
&self.signature
}
/// Returns the total number of accounts loaded in this Transaction
pub fn get_number_of_accounts(&self) -> IndexOfAccount {
self.accounts.len() as IndexOfAccount
}
/// Searches for an account by its key
pub fn get_key_of_account_at_index(
&self,
index_in_transaction: IndexOfAccount,
) -> Result<&Pubkey, InstructionError> {
self.account_keys
.get(index_in_transaction as usize)
.ok_or(InstructionError::NotEnoughAccountKeys)
}
/// Searches for an account by its key
#[cfg(not(target_os = "solana"))]
pub fn get_account_at_index(
&self,
index_in_transaction: IndexOfAccount,
) -> Result<&RefCell<AccountSharedData>, InstructionError> {
self.accounts
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
.get(index_in_transaction)
.ok_or(InstructionError::NotEnoughAccountKeys)
}
/// Searches for an account by its key
pub fn find_index_of_account(&self, pubkey: &Pubkey) -> Option<IndexOfAccount> {
self.account_keys
.iter()
.position(|key| key == pubkey)
.map(|index| index as IndexOfAccount)
}
/// Searches for a program account by its key
pub fn find_index_of_program_account(&self, pubkey: &Pubkey) -> Option<IndexOfAccount> {
self.account_keys
.iter()
.rposition(|key| key == pubkey)
.map(|index| index as IndexOfAccount)
}
/// Gets the max length of the InstructionContext trace
pub fn get_instruction_trace_capacity(&self) -> usize {
self.instruction_trace_capacity
}
/// Returns the instruction trace length.
///
/// Not counting the last empty InstructionContext which is always pre-reserved for the next instruction.
/// See also `get_next_instruction_context()`.
pub fn get_instruction_trace_length(&self) -> usize {
self.instruction_trace.len().saturating_sub(1)
}
/// Gets an InstructionContext by its index in the trace
pub fn get_instruction_context_at_index_in_trace(
&self,
index_in_trace: usize,
) -> Result<&InstructionContext, InstructionError> {
self.instruction_trace
.get(index_in_trace)
.ok_or(InstructionError::CallDepth)
}
2022-04-12 17:52:47 -07:00
/// Gets an InstructionContext by its nesting level in the stack
pub fn get_instruction_context_at_nesting_level(
&self,
nesting_level: usize,
2022-04-12 17:52:47 -07:00
) -> Result<&InstructionContext, InstructionError> {
let index_in_trace = *self
.instruction_stack
.get(nesting_level)
.ok_or(InstructionError::CallDepth)?;
let instruction_context = self.get_instruction_context_at_index_in_trace(index_in_trace)?;
debug_assert_eq!(instruction_context.nesting_level, nesting_level);
Ok(instruction_context)
}
/// Gets the max height of the InstructionContext stack
pub fn get_instruction_stack_capacity(&self) -> usize {
self.instruction_stack_capacity
}
/// Gets instruction stack height, top-level instructions are height
/// `solana_sdk::instruction::TRANSACTION_LEVEL_STACK_HEIGHT`
pub fn get_instruction_context_stack_height(&self) -> usize {
self.instruction_stack.len()
}
/// Returns the current InstructionContext
pub fn get_current_instruction_context(&self) -> Result<&InstructionContext, InstructionError> {
let level = self
.get_instruction_context_stack_height()
.checked_sub(1)
.ok_or(InstructionError::CallDepth)?;
self.get_instruction_context_at_nesting_level(level)
}
/// Returns the InstructionContext to configure for the next invocation.
///
/// The last InstructionContext is always empty and pre-reserved for the next instruction.
pub fn get_next_instruction_context(
&mut self,
) -> Result<&mut InstructionContext, InstructionError> {
self.instruction_trace
.last_mut()
.ok_or(InstructionError::CallDepth)
}
/// Pushes the next InstructionContext
#[cfg(not(target_os = "solana"))]
pub fn push(&mut self) -> Result<(), InstructionError> {
let nesting_level = self.get_instruction_context_stack_height();
let caller_instruction_context = self
.instruction_trace
.last()
.ok_or(InstructionError::CallDepth)?;
let callee_instruction_accounts_lamport_sum =
self.instruction_accounts_lamport_sum(caller_instruction_context)?;
if !self.instruction_stack.is_empty()
&& self.is_early_verification_of_account_modifications_enabled()
{
let caller_instruction_context = self.get_current_instruction_context()?;
let original_caller_instruction_accounts_lamport_sum =
caller_instruction_context.instruction_accounts_lamport_sum;
let current_caller_instruction_accounts_lamport_sum =
self.instruction_accounts_lamport_sum(caller_instruction_context)?;
if original_caller_instruction_accounts_lamport_sum
!= current_caller_instruction_accounts_lamport_sum
{
return Err(InstructionError::UnbalancedInstruction);
}
}
{
2023-07-13 10:14:33 -07:00
let instruction_context = self.get_next_instruction_context()?;
instruction_context.nesting_level = nesting_level;
instruction_context.instruction_accounts_lamport_sum =
callee_instruction_accounts_lamport_sum;
}
let index_in_trace = self.get_instruction_trace_length();
if index_in_trace >= self.instruction_trace_capacity {
return Err(InstructionError::MaxInstructionTraceLengthExceeded);
}
self.instruction_trace.push(InstructionContext::default());
if nesting_level >= self.instruction_stack_capacity {
return Err(InstructionError::CallDepth);
}
self.instruction_stack.push(index_in_trace);
Ok(())
}
/// Pops the current InstructionContext
#[cfg(not(target_os = "solana"))]
pub fn pop(&mut self) -> Result<(), InstructionError> {
if self.instruction_stack.is_empty() {
return Err(InstructionError::CallDepth);
}
// Verify (before we pop) that the total sum of all lamports in this instruction did not change
let detected_an_unbalanced_instruction =
if self.is_early_verification_of_account_modifications_enabled() {
self.get_current_instruction_context()
.and_then(|instruction_context| {
// Verify all executable accounts have no outstanding refs
for account_index in instruction_context.program_accounts.iter() {
self.get_account_at_index(*account_index)?
.try_borrow_mut()
.map_err(|_| InstructionError::AccountBorrowOutstanding)?;
}
self.instruction_accounts_lamport_sum(instruction_context)
.map(|instruction_accounts_lamport_sum| {
instruction_context.instruction_accounts_lamport_sum
!= instruction_accounts_lamport_sum
})
})
} else {
Ok(false)
};
// Always pop, even if we `detected_an_unbalanced_instruction`
self.instruction_stack.pop();
if detected_an_unbalanced_instruction? {
Err(InstructionError::UnbalancedInstruction)
} else {
Ok(())
}
}
/// Gets the return data of the current InstructionContext or any above
pub fn get_return_data(&self) -> (&Pubkey, &[u8]) {
(&self.return_data.program_id, &self.return_data.data)
}
/// Set the return data of the current InstructionContext
pub fn set_return_data(
&mut self,
program_id: Pubkey,
data: Vec<u8>,
) -> Result<(), InstructionError> {
self.return_data = TransactionReturnData { program_id, data };
Ok(())
}
/// Calculates the sum of all lamports within an instruction
#[cfg(not(target_os = "solana"))]
fn instruction_accounts_lamport_sum(
&self,
instruction_context: &InstructionContext,
) -> Result<u128, InstructionError> {
if !self.is_early_verification_of_account_modifications_enabled() {
return Ok(0);
}
let mut instruction_accounts_lamport_sum: u128 = 0;
for instruction_account_index in 0..instruction_context.get_number_of_instruction_accounts()
{
if instruction_context
.is_instruction_account_duplicate(instruction_account_index)?
.is_some()
{
continue; // Skip duplicate account
}
let index_in_transaction = instruction_context
.get_index_of_instruction_account_in_transaction(instruction_account_index)?;
instruction_accounts_lamport_sum = (self
.get_account_at_index(index_in_transaction)?
.try_borrow()
.map_err(|_| InstructionError::AccountBorrowOutstanding)?
.lamports() as u128)
.checked_add(instruction_accounts_lamport_sum)
.ok_or(InstructionError::ArithmeticOverflow)?;
}
Ok(instruction_accounts_lamport_sum)
}
/// Returns the accounts resize delta
pub fn accounts_resize_delta(&self) -> Result<i64, InstructionError> {
self.accounts_resize_delta
.try_borrow()
.map_err(|_| InstructionError::GenericError)
.map(|value_ref| *value_ref)
}
/// Enables enforcing a maximum accounts data allocation size per transaction
#[cfg(not(target_os = "solana"))]
pub fn enable_cap_accounts_data_allocations_per_transaction(&mut self) {
self.is_cap_accounts_data_allocations_per_transaction_enabled = true;
}
}
/// Return data at the end of a transaction
#[derive(Clone, Debug, Default, Deserialize, PartialEq, Eq, Serialize)]
pub struct TransactionReturnData {
pub program_id: Pubkey,
pub data: Vec<u8>,
}
/// Loaded instruction shared between runtime and programs.
///
/// This context is valid for the entire duration of a (possibly cross program) instruction being processed.
#[derive(Debug, Clone, Default, Eq, PartialEq)]
pub struct InstructionContext {
nesting_level: usize,
instruction_accounts_lamport_sum: u128,
program_accounts: Vec<IndexOfAccount>,
instruction_accounts: Vec<InstructionAccount>,
instruction_data: Vec<u8>,
}
impl InstructionContext {
/// Used together with TransactionContext::get_next_instruction_context()
#[cfg(not(target_os = "solana"))]
pub fn configure(
&mut self,
program_accounts: &[IndexOfAccount],
instruction_accounts: &[InstructionAccount],
instruction_data: &[u8],
) {
self.program_accounts = program_accounts.to_vec();
self.instruction_accounts = instruction_accounts.to_vec();
self.instruction_data = instruction_data.to_vec();
}
/// How many Instructions were on the stack after this one was pushed
///
/// That is the number of nested parent Instructions plus one (itself).
pub fn get_stack_height(&self) -> usize {
self.nesting_level.saturating_add(1)
}
/// Number of program accounts
pub fn get_number_of_program_accounts(&self) -> IndexOfAccount {
self.program_accounts.len() as IndexOfAccount
}
/// Number of accounts in this Instruction (without program accounts)
pub fn get_number_of_instruction_accounts(&self) -> IndexOfAccount {
self.instruction_accounts.len() as IndexOfAccount
}
/// Assert that enough account were supplied to this Instruction
pub fn check_number_of_instruction_accounts(
&self,
expected_at_least: IndexOfAccount,
) -> Result<(), InstructionError> {
if self.get_number_of_instruction_accounts() < expected_at_least {
Err(InstructionError::NotEnoughAccountKeys)
} else {
Ok(())
}
}
/// Data parameter for the programs `process_instruction` handler
pub fn get_instruction_data(&self) -> &[u8] {
&self.instruction_data
}
/// Searches for a program account by its key
pub fn find_index_of_program_account(
&self,
transaction_context: &TransactionContext,
pubkey: &Pubkey,
) -> Option<IndexOfAccount> {
self.program_accounts
.iter()
.position(|index_in_transaction| {
transaction_context
.account_keys
.get(*index_in_transaction as usize)
== Some(pubkey)
})
.map(|index| index as IndexOfAccount)
}
/// Searches for an instruction account by its key
pub fn find_index_of_instruction_account(
&self,
transaction_context: &TransactionContext,
pubkey: &Pubkey,
) -> Option<IndexOfAccount> {
self.instruction_accounts
.iter()
.position(|instruction_account| {
transaction_context
.account_keys
.get(instruction_account.index_in_transaction as usize)
== Some(pubkey)
})
.map(|index| index as IndexOfAccount)
}
/// Translates the given instruction wide program_account_index into a transaction wide index
pub fn get_index_of_program_account_in_transaction(
&self,
program_account_index: IndexOfAccount,
) -> Result<IndexOfAccount, InstructionError> {
Ok(*self
.program_accounts
.get(program_account_index as usize)
.ok_or(InstructionError::NotEnoughAccountKeys)?)
}
/// Translates the given instruction wide instruction_account_index into a transaction wide index
pub fn get_index_of_instruction_account_in_transaction(
&self,
instruction_account_index: IndexOfAccount,
) -> Result<IndexOfAccount, InstructionError> {
Ok(self
.instruction_accounts
.get(instruction_account_index as usize)
.ok_or(InstructionError::NotEnoughAccountKeys)?
.index_in_transaction as IndexOfAccount)
}
/// Returns `Some(instruction_account_index)` if this is a duplicate
/// and `None` if it is the first account with this key
pub fn is_instruction_account_duplicate(
&self,
instruction_account_index: IndexOfAccount,
) -> Result<Option<IndexOfAccount>, InstructionError> {
let index_in_callee = self
.instruction_accounts
.get(instruction_account_index as usize)
.ok_or(InstructionError::NotEnoughAccountKeys)?
.index_in_callee;
Ok(if index_in_callee == instruction_account_index {
None
} else {
Some(index_in_callee)
})
}
/// Gets the key of the last program account of this Instruction
pub fn get_last_program_key<'a, 'b: 'a>(
&'a self,
transaction_context: &'b TransactionContext,
) -> Result<&'b Pubkey, InstructionError> {
self.get_index_of_program_account_in_transaction(
self.get_number_of_program_accounts().saturating_sub(1),
)
.and_then(|index_in_transaction| {
transaction_context.get_key_of_account_at_index(index_in_transaction)
})
}
fn try_borrow_account<'a, 'b: 'a>(
&'a self,
transaction_context: &'b TransactionContext,
index_in_transaction: IndexOfAccount,
index_in_instruction: IndexOfAccount,
) -> Result<BorrowedAccount<'a>, InstructionError> {
let account = transaction_context
.accounts
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
.get(index_in_transaction)
.ok_or(InstructionError::MissingAccount)?
.try_borrow_mut()
.map_err(|_| InstructionError::AccountBorrowFailed)?;
Ok(BorrowedAccount {
transaction_context,
instruction_context: self,
index_in_transaction,
index_in_instruction,
account,
})
}
/// Gets the last program account of this Instruction
pub fn try_borrow_last_program_account<'a, 'b: 'a>(
&'a self,
transaction_context: &'b TransactionContext,
) -> Result<BorrowedAccount<'a>, InstructionError> {
let result = self.try_borrow_program_account(
transaction_context,
self.get_number_of_program_accounts().saturating_sub(1),
);
debug_assert!(result.is_ok());
result
}
/// Tries to borrow a program account from this Instruction
pub fn try_borrow_program_account<'a, 'b: 'a>(
&'a self,
transaction_context: &'b TransactionContext,
program_account_index: IndexOfAccount,
) -> Result<BorrowedAccount<'a>, InstructionError> {
let index_in_transaction =
self.get_index_of_program_account_in_transaction(program_account_index)?;
self.try_borrow_account(
transaction_context,
index_in_transaction,
program_account_index,
)
}
/// Gets an instruction account of this Instruction
pub fn try_borrow_instruction_account<'a, 'b: 'a>(
&'a self,
transaction_context: &'b TransactionContext,
instruction_account_index: IndexOfAccount,
) -> Result<BorrowedAccount<'a>, InstructionError> {
let index_in_transaction =
self.get_index_of_instruction_account_in_transaction(instruction_account_index)?;
self.try_borrow_account(
transaction_context,
index_in_transaction,
self.get_number_of_program_accounts()
.saturating_add(instruction_account_index),
)
}
/// Returns whether an instruction account is a signer
pub fn is_instruction_account_signer(
&self,
instruction_account_index: IndexOfAccount,
) -> Result<bool, InstructionError> {
Ok(self
.instruction_accounts
.get(instruction_account_index as usize)
.ok_or(InstructionError::MissingAccount)?
.is_signer)
}
/// Returns whether an instruction account is writable
pub fn is_instruction_account_writable(
&self,
instruction_account_index: IndexOfAccount,
) -> Result<bool, InstructionError> {
Ok(self
.instruction_accounts
.get(instruction_account_index as usize)
.ok_or(InstructionError::MissingAccount)?
.is_writable)
}
/// Calculates the set of all keys of signer instruction accounts in this Instruction
pub fn get_signers(
&self,
transaction_context: &TransactionContext,
) -> Result<HashSet<Pubkey>, InstructionError> {
let mut result = HashSet::new();
for instruction_account in self.instruction_accounts.iter() {
if instruction_account.is_signer {
result.insert(
*transaction_context
.get_key_of_account_at_index(instruction_account.index_in_transaction)?,
);
}
}
Ok(result)
}
}
/// Shared account borrowed from the TransactionContext and an InstructionContext.
#[derive(Debug)]
pub struct BorrowedAccount<'a> {
transaction_context: &'a TransactionContext,
instruction_context: &'a InstructionContext,
index_in_transaction: IndexOfAccount,
index_in_instruction: IndexOfAccount,
account: RefMut<'a, AccountSharedData>,
}
impl<'a> BorrowedAccount<'a> {
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
/// Returns the transaction context
pub fn transaction_context(&self) -> &TransactionContext {
self.transaction_context
}
/// Returns the index of this account (transaction wide)
#[inline]
pub fn get_index_in_transaction(&self) -> IndexOfAccount {
self.index_in_transaction
}
/// Returns the public key of this account (transaction wide)
#[inline]
pub fn get_key(&self) -> &Pubkey {
self.transaction_context
.get_key_of_account_at_index(self.index_in_transaction)
.unwrap()
}
/// Returns the owner of this account (transaction wide)
#[inline]
pub fn get_owner(&self) -> &Pubkey {
self.account.owner()
}
/// Assignes the owner of this account (transaction wide)
#[cfg(not(target_os = "solana"))]
pub fn set_owner(&mut self, pubkey: &[u8]) -> Result<(), InstructionError> {
if self
.transaction_context
.is_early_verification_of_account_modifications_enabled()
{
// Only the owner can assign a new owner
if !self.is_owned_by_current_program() {
return Err(InstructionError::ModifiedProgramId);
}
// and only if the account is writable
if !self.is_writable() {
return Err(InstructionError::ModifiedProgramId);
}
// and only if the account is not executable
if self.is_executable() {
return Err(InstructionError::ModifiedProgramId);
}
// and only if the data is zero-initialized or empty
if !is_zeroed(self.get_data()) {
return Err(InstructionError::ModifiedProgramId);
}
// don't touch the account if the owner does not change
if self.get_owner().to_bytes() == pubkey {
return Ok(());
}
self.touch()?;
}
self.account.copy_into_owner_from_slice(pubkey);
Ok(())
}
/// Returns the number of lamports of this account (transaction wide)
#[inline]
pub fn get_lamports(&self) -> u64 {
self.account.lamports()
}
/// Overwrites the number of lamports of this account (transaction wide)
#[cfg(not(target_os = "solana"))]
pub fn set_lamports(&mut self, lamports: u64) -> Result<(), InstructionError> {
if self
.transaction_context
.is_early_verification_of_account_modifications_enabled()
{
// An account not owned by the program cannot have its balance decrease
if !self.is_owned_by_current_program() && lamports < self.get_lamports() {
return Err(InstructionError::ExternalAccountLamportSpend);
}
// The balance of read-only may not change
if !self.is_writable() {
return Err(InstructionError::ReadonlyLamportChange);
}
// The balance of executable accounts may not change
if self.is_executable() {
return Err(InstructionError::ExecutableLamportChange);
}
// don't touch the account if the lamports do not change
if self.get_lamports() == lamports {
return Ok(());
}
self.touch()?;
}
self.account.set_lamports(lamports);
Ok(())
}
/// Adds lamports to this account (transaction wide)
#[cfg(not(target_os = "solana"))]
pub fn checked_add_lamports(&mut self, lamports: u64) -> Result<(), InstructionError> {
self.set_lamports(
self.get_lamports()
.checked_add(lamports)
.ok_or(InstructionError::ArithmeticOverflow)?,
)
}
/// Subtracts lamports from this account (transaction wide)
#[cfg(not(target_os = "solana"))]
pub fn checked_sub_lamports(&mut self, lamports: u64) -> Result<(), InstructionError> {
self.set_lamports(
self.get_lamports()
.checked_sub(lamports)
.ok_or(InstructionError::ArithmeticOverflow)?,
)
}
/// Returns a read-only slice of the account data (transaction wide)
#[inline]
pub fn get_data(&self) -> &[u8] {
self.account.data()
}
/// Returns a writable slice of the account data (transaction wide)
#[cfg(not(target_os = "solana"))]
pub fn get_data_mut(&mut self) -> Result<&mut [u8], InstructionError> {
self.can_data_be_changed()?;
self.touch()?;
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
self.make_data_mut();
Ok(self.account.data_as_mut_slice())
}
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
/// Returns the spare capacity of the vector backing the account data.
///
/// This method should only ever be used during CPI, where after a shrinking
/// realloc we want to zero the spare capacity.
#[cfg(not(target_os = "solana"))]
pub fn spare_data_capacity_mut(&mut self) -> Result<&mut [MaybeUninit<u8>], InstructionError> {
debug_assert!(!self.account.is_shared());
Ok(self.account.spare_data_capacity_mut())
}
/// Overwrites the account data and size (transaction wide).
///
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
/// You should always prefer set_data_from_slice(). Calling this method is
/// currently safe but requires some special casing during CPI when direct
/// account mapping is enabled.
#[cfg(all(
not(target_os = "solana"),
any(test, feature = "dev-context-only-utils")
))]
pub fn set_data(&mut self, data: Vec<u8>) -> Result<(), InstructionError> {
self.can_data_be_resized(data.len())?;
self.can_data_be_changed()?;
self.touch()?;
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
self.update_accounts_resize_delta(data.len())?;
self.account.set_data(data);
Ok(())
}
/// Overwrites the account data and size (transaction wide).
///
/// Call this when you have a slice of data you do not own and want to
/// replace the account data with it.
#[cfg(not(target_os = "solana"))]
pub fn set_data_from_slice(&mut self, data: &[u8]) -> Result<(), InstructionError> {
self.can_data_be_resized(data.len())?;
self.can_data_be_changed()?;
self.touch()?;
self.update_accounts_resize_delta(data.len())?;
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
// Calling make_data_mut() here guarantees that set_data_from_slice()
// copies in places, extending the account capacity if necessary but
// never reducing it. This is required as the account migh be directly
// mapped into a MemoryRegion, and therefore reducing capacity would
// leave a hole in the vm address space. After CPI or upon program
// termination, the runtime will zero the extra capacity.
self.make_data_mut();
self.account.set_data_from_slice(data);
Ok(())
}
/// Resizes the account data (transaction wide)
///
/// Fills it with zeros at the end if is extended or truncates at the end otherwise.
#[cfg(not(target_os = "solana"))]
pub fn set_data_length(&mut self, new_length: usize) -> Result<(), InstructionError> {
self.can_data_be_resized(new_length)?;
self.can_data_be_changed()?;
// don't touch the account if the length does not change
if self.get_data().len() == new_length {
return Ok(());
}
self.touch()?;
self.update_accounts_resize_delta(new_length)?;
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
self.account.resize(new_length, 0);
Ok(())
}
/// Appends all elements in a slice to the account
#[cfg(not(target_os = "solana"))]
pub fn extend_from_slice(&mut self, data: &[u8]) -> Result<(), InstructionError> {
let new_len = self.get_data().len().saturating_add(data.len());
self.can_data_be_resized(new_len)?;
self.can_data_be_changed()?;
if data.is_empty() {
return Ok(());
}
self.touch()?;
self.update_accounts_resize_delta(new_len)?;
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
// Even if extend_from_slice never reduces capacity, still realloc using
// make_data_mut() if necessary so that we grow the account of the full
// max realloc length in one go, avoiding smaller reallocations.
self.make_data_mut();
self.account.extend_from_slice(data);
Ok(())
}
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
/// Reserves capacity for at least additional more elements to be inserted
/// in the given account. Does nothing if capacity is already sufficient.
#[cfg(not(target_os = "solana"))]
pub fn reserve(&mut self, additional: usize) -> Result<(), InstructionError> {
// Note that we don't need to call can_data_be_changed() here nor
// touch() the account. reserve() only changes the capacity of the
// memory that holds the account but it doesn't actually change content
// nor length of the account.
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
self.make_data_mut();
self.account.reserve(additional);
Ok(())
}
/// Returns the number of bytes the account can hold without reallocating.
#[cfg(not(target_os = "solana"))]
pub fn capacity(&self) -> usize {
self.account.capacity()
}
/// Returns whether the underlying AccountSharedData is shared.
///
/// The data is shared if the account has been loaded from the accounts database and has never
/// been written to. Writing to an account unshares it.
///
/// During account serialization, if an account is shared it'll get mapped as CoW, else it'll
/// get mapped directly as writable.
#[cfg(not(target_os = "solana"))]
pub fn is_shared(&self) -> bool {
self.account.is_shared()
}
#[cfg(not(target_os = "solana"))]
fn make_data_mut(&mut self) {
// if the account is still shared, it means this is the first time we're
// about to write into it. Make the account mutable by copying it in a
// buffer with MAX_PERMITTED_DATA_INCREASE capacity so that if the
// transaction reallocs, we don't have to copy the whole account data a
// second time to fullfill the realloc.
//
direct mapping: misc fixes (#32649) * transaction_context: update make_data_mut comment * bpf_loader: cpi: pass SerializeAccountMetadata to CallerAccount::from* We now have a way to provide CallerAccount with trusted values coming from our internal serialization code and not from untrusted vm space * bpf_loader: direct_mapping: enforce account info pointers to be immutable When direct mapping is enabled, we might need to update account data memory regions across CPI calls. Since the only way we have to retrieve the regions is based on their vm addresses, we enforce vm addresses to be stable. Accounts can still be mutated and resized of course, but it must be done in place. This also locks all other AccountInfo pointers, since there's no legitimate reason to make them point to anything else. * bpf_loader: cpi: access ref_to_len_in_vm through VmValue Direct mapping needs to translate vm values at each access since permissions of the underlying memory might have changed. * direct mapping: improve memory permission tracking across CPI calls Ensure that the data and realloc regions of an account always track the account's permissions. In order to do this, we also need to split realloc regions in their own self contained regions, where before we had: [account fields][account data][account realloc + more account fields + next account fields][next account data][...] we now have: [account fields][account data][account realloc][more account fields + next account fields][next account data][...] Tested in TEST_[FORBID|ALLOW]_WRITE_AFTER_OWNERSHIP_CHANGE* Additionally when direct mapping is on, we must update all perms at once before doing account data updates. Otherwise, updating an account might write into another account whose perms we haven't updated yet. Tested in TEST_FORBID_LEN_UPDATE_AFTER_OWNERSHIP_CHANGE. * bpf_loader: serialization: address review comment don't return vm_addr from push_account_region * bpf_loader: rename push_account_region to push_account_data_region * cpi: fix slow edge case zeroing extra account capacity after shrinking an account When returning from CPI we need to zero all the account memory up to the original length only if we know we're potentially dealing with uninitialized memory. When we know that the spare capacity has deterministic content, we only need to zero new_len..prev_len. This fixes a slow edge case that was triggerable by the following scenario: - load a large account (say 10MB) into the vm - shrink to 10 bytes - would memset 10..10MB - shrink to 9 bytes - would memset 9..10MB - shrink to 8 bytes - would memset 8..10MB - ... Now instead in the scenario above the following will happen: - load a large account (say 10MB) into the vm - shrink to 10 bytes - memsets 10..10MB - shrink to 9 bytes - memsets 9..10 - shrink to 8 bytes - memset 8..9 - ... * bpf_loader: add account_data_region_memory_state helper Shared between serialization and CPI to figure out the MemoryState of an account. * cpi: direct_mapping: error out if ref_to_len_in_vm points to account memory If ref_to_len_in_vm is allowed to be in account memory, calles could mutate it, essentially letting callees directly mutate callers memory. * bpf_loader: direct_mapping: map AccessViolation -> InstructionError Return the proper ReadonlyDataModified / ExecutableDataModified / ExternalAccountDataModified depending on where the violation occurs * bpf_loader: cpi: remove unnecessary infallible slice::get call
2023-08-30 02:57:24 -07:00
// NOTE: The account memory region CoW code in bpf_loader::create_vm() implements the same
// logic and must be kept in sync.
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
if self.account.is_shared() {
self.account.reserve(MAX_PERMITTED_DATA_INCREASE);
}
}
/// Deserializes the account data into a state
#[cfg(not(target_os = "solana"))]
pub fn get_state<T: serde::de::DeserializeOwned>(&self) -> Result<T, InstructionError> {
self.account
.deserialize_data()
.map_err(|_| InstructionError::InvalidAccountData)
}
/// Serializes a state into the account data
#[cfg(not(target_os = "solana"))]
pub fn set_state<T: serde::Serialize>(&mut self, state: &T) -> Result<(), InstructionError> {
let data = self.get_data_mut()?;
let serialized_size =
bincode::serialized_size(state).map_err(|_| InstructionError::GenericError)?;
if serialized_size > data.len() as u64 {
return Err(InstructionError::AccountDataTooSmall);
}
bincode::serialize_into(&mut *data, state).map_err(|_| InstructionError::GenericError)?;
Ok(())
}
// Returns whether or the lamports currently in the account is sufficient for rent exemption should the
// data be resized to the given size
#[cfg(not(target_os = "solana"))]
pub fn is_rent_exempt_at_data_length(&self, data_length: usize) -> bool {
self.transaction_context
.rent
.unwrap_or_default()
.is_exempt(self.get_lamports(), data_length)
}
/// Returns whether this account is executable (transaction wide)
#[inline]
pub fn is_executable(&self) -> bool {
self.account.executable()
}
/// Configures whether this account is executable (transaction wide)
#[cfg(not(target_os = "solana"))]
pub fn set_executable(&mut self, is_executable: bool) -> Result<(), InstructionError> {
if let Some(rent) = self.transaction_context.rent {
// To become executable an account must be rent exempt
if !rent.is_exempt(self.get_lamports(), self.get_data().len()) {
return Err(InstructionError::ExecutableAccountNotRentExempt);
}
// Only the owner can set the executable flag
if !self.is_owned_by_current_program() {
return Err(InstructionError::ExecutableModified);
}
// and only if the account is writable
if !self.is_writable() {
return Err(InstructionError::ExecutableModified);
}
// one can not clear the executable flag
if self.is_executable() && !is_executable {
return Err(InstructionError::ExecutableModified);
}
// don't touch the account if the executable flag does not change
if self.is_executable() == is_executable {
return Ok(());
}
self.touch()?;
}
self.account.set_executable(is_executable);
Ok(())
}
/// Returns the rent epoch of this account (transaction wide)
#[cfg(not(target_os = "solana"))]
#[inline]
pub fn get_rent_epoch(&self) -> u64 {
self.account.rent_epoch()
}
/// Returns whether this account is a signer (instruction wide)
pub fn is_signer(&self) -> bool {
if self.index_in_instruction < self.instruction_context.get_number_of_program_accounts() {
return false;
}
self.instruction_context
.is_instruction_account_signer(
self.index_in_instruction
.saturating_sub(self.instruction_context.get_number_of_program_accounts()),
)
.unwrap_or_default()
}
/// Returns whether this account is writable (instruction wide)
pub fn is_writable(&self) -> bool {
if self.index_in_instruction < self.instruction_context.get_number_of_program_accounts() {
return false;
}
self.instruction_context
.is_instruction_account_writable(
self.index_in_instruction
.saturating_sub(self.instruction_context.get_number_of_program_accounts()),
)
.unwrap_or_default()
}
/// Returns true if the owner of this account is the current `InstructionContext`s last program (instruction wide)
pub fn is_owned_by_current_program(&self) -> bool {
self.instruction_context
.get_last_program_key(self.transaction_context)
.map(|key| key == self.get_owner())
.unwrap_or_default()
}
/// Returns an error if the account data can not be mutated by the current program
#[cfg(not(target_os = "solana"))]
pub fn can_data_be_changed(&self) -> Result<(), InstructionError> {
if !self
.transaction_context
.is_early_verification_of_account_modifications_enabled()
{
return Ok(());
}
// Only non-executable accounts data can be changed
if self.is_executable() {
return Err(InstructionError::ExecutableDataModified);
}
// and only if the account is writable
if !self.is_writable() {
return Err(InstructionError::ReadonlyDataModified);
}
// and only if we are the owner
if !self.is_owned_by_current_program() {
return Err(InstructionError::ExternalAccountDataModified);
}
Ok(())
}
/// Returns an error if the account data can not be resized to the given length
#[cfg(not(target_os = "solana"))]
pub fn can_data_be_resized(&self, new_length: usize) -> Result<(), InstructionError> {
if !self
.transaction_context
.is_early_verification_of_account_modifications_enabled()
{
return Ok(());
}
let old_length = self.get_data().len();
// Only the owner can change the length of the data
if new_length != old_length && !self.is_owned_by_current_program() {
return Err(InstructionError::AccountDataSizeChanged);
}
// The new length can not exceed the maximum permitted length
if new_length > MAX_PERMITTED_DATA_LENGTH as usize {
return Err(InstructionError::InvalidRealloc);
}
if self
.transaction_context
.is_cap_accounts_data_allocations_per_transaction_enabled
{
// The resize can not exceed the per-transaction maximum
let length_delta = (new_length as i64).saturating_sub(old_length as i64);
if self
.transaction_context
.accounts_resize_delta()?
.saturating_add(length_delta)
> MAX_PERMITTED_ACCOUNTS_DATA_ALLOCATIONS_PER_TRANSACTION
{
return Err(InstructionError::MaxAccountsDataAllocationsExceeded);
}
}
Ok(())
}
#[cfg(not(target_os = "solana"))]
fn touch(&self) -> Result<(), InstructionError> {
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
self.transaction_context
.accounts()
.touch(self.index_in_transaction)
}
#[cfg(not(target_os = "solana"))]
fn update_accounts_resize_delta(&mut self, new_len: usize) -> Result<(), InstructionError> {
let mut accounts_resize_delta = self
.transaction_context
.accounts_resize_delta
.try_borrow_mut()
.map_err(|_| InstructionError::GenericError)?;
*accounts_resize_delta = accounts_resize_delta
.saturating_add((new_len as i64).saturating_sub(self.get_data().len() as i64));
Ok(())
}
}
/// Everything that needs to be recorded from a TransactionContext after execution
#[cfg(not(target_os = "solana"))]
pub struct ExecutionRecord {
pub accounts: Vec<TransactionAccount>,
pub return_data: TransactionReturnData,
pub touched_account_count: u64,
pub accounts_resize_delta: i64,
}
/// Used by the bank in the runtime to write back the processed accounts and recorded instructions
#[cfg(not(target_os = "solana"))]
impl From<TransactionContext> for ExecutionRecord {
fn from(context: TransactionContext) -> Self {
let accounts = Rc::try_unwrap(context.accounts)
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
.expect("transaction_context.accounts has unexpectd outstanding refs");
let touched_account_count = accounts.touched_count() as u64;
let accounts = accounts.into_accounts();
Self {
accounts: Vec::from(Pin::into_inner(context.account_keys))
.into_iter()
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
.zip(accounts)
.collect(),
return_data: context.return_data,
touched_account_count,
accounts_resize_delta: RefCell::into_inner(context.accounts_resize_delta),
}
}
}
#[cfg(not(target_os = "solana"))]
fn is_zeroed(buf: &[u8]) -> bool {
const ZEROS_LEN: usize = 1024;
const ZEROS: [u8; ZEROS_LEN] = [0; ZEROS_LEN];
let mut chunks = buf.chunks_exact(ZEROS_LEN);
#[allow(clippy::indexing_slicing)]
{
chunks.all(|chunk| chunk == &ZEROS[..])
&& chunks.remainder() == &ZEROS[..chunks.remainder().len()]
}
}