solana/programs/bpf_loader/src/serialization.rs

1096 lines
43 KiB
Rust
Raw Normal View History

2022-03-02 14:50:16 -08:00
#![allow(clippy::integer_arithmetic)]
use {
byteorder::{ByteOrder, LittleEndian},
solana_rbpf::{
aligned_memory::{AlignedMemory, Pod},
ebpf::{HOST_ALIGN, MM_INPUT_START},
memory_region::MemoryRegion,
},
solana_sdk::{
bpf_loader_deprecated,
entrypoint::{BPF_ALIGN_OF_U128, MAX_PERMITTED_DATA_INCREASE, NON_DUP_MARKER},
instruction::InstructionError,
pubkey::Pubkey,
system_instruction::MAX_PERMITTED_DATA_LENGTH,
transaction_context::{
BorrowedAccount, IndexOfAccount, InstructionContext, TransactionContext,
},
},
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
std::mem::{self, size_of},
};
/// Maximum number of instruction accounts that can be serialized into the
/// SBF VM.
const MAX_INSTRUCTION_ACCOUNTS: u8 = NON_DUP_MARKER;
enum SerializeAccount<'a> {
Account(IndexOfAccount, BorrowedAccount<'a>),
Duplicate(IndexOfAccount),
}
struct Serializer {
pub buffer: AlignedMemory<HOST_ALIGN>,
regions: Vec<MemoryRegion>,
vaddr: u64,
region_start: usize,
aligned: bool,
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
copy_account_data: bool,
}
impl Serializer {
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
fn new(size: usize, start_addr: u64, aligned: bool, copy_account_data: bool) -> Serializer {
Serializer {
buffer: AlignedMemory::with_capacity(size),
regions: Vec::new(),
region_start: 0,
vaddr: start_addr,
aligned,
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
copy_account_data,
}
}
fn fill_write(&mut self, num: usize, value: u8) -> std::io::Result<()> {
self.buffer.fill_write(num, value)
}
pub fn write<T: Pod>(&mut self, value: T) {
self.debug_assert_alignment::<T>();
// Safety:
// in serialize_parameters_(aligned|unaligned) first we compute the
// required size then we write into the newly allocated buffer. There's
// no need to check bounds at every write.
//
// AlignedMemory::write_unchecked _does_ debug_assert!() that the capacity
// is enough, so in the unlikely case we introduce a bug in the size
// computation, tests will abort.
unsafe {
self.buffer.write_unchecked(value);
}
}
fn write_all(&mut self, value: &[u8]) {
// Safety:
// see write() - the buffer is guaranteed to be large enough
unsafe {
self.buffer.write_all_unchecked(value);
}
}
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
fn write_account(&mut self, account: &mut BorrowedAccount<'_>) -> Result<(), InstructionError> {
if self.copy_account_data {
self.write_all(account.get_data());
} else {
self.push_account_region(account)?;
}
if self.aligned {
let align_offset =
(account.get_data().len() as *const u8).align_offset(BPF_ALIGN_OF_U128);
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
if self.copy_account_data {
self.fill_write(MAX_PERMITTED_DATA_INCREASE + align_offset, 0)
.map_err(|_| InstructionError::InvalidArgument)?;
} else {
// The deserialization code is going to align the vm_addr to
// BPF_ALIGN_OF_U128. Always add one BPF_ALIGN_OF_U128 worth of
// padding and shift the start of the next region, so that once
// vm_addr is aligned, the corresponding host_addr is aligned
// too.
self.fill_write(MAX_PERMITTED_DATA_INCREASE + BPF_ALIGN_OF_U128, 0)
.map_err(|_| InstructionError::InvalidArgument)?;
self.region_start += BPF_ALIGN_OF_U128.saturating_sub(align_offset);
}
}
Ok(())
}
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
fn push_account_region(
&mut self,
account: &mut BorrowedAccount<'_>,
) -> Result<(), InstructionError> {
self.push_region();
let account_len = account.get_data().len();
if account_len > 0 {
let region = if account.can_data_be_changed().is_ok() {
if account.is_shared() {
// If the account is still shared it means it wasn't written to yet during this
// transaction. We map it as CoW and it'll be copied the first time something
// tries to write into it.
let index_in_transaction = account.get_index_in_transaction();
MemoryRegion::new_cow(
account.get_data(),
self.vaddr,
index_in_transaction as u64,
)
} else {
// The account isn't shared anymore, meaning it was written to earlier during
// this transaction. We can just map as writable no need for any fancy CoW
// business.
MemoryRegion::new_writable(account.get_data_mut()?, self.vaddr)
}
} else {
MemoryRegion::new_readonly(account.get_data(), self.vaddr)
};
self.vaddr += region.len;
self.regions.push(region);
}
Ok(())
}
fn push_region(&mut self) {
let range = self.region_start..self.buffer.len();
let region = MemoryRegion::new_writable(
self.buffer.as_slice_mut().get_mut(range.clone()).unwrap(),
self.vaddr,
);
self.regions.push(region);
self.region_start = range.end;
self.vaddr += range.len() as u64;
}
fn finish(mut self) -> (AlignedMemory<HOST_ALIGN>, Vec<MemoryRegion>) {
self.push_region();
debug_assert_eq!(self.region_start, self.buffer.len());
(self.buffer, self.regions)
}
fn debug_assert_alignment<T>(&self) {
debug_assert!(
!self.aligned
|| self
.buffer
.as_slice()
.as_ptr_range()
.end
.align_offset(mem::align_of::<T>())
== 0
);
}
}
pub fn serialize_parameters(
transaction_context: &TransactionContext,
instruction_context: &InstructionContext,
should_cap_ix_accounts: bool,
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
copy_account_data: bool,
) -> Result<(AlignedMemory<HOST_ALIGN>, Vec<MemoryRegion>, Vec<usize>), InstructionError> {
let num_ix_accounts = instruction_context.get_number_of_instruction_accounts();
if should_cap_ix_accounts && num_ix_accounts > MAX_INSTRUCTION_ACCOUNTS as IndexOfAccount {
return Err(InstructionError::MaxAccountsExceeded);
}
let is_loader_deprecated = *instruction_context
.try_borrow_last_program_account(transaction_context)?
.get_owner()
== bpf_loader_deprecated::id();
let num_accounts = instruction_context.get_number_of_instruction_accounts() as usize;
let mut accounts = Vec::with_capacity(num_accounts);
let mut account_lengths: Vec<usize> = Vec::with_capacity(num_accounts);
for instruction_account_index in 0..instruction_context.get_number_of_instruction_accounts() {
if let Some(index) =
instruction_context.is_instruction_account_duplicate(instruction_account_index)?
{
accounts.push(SerializeAccount::Duplicate(index));
// unwrap here is safe: if an account is a duplicate, we must have
// seen the original already
account_lengths.push(*account_lengths.get(index as usize).unwrap());
} else {
let account = instruction_context
.try_borrow_instruction_account(transaction_context, instruction_account_index)?;
account_lengths.push(account.get_data().len());
accounts.push(SerializeAccount::Account(
instruction_account_index,
account,
));
};
}
if is_loader_deprecated {
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
serialize_parameters_unaligned(
transaction_context,
instruction_context,
accounts,
copy_account_data,
)
} else {
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
serialize_parameters_aligned(
transaction_context,
instruction_context,
accounts,
copy_account_data,
)
}
.map(|(buffer, regions)| (buffer, regions, account_lengths))
}
pub fn deserialize_parameters(
transaction_context: &TransactionContext,
instruction_context: &InstructionContext,
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
copy_account_data: bool,
buffer: &[u8],
account_lengths: &[usize],
) -> Result<(), InstructionError> {
let is_loader_deprecated = *instruction_context
.try_borrow_last_program_account(transaction_context)?
.get_owner()
== bpf_loader_deprecated::id();
if is_loader_deprecated {
deserialize_parameters_unaligned(
transaction_context,
instruction_context,
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
copy_account_data,
buffer,
account_lengths,
)
} else {
deserialize_parameters_aligned(
transaction_context,
instruction_context,
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
copy_account_data,
buffer,
account_lengths,
)
}
}
fn serialize_parameters_unaligned(
transaction_context: &TransactionContext,
instruction_context: &InstructionContext,
accounts: Vec<SerializeAccount>,
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
copy_account_data: bool,
) -> Result<(AlignedMemory<HOST_ALIGN>, Vec<MemoryRegion>), InstructionError> {
// Calculate size in order to alloc once
let mut size = size_of::<u64>();
for account in &accounts {
size += 1; // dup
match account {
SerializeAccount::Duplicate(_) => {}
SerializeAccount::Account(_, account) => {
size += size_of::<u8>() // is_signer
+ size_of::<u8>() // is_writable
+ size_of::<Pubkey>() // key
+ size_of::<u64>() // lamports
+ size_of::<u64>() // data len
+ size_of::<Pubkey>() // owner
+ size_of::<u8>() // executable
+ size_of::<u64>(); // rent_epoch
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
if copy_account_data {
size += account.get_data().len();
}
}
}
}
size += size_of::<u64>() // instruction data len
+ instruction_context.get_instruction_data().len() // instruction data
+ size_of::<Pubkey>(); // program id
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
let mut s = Serializer::new(size, MM_INPUT_START, false, copy_account_data);
s.write::<u64>((accounts.len() as u64).to_le());
for account in accounts {
match account {
SerializeAccount::Duplicate(position) => s.write(position as u8),
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
SerializeAccount::Account(_, mut account) => {
s.write::<u8>(NON_DUP_MARKER);
s.write::<u8>(account.is_signer() as u8);
s.write::<u8>(account.is_writable() as u8);
s.write_all(account.get_key().as_ref());
s.write::<u64>(account.get_lamports().to_le());
s.write::<u64>((account.get_data().len() as u64).to_le());
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
s.write_account(&mut account)?;
s.write_all(account.get_owner().as_ref());
s.write::<u8>(account.is_executable() as u8);
2022-11-09 11:39:38 -08:00
s.write::<u64>((account.get_rent_epoch()).to_le());
}
};
}
s.write::<u64>((instruction_context.get_instruction_data().len() as u64).to_le());
s.write_all(instruction_context.get_instruction_data());
s.write_all(
instruction_context
.try_borrow_last_program_account(transaction_context)?
.get_key()
.as_ref(),
);
Ok(s.finish())
}
pub fn deserialize_parameters_unaligned(
transaction_context: &TransactionContext,
instruction_context: &InstructionContext,
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
copy_account_data: bool,
buffer: &[u8],
account_lengths: &[usize],
) -> Result<(), InstructionError> {
let mut start = size_of::<u64>(); // number of accounts
for (instruction_account_index, pre_len) in
(0..instruction_context.get_number_of_instruction_accounts()).zip(account_lengths.iter())
{
let duplicate =
instruction_context.is_instruction_account_duplicate(instruction_account_index)?;
start += 1; // is_dup
if duplicate.is_none() {
let mut borrowed_account = instruction_context
.try_borrow_instruction_account(transaction_context, instruction_account_index)?;
start += size_of::<u8>(); // is_signer
start += size_of::<u8>(); // is_writable
start += size_of::<Pubkey>(); // key
let lamports = LittleEndian::read_u64(
2022-03-10 11:48:33 -08:00
buffer
.get(start..)
.ok_or(InstructionError::InvalidArgument)?,
);
if borrowed_account.get_lamports() != lamports {
borrowed_account.set_lamports(lamports)?;
}
start += size_of::<u64>() // lamports
+ size_of::<u64>(); // data length
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
if copy_account_data {
let data = buffer
.get(start..start + pre_len)
.ok_or(InstructionError::InvalidArgument)?;
// The redundant check helps to avoid the expensive data comparison if we can
match borrowed_account
.can_data_be_resized(data.len())
.and_then(|_| borrowed_account.can_data_be_changed())
{
Ok(()) => borrowed_account.set_data_from_slice(data)?,
Err(err) if borrowed_account.get_data() != data => return Err(err),
_ => {}
}
start += pre_len; // data
}
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
start += size_of::<Pubkey>() // owner
+ size_of::<u8>() // executable
+ size_of::<u64>(); // rent_epoch
}
}
Ok(())
}
fn serialize_parameters_aligned(
transaction_context: &TransactionContext,
instruction_context: &InstructionContext,
accounts: Vec<SerializeAccount>,
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
copy_account_data: bool,
) -> Result<(AlignedMemory<HOST_ALIGN>, Vec<MemoryRegion>), InstructionError> {
// Calculate size in order to alloc once
let mut size = size_of::<u64>();
for account in &accounts {
size += 1; // dup
match account {
SerializeAccount::Duplicate(_) => size += 7, // padding to 64-bit aligned
SerializeAccount::Account(_, account) => {
let data_len = account.get_data().len();
size += size_of::<u8>() // is_signer
+ size_of::<u8>() // is_writable
+ size_of::<u8>() // executable
+ size_of::<u32>() // original_data_len
+ size_of::<Pubkey>() // key
+ size_of::<Pubkey>() // owner
+ size_of::<u64>() // lamports
+ size_of::<u64>() // data len
+ MAX_PERMITTED_DATA_INCREASE
+ size_of::<u64>(); // rent epoch
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
if copy_account_data {
size += data_len + (data_len as *const u8).align_offset(BPF_ALIGN_OF_U128);
} else {
size += BPF_ALIGN_OF_U128;
}
}
}
}
size += size_of::<u64>() // data len
+ instruction_context.get_instruction_data().len()
+ size_of::<Pubkey>(); // program id;
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
let mut s = Serializer::new(size, MM_INPUT_START, true, copy_account_data);
// Serialize into the buffer
s.write::<u64>((accounts.len() as u64).to_le());
for account in accounts {
match account {
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
SerializeAccount::Account(_, mut borrowed_account) => {
s.write::<u8>(NON_DUP_MARKER);
s.write::<u8>(borrowed_account.is_signer() as u8);
s.write::<u8>(borrowed_account.is_writable() as u8);
s.write::<u8>(borrowed_account.is_executable() as u8);
s.write_all(&[0u8, 0, 0, 0]);
s.write_all(borrowed_account.get_key().as_ref());
s.write_all(borrowed_account.get_owner().as_ref());
s.write::<u64>(borrowed_account.get_lamports().to_le());
s.write::<u64>((borrowed_account.get_data().len() as u64).to_le());
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
s.write_account(&mut borrowed_account)?;
2022-11-09 11:39:38 -08:00
s.write::<u64>((borrowed_account.get_rent_epoch()).to_le());
}
SerializeAccount::Duplicate(position) => {
s.write::<u8>(position as u8);
s.write_all(&[0u8, 0, 0, 0, 0, 0, 0]);
}
};
}
s.write::<u64>((instruction_context.get_instruction_data().len() as u64).to_le());
s.write_all(instruction_context.get_instruction_data());
s.write_all(
instruction_context
.try_borrow_last_program_account(transaction_context)?
.get_key()
.as_ref(),
);
Ok(s.finish())
}
pub fn deserialize_parameters_aligned(
transaction_context: &TransactionContext,
instruction_context: &InstructionContext,
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
copy_account_data: bool,
buffer: &[u8],
account_lengths: &[usize],
) -> Result<(), InstructionError> {
let mut start = size_of::<u64>(); // number of accounts
for (instruction_account_index, pre_len) in
(0..instruction_context.get_number_of_instruction_accounts()).zip(account_lengths.iter())
{
let duplicate =
instruction_context.is_instruction_account_duplicate(instruction_account_index)?;
start += size_of::<u8>(); // position
if duplicate.is_some() {
start += 7; // padding to 64-bit aligned
} else {
let mut borrowed_account = instruction_context
.try_borrow_instruction_account(transaction_context, instruction_account_index)?;
start += size_of::<u8>() // is_signer
+ size_of::<u8>() // is_writable
+ size_of::<u8>() // executable
+ size_of::<u32>() // original_data_len
+ size_of::<Pubkey>(); // key
let owner = buffer
.get(start..start + size_of::<Pubkey>())
.ok_or(InstructionError::InvalidArgument)?;
start += size_of::<Pubkey>(); // owner
let lamports = LittleEndian::read_u64(
2022-03-10 11:48:33 -08:00
buffer
.get(start..)
.ok_or(InstructionError::InvalidArgument)?,
);
if borrowed_account.get_lamports() != lamports {
borrowed_account.set_lamports(lamports)?;
}
start += size_of::<u64>(); // lamports
2022-03-10 11:48:33 -08:00
let post_len = LittleEndian::read_u64(
buffer
.get(start..)
.ok_or(InstructionError::InvalidArgument)?,
) as usize;
start += size_of::<u64>(); // data length
if post_len.saturating_sub(*pre_len) > MAX_PERMITTED_DATA_INCREASE
|| post_len > MAX_PERMITTED_DATA_LENGTH as usize
{
return Err(InstructionError::InvalidRealloc);
}
// The redundant check helps to avoid the expensive data comparison if we can
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
let alignment_offset = (*pre_len as *const u8).align_offset(BPF_ALIGN_OF_U128);
if copy_account_data {
let data = buffer
.get(start..start + post_len)
.ok_or(InstructionError::InvalidArgument)?;
match borrowed_account
.can_data_be_resized(post_len)
.and_then(|_| borrowed_account.can_data_be_changed())
{
Ok(()) => borrowed_account.set_data_from_slice(data)?,
Err(err) if borrowed_account.get_data() != data => return Err(err),
_ => {}
}
start += *pre_len; // data
} else {
// See Serializer::write_account() as to why we have this
// padding before the realloc region here.
start += BPF_ALIGN_OF_U128.saturating_sub(alignment_offset);
let data = buffer
.get(start..start + MAX_PERMITTED_DATA_INCREASE)
.ok_or(InstructionError::InvalidArgument)?;
match borrowed_account
.can_data_be_resized(post_len)
.and_then(|_| borrowed_account.can_data_be_changed())
{
Ok(()) => {
borrowed_account.set_data_length(post_len)?;
let allocated_bytes = post_len.saturating_sub(*pre_len);
if allocated_bytes > 0 {
borrowed_account
.get_data_mut()?
.get_mut(*pre_len..pre_len.saturating_add(allocated_bytes))
.ok_or(InstructionError::InvalidArgument)?
.copy_from_slice(
data.get(0..allocated_bytes)
.ok_or(InstructionError::InvalidArgument)?,
);
}
}
Err(err) if borrowed_account.get_data().len() != post_len => return Err(err),
_ => {}
}
}
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
start += MAX_PERMITTED_DATA_INCREASE;
start += alignment_offset;
start += size_of::<u64>(); // rent_epoch
if borrowed_account.get_owner().to_bytes() != owner {
// Change the owner at the end so that we are allowed to change the lamports and data before
borrowed_account.set_owner(owner)?;
}
}
}
Ok(())
}
#[cfg(test)]
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
#[allow(clippy::indexing_slicing)]
mod tests {
use {
super::*,
solana_program_runtime::with_mock_invoke_context,
solana_sdk::{
account::{Account, AccountSharedData, WritableAccount},
account_info::AccountInfo,
bpf_loader,
entrypoint::deserialize,
transaction_context::InstructionAccount,
},
std::{
cell::RefCell,
rc::Rc,
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
slice::{self, from_raw_parts, from_raw_parts_mut},
},
};
#[test]
fn test_serialize_parameters_with_many_accounts() {
struct TestCase {
num_ix_accounts: usize,
append_dup_account: bool,
should_cap_ix_accounts: bool,
expected_err: Option<InstructionError>,
name: &'static str,
}
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
for copy_account_data in [true] {
for TestCase {
num_ix_accounts,
append_dup_account,
should_cap_ix_accounts,
expected_err,
name,
} in [
TestCase {
name: "serialize max accounts without cap",
num_ix_accounts: usize::from(MAX_INSTRUCTION_ACCOUNTS),
should_cap_ix_accounts: false,
append_dup_account: false,
expected_err: None,
},
TestCase {
name: "serialize max accounts and append dup without cap",
num_ix_accounts: usize::from(MAX_INSTRUCTION_ACCOUNTS),
should_cap_ix_accounts: false,
append_dup_account: true,
expected_err: None,
},
TestCase {
name: "serialize max accounts with cap",
num_ix_accounts: usize::from(MAX_INSTRUCTION_ACCOUNTS),
should_cap_ix_accounts: true,
append_dup_account: false,
expected_err: None,
},
TestCase {
name: "serialize too many accounts with cap",
num_ix_accounts: usize::from(MAX_INSTRUCTION_ACCOUNTS) + 1,
should_cap_ix_accounts: true,
append_dup_account: false,
expected_err: Some(InstructionError::MaxAccountsExceeded),
},
TestCase {
name: "serialize too many accounts and append dup with cap",
num_ix_accounts: usize::from(MAX_INSTRUCTION_ACCOUNTS),
should_cap_ix_accounts: true,
append_dup_account: true,
expected_err: Some(InstructionError::MaxAccountsExceeded),
},
// This test case breaks parameter deserialization and can be cleaned up
// when should_cap_ix_accounts is enabled.
//
// TestCase {
// name: "serialize too many accounts and append dup without cap",
// num_ix_accounts: usize::from(MAX_INSTRUCTION_ACCOUNTS) + 1,
// should_cap_ix_accounts: false,
// append_dup_account: true,
// expected_err: None,
// },
] {
let program_id = solana_sdk::pubkey::new_rand();
let mut transaction_accounts = vec![(
program_id,
AccountSharedData::from(Account {
lamports: 0,
data: vec![],
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
owner: bpf_loader::id(),
executable: true,
rent_epoch: 0,
}),
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
)];
for _ in 0..num_ix_accounts {
transaction_accounts.push((
Pubkey::new_unique(),
AccountSharedData::from(Account {
lamports: 0,
data: vec![],
owner: program_id,
executable: false,
rent_epoch: 0,
}),
));
}
let mut instruction_accounts: Vec<_> = (0..num_ix_accounts as IndexOfAccount)
.map(|index_in_callee| InstructionAccount {
index_in_transaction: index_in_callee + 1,
index_in_caller: index_in_callee + 1,
index_in_callee,
is_signer: false,
is_writable: false,
})
.collect();
if append_dup_account {
instruction_accounts.push(instruction_accounts.last().cloned().unwrap());
}
let program_indices = [0];
let instruction_data = vec![];
with_mock_invoke_context!(
invoke_context,
transaction_context,
transaction_accounts
);
invoke_context
.transaction_context
.get_next_instruction_context()
.unwrap()
.configure(&program_indices, &instruction_accounts, &instruction_data);
invoke_context.push().unwrap();
let instruction_context = invoke_context
.transaction_context
.get_current_instruction_context()
.unwrap();
let serialization_result = serialize_parameters(
invoke_context.transaction_context,
instruction_context,
should_cap_ix_accounts,
copy_account_data,
);
assert_eq!(
serialization_result.as_ref().err(),
expected_err.as_ref(),
"{name} test case failed",
);
if expected_err.is_some() {
continue;
}
let (mut serialized, regions, _account_lengths) = serialization_result.unwrap();
let mut serialized_regions = concat_regions(&regions);
let (de_program_id, de_accounts, de_instruction_data) = unsafe {
deserialize(
if copy_account_data {
serialized.as_slice_mut()
} else {
serialized_regions.as_slice_mut()
}
.first_mut()
.unwrap() as *mut u8,
)
};
assert_eq!(de_program_id, &program_id);
assert_eq!(de_instruction_data, &instruction_data);
for account_info in de_accounts {
let index_in_transaction = invoke_context
.transaction_context
.find_index_of_account(account_info.key)
.unwrap();
let account = invoke_context
.transaction_context
.get_account_at_index(index_in_transaction)
.unwrap()
.borrow();
assert_eq!(account.lamports(), account_info.lamports());
assert_eq!(account.data(), &account_info.data.borrow()[..]);
assert_eq!(account.owner(), account_info.owner);
assert_eq!(account.executable(), account_info.executable);
assert_eq!(account.rent_epoch(), account_info.rent_epoch);
}
}
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
}
}
#[test]
fn test_serialize_parameters() {
for copy_account_data in [false, true] {
let program_id = solana_sdk::pubkey::new_rand();
let transaction_accounts = vec![
(
program_id,
AccountSharedData::from(Account {
lamports: 0,
data: vec![],
owner: bpf_loader::id(),
executable: true,
rent_epoch: 0,
}),
),
(
solana_sdk::pubkey::new_rand(),
AccountSharedData::from(Account {
lamports: 1,
data: vec![1u8, 2, 3, 4, 5],
owner: bpf_loader::id(),
executable: false,
rent_epoch: 100,
}),
),
(
solana_sdk::pubkey::new_rand(),
AccountSharedData::from(Account {
lamports: 2,
data: vec![11u8, 12, 13, 14, 15, 16, 17, 18, 19],
owner: bpf_loader::id(),
executable: true,
rent_epoch: 200,
}),
),
(
solana_sdk::pubkey::new_rand(),
AccountSharedData::from(Account {
lamports: 3,
data: vec![],
owner: bpf_loader::id(),
executable: false,
rent_epoch: 3100,
}),
),
(
solana_sdk::pubkey::new_rand(),
AccountSharedData::from(Account {
lamports: 4,
data: vec![1u8, 2, 3, 4, 5],
owner: bpf_loader::id(),
executable: false,
rent_epoch: 100,
}),
),
(
solana_sdk::pubkey::new_rand(),
AccountSharedData::from(Account {
lamports: 5,
data: vec![11u8, 12, 13, 14, 15, 16, 17, 18, 19],
owner: bpf_loader::id(),
executable: true,
rent_epoch: 200,
}),
),
(
solana_sdk::pubkey::new_rand(),
AccountSharedData::from(Account {
lamports: 6,
data: vec![],
owner: bpf_loader::id(),
executable: false,
rent_epoch: 3100,
}),
),
];
let instruction_accounts: Vec<InstructionAccount> = [1, 1, 2, 3, 4, 4, 5, 6]
.into_iter()
.enumerate()
.map(
|(index_in_instruction, index_in_transaction)| InstructionAccount {
index_in_transaction,
index_in_caller: index_in_transaction,
index_in_callee: index_in_transaction - 1,
is_signer: false,
is_writable: index_in_instruction >= 4,
},
)
.collect();
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
let instruction_data = vec![1u8, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11];
let program_indices = [0];
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
let mut original_accounts = transaction_accounts.clone();
with_mock_invoke_context!(invoke_context, transaction_context, transaction_accounts);
invoke_context
.transaction_context
.get_next_instruction_context()
.unwrap()
.configure(&program_indices, &instruction_accounts, &instruction_data);
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
invoke_context.push().unwrap();
let instruction_context = invoke_context
.transaction_context
.get_current_instruction_context()
.unwrap();
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
// check serialize_parameters_aligned
let (mut serialized, regions, account_lengths) = serialize_parameters(
invoke_context.transaction_context,
instruction_context,
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
true,
copy_account_data,
)
.unwrap();
let mut serialized_regions = concat_regions(&regions);
if copy_account_data {
assert_eq!(serialized.as_slice(), serialized_regions.as_slice());
}
let (de_program_id, de_accounts, de_instruction_data) = unsafe {
deserialize(
if copy_account_data {
serialized.as_slice_mut()
} else {
serialized_regions.as_slice_mut()
}
.first_mut()
.unwrap() as *mut u8,
)
};
assert_eq!(&program_id, de_program_id);
assert_eq!(instruction_data, de_instruction_data);
assert_eq!(
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
(de_instruction_data.first().unwrap() as *const u8).align_offset(BPF_ALIGN_OF_U128),
0
);
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
for account_info in de_accounts {
let index_in_transaction = invoke_context
.transaction_context
.find_index_of_account(account_info.key)
.unwrap();
let account = invoke_context
.transaction_context
.get_account_at_index(index_in_transaction)
.unwrap()
.borrow();
assert_eq!(account.lamports(), account_info.lamports());
assert_eq!(account.data(), &account_info.data.borrow()[..]);
assert_eq!(account.owner(), account_info.owner);
assert_eq!(account.executable(), account_info.executable);
assert_eq!(account.rent_epoch(), account_info.rent_epoch);
assert_eq!(
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
(*account_info.lamports.borrow() as *const u64).align_offset(BPF_ALIGN_OF_U128),
0
);
assert_eq!(
account_info
.data
.borrow()
.as_ptr()
.align_offset(BPF_ALIGN_OF_U128),
0
);
}
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
deserialize_parameters(
invoke_context.transaction_context,
instruction_context,
copy_account_data,
serialized.as_slice(),
&account_lengths,
)
.unwrap();
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
for (index_in_transaction, (_key, original_account)) in
original_accounts.iter().enumerate()
{
let account = invoke_context
.transaction_context
.get_account_at_index(index_in_transaction as IndexOfAccount)
.unwrap()
.borrow();
assert_eq!(&*account, original_account);
}
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
// check serialize_parameters_unaligned
original_accounts
.first_mut()
.unwrap()
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
.1
.set_owner(bpf_loader_deprecated::id());
invoke_context
.transaction_context
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
.get_account_at_index(0)
.unwrap()
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
.borrow_mut()
.set_owner(bpf_loader_deprecated::id());
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
let (mut serialized, regions, account_lengths) = serialize_parameters(
invoke_context.transaction_context,
instruction_context,
true,
copy_account_data,
)
.unwrap();
let mut serialized_regions = concat_regions(&regions);
let (de_program_id, de_accounts, de_instruction_data) = unsafe {
deserialize_unaligned(
if copy_account_data {
serialized.as_slice_mut()
} else {
serialized_regions.as_slice_mut()
}
.first_mut()
.unwrap() as *mut u8,
)
};
assert_eq!(&program_id, de_program_id);
assert_eq!(instruction_data, de_instruction_data);
for account_info in de_accounts {
let index_in_transaction = invoke_context
.transaction_context
.find_index_of_account(account_info.key)
.unwrap();
let account = invoke_context
.transaction_context
.get_account_at_index(index_in_transaction)
.unwrap()
.borrow();
assert_eq!(account.lamports(), account_info.lamports());
assert_eq!(account.data(), &account_info.data.borrow()[..]);
assert_eq!(account.owner(), account_info.owner);
assert_eq!(account.executable(), account_info.executable);
assert_eq!(account.rent_epoch(), account_info.rent_epoch);
}
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
deserialize_parameters(
invoke_context.transaction_context,
instruction_context,
copy_account_data,
serialized.as_slice(),
&account_lengths,
)
.unwrap();
for (index_in_transaction, (_key, original_account)) in
original_accounts.iter().enumerate()
{
let account = invoke_context
.transaction_context
.get_account_at_index(index_in_transaction as IndexOfAccount)
.unwrap()
.borrow();
assert_eq!(&*account, original_account);
}
}
}
// the old bpf_loader in-program deserializer bpf_loader::id()
#[allow(clippy::type_complexity)]
pub unsafe fn deserialize_unaligned<'a>(
input: *mut u8,
) -> (&'a Pubkey, Vec<AccountInfo<'a>>, &'a [u8]) {
let mut offset: usize = 0;
// number of accounts present
#[allow(clippy::cast_ptr_alignment)]
let num_accounts = *(input.add(offset) as *const u64) as usize;
offset += size_of::<u64>();
// account Infos
let mut accounts = Vec::with_capacity(num_accounts);
for _ in 0..num_accounts {
let dup_info = *(input.add(offset) as *const u8);
offset += size_of::<u8>();
if dup_info == NON_DUP_MARKER {
#[allow(clippy::cast_ptr_alignment)]
let is_signer = *(input.add(offset) as *const u8) != 0;
offset += size_of::<u8>();
#[allow(clippy::cast_ptr_alignment)]
let is_writable = *(input.add(offset) as *const u8) != 0;
offset += size_of::<u8>();
let key: &Pubkey = &*(input.add(offset) as *const Pubkey);
offset += size_of::<Pubkey>();
#[allow(clippy::cast_ptr_alignment)]
let lamports = Rc::new(RefCell::new(&mut *(input.add(offset) as *mut u64)));
offset += size_of::<u64>();
#[allow(clippy::cast_ptr_alignment)]
let data_len = *(input.add(offset) as *const u64) as usize;
offset += size_of::<u64>();
let data = Rc::new(RefCell::new({
from_raw_parts_mut(input.add(offset), data_len)
}));
offset += data_len;
let owner: &Pubkey = &*(input.add(offset) as *const Pubkey);
offset += size_of::<Pubkey>();
#[allow(clippy::cast_ptr_alignment)]
let executable = *(input.add(offset) as *const u8) != 0;
offset += size_of::<u8>();
#[allow(clippy::cast_ptr_alignment)]
let rent_epoch = *(input.add(offset) as *const u64);
offset += size_of::<u64>();
accounts.push(AccountInfo {
2021-04-18 10:27:36 -07:00
key,
is_signer,
is_writable,
lamports,
data,
owner,
executable,
rent_epoch,
});
} else {
// duplicate account, clone the original
2022-03-10 11:48:33 -08:00
accounts.push(accounts.get(dup_info as usize).unwrap().clone());
}
}
// instruction data
#[allow(clippy::cast_ptr_alignment)]
let instruction_data_len = *(input.add(offset) as *const u64) as usize;
offset += size_of::<u64>();
let instruction_data = { from_raw_parts(input.add(offset), instruction_data_len) };
offset += instruction_data_len;
// program Id
let program_id: &Pubkey = &*(input.add(offset) as *const Pubkey);
(program_id, accounts, instruction_data)
}
Account data direct mapping (#28053) * AccountSharedData: make data_mut() private This ensures that the inner Vec is never handed out. This is in preparation of enforcing that the capacity of the inner vec never shrinks, which is required for direct mapping. * Adds the feature bpf_account_data_direct_mapping. * Remaps EbpfError::AccessViolation into InstructionError::ReadonlyDataModified. * WIP: Memory regions for each instruction account in create_vm(). * Fix serialization benches, run both copy and !copy variants * rbpf-cli: fix build * BorrowedAccount: ensure that account capacity is never reduced Accounts can be directly mapped in address space. Their capacity can't be reduced mid transaction as that would create holes in vm address space that point to invalid host memory. * bpf_load: run serialization tests for both copy and !copy account data * bpf_loader: add Serializer::write_account * fix lints * BorrowedAccount: make_data_mut is host only * Fix unused import warning * Fix lints * cpi: add explicit direct_mapping arg to update_(callee|caller)_account * cpi: rename account_data_or_only_realloc_padding to serialized_data * cpi: add CallerAccount::original_data_len comment * cpi: add update_callee_account direct_mapping test * cpi: add test_update_caller_account_data_direct_mapping and fix bug We used to have a bug in zeroing data when shrinking account, where we zeroed the spare account capacity but not the realloc padding. * cpi: add tests for mutated readonly accounts * cpi: update_caller_account doesn't need to change .serialized_data when direct_mapping is on * cpi: update_caller_account: ensure that account capacity is always enough Introduce a better way to ensure that account capacity never goes below what might be mapped in memory regions. * cpi: zero account capacity using the newly introduced BorrowedAccount::spare_data_capacity_mut() Before we were using BorrowedAccount::get_data_mut() to get the base pointer to the account data, then we were slicing the spare capacity from it. Calling get_data_mut() doesn't work if an account has been closed tho, since the current program doesn't own the account anymore and therefore get_data_mut() errors out. * bpf_loader: fix same lint for the umpteenth time * bpf_loader: map AccessViolation to ReadonlyDataModified only for account region violations * programs/sbf: realloc: add test for large write after realloc Add a test that after a realloc does a large write that spans the original account length and the realloc area. This ensures that memory mapping works correctly across the boundary. * programs/sbf: run test_program_sbf_realloc with both direct_mapping on and off By default test banks test with all features on. This ensures we keep testing the existing code until the new feature is enabled. * bpf_loader: tweak memcmp syscall Split the actual memcmp code in a separate function. Remove check indexing the slices since the slices are guaranteed to have the correct length by construction. * bpf_loader: tweak the memset syscall Use slice::fill, which is effectively memset. * bpf_loader: syscalls: update mem syscalls to work with non contiguous memory With direct mapping enabled, accounts can now span multiple memory regions. * fix lint, rebase mem_ops * Implement CoW for writable accounts * Fix CI * Move CoW to the MemoryMapping level * Update after rbpf API change * Fix merge screwup * Add create_vm macro. Fix benches. * cpi: simplify update_caller_account Simplify the logic to update a caller's memory region when a callee causes an account data pointer to change (eg during CoW) * benches/bpf_loader: move serialization out of create_vm bench * benches/bpf_loader: don't copy accounts when direct mapping is on * Fix review nits * bpf_loader: mem_ops: handle u64 overflow in MemoryChunkIterator::new When starting at u64::MAX, the chunk iterator would always return the empty sequence (None on the first next()) call, instead of returning a memory access violation. Use checked instead of saturating arithmetic to detect the condition and error out. This commit also adds more tests around boundary conditions. * Fix loader-v3 tests: data_mut => data_as_mut_slice * Fix CI * bpf_loader: fix tuner bench: account must be writable With direct mapping on, invalid writes are caught early meaning the tuner would fail on the first store and not consume the whole budget like the benchmark expects. --------- Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
2023-04-28 13:54:39 -07:00
fn concat_regions(regions: &[MemoryRegion]) -> AlignedMemory<HOST_ALIGN> {
let len = regions.iter().fold(0, |len, region| len + region.len) as usize;
let mut mem = AlignedMemory::zero_filled(len);
for region in regions {
let host_slice = unsafe {
slice::from_raw_parts(region.host_addr.get() as *const u8, region.len as usize)
};
mem.as_slice_mut()[(region.vm_addr - MM_INPUT_START) as usize..][..region.len as usize]
.copy_from_slice(host_slice)
}
mem
}
}