* Add failing test for precompile transition
* Skip adding builtins if they will be removed
* cargo clean
* nits
* fix abi check
* remove workaround
Co-authored-by: Jack May <jack@solana.com>
* Moves TransactionContext::get_program_key() to InstructionContext::get_program_key().
* Removes TransactionContext::get_loader_key().
* Test full program and loader executable account chain in BPF loader.
* Add simulation detection countermeasures
* Add program and test using TestValidator
* Remove incinerator deposit
* Remove incinerator
* Update Cargo.lock
* Add more features to simulation bank
* Update Cargo.lock per rebase
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
* Moves stack_height of the instruction_trace into InstructionContext.
* Uses nesting_level instead of stack_height internally in transaction_context.rs
* Uses the instruction_trace instead of the instruction_context_stack internally.
* Adds feature gate: record_instruction_in_transaction_context_push.
* Adds BorrowedAccount::check_sysvar().
* Adds BorrowedAccount::get_data_mut().
* Implements account resizing in BorrowedAccount.
* Exposes is_signer() and is_writable() in InstructionContext.
* Removes AccountMeta and get_instruction_accounts_metas().
* Makes throwing errors in BorrowedAccount optional.
* Removes result return values from BorrowedAccount.
* 10461 Reject close of vote accounts unless it earned no credits in the previous epoch. This is checked by comparing current epoch (from clock sysvar) with the most recent epoch with credits in vote state.
* Fixup typo
* Add new feature
* Add new TransactionError
* Add framework for checking account state before and after transaction processing
* Fail transactions that leave new rent-paying accounts
* Only check rent-state of writable tx accounts
* Review comments: combine process_result success behavior; log and metrics before feature activation
* Fix tests that assume rent-exempt accounts are okay
* Remove test no longer relevant
* Remove native/sysvar special case
* Move metrics submission to report legacy->legacy rent paying transitions as well
* Refactor Bank::load_and_execute_transactions
* Refactor: improve type safety of TransactionExecutionResult
* Add enum for extra type safety in execution results
* feedback
* Makes error handling in BorrowedAccount optional.
Adds BorrowedAccount ::get_rent_epoch().
Exposes InstructionContext::get_index_in_transaction().
Turns accounts and account_keys into pinned boxed slices.
* Introduces "unsafe" to InvokeContext::push().
* Turns &TransactionContext into &mut TransactionContext in InvokeContext.
* Push and pop InstructionContext in InvokeContext.
Makes test_process_cross_program and test_native_invoke symmetric.
Removes the borrow check from test_invoke_context_verify.
* Removes keyed_accounts from prepare_instruction()
* Removes usage of invoke_stack.
* Removes keyed_accounts from program-test.
* Removes caller_write_privileges.
* Removes keyed_accounts from BPF parameter (de-)serialization.
* Add more-legitimate conversion from legacy Transaction to SanitizedTransaction
* Add Banks method with preflight checks
* Expose BanksClient method with preflight checks
* Unwrap simulation err
* Add Bank simulation method that works on unfrozen Banks
* Add simpler api
* Better name: BanksTransactionResultWithSimulation
* Adds TransactionContext, InstructionContext and BorrowedAccount.
* Redirects the usage of accounts in InvokeContext through TransactionContext.
Also use the types declared in transaction_context.rs everywhere.
* Adjusts all affected tests.
* Unifies all InstructionRecorders of a transaction into one.
* Stops explicitly compiling CPI instructions for recording,
uses the indices gathered from instruction_accounts instead.
* Removes Rc from Rc<RefCell<AccountSharedData>> in the program-runtime.
* Adjusts tests in bpf_loader, system_instruction_processor, config_processor, vote_instruction and stake_instruction
* use cost model to limit new account creation
* handle every system instruction
* remove &
* simplify match
* simplify match
* add datapoint for account data size
* add postgres error handling
* handle accounts:unlock_accounts
* Moves create_message(), native_invoke() and process_cross_program_instruction()
from the InstructionProcessor to the InvokeContext so that they can have a useful "self" parameter.
* Moves InstructionProcessor into InvokeContext and Bank.
* Moves ExecuteDetailsTimings into its own file.
* Moves Executor into invoke_context.rs
* Moves PreAccount into its own file.
* impl AbiExample for BuiltinPrograms
Allows the use of GPU acceleration in verifying the signatures in Entry's after deserialization in the replay stage
Co-authored-by: Stephen Akridge <sakridge@gmail.com>
Co-authored-by: Ryan Leung <ryan.leung@solana.com>
with results code path.
- fix a bug that could unlock accounts that weren't locked
- add test to the refactored function
- skip enumerating transaction accounts if qos results is an error
- add #[must_use] annotation
- avoid clone error in results
- add qos error code to unlock_accounts match statement
- remove unnecessary AbiExample
* Moves the Executor dyn Trait to instruction_processor.rs
* Moves the Logger dyn Trait as well as the ic_msg and ic_logger_msg macros to log_collector.rs,
and moves the stable_log to stable_log.rs
* Moves the ComputeMeter dyn Trait to invoke_context.rs
* Moves the InvokeContext dyn Trait and the ProcessInstructionWithContext type to invoke_context.rs
* Updates cargo files.
* Re-export InvokeContext in program-test
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
* - cache calculated transaction cost to allow sharing;
- atomic cost tracking op;
- only lock accounts for transactions eligible for current block;
- moved qos service and stats reporting to its own model;
- add cost_weight default to neutral (as 1), vote has zero weight;
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Update core/src/qos_service.rs
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Update core/src/qos_service.rs
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Move blockhash and fee_calculator in ThisInvokeContext instead of using a reference.
* Moves tx_wide_compute_cap into InvokeContext::push().
* Adds ThisInvokeContext::new_mock() constructor.
* Adds missing loader account in uses of MockInvokeContext.
* Use keyed_account_at_index() when accessing keyed_accounts.
* Makes sysvar interface consistent between ThisInvokeContext and MockInvokeContext,
in order to add InvokeContext::get_sysvars().
* Adds InvokeContext::set_blockhash() and InvokeContext ::set_fee_calculator().
* Adds new_mock_with_features.
* Makes ancestors optional in ThisInvokeContext.
* Adds prepare_mock_invoke_context() and mock_process_instruction().
- decouple cost_model from cost_tracker; allowing one cost_model
instance being shared within a validator;
- update cost_model api to calculate_cost(&self...)->transaction_cost
* add filler accounts to bloat validator and predict failure
* assert no accounts match filler
* cleanup magic numbers
* panic if can't load from snapshot with filler accounts specified
* some renames
* renames
* into_par_iter
* clean filler accts, too
* Adds first_instruction_account parameter to process_instruction().
* Removes InvokeContext::remove_first_keyed_account() from all BPF loaders.
* Removes InvokeContext::remove_first_keyed_account() from all builtin programs.
* Removes InvokeContext::remove_first_keyed_account() from all mock ups.
* Deprecates InvokeContext::remove_first_keyed_account().
* Documents index base of keyed_account_at_index().
* Adds dynamic offset to call sites of "keyed_account_at_index()".
* Add separate vote processing tpu port
* Add feature to send to tpu vote port
* Add vote rejecting sigverify mode
* use packet.meta.is_simple_vote_tx in place of deserialization
* consolidate code that identifies vote tx atcommon path for cpu and gpu
* new key for feature set
* banking forward tpu vote
* add tpu vote port to dockerfile and other review changes
* Simplify thread id compare
* fix a test; updated cluster_info ABI change
Co-authored-by: Tao Zhu <tao@solana.com>
Co-authored-by: sakridge <sakridge@gmail.com>
Summary of Changes
Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows
Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.
The code comprises 4 major parts:
accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
* Inlines MessageProcessor::execute_instruction() in MessageProcessor::process_message().
* Moves MessageProcessor::create_pre_accounts() into ThisInvokeContext::push().
* Move instruction_recorders slice into InvokeContext.
* Makes account_indices optional in InvokeContext::push().
* Separates initial InvokeContext::push() from ThisInvokeContext::new().
* invoke_context.pop() the base invocation frame.
* Zip message.instructions.iter() and program_indices.iter().
* Moves ThisInvokeContext::new() to the beginning of the loop inside MessageProcessor::process_message().
* Hoists ThisInvokeContext::new() out of loop inside MessageProcessor::process_message().
* Removes unnecessary clone() from ThisInvokeContext::new() in MessageProcessor ::process_message().
* Stop ignoring errors from MessageProcessor::create_pre_accounts().
* Moves MessageProcessor::verify_account_references() and MessageProcessor::verify() into InvokeContext::verify().
* Inlines MessageProcessor::execute_instruction() in MessageProcessor::process_message().
* Moves MessageProcessor::create_pre_accounts() into ThisInvokeContext::push().
* Hoists ThisInvokeContext::new() out of loop inside MessageProcessor::process_message().
* Moves MessageProcessor::verify_account_references() and MessageProcessor::verify() into InvokeContext::verify().
* Appends loaders / executable_accounts to accounts in transaction loading.
* Adds indices to loaders / executable_accounts.
* Moves MessageProcessor::create_keyed_accounts() into InvokeContext::push().
* Removes "executable_accounts",
now referenced by transaction wide index into "accounts".
* Removes create_pre_accounts() from InstructionProcessor,
as it is already in MessageProcessor.
* Collect program account indices directly in load_executable_accounts().
* Return error if Transaction locks an executable as writable
* Return error if a ProgramData account is writable but the upgradable loader isn't present
* Remove unreachable clause
* Fixup bpf tests
* Review comments
* Add new TransactionError
* Disallow writes to any upgradeable-loader account when loader not present; remove is_upgradeable_loader_present exception for all other executables
* Handle cleaning zero-lamport accounts
Handle cleaning zero-lamport accounts in slots higher than the last full
snapshot slot. This is part of the Incremental Snapshot work.
Fixes#18825
* added realtime cost checking logic to reject block that would exceed max limit:
- defines max limits at block_cost_limits.rs
- right after each bath's execution, accumulate its cost and check again
limit, return error if limit is exceeded
* update abi that changed due to adding additional TransactionError
* To avoid counting stats mltiple times, only accumulate execute-timing when a bank is completed
* gate it by a feature
* move cost const def into block_cost_limits.rs
* redefine the cost for signature and account access, removed signer part as it is not well defined for now
* check if per_program_timings of execute_timings before sending
* Add deprecated tag to Bank::get_blockhash_last_valid_slot
* Update SendTransactionService to use last_valid_block_height
* Update solana-tokens to use last_valid_block_height
* Remove dangling file
* Update solana program to use last_valid_block_height
* Update Banks crates to use last_valid_block_height
Filtering out storages for incremental snapshots will be needed by the
background services for incremental snapshot support, but there is not a
Bank at that point. Since the filtering doesn't apply only to Bank, and
more to snapshots, move the functionality into snapshot_utils.
This commit adds high-level functions for creating and loading-from
incremental snapshots, plus all low-level functions required to perform
those tasks. This commit **does not** add taking incremental snapshots
as part of a running validator, nor starting up a node with an
incremental snapshot; just laying ground work.
Additionally, `snapshot_utils` and `serde_snapshot` have been
refactored to use a common code paths for the different snapshots.
Also of note, some renaming has happened:
1. Snapshots are now either `full_` or `incremental_` throughout the
codebase. If not specified, the code applies to both.
2. Bank snapshots now are called "bank snapshots"
(before they were called "slot snapshots", "bank snapshots", or
just "snapshots"). The one exception is within `Bank`, where they
are still just "snapshots", because they are already "bank
snapshots".
3. Snapshot archives now have `_archive` in the code. This
should clear up an ambiguity between bank snapshots and snapshot
archives.
* feature: add new token program feature
* Fixup test
* Update to spl-token v3.2.0
* Update Cargo.lock + fmt
* Update token program version in fetch-spl.sh
* Bump associated token program to 1.0.3
* Add aToken so
* Move transaction sanitization earlier in the pipeline
* Renamed HashedTransaction to SanitizedTransaction
* Implement deref for sanitized transaction
* bring back process_transactions test method
* Use sanitized transactions for cost model calculation
* Changes ThisInvokeContext::get_account() to use accounts instead of pre_accounts.
* Adds explicit keys to accounts to make them symmetric to account_deps.
* Appends account_deps to accounts in transaction loading and removes account_deps everywhere else.
1. Added both options for measuring space usage using total accounts usage and for individual store shrink ratio using an enum. Validator CLI options: --accounts-shrink-optimize-total-space and --accounts-shrink-ratio
2. Added code for selecting candidates based on total usage in a separate function select_candidates_by_total_usage
3. Added unit tests for the new functions added
4. The default implementations is kept at 0.8 shrink ratio with --accounts-shrink-optimize-total-space set to true
Fixes#17544
* Accounts dumping logic
* Add test for interaction between cache flush and remove_unrooted_slot()
* Update comments
* Rename
* renaming
* Add more comments
* Renaming
* Fixup test and bad check
* Upgrade Rust to 1.52.0
update nightly_version to newly pushed docker image
fix clippy lint errors
1.52 comes with grcov 0.8.0, include this version to script
* upgrade to Rust 1.52.1
* disabling Serum from downstream projects until it is upgraded to Rust 1.52.1
* AccountSecondaryIndexes.include/exclude
* use normal scan if key is not indexed
* add a test to ask for a scan for an excluded secondary index
* fix up cli args
* distribute_rent_to_validators checked_add_lamports unwrap
* make rent disappear on add failure
* add pubkey to message
* update message text
* don't store account that we failed to transfer to
* format
In a scenario where a bank is unrooted and dropped, any keys that exist
_only_ in that bank are now cleaned up.
This work was originally based on PR #15106.
* add/sub lamports
* make add/sub return Result
* sample replacements
* cleanup
* fix up a few tests as examples
* move enum, cleanup, impl from
* fmt
* cleanup
* add lamports.rs
* Account for possibility of cache flush in load()
* More cleaning
* More cleaning
* Remove unused method and some comment cleaning
* Fix typo
* Make the detected impossible purge race panic()!
* Finally revert to original .expect()
* Fix typos...
* Add assertion for max_root for easier reasoning
* Reframe races with LoadHint as possible opt.
* Fix test
* Make race bug tests run longer for less flaky
* Delay the clone-in-lock slow path even for RPC
* Make get_account panic-free & add its onchain ver.
* Fix rebase conflicts...
* Clean up
* Clean up comment
* Revert fn name change
* Fix flaky test...
* fmt...
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
Account data hashing used to use different ways of hashing on different
clusters. That is no longer the case, but the old code still existed.
This commit removes that old, now used code.
**NOTE** The golden hash values in bank.rs needed to be updated. Since
the original code that selected the hash algorithm used `if >` instead
of `if >=`, this meant that the genesis block's hash _always_ used the
old hashing method, which is no longer valid.
Validated by running `cargo test` successfully.
* Track transaction check time separately from account loads
* banking packet process metrics
* Remove signature clone in status cache lookup
* Reduce allocations when converting packets to transactions
* Add blake3 hash of transaction messages in status cache
* Bug fixes
* fix tests and run fmt
* Address feedback
* fix simd tx entry verification
* Fix rebase
* Feedback
* clean up
* Add tests
* Remove feature switch and fall back to signature check
* Bump programs/bpf Cargo.lock
* clippy
* nudge benches
* Bump `BankSlotDelta` frozen ABI hash`
* Add blake3 to sdk/programs/Cargo.lock
* nudge bpf tests
* short circuit status cache checks
Co-authored-by: Trent Nelson <trent@solana.com>
* Drop write lock on sysvars
* adds env var for demoting sysvar write lock demotion
* moves demote logic to is_writable
* feature gates sysvar write lock demotion
* adds builtins to write lock demotion
* adds system program id to builtins
* adds Feature111...
* adds an abi-freeze test
* mvines set of builtin program keys
Co-authored-by: Michael Vines <mvines@gmail.com>
* update tests
* adds bpf loader keys
* Add test sysvar
* Plumb demote_sysvar to is_writable
* more plumbing of demote_sysvar_write_locks to is_writable
* patches test_program_bpf_instruction_introspection
* hard codes demote_sysvar_write_locks to false for serialization/encoding methods
* Revert "hard codes demote_sysvar_write_locks to false for serialization/encoding methods"
This reverts commit ae3e2d2e777437bddd753933097a210dcbc1b1fc.
* change the hardcoded ones to demote_sysvar_write_locks=true
* Use data_as_mut_slice
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
* Add some code for special local testing
* Add comment to store_account_and_update_capitalization
* Simplify account.rent_epoch handling for sysvar rent
* Introduce *_for_test functions
* Add deprecation messages to existing api
* Move to module-level clippy allowance
* Fix stake_weighted_timestamp math
* Fix genesis_config
* Fix shred_version
* Fix hard_forks
* Fix process_instruction
* Add ArithmeticOverflow ix error
* Fix nonce_keyed_account
* Update BankSlotDelta frozen abi due to new ix error
* sdk: Add Borsh to Pubkey
* Add serialization error for easier borsh integration
* Add Borsh usage to banks-client and sdk
* Rename SerializationError -> IOError
* Add new errors to proto
* Update Cargo lock
* Update Cargo.lock based on CI
* Clippy
* Update ABI on bank
* Address review feedback
* Update sanity program instruction count test
* cleanup if
* pass expected capitalization to hash calculation to improve assert message
* fix bank function
* one more level
* calculate_accounts_hash_helper
* add slot to error message
* success
* Simpler cap by including sysvars and native programs
* Fix tests
* Add comment
* revert some unrelated code
* Update test_bank_update_sysvar_account for cap.
* Test cap. for add_native_program using new helper
* Improve the cap adjustment with new tests
* Fix typo...
* Adjust test for improved code coverage
* Rename simpler_capitalization => simple_capitalization
* More rename and bonus commenting
* Move bank drop to AccountsBackgroundService
* Send to ABS on drop instead, protects against other places banks are dropped
* Fix Abi
* test
Co-authored-by: Carl Lin <carl@solana.com>
* Adds a CLI option to the validator to enable just-in-time compilation of BPF.
* Refactoring to use bpf_loader_program instead of feature_set to pass JIT flag from the validator CLI to the executor.
Gossip and other places repeatedly de-serialize vote-state stored in
vote accounts. Ideally the first de-serialization should cache the
result.
This commit adds new VoteAccount type which lazily de-serializes
VoteState from Account data and caches the result internally.
Serialize and Deserialize traits are manually implemented to match
existing code. So, despite changes to frozen_abi, this commit should be
backward compatible.
* runtime: Add `FeeCalculator` resolution method to `HashAgeKind`
* runtime: Plumb fee-collected accounts for failed nonce tx rollback
* runtime: Use fee-collected nonce/fee account for nonced TX error rollback
* runtime: Add test for failed nonced TX accounts rollback
* Fee payer test
* fixup: replace nonce account when it pays the fee
* fixup: nonce fee-payer collect test
* fixup: fixup: clippy/fmt for replace...
* runtime: Test for `HashAgeKind::fee_calculator()`
* Clippy
Co-authored-by: Trent Nelson <trent@solana.com>
* Add new inflation feature-ids, and full_inflation default values
* Compute inflation start from full_inflation activation
* Include pico_inflation in inflation start computation
* Add full-inflation constructor
* Align inflation taper with rewards accrual start and catch overflow edge case
* Fix fragile tests in prep of stake rewrite pr
* Restore BOOTSTRAP_VALIDATOR_LAMPORTS where appropriate
* Further clean up
* Further clean up
* Aligh with other call site change
* Remove false warn!
* fix ci!
* Fix slow/stuck unstaking due to toggling in epoch
* nits
* nits
* Add stake_program_v2 feature status check to cli
Co-authored-by: Tyera Eulberg <tyera@solana.com>
* Test for different ancestors with mismatch bank hash
* Test cleanup
* Remove nondeterministic ancestor check
* Update timestamp bounding feature key
* Update design doc
* Filter recent_timestamps to nodes voting within the last epoch
Co-authored-by: Stephen Akridge <sakridge@gmail.com>
* Output more inflation calc details in ledger-tool
* Fix broken ci...
* Rename confusing variables
* Fix panic by wrapping PointValue with Opiton...
* Minor modifications
* Remove explict needless flush; Drop already does
* Yet another csv field adjustments
* Add data_size and rename epochs to earned_epochs
* Introduce null_tracer
* Unwrap Option in new_from_parent_with_tracer
* Don't shorten identifiers
* Allow irrefutable_let_patterns temporalily
* More null_tracer
* More field adjustments
* Re-use accounts_db stores
Creating files and dropping mmap areas can be expensive
* Add test for storage finder
Can encounter an infinite loop when the store is too small, but
smaller than the normal store size.
* Fix storage finding
* Check for strong_count == 1
* try_recycle helper
Gavin Chan
Justin Starry
Tyera Eulberg
buffalu
Jack May
Alexander Meißner
Michael Vines
Brooks Prumo
Jeff Washington (jwash)
carllin
Will Hickey
Jon Cinque
Tao Zhu
Tyera Eulberg
Trent Nelson
behzad nouri
segfaultdoctor
Ashwin Sekar
ryleung-solana
Tao Zhu
sakridge
Jeff Washington (jwash)
Lijun Wang
Sean Young
Ryo Onodera
jon-chuang
Greg Fitzgerald
carllin
Eric Williams