* Makes members of SyscallContext public.
Removes check_aligned and check_size from SyscallContext.
* Replaces InvokeContext::set_syscall_context() in tests with mock_create_vm!().
* Passes SyscallContext directly to InvokeContext::set_syscall_context().
* Merges TraceLogStackFrame into SyscallContext.
* Removes the create_vm!() macro.
* Moves BpfAllocator from bpf_loader into program_runtime.
* Frees BpfAllocator from Rc<RefCell<>>.
* Removes unused code from BpfAllocator.
* Consume CUs for heap before doing the allocation.
* Exposes syscalls in rbpf-cli.
* Adds debugging_features parameter to load_program_from_bytes() and load_program_from_account().
* Removes test_bpf_loader_non_terminating_program() as that is already tested in RBPF.
* Moves stack and heap allocation back onto the program runtime stack.
* Replaces assign_program() by replenish() in load_and_get_programs_from_cache().
* Replaces LoadedProgramEntry by a boolean.
* Only lock the global cache once all missing programs have been loaded
* Factors out call of self.rc.accounts.filter_executable_program_accounts().
* add feature gate
* builtins consume statically defined units at beginning of process_instruction()
* Add new instructionError; return error if builtin did not consume units to enforce builtin to consume units;
* updated related tests
* updated ProgramTest with deactivated native_programs_consume_cu feature to continue support existing mock/test programs that do not consume units
* include loaded accounts data size limit in transaction base fee calculation
* citing compute_budget for heap cost;
* update sbf tests
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* add default_cost as mandatory field for Builtin
* updated tests
* set zkp program default to VerifyTransfer CUs
---------
Co-authored-by: Jon Cinque <joncinque@pm.me>
1. replace hardcoded loaded accounts data size limit with compute budget instruction;
2. new transaction error for invalid account data size limit
3. test requested limit with combination of features and transactions
* Find and load missing programs in LoadedPrograms cache
- filter program accounts in a transaction batch
- filter the accounts that are missing in LoadedPrograms cache
- load the programs before processing the transactions
- unit tests
* address review comments
* fix clippy
* address review comments
* fix test
* fix more tests
* Use three separate HashMaps instead of the enum TxBankExecutorCacheDiff.
* Replaces all places which deploy programs by a macro.
* Adds a feature gate.
* Adjust tests.
* Makes undeployment visible immediately.
* Add address lookup tables to minimized snapshot
* Add comment for future posterity
* Add reference to the issue
* Adjust comment a bit
Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>
---------
Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>
chore: Move bank new epoch metric into separate file
bank.rs is very large, so move the metrics reporting code into a
separate file. This reduces line count and also makes the actual logic
in bank.rs more readable by removing a big block of boilerplate code.
There are more metrics that we can do this for, but doing them one at a
time to avoid large diffs.
Problem
The plugins need to know when all transactions for a block have been all notified to serve getBlock request correctly. As block and transaction notifications are sent asynchronously to each other it will be difficult.
Summary of Changes
Include the executed transaction count in block notification which can be used to check if all transactions have been notified.
* check android builds
* feature: set rent_epoch to Epoch::MAX
* tweaks
* Update runtime/src/rent_collector.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* simplify changes to tests
* back out some test changes
* calculate_rent_result passes through Exempt
* move calc outside loop
* if rent epoch is already max, use 'NoRentCollectionNow'
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Support versioned transactions in program test framework
* use working bank
* Update to process_transaction_with_metadata
* Migrate client apis from Transaction to Into<VersionedTransaction>
* feedback
- Add new compute-budget instruction to set transaction-wide accounts data size limit
- Set default accounts data limit to 10MB, and max to 100MB, per transaction;
- Add getters to make changing default and/or max values easier in the future with feature gates;
- added error counter for transactions exceed data size limit
* Moves CreateMetrics into the program-runtime crate.
* Moves the Executor trait into executor.rs
* Removes the first_instruction_account parameter from Executor::execute().
* removes feature gate code preserving rent_epoch for rent exempt accounts
* update test expected hashes
Co-authored-by: jeff washington <jeff.washington@solana.com>
If a validator starts with a snapshot from the wrong cluster, an assert
will trigger complaining about the genesis_creation_time encoded in the
snapshot differing from the genesis_creation_time embedded in
genesis.bin.
This could be a fairly easy mistake to make so add a help string to the
assertion to point operators in the right direction.
* Split out voting and banking threads in banking stage
Additionally this allows us to aggressively prune the buffer for voting threads
as with the new vote state only the latest vote from each validator is
necessary.
* Update local cluster test to use new Vote ix
* Encapsulate transaction storage filtering better
* Address pr comments
* Commit cargo lock change
* clippy
* Remove unsafe impls
* pr comments
* compute_sanitized_transaction -> build_sanitized_transaction
* &Arc -> Arc
* Move test
* Refactor metrics enums
* clippy
* Moves the integration test test_bpf_loader_upgradeable_deploy_with_max_len() from the bpf_loader into the runtime.
* Adds Bank::create_executor().
* Adds a test for Bank::create_executor().
* Moves CachedExecutors, related structs, consts and tests into the program-runtime crate.
* Moves TransactionExecutor, related enum and type defs into executor_cache mod.
* BorrowedAccount: add set_data_from_slice(), make set_data() take owned values
set_data() used to take a slice and would force alloc+copy if the caller
has owned values (eg account creation, account lookup table).
Expose set_data_from_slice() for callers that have slices, and switch
set_data() to taking an owned Vec.
* BorrowAccount: refactor common accounts_update_delta code in helper method
* BorrowedAccount: add extend_from_slice()
This allows avoiding copies appending entries to account lookup tables.
* BorrowedAccount: remove unnecessary ifs around update_accounts_resize_delta
* Replaces InstructionContext::new() by InstructionContext::configure().
Adds TransactionContext::get_next_instruction_context().
* Switch back to using references as parameters.
* Hoists InstructionContext::configure() from TransactionContext::push() into InvokeContext::push().
* Moves InstructionContext::config() to the beginning of InvokeContext::push().
* Hoists InstructionContext::configure() from InvokeContext::push() into InvokeContext::process_instruction().
* Review feedback: Updates docu comments.
* refactor: extract store_stake_accounts fn
* refactor: extract store_vote_account fn
* refactor: extract reward history update fn
* remove avg point value from pay_valiator fn. not used
* clippy: slice
* clippy: slice
* remove abort() from test-validator (#27124)
* chore: bump bytes from 1.1.0 to 1.2.1 (#27172)
* chore: bump bytes from 1.1.0 to 1.2.1
Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.1.0 to 1.2.1.
- [Release notes](https://github.com/tokio-rs/bytes/releases)
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/bytes/compare/v1.1.0...v1.2.1)
---
updated-dependencies:
- dependency-name: bytes
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
* Share Ancestors API get with contains_key (#27161)
consolidate similar fns
* Rename to `MAX_BLOCK_ACCOUNTS_DATA_SIZE_DELTA` (#27175)
* chore: bump libc from 0.2.129 to 0.2.131 (#27162)
* chore: bump libc from 0.2.129 to 0.2.131
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.129 to 0.2.131.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.129...0.2.131)
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
* reverts wide fanout in broadcast when the root node is down (#26359)
A change included in
https://github.com/solana-labs/solana/pull/20480
was that when the root node in turbine broadcast tree is down, the
leader will broadcast the shred to all nodes in the first layer.
The intention was to mitigate the impact of dead nodes on shreds
propagation, because if the root node is down, then the entire cluster
will miss out the shred.
On the other hand, if x% of stake is down, this will cause 200*x% + 1
packets/shreds ratio at the broadcast stage which might contribute to
line-rate saturation and packet drop.
To avoid this bandwidth saturation issue, this commit reverts that logic
and always broadcasts shreds from the leader only to the root node.
As before we rely on erasure codes to recover shreds lost due to staked
nodes being offline.
* add getTokenLargestAccounts rpc method to rust client (#26840)
* add get token largest accounts rpc call to client
* split to include with commitment
* Bump spl-token-2022 (#27181)
* Bump token-2022 to 0.4.3
* Allow cargo to bump stuff to v1.11.5
* VoteProgram.safeWithdraw function to safeguard against accidental vote account closures (#26586)
feat: safe withdraw function
Co-authored-by: aschonfeld <andrew@proofofalpha.io>
* chore: bump futures from 0.3.21 to 0.3.23 (#27182)
* chore: bump futures from 0.3.21 to 0.3.23
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.21 to 0.3.23.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.21...0.3.23)
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
* chore: bump nix from 0.24.2 to 0.25.0 (#27179)
* chore: bump nix from 0.24.2 to 0.25.0
Bumps [nix](https://github.com/nix-rust/nix) from 0.24.2 to 0.25.0.
- [Release notes](https://github.com/nix-rust/nix/releases)
- [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nix-rust/nix/compare/v0.24.2...v0.25.0)
---
updated-dependencies:
- dependency-name: nix
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
* Parse ConfidentialTransaction instructions (#26825)
Parse ConfidentialTransfer instructions
* snapshots: serialize version file first (#27192)
serialize version file first
* serialize incremental_snapshot_hash (#26839)
* serialize incremental_snapshot_hash
* pr feedback
* derives Error trait for ClusterInfoError and core::result::Error (#27208)
* Add clean_accounts_for_tests() (#27200)
* Rust v1.63.0 (#27148)
* Upgrade to Rust v1.63.0
* Add nightly_clippy_allows
* Resolve some new clippy nightly lints
* Increase QUIC packets completion timeout
Co-authored-by: Michael Vines <mvines@gmail.com>
* docs: updated "transaction fees" page (#26861)
* docs: transaction fees, compute units, compute budget
* docs: added messages definition
* Revert "docs: added messages definition"
This reverts commit 3c56156dfaaf17158c5eafbc5877080a83607a06.
* docs: added messages definition
* Update docs/src/transaction_fees.md
Co-authored-by: Jacob Creech <82475023+jacobcreech@users.noreply.github.com>
* fix: updates from feedback
Co-authored-by: Jacob Creech <82475023+jacobcreech@users.noreply.github.com>
* sdk: Fix args after "--" in build-bpf and test-bpf (#27221)
* Flaky Unit Test test_rpc_subscriptions (#27214)
Increase unit test timeout from 5 seconds to 10 seconds
* chore: only buildkite pipelines use sccache in docker-run.sh (#27204)
chore: only buildkite ci use sccache
* clean feature: `prevent_calling_precompiles_as_programs` (#27100)
* clean feature: prevent_calling_precompiles_as_programs
* fix tests
* fix test
* remove comment
* fix test
* feedback
* Add get_account_with_commitment to BenchTpsClient (#27176)
* Fix a corner-case panic in get_entries_in_data_block() (#27195)
#### Problem
get_entries_in_data_block() panics when there's inconsistency between
slot_meta and data_shred.
However, as we don't lock on reads, reading across multiple column families is
not atomic (especially for older slots) and thus does not guarantee consistency
as the background cleanup service could purge the slot in the middle. Such
panic was reported in #26980 when the validator serves a high load of RPC calls.
#### Summary of Changes
This PR makes get_entries_in_data_block() panic only when the inconsistency
between slot-meta and data-shred happens on a slot older than lowest_cleanup_slot.
* Verify snapshot slot deltas (#26666)
* store-tool: log lamports for each account (#27168)
log lamports for each account
* add an assert for a debug feature to avoid wasted time (#27210)
* remove redundant call that bumps age to future (#27215)
* Use from_secs api to create duration (#27222)
use from_secs api to create duration
* reorder slot # in debug hash data path (#27217)
* create helper fn for clarity (#27216)
* Verifying snapshot bank must always specify the snapshot slot (#27234)
* Remove `Bank::ensure_no_storage_rewards_pool()` (#26468)
* cli: Add subcommands for address lookup tables (#27123)
* cli: Add subcommand for creating address lookup tables
* cli: Add additional subcommands for address lookup tables
* short commands
* adds hash domain to ping-pong protocol (#27193)
In order to maintain backward compatibility, for now the responding node
will hash the token both with and without domain so that the other node
will accept the response regardless of its upgrade status.
Once the cluster has upgraded to the new code, we will remove the legacy
domain = false case.
* Revert "Rust v1.63.0 (#27148)" (#27245)
This reverts commit a2e7bdf50a.
* correct double negation (#27240)
* Enable QUIC client by default. Add arg to disable QUIC client. (Forward port #26927) (#27194)
Enable QUIC client by default. Add arg to disable QUIC client.
* Enable QUIC client by default. Add arg to disable QUIC client.
* Deprecate --disable-quic-servers arg
* Add #[ignore] annotation to failing tests
* slots_connected: check if the range is connected (>= ending_slot) (#27152)
* create-snapshot check if snapshot slot exists (#27153)
* Add Bank::clean_accounts_for_tests() (#27209)
* Call `AccountsDb::shrink_all_slots()` directly (#27235)
* add ed25519_program to built-in instruction cost list (#27199)
* add ed25519_program to built-in instruction cost list
* Remove unnecessary and stale comment
* simple refactorings to disk idx (#27238)
* add _inclusive for clarity (#27239)
* eliminate unnecessary ZERO_RAW_LAMPORTS_SENTINEL (#27218)
* make test code more clear (#27260)
* banking stage: actually aggregate tracer packet stats (#27118)
* aggregated_tracer_packet_stats_option was alwasys None
* Actually accumulate tracer packet stats
* Refactor epoch reward 1 (#27253)
* refactor: extract store_stake_accounts fn
* clippy: slice
Co-authored-by: haoran <haoran@mbook>
* recovers merkle shreds from erasure codes (#27136)
The commit
* Identifies Merkle shreds when recovering from erasure codes and
dispatches specialized code to reconstruct shreds.
* Coding shred headers are added to recovered erasure shards.
* Merkle tree is reconstructed for the erasure batch and added to
recovered shreds.
* The common signature (for the root of Merkle tree) is attached to all
recovered shreds.
* Simplify `Bank::clean_accounts()` by removing params (#27254)
* Account files remove (#26910)
* Create a new function cleanup_accounts_paths, a trivial change
* Remove account files asynchronously
* Update and simplify the implementation after the validator test runs.
* Fixes after testing on the dev device
* Discard tokio. Use thread instead
* Fix comments format
* Fix config type to pass the github test
* Fix failed tests. Handle the case of non-existing path
* Final cleanup, addressing the review comments
Avoided OsString.
Made the function more generic with "impl AsRef<Path>"
Co-authored-by: Jeff Washington <jeff.washington@solana.com>
* Refactor: Flattens `TransactionContext::instruction_trace` (#27109)
* Flattens TransactionContext::instruction_trace.
* Stop the search at transaction level.
* Renames get_instruction_context_at => get_instruction_context_at_nesting_level.
* Removes TransactionContext::get_instruction_trace().
Adds TransactionContext::get_instruction_trace_length() and TransactionContext::get_instruction_context_at_index().
* Have TransactionContext::instruction_accounts_lamport_sum() accept an iterator instead of a slice.
* Removes instruction_trace from ExecutionRecord.
* make InstructionContext::new() private
* Parallel insertion of dirty store keys during clean (#27058)
parallelize dirty store key insertion
* Refactor epoch reward 2 (#27257)
* refactor: extract store_stake_accounts fn
* refactor: extract store_vote_account fn
* clippy: slice
* clippy: slice
* fix merge error
Co-authored-by: haoran <haoran@mbook>
* Standardize thread names
Tenets:
1. Limit thread names to 15 characters
2. Prefix all Solana-controlled threads with "sol"
3. Use Camel case. It's more character dense than Snake or Kebab case
* cleanup comment on filter_zero_lamport_clean_for_incremental_snapshots (#27273)
* remove inaccurate log (#27255)
* patches metrics for invalid cached vote/stake accounts (#27266)
patches invalid cached vote/stake accounts metrics
Invalid cached vote accounts is overcounting actual mismatches, and
invalid cached stake accounts is undercounting.
* Refactor epoch reward 3 (#27259)
* refactor: extract store_stake_accounts fn
* refactor: extract store_vote_account fn
* refactor: extract reward history update fn
* clippy: slice
* clippy: slice
Co-authored-by: haoran <haoran@mbook>
* fix merges
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: haoran <haoran@mbook>
Co-authored-by: Jeff Biseda <jbiseda@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Brooks Prumo <brooks@solana.com>
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Co-authored-by: AJ Taylor <anthonjtaylor@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
Co-authored-by: Andrew Schonfeld <andrew.schonfeld1@gmail.com>
Co-authored-by: aschonfeld <andrew@proofofalpha.io>
Co-authored-by: apfitzge <apfitzge@users.noreply.github.com>
Co-authored-by: Jeff Washington (jwash) <wash678@gmail.com>
Co-authored-by: Brennan Watt <brennan.watt@solana.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
Co-authored-by: Nick Frostbutter <75431177+nickfrosty@users.noreply.github.com>
Co-authored-by: Jacob Creech <82475023+jacobcreech@users.noreply.github.com>
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
Co-authored-by: Yihau Chen <a122092487@gmail.com>
Co-authored-by: Justin Starry <justin@solana.com>
Co-authored-by: kirill lykov <kirill.lykov@solana.com>
Co-authored-by: Yueh-Hsuan Chiang <93241502+yhchiang-sol@users.noreply.github.com>
Co-authored-by: leonardkulms <42893075+leonardkulms@users.noreply.github.com>
Co-authored-by: Will Hickey <will.hickey@solana.com>
Co-authored-by: Tao Zhu <82401714+taozhu-chicago@users.noreply.github.com>
Co-authored-by: Xiang Zhu <xzhu70@gmail.com>
Co-authored-by: Jeff Washington <jeff.washington@solana.com>
Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
* Flattens TransactionContext::instruction_trace.
* Stop the search at transaction level.
* Renames get_instruction_context_at => get_instruction_context_at_nesting_level.
* Removes TransactionContext::get_instruction_trace().
Adds TransactionContext::get_instruction_trace_length() and TransactionContext::get_instruction_context_at_index().
* Have TransactionContext::instruction_accounts_lamport_sum() accept an iterator instead of a slice.
* Removes instruction_trace from ExecutionRecord.
* make InstructionContext::new() private
* Concurrent replay slots
* Split out concurrent and single bank replay paths
* Sub function processing of replay results for readability
* Add feature switch for concurrent replay
* Adjusts test cases for stricter requirements.
* Removes account reset in deserialization test.
* Removes verify related test cases.
* Replicates account modification verification logic of PreAccount in BorrowedAccount.
* Adds TransactionContext::account_touched_flags.
* Adds account modification verification to the BPF ABIv0 and ABIv1 deserialization, CPI syscall and program-test.
* Replicates the total sum of all lamports verification of PreAccounts in InstructionContext
* Check that the callers instruction balance is maintained during a call / push.
* Replicates PreAccount statistics in TransactionContext.
* Disable verify() and verify_and_update() if the feature enable_early_verification_of_account_modifications is enabled.
* Moves Option<Rent> of enable_early_verification_of_account_modifications into TransactionContext::new().
* Relaxes AccountDataMeter related test cases.
* Don't touch the account if nothing changes.
* Adds two tests to trigger InstructionError::UnbalancedInstruction.
Co-authored-by: Justin Starry <justin@solana.com>
https://github.com/solana-labs/solana/pull/22292
prevents rent paying account creation going forward. As a result
rent_epoch field for rent exempt accounts is redundant, and advancing
this field will incur expensive account rewrites and cause discrepancy
between accounts-db and cached vote/stake accounts.
This commit adds a feature which upon activation preserves rent_epoch
field for rent exempt accounts so that the field is frozen and is no
longer advanced.
Alexander Meißner
Trent Nelson
Brennan
HaoranYi
Pankaj Garg
Andrew Fitzgerald
Tao Zhu
Brooks
Ryo Onodera
Alessandro Decina
Jeff Washington (jwash)
Tyera
behzad nouri
steviez
Lijun Wang
Illia Bobyr
Justin Starry
K-anon
Ashwin Sekar
Ian Macalinao
Michael Vines
Christian Kamm
Jeff Washington (jwash)
Jack May
Nicholas Clarke