Commit Graph

793 Commits

Author SHA1 Message Date
Joe C 03ef611f0c
program-runtime: hoist `RuntimeConfig` up to SVM (#630)
program-runtime: hoist `RuntimeConfig` out into SVM
2024-04-07 10:45:57 -05:00
Alexander Meißner 55c05c5ea5
Fix - `FailedVerification` and `Closed` tombstones (#419)
* Only the verifier can cause FailedVerification, everything else is Closed

* Removes the environments parameter from load_program_accounts().

* cargo fmt

* Simplify invocation of deployed program

* Attempt to invoke a program before it is deployed

* Attempt to invoke a buffer before it is used in a deployment

* Escalates Option return value of load_program_accounts() to load_program_with_pubkey().

* Review feedback
2024-04-05 13:03:18 +02:00
Alessandro Decina f36a45a971
ledger-tool: name rayon threads (#464)
ledger-tool: name and shame rayon threads
2024-03-29 06:13:01 +11:00
Pankaj Garg 91b1ee3df6
Fix: deploy program on last slot of epoch during environment change (#101)
* Fix: deploy program on last slot of epoch during environment change

* solana-runtime: deploy at last epoch slot test

* disable deployment of sol_alloc_free

* Move tx-batch-constructor to its own function

* use new_from_cache

---------

Co-authored-by: Alessandro Decina <alessandro.d@gmail.com>
2024-03-22 07:43:28 -07:00
HaoranYi 0f8f8cd970
Revert deprecate executable feature (#309)
* revert deprecate executable feature

* add native loader account transfer test

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-03-20 10:54:15 -05:00
Pankaj Garg cc3afa5588
Remove public visibility of program cache from bank (#279) 2024-03-17 15:29:20 -07:00
Yihau Chen 51dc7e6fb7
[anza migration]: add 'agave=info' to default log level (#223) 2024-03-14 20:35:33 +08:00
steviez 5ed30beb2a
ledger-tool: Allow compute-slot-cost to operate on dead slots (#213)
Make this command accept the --allow-dead-slots arg as well
2024-03-13 11:29:05 -05:00
Brooks 88f6a7a459
Removes holding storages in AccountsHashVerifier for fastboot (#120) 2024-03-11 17:09:26 -04:00
Dmitri Makarov ba43f74dcf
[SVM] Move RuntimeConfig to program-runtime (#96)
RuntimeConfig doesn't use anything SVM specific and logically belongs
in program runtime rather than SVM.  This change moves the definition
of RuntimeConfig struct from the SVM crate to program-runtime and
adjusts `use` statements accordingly.
2024-03-07 10:16:16 -08:00
Brooks 93f5b514fa
Adds StartingSnapshotStorages to AccountsHashVerifier (#58) 2024-03-04 16:32:51 -05:00
Yihau Chen 3f9a7a52ea [anza migration] rename crates (#10)
* rename geyser-plugin-interface

* rename cargo registry

* rename watchtower

* rename ledger tool

* rename validator

* rename install

* rename geyser plugin interface when patch
2024-03-03 12:31:24 +08:00
Tyera a7f9fe103f
Split runtime utilities out of stake_state.rs (#35386)
* Add points module

* Add rewards module

* Hide rewards doc

* Fixup ledger-tool imports
2024-03-01 09:08:55 -07:00
Sean Young 9bb59aa30f
ledger-tool: verify: add --record-slots and --verify-slots (#34246)
ledger-tool: verify: add --verify-slots and --verify-slots-details

This adds:

    --record-slots <FILENAME>
	Write the slot hashes to this file.

    --record-slots-config hash-only|accounts
	Store the bank (=accounts) json file, or not.

    --verify-slots <FILENAME>
        Verify slot hashes against this file.

The first case can be used to dump a list of (slot, hash) to a json file
during a replay. The second case can be used to check slot hashes against
previously recorded values.

This is useful for debugging consensus failures, eg:

    # on good commit/branch
    ledger-tool verify --record-slots good.json --record-slots-config=accounts

    # on bad commit or potentially consensus breaking branch
    ledger-tool verify --verify-slots good.json

On a hash mismatch an error will be logged with the expected hash vs the
computed hash.
2024-03-01 08:39:30 +00:00
Alexander Meißner 74758d9fbf
Refactor - Move recompilation out of program loading (#35297)
Moves recompilation specifics out of load_program().
2024-02-23 17:25:32 +01:00
steviez bfcd4c8656
ledger-tool: Use error handling in blockstore command code (#35157)
There are lots of operations that could fail, including lots of the
Blockstore calls. The old code matched on Ok(_) or did unwrap()'s
which clutter the code and increase indentation.

This change wraps the entire command in a function that returns a
Result. The wrapper then does a single unwrap_or_else() and prints
any error message. Everywhere else is now free to use the ? operator
2024-02-22 00:19:28 -06:00
Ryo Onodera 024d6ecc4f
Add --unified-scheduler-handler-threads (#35195)
* Add --unified-scheduler-handler-threads

* Adjust value name

* Warn if the flag was ignored

* Tweak message a bit
2024-02-22 09:05:17 +09:00
steviez a1c39a3c22
List the default value for `--accounts` in CLI help (#35254) 2024-02-20 13:12:41 -06:00
Brooks e656e46b24
Makes help's default for `--accounts-index-path` consistent (#35255) 2024-02-20 18:32:06 +00:00
Tyera 9a69e3aa7a
ledger-tool: add warn log if capitalization changes during create-snapshot (#35155)
* Add warn log if capitalization changes during create-snapshot

* Add enable-capitalization-change flag

* Print capitalization message at end
2024-02-16 11:33:15 -07:00
steviez 41f97d7d09
ledger-tool: Add additional modes for accounts subcommand (#34925)
- Add mode to output individual pubkeys
- Add mode to output program accounts
2024-02-08 20:43:11 -06:00
Pankaj Garg 46b9586630
SVM: Move SVM code to its own crate folder (#35119) 2024-02-06 16:06:32 -08:00
Pankaj Garg 3cf5dd2afb
SVM: Move RuntimeConfig to svm folder (#35085) 2024-02-05 13:49:36 -08:00
Brooks f62293918d
Moves the async deleter code to accounts-db (#35040) 2024-02-02 09:21:26 -05:00
Brooks daa2449ad4
Removes RwLock on AccountsDb::shrink_paths (#35027) 2024-02-01 09:35:34 -05:00
steviez b3ea62fba3
ledger-tool: Change --snapshot-archive-path to --snapshots (#35019)
This change makes solana-ledger-tool match solana-validator CLI; the old
flag --snapshot-archive-path is retained as an argument alias and can
still be used for the sake of backwards compatibility.
2024-01-30 22:50:37 -04:00
Brooks e1260a9604
Removes unused parameters from AccountsHashVerifier::new() (#34976) 2024-01-26 21:52:05 +00:00
steviez 3add40fc07
ledger-tool: Refactor accounts subcommand output code (#34915)
The accounts command currently dumps every single account in the
AccountsDb. This is obviously a lot of output, so a previous change
streamed the accounts instead of collecting and dumping at the end.

The streaming approach is much more performant, but the implementation
is non-trivial. This change
- Moves the accounts output code to output.rs
- Refactor the logic to several objects that implment the functionality
- Adjust the json output to also include the summary

This change lays the groundwork for cleanly adding several more flags
that will allow for querying different subsets of accounts.
2024-01-26 00:55:05 -06:00
steviez 89fd6acb8f
ledger-tool: Minor cleanup on --ignore-ulimit-nofile-error flag (#34944)
This argument is a flag and doesn't take a value; however, it had the
.value_name() modifier set with "FORMAT". This could be confusing so
remove .value_name() and add .takes_value(false)
2024-01-25 22:57:26 -06:00
Brooks b150de6d10
Replaces fs-err in clean_orphaned_account_snapshot_dirs() (#34902)
* Replaces fs-err in clean_orphaned_account_snapshot_dirs()

* pr: revert info message format changes
2024-01-23 19:46:02 +00:00
Brooks 8ff511e8fa
Moves create_and_canonicalize_directories() into accounts-db utils (#34882) 2024-01-23 06:46:27 -05:00
Brooks 2f744f1639
Moves create_all_accounts_run_and_snapshot_dirs() into accounts-db utils (#34877) 2024-01-22 18:18:43 -05:00
steviez 3dd348802f
Bubble up genesis load errors instead of exiting (#34851)
The function open_genesis_config() performs several operations that
could fail. If any of these fail, the process exits immediately.

Instead of exiting immediately, bubble up the error and let the caller
decide the appropriate action. solana-validator and solana-ledger-tool
will functionally be unchanged, but this consolidates startup failures
for both of these processes.
2024-01-19 10:25:46 -05:00
steviez a203f1489e
ledger-tool: Condense ProcessOptions parsing logic (#34694)
The code to parse process options was repeated in several (8) commands
that replay block processing. So, move the argument parsing into a
common helper that can be used by all of those commands.
2024-01-11 13:24:21 -06:00
Tyera 166be2995e
Bigtable: update google proto files and allow configuration of max_message_size (#34740)
* Update proto files with tonic-build v0.9.2

* Manually ignore invalid doc-tests

* Add new ReadRowsRequest fields

* Add LedgerStorageConfig::max_message_size and default value

* Add BigtableConnection::max_message_size and use on client creation

* Add max_message_size to RpcBigtableConfig and make const pub

* Add solana-validator cli arg
2024-01-10 21:20:15 -07:00
steviez 7ee9d9a1f2
ledger-tool: Move blockstore commands to blockstore subcommand (#34597)
solana-ledger-tool is a bit of a kitchen sink and currently has
upwards of 30 commands at the top level of the CLI. UI aside, the
contents of ledger-tool/src/main.rs are somewhat cluttered.

To reduce clutter and introduce a more hierarchical structure, this
change introduces a new blockstore subcommand and moves all commands
that only touch the blockstore to a blockstore subcommand. For example,
  solana-ledger-tool slot X 
is now invoked like
  solana-ledger-tool blockstore slot X

However, given how long the tool has existed, support is retained for
calling these old commands without the extra blockstore command. But,
the commands are not listed in the help in order to steer new users to
use the newer calling structure.
2024-01-10 15:42:09 -06:00
steviez 48391152ae
ledger-tool: Minor cleanup on accountsdb config argument parsing (#34671)
- Avoid repeated parsing + use of values_t_or_exit!()
- Move associated items closer to show grouping
2024-01-09 16:42:15 -06:00
steviez 9fe20376b7
ledger-tool: Adjust logic to obtain TransactionStatusService Blockstore (#34646)
TransactionStatusService needs Primary access in order to write
transaction status into the Blockstore if enable_rpc_transaction_history
is set to True. The current logic attempts to get Primary access for the
service.

However, in the event that this function had been called with a
Blockstore that already had Primary access, this second attempt to get
Primary access would fail. So, only attempt to open with Primary access
when necessary AND when the current access level is not sufficient.
2024-01-04 10:30:58 -06:00
steviez 744c2cbe04
ledger-tool: Switch subcommand dispatch from if-else to match (#34644)
A future change will add more cases to this if-else if-...-else chain.
Using a match statement will be easier to follow then a very long
if-else if-... chain.

This change was broken out in order to have a higher signal to noise
ratio in the subsequent change.
2024-01-03 21:53:27 -06:00
steviez 0b49d82d3e
ledger-tool: Move blockstore arg parsing to open_blockstore() (#34596)
The open_blockstore() helper currently takes multiple configurable
options. While the arguments are parsed at a high enough scope in
main.rs to avoid repeated calls, this parsing is duplicated in the
separate subcommand files (program.rs and bigtable.rs).

The repeated parsing is redundant, and also prone to having to missing
an arg (as was the case with bigtable not having wal_recovery_mode).

So, this PR consolidates the parsing to a single function and uses that
function across the previous callers.
2024-01-03 16:04:51 -06:00
steviez 1d93732a60
clippy: ledger-tool lints (#34640)
warning: `flatten()` will run forever if the iterator repeatedly produces an `Err`
    --> ledger-tool/src/main.rs:2649:39
     |
2649 |                 for line in f.lines().flatten() {
     |                                       ^^^^^^^^^ help: replace with: `map_while(Result::ok)`
     |
note: this expression returning a `std::io::Lines` may produce an infinite number of `Err` in case of a read error
    --> ledger-tool/src/main.rs:2649:29
     |
2649 |                 for line in f.lines().flatten() {
     |                             ^^^^^^^^^
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#lines_filter_map_ok
     = note: `#[warn(clippy::lines_filter_map_ok)]` on by default

warning: `solana-ledger-tool` (bin "solana-ledger-tool" test) generated 1 warning
2024-01-03 13:35:05 -06:00
steviez 5b91e299da
ledger-tool: Move ledger output methods to output.rs (#34595)
We have a dedicated file for CLI output, so move the various output
functions there to declutter main.rs file
2024-01-03 09:05:16 -06:00
HaoranYi 531d69fd10
pass feature_set to BorrowedAccount (#34542)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-12-21 20:03:17 +01:00
Tyera 7d209a448d
ledger-tool: support v0 transactions in blocks from bigtable (#34544)
Support v0 transactions
2023-12-20 19:54:39 +00:00
Tyera cc0e5f7a13
ledger-tool: add show-entries option to bigtable block (#34536)
* Add cli flag to show entry data

* Add display structs

* Add writeln_entry helper fn

* Add entry conversion method

* Populate Display for CliBlockWithEntries

* Add ctor from flattened block and entries iterator

* Support show_entries
2023-12-20 09:53:19 -07:00
Ryo Onodera d2b5afc410
Finish unified scheduler plumbing with min impl (#34300)
* Finalize unified scheduler plumbing with min impl

* Fix comment

* Rename leftover type name...

* Make logging text less ambiguous

* Make PhantomData simplyer without already used S

* Make TaskHandler stateless again

* Introduce HandlerContext to simplify TaskHandler

* Add comment for coexistence of Pool::{new,new_dyn}

* Fix grammar

* Remove confusing const for upcoming changes

* Demote InstalledScheduler::context() into dcou

* Delay drop of context up to return_to_pool()-ing

* Revert "Demote InstalledScheduler::context() into dcou"

This reverts commit 049a126c905df0ba8ad975c5cb1007ae90a21050.

* Revert "Delay drop of context up to return_to_pool()-ing"

This reverts commit 60b1bd2511a714690b0b2331e49bc3d0c72e3475.

* Make context handling really type-safe

* Update comment

* Fix grammar...

* Refine type aliases for boxed traits

* Swap the tuple order for readability & semantics

* Simplify PooledScheduler::result_with_timings type

* Restore .in_sequence()

* Use where for aesthetics

* Simplify if...

* Fix typo...

* Polish ::schedule_execution() a bit

* Fix rebase conflicts..

* Make test more readable

* Fix test failures after rebase...
2023-12-19 09:50:41 +09:00
Tyera 84a079e6bc
ledger-tool: make read_only consistent across bigtable subcommands (#34513)
* Make read_only consistent across bigtable subcommands

* Update compare reference-bigtable setting
2023-12-18 16:58:21 -07:00
Tyera 74d02acafd
ledger-tool: Use OutputFormat printer in program subcommand (#34475)
Use OutputFormat printer
2023-12-15 10:38:44 -07:00
steviez eaec42280a
ledger-tool: Disable os memory reporting on verify command by default (#34469)
Negate the name of the arg and change the meaning such that it is now
set-to-enable instead of set-to-disable
2023-12-15 09:52:38 -06:00
Tyera a58e462403
Remove ledger-tool-specific output format enum (#34473) 2023-12-14 23:23:37 -07:00