Commit Graph

494 Commits

Author SHA1 Message Date
Alexander Meißner ee2c2ef6c7
Cleanup - require_static_program_ids_in_transaction (#31767)
require_static_program_ids_in_transaction
2023-06-07 17:12:41 +02:00
Brooks 7b37a1672f
Enables fastboot for ledger-tool (#31921) 2023-06-06 18:32:24 -04:00
Pankaj Garg c86e160438
Add runtime environment to FailedVerification tombstones (#31991)
* Add runtime environment to FailedVerification tombstones

* modify default variant

* prune update

* add DelayVisibility in prune_feature_set_transition
2023-06-06 22:24:39 +00:00
Illia Bobyr 4353ac6797
Pass Arc<AtomicBool> by value, not by reference. (#31916)
`Arc` is already a reference internally, so it does not seem to be
beneficial to pass a reference to it.  Just adds an extra layer of
indirection.

Functions that need to be able to increment `Arc` reference count need
to take `Arc<AtomicBool>`, but those that just want to read the
`AtomicBool` value can accept `&AtomicBool`, making them a bit more
generic.

This change focuses specifically on `Arc<AtomicBool>`.  There are other
uses of `&Arc<T>` in the code base that could be converted in a similar
manner.  But it would make the change even larger.
2023-06-01 17:25:48 -07:00
steviez 1d6b03358f
ledger-tool: Rename load_frozen_forks() and adjust error messages (#31903)
This function has morphed as it has been updated, and the existing
function name is a bit of misnomer. Update the function name to use
terminology that I think is more clear and more consistent with what we
use core/src/validator.rs. Additionally, adjust the error messages to be
more clear about what exactly is wrong for when we can't process.
2023-05-31 21:12:49 -05:00
steviez debe794987
Replace improper &Arc<...> with Arc<...> in Bank and Accounts (#31892)
The callstack updated in this PR passed an &Arc<...> down only to have
the bottom level clone the reference. Thus, we are giving shared
ownership so the reference is a bit redundant and arguably obscures the
intention to clone further down the callstack.
2023-05-31 12:36:44 -05:00
Brooks 9220e53e23
ledger-tool: Clean up load_bank_forks() (#31870) 2023-05-31 06:50:33 -04:00
Jeff Washington (jwash) aff9c03e7a
fix cli args (#31804) 2023-05-25 07:47:00 -07:00
Jeff Washington (jwash) 461342cdd4
add cli args and use of partitioned rewards config (#31800)
* add cli args and use of partitioned rewards config

* update comments
2023-05-24 10:54:09 -07:00
Andrew Fitzgerald f52ded35f4
async delete contents but leave directory (#31737)
* async delete contents but leave directory

* Clarified comment
2023-05-23 15:33:09 -07:00
Alexander Meißner 83f692ce67
Refactor - Share RBPF config and syscall registry globally (#31727)
* Adds LoadedPrograms::program_runtime_environment_v1.

* No need to explicitly remove precompiles from program_accounts_map.
2023-05-20 20:28:00 +02:00
Alexander Meißner 520c647918
Refactor - Merges `BuiltinPrograms` into `LoadedPrograms` (#31654)
* Merges BuiltinPrograms into LoadedPrograms.

* Prevents built-ins from being pruned.
2023-05-18 22:18:28 +02:00
Pankaj Garg b03422968b
Remove Rc and RefCell to carry program cache in invoke_context (#31684) 2023-05-17 06:10:50 -07:00
Pankaj Garg 9b860abfcc
Fix warning about unused mut in ledger-tool (#31686) 2023-05-17 00:17:42 +00:00
Trent Nelson ad67fd5be5
validator: remove optional remote accounts hash consistency check (#31279) 2023-05-16 14:23:13 -06:00
steviez 02679eb5f9
ledger-tool: Move ELF magic numbers into constant with comment (#31655) 2023-05-15 19:32:59 +00:00
Dmitri Makarov 950892e503
Fix ledger-tool program command for non-runnable subcommands (#31596)
* Fix ledger-tool program command for non-runnable subcommands

These subcommands (cfg and disassemble) don't need to load a ledger
database for running, and neither do they need an input file
specifying instruction accounts etc. This change creates a simplified
invoke_context for processing the static actions of ledger-tool
program subcommand.

* Factor out program file loading
2023-05-15 14:23:54 -04:00
Alexander Meißner 5c8b5a2a68
Bump solana_rbpf to v0.4.0 (#31594)
* Moves "disable_deploy_of_alloc_free_syscall" parameter inside create_loader().

* Removes the "is_abi_v2" flag.

* Bumps solana_rbpf to v0.4.0
2023-05-12 18:07:13 +02:00
Tyera 3f70ddb2c5
Add entry notification service for geyser (#31290)
* Move entry_notifier_interface

* Add EntryNotifierService

* Use descriptive struct in sender/receiver

* Optionally initialize EntryNotifierService in validator

* Plumb EntryNotfierSender into Tvu, blockstore_processor

* Plumb EntryNotfierSender into Tpu

* Only return one option when constructing EntryNotifierService
2023-05-10 17:20:51 -06:00
Dmitri Makarov fb79e2bbb8
Reorganize ledger-tool run sub-command (#31572)
* Rename ledger-tool run subcommand to program subcommand

* Reorganize ledger-tool program command to multiple subcommands
2023-05-10 15:25:38 -04:00
Pankaj Garg 2210af60ee
Replace executor cache usage with LoadedPrograms cache (#31462)
* Replace executor cache usage with LoadedPrograms cache

* clippy fixes

* update cache with updated programs

* fixes

* more cleanup

* update tx batch cache with the tx results

* address review comments

* handle program closing backward compatibility

* handle unloaded programs during extraction
2023-05-09 15:44:38 -07:00
Brooks 3bb2e3b546
Purges incomplete snapshot dirs at startup (#31555) 2023-05-09 14:08:12 -04:00
Brooks 6cf6b8cdf7
clippy: Replace `if let Some() = res.ok()` with `if let Ok() = res` (#31525) 2023-05-08 09:36:06 -04:00
Dmitri Makarov f09e791a09
Add debugging/tracing capabilities to ledger-tool, previously rbp… (#31346)
* Add debug and trace capabilities to ledger-tool, previously rbpf-cli

* Dump nested logs from saved traces in invoke_context

* Change the ledger-tool run input format to represent keys as strings
2023-05-05 05:14:41 -04:00
Dmitri Makarov e03826b6d8
Refactor ledger-tool moving some of functions to a separate module (#31359)
* Refactor ledger-tool moving some of functions to a separate module

* Rename utils to ledger_utils, move arg parsing funcs to a new module
2023-04-27 14:11:04 -04:00
Brooks c73b6545e6
Spins up AccountsHashVerifier in ledger-tool (#31358) 2023-04-26 12:07:35 -04:00
Alexander Meißner 7002c568fb
Cleanup - Removes the CLI option "no-bpf-jit" (#31329)
* Enables JIT everywhere by default.

* Removes the CLI argument "no-bpf-jit" and its plumbering through out the validator code base.

* Removes with_jit bpf_loader variants.

* Removes the to_builtin!() macro.
2023-04-25 19:04:11 +02:00
Brooks ca1bde3591
Use Arc instead of &Arc in AccountsBackgroundService::new (#31268) 2023-04-19 11:10:41 -04:00
Trent Nelson f34a6bcfce
runtime: transpose `VoteAccount::vote_state()` return to improve ergonomics (#31256) 2023-04-18 14:48:52 -06:00
steviez f3935f679e
ledger-tool: Add deprecation warning for --no-compaction (#31178)
The argument is deprecated with a change in default behavior but this
may not be obvious to a user; the warning makes this a bit more visible.
2023-04-15 00:18:03 -05:00
Brooks d43e19bb03
Refactors the Full/Incremental SnapshotHash types (#31186) 2023-04-13 18:01:27 -04:00
Jeff Washington (jwash) 3d6248d76a
ledger-tool verify cli for final hash calc (#31153) 2023-04-12 07:30:45 -07:00
steviez cc8e531a5d
Enforce a minimum of 1 on full and incremental snapshot retention (#30968) 2023-03-30 10:16:36 -05:00
steviez 86cf0ad758
ledger-tool: Skip compaction in purge command by default (#30891)
Performing a manual ledger compaction can be very time consuming.
Although the compaction will optimize storage use, swapping to skip
compaction favors the command completing faster. Additionally, we no
longer do regular compaction in the validator (via LedgerCleanupService)
and instead make use of range deletion which is much more efficient.

Given the above, this changes alters the default behavior to skip ledger
compaction by default, and introduces a flag should a user really want
to run compaction.
2023-03-29 10:34:00 -05:00
steviez 24be8508bb
ledger-tool: Use common AccountsDbConfig for all functions (#30707)
PR #29975 previously combined some repeated logic around parsing
AccountsDb configuration options. That PR consolidated logic between
verify and create-snapshot, but left alone all of the other commands
that load up and process Bank(s).

Make all functions that will spin up Banks / AccountsDb share this
common function. Additionally, expose some of the AccountsDb
configuration flags to the other commands such as accounts, graph, etc.
2023-03-24 12:46:09 -05:00
steviez 8db35eb7e5
ledger-tool: Add flag to find non-vote optimistic slots (#30580)
In cluster restart scenarios, it is desirable to know the latest
optimistic slot(s), which the subcommand latest-optimistic-slots will
return. However, it is also useful to know whether slots contain only
votes or if they contain votes and user transactions.

This PR adds an extra column of output to show whether an optimistically
confirmed slot is vote only (contains zero non-vote transactions).
Additionally, a flag has been added to enable filtering output to
exclude vote only slots.
2023-03-23 14:13:41 -07:00
Ryo Onodera 6c444df9e0
Add --block-{verification,production}-method flags (noop atm) (#30746)
* Add --{replaying,banking}-backend flags (noop atm)

* Greatly simplify enums with strum

* Update programs/sbf/Cargo.lock...

* Rely on Display, removing Debug

* constify cli_names()

* Don't allow omitting bankend value

* Rename to --block-{verification,production}-method

* Use more specific name

* Actually support missing value....

* Remove strictly-unnecessary flags

* Use lazy_static! instead of abusing DefaultArgs...
2023-03-23 12:57:28 +09:00
Illia Bobyr 809041b151
poh_verify => run_verification: Rename to be more accurate (#30811)
`poh_verify` actually disables transaction signature, tick count and
built in program argument verifications as well.  It is somewhat
confusing to call it `poh_verify`.
2023-03-22 11:03:30 -07:00
Ryo Onodera 721719d776
Add solana_clap_utils::hidden_unless_forced() (#30843)
* Add solana_clap_utils::hidden_unless_forced()

* Use more descriptive env name

Co-authored-by: mvines <mvines@gmail.com>

---------

Co-authored-by: mvines <mvines@gmail.com>
2023-03-22 08:59:17 +00:00
Xiang Zhu 8e3a30c22c
Clean orphaned account snapshot dirs (#30645)
* Clean up orphaned account snapshot hardlink dirs

* fix compilation issues

* debugged, now working.  seeing the orphaned directories deleted

* change back to eprintln + exit for account_path error

* changed eprintln to panic for now

* add test_clean_orphaned_account_snapshot_dirs for codecov check

* address a few comments and nit isseus

* directly unzip, skipped the intermediate array of tuples

* let set_up_account_run_and_snapshot_paths return Result

* 'proper' typo, and comment on return

* use map_err

* use for loop in clean_orphaned_account_snapshot_dirs, removed panic

* add test_set_up_account_run_and_snapshot_paths

* minor, replace .for_each with .all

* rename set_up_account_run_and_snapshot_paths to create_all_accounts_run_and_snapshot_dirs

* remove unnecessary closure return type

* change to for loop

* change match to unwrap_or_else

* remove create_dir_all(&account_path) in create_all

* minor comment cleanup
2023-03-17 15:22:10 -07:00
Tyera 20223b2557
Revert ledger-tool changes, fix master (#30763)
Revert 10f49d4 for ledger-tool
2023-03-16 19:30:10 -06:00
steviez f1cd64f4a1
ledger-tool: chore: Cleanup snapshot retention constants (#30745)
Declare constants at top of function with comment, and stringify the
constants when needed to be in line with how other defaults are done in
ledger-tool main function.
2023-03-17 09:25:58 +08:00
cavemanloverboy 10f49d4e26
Geyser Runtime Reload (#30352)
Support dynamic geyser plugin load, unload, and listing through admin RPC.
2023-03-16 17:03:00 -07:00
Tyera 9d792c1848
Ledger-tool: add param to deactivate features in snapshot creation (#30755)
* Add separate parameter for deactivating features on snapshot creation

* Add debug logs for account/feature removals

* Add the word gate
2023-03-16 15:54:05 -06:00
steviez db94e4564f
ledger-tool: Avoid creating SanitizedTransaction in print command (#30575)
The print command looks up program_ids from transactions by creating
SanitizedTransactions. But, print doesn't have access to a Bank so
transactions that include an address table lookup can't be sanitized.

Sanitizing the transaction isn't necessary, so this change just looks up
the program id from unsanitized transactions.
2023-03-03 14:18:57 -06:00
Brooks 4ba80ad785
Inline format args (#30364)
clippy fixes
2023-02-16 17:00:43 +00:00
steviez 5ad8c8bd20
fixup! lt: Reallow custom accounts path with Secondary access (#30228) (#30236)
The previous change used Primary access; however, it is better to use
PrimaryForMaintenance in this case to disallow any potential compaction
from happening.
2023-02-10 14:09:14 -06:00
Ryo Onodera 3e6162e69e
Add address lookup tables to minimized snapshot (#30158)
* Add address lookup tables to minimized snapshot

* Add comment for future posterity

* Add reference to the issue

* Adjust comment a bit

Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>

---------

Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>
2023-02-10 14:46:02 +09:00
steviez 64d2809244
ledger-tool: Reallow custom accounts path with Secondary access (#30228)
Previously, ledger-tool had a guardrail to disallow a custom accounts
path when access mode to the blockstore was Secondary. This was to avoid
potentially pulling the accounts out from underneath solana-validator.
When ledger-tool switched over to use Secondary blockstore access for
all commands that do not need write access, this removed the ability to
use custom accounts paths with ledger-tool at all for these commands.
Custom accounts paths are desirable, especially if that custom path is
in tmpfs to speed up processing.

With this change, when a custom accounts path is passed for a command
using Secondary access, ledger-tool now checks if Primary access is
being held by another process. If not, allow processing to proceed
with the custom accounts path.

The above check isn't fullproof, but it is about equal to the check that
previously existed when ledger-tool would run in Primary access mode
when it didn't need to.
2023-02-09 19:53:31 -06:00
Benno Fünfstück e8c43aa0d1
ledger-tool: Notify geyser of transactions (#29933)
ledger-tool: Notify geyser of transactions
2023-02-09 12:34:20 -08:00