Commit Graph

693 Commits

Author SHA1 Message Date
steviez d5ad29d837
Make Blockstore::scan_and_fix_roots() take optional start/stop slots (#32289)
The optional args allow reuse by ledger-tool repair roots command Also,
hold cleanup lock for duration of Blockstore::scan_and_fix_roots().

This prevents a scenario where scan_and_fix_roots() could identify a
slot as needing to be marked root, that slot getting cleaned by
LedgerCleanupService, and then scan_and_fix_roots() marking the slot as
root on the now purged slot.
2023-06-28 22:32:03 -05:00
steviez 67f5a42bd5
ledger-tool: Adjust the help messages for force-update-to-open (#32318)
The previous help message was pretty generic as it covered two potential
error scenarios. This change gives a more detailed error message to each
of those errors.
2023-06-28 18:40:20 -05:00
Brooks ee6684b2eb
Adds UseSnapshotArchivesAtStartup::WhenNewest (#32284) 2023-06-27 14:26:40 -04:00
Brooks 5f1b5b877a
Replace boot_from_local_state with use_snapshot_archives_at_startup (#32260) 2023-06-26 12:44:25 -04:00
Brooks 64ecfaf769
Renames --boot-from-local-state to --use-snapshot-archives-at-startup (#32217) 2023-06-23 18:34:18 -04:00
steviez 20a7cdd43d
Restrict access to Bank's HardForks (#32180)
* Restrict access to Bank's HardForks

Callers could previously obtain a a lock to read/write HardForks from
any Bank. This would allow any caller to modify, and creates the
opportunity for inconsistent handling of what is considered a valid hard
fork (ie too old).

This PR adds a function to Bank so consistent sanity checks can be
applied; the caller will already have a Bank as that is where they would
have obtained the HardForks from in the first place. Additionally,
change the getter to return a copy of HardForks (simple Vec).

* Allow hard fork at bank slot if bank is not yet frozen
2023-06-20 23:44:43 -05:00
Brooks 2aac9fca35
Adds --boot-from-local-state to ledger-tool create-snapshot (#32032) 2023-06-16 12:58:57 -04:00
Alexander Meißner 3f13cd353e
Fix - LoadedProgramType::Closed (#31922)
* Makes Bank::load_program() return correct tombstones.

* Removes early TX failure caused by closed and invalid programs.

* Adjusts the feature gate of simplify_writable_program_account_check.
2023-06-08 15:40:18 +02:00
Alexander Meißner ee2c2ef6c7
Cleanup - require_static_program_ids_in_transaction (#31767)
require_static_program_ids_in_transaction
2023-06-07 17:12:41 +02:00
Brooks 7b37a1672f
Enables fastboot for ledger-tool (#31921) 2023-06-06 18:32:24 -04:00
Pankaj Garg c86e160438
Add runtime environment to FailedVerification tombstones (#31991)
* Add runtime environment to FailedVerification tombstones

* modify default variant

* prune update

* add DelayVisibility in prune_feature_set_transition
2023-06-06 22:24:39 +00:00
Illia Bobyr 4353ac6797
Pass Arc<AtomicBool> by value, not by reference. (#31916)
`Arc` is already a reference internally, so it does not seem to be
beneficial to pass a reference to it.  Just adds an extra layer of
indirection.

Functions that need to be able to increment `Arc` reference count need
to take `Arc<AtomicBool>`, but those that just want to read the
`AtomicBool` value can accept `&AtomicBool`, making them a bit more
generic.

This change focuses specifically on `Arc<AtomicBool>`.  There are other
uses of `&Arc<T>` in the code base that could be converted in a similar
manner.  But it would make the change even larger.
2023-06-01 17:25:48 -07:00
steviez 1d6b03358f
ledger-tool: Rename load_frozen_forks() and adjust error messages (#31903)
This function has morphed as it has been updated, and the existing
function name is a bit of misnomer. Update the function name to use
terminology that I think is more clear and more consistent with what we
use core/src/validator.rs. Additionally, adjust the error messages to be
more clear about what exactly is wrong for when we can't process.
2023-05-31 21:12:49 -05:00
steviez debe794987
Replace improper &Arc<...> with Arc<...> in Bank and Accounts (#31892)
The callstack updated in this PR passed an &Arc<...> down only to have
the bottom level clone the reference. Thus, we are giving shared
ownership so the reference is a bit redundant and arguably obscures the
intention to clone further down the callstack.
2023-05-31 12:36:44 -05:00
Brooks 9220e53e23
ledger-tool: Clean up load_bank_forks() (#31870) 2023-05-31 06:50:33 -04:00
Jeff Washington (jwash) aff9c03e7a
fix cli args (#31804) 2023-05-25 07:47:00 -07:00
Jeff Washington (jwash) 461342cdd4
add cli args and use of partitioned rewards config (#31800)
* add cli args and use of partitioned rewards config

* update comments
2023-05-24 10:54:09 -07:00
Andrew Fitzgerald f52ded35f4
async delete contents but leave directory (#31737)
* async delete contents but leave directory

* Clarified comment
2023-05-23 15:33:09 -07:00
Alexander Meißner 83f692ce67
Refactor - Share RBPF config and syscall registry globally (#31727)
* Adds LoadedPrograms::program_runtime_environment_v1.

* No need to explicitly remove precompiles from program_accounts_map.
2023-05-20 20:28:00 +02:00
Alexander Meißner 520c647918
Refactor - Merges `BuiltinPrograms` into `LoadedPrograms` (#31654)
* Merges BuiltinPrograms into LoadedPrograms.

* Prevents built-ins from being pruned.
2023-05-18 22:18:28 +02:00
Pankaj Garg b03422968b
Remove Rc and RefCell to carry program cache in invoke_context (#31684) 2023-05-17 06:10:50 -07:00
Pankaj Garg 9b860abfcc
Fix warning about unused mut in ledger-tool (#31686) 2023-05-17 00:17:42 +00:00
Trent Nelson ad67fd5be5
validator: remove optional remote accounts hash consistency check (#31279) 2023-05-16 14:23:13 -06:00
steviez 02679eb5f9
ledger-tool: Move ELF magic numbers into constant with comment (#31655) 2023-05-15 19:32:59 +00:00
Dmitri Makarov 950892e503
Fix ledger-tool program command for non-runnable subcommands (#31596)
* Fix ledger-tool program command for non-runnable subcommands

These subcommands (cfg and disassemble) don't need to load a ledger
database for running, and neither do they need an input file
specifying instruction accounts etc. This change creates a simplified
invoke_context for processing the static actions of ledger-tool
program subcommand.

* Factor out program file loading
2023-05-15 14:23:54 -04:00
Alexander Meißner 5c8b5a2a68
Bump solana_rbpf to v0.4.0 (#31594)
* Moves "disable_deploy_of_alloc_free_syscall" parameter inside create_loader().

* Removes the "is_abi_v2" flag.

* Bumps solana_rbpf to v0.4.0
2023-05-12 18:07:13 +02:00
Tyera 3f70ddb2c5
Add entry notification service for geyser (#31290)
* Move entry_notifier_interface

* Add EntryNotifierService

* Use descriptive struct in sender/receiver

* Optionally initialize EntryNotifierService in validator

* Plumb EntryNotfierSender into Tvu, blockstore_processor

* Plumb EntryNotfierSender into Tpu

* Only return one option when constructing EntryNotifierService
2023-05-10 17:20:51 -06:00
Dmitri Makarov fb79e2bbb8
Reorganize ledger-tool run sub-command (#31572)
* Rename ledger-tool run subcommand to program subcommand

* Reorganize ledger-tool program command to multiple subcommands
2023-05-10 15:25:38 -04:00
Pankaj Garg 2210af60ee
Replace executor cache usage with LoadedPrograms cache (#31462)
* Replace executor cache usage with LoadedPrograms cache

* clippy fixes

* update cache with updated programs

* fixes

* more cleanup

* update tx batch cache with the tx results

* address review comments

* handle program closing backward compatibility

* handle unloaded programs during extraction
2023-05-09 15:44:38 -07:00
Brooks 3bb2e3b546
Purges incomplete snapshot dirs at startup (#31555) 2023-05-09 14:08:12 -04:00
Brooks 6cf6b8cdf7
clippy: Replace `if let Some() = res.ok()` with `if let Ok() = res` (#31525) 2023-05-08 09:36:06 -04:00
Dmitri Makarov f09e791a09
Add debugging/tracing capabilities to ledger-tool, previously rbp… (#31346)
* Add debug and trace capabilities to ledger-tool, previously rbpf-cli

* Dump nested logs from saved traces in invoke_context

* Change the ledger-tool run input format to represent keys as strings
2023-05-05 05:14:41 -04:00
Dmitri Makarov e03826b6d8
Refactor ledger-tool moving some of functions to a separate module (#31359)
* Refactor ledger-tool moving some of functions to a separate module

* Rename utils to ledger_utils, move arg parsing funcs to a new module
2023-04-27 14:11:04 -04:00
Brooks c73b6545e6
Spins up AccountsHashVerifier in ledger-tool (#31358) 2023-04-26 12:07:35 -04:00
Alexander Meißner 7002c568fb
Cleanup - Removes the CLI option "no-bpf-jit" (#31329)
* Enables JIT everywhere by default.

* Removes the CLI argument "no-bpf-jit" and its plumbering through out the validator code base.

* Removes with_jit bpf_loader variants.

* Removes the to_builtin!() macro.
2023-04-25 19:04:11 +02:00
Brooks ca1bde3591
Use Arc instead of &Arc in AccountsBackgroundService::new (#31268) 2023-04-19 11:10:41 -04:00
Trent Nelson f34a6bcfce
runtime: transpose `VoteAccount::vote_state()` return to improve ergonomics (#31256) 2023-04-18 14:48:52 -06:00
steviez f3935f679e
ledger-tool: Add deprecation warning for --no-compaction (#31178)
The argument is deprecated with a change in default behavior but this
may not be obvious to a user; the warning makes this a bit more visible.
2023-04-15 00:18:03 -05:00
Brooks d43e19bb03
Refactors the Full/Incremental SnapshotHash types (#31186) 2023-04-13 18:01:27 -04:00
Jeff Washington (jwash) 3d6248d76a
ledger-tool verify cli for final hash calc (#31153) 2023-04-12 07:30:45 -07:00
steviez cc8e531a5d
Enforce a minimum of 1 on full and incremental snapshot retention (#30968) 2023-03-30 10:16:36 -05:00
steviez 86cf0ad758
ledger-tool: Skip compaction in purge command by default (#30891)
Performing a manual ledger compaction can be very time consuming.
Although the compaction will optimize storage use, swapping to skip
compaction favors the command completing faster. Additionally, we no
longer do regular compaction in the validator (via LedgerCleanupService)
and instead make use of range deletion which is much more efficient.

Given the above, this changes alters the default behavior to skip ledger
compaction by default, and introduces a flag should a user really want
to run compaction.
2023-03-29 10:34:00 -05:00
steviez 24be8508bb
ledger-tool: Use common AccountsDbConfig for all functions (#30707)
PR #29975 previously combined some repeated logic around parsing
AccountsDb configuration options. That PR consolidated logic between
verify and create-snapshot, but left alone all of the other commands
that load up and process Bank(s).

Make all functions that will spin up Banks / AccountsDb share this
common function. Additionally, expose some of the AccountsDb
configuration flags to the other commands such as accounts, graph, etc.
2023-03-24 12:46:09 -05:00
steviez 8db35eb7e5
ledger-tool: Add flag to find non-vote optimistic slots (#30580)
In cluster restart scenarios, it is desirable to know the latest
optimistic slot(s), which the subcommand latest-optimistic-slots will
return. However, it is also useful to know whether slots contain only
votes or if they contain votes and user transactions.

This PR adds an extra column of output to show whether an optimistically
confirmed slot is vote only (contains zero non-vote transactions).
Additionally, a flag has been added to enable filtering output to
exclude vote only slots.
2023-03-23 14:13:41 -07:00
Ryo Onodera 6c444df9e0
Add --block-{verification,production}-method flags (noop atm) (#30746)
* Add --{replaying,banking}-backend flags (noop atm)

* Greatly simplify enums with strum

* Update programs/sbf/Cargo.lock...

* Rely on Display, removing Debug

* constify cli_names()

* Don't allow omitting bankend value

* Rename to --block-{verification,production}-method

* Use more specific name

* Actually support missing value....

* Remove strictly-unnecessary flags

* Use lazy_static! instead of abusing DefaultArgs...
2023-03-23 12:57:28 +09:00
Illia Bobyr 809041b151
poh_verify => run_verification: Rename to be more accurate (#30811)
`poh_verify` actually disables transaction signature, tick count and
built in program argument verifications as well.  It is somewhat
confusing to call it `poh_verify`.
2023-03-22 11:03:30 -07:00
Ryo Onodera 721719d776
Add solana_clap_utils::hidden_unless_forced() (#30843)
* Add solana_clap_utils::hidden_unless_forced()

* Use more descriptive env name

Co-authored-by: mvines <mvines@gmail.com>

---------

Co-authored-by: mvines <mvines@gmail.com>
2023-03-22 08:59:17 +00:00
Xiang Zhu 8e3a30c22c
Clean orphaned account snapshot dirs (#30645)
* Clean up orphaned account snapshot hardlink dirs

* fix compilation issues

* debugged, now working.  seeing the orphaned directories deleted

* change back to eprintln + exit for account_path error

* changed eprintln to panic for now

* add test_clean_orphaned_account_snapshot_dirs for codecov check

* address a few comments and nit isseus

* directly unzip, skipped the intermediate array of tuples

* let set_up_account_run_and_snapshot_paths return Result

* 'proper' typo, and comment on return

* use map_err

* use for loop in clean_orphaned_account_snapshot_dirs, removed panic

* add test_set_up_account_run_and_snapshot_paths

* minor, replace .for_each with .all

* rename set_up_account_run_and_snapshot_paths to create_all_accounts_run_and_snapshot_dirs

* remove unnecessary closure return type

* change to for loop

* change match to unwrap_or_else

* remove create_dir_all(&account_path) in create_all

* minor comment cleanup
2023-03-17 15:22:10 -07:00
Tyera 20223b2557
Revert ledger-tool changes, fix master (#30763)
Revert 10f49d4 for ledger-tool
2023-03-16 19:30:10 -06:00
steviez f1cd64f4a1
ledger-tool: chore: Cleanup snapshot retention constants (#30745)
Declare constants at top of function with comment, and stringify the
constants when needed to be in line with how other defaults are done in
ledger-tool main function.
2023-03-17 09:25:58 +08:00