Commit Graph

826 Commits

Author SHA1 Message Date
mergify[bot] 654cf17c68
v2.0: ledger-tool: Fix create-snapshot default value for output_directory (backport of #3148) (#3153)
ledger-tool: Fix create-snapshot default value for output_directory (#3148)

The arguments to specify full and incremental snapshot archives paths
used to be a global argument; these were moved to only be instantiated
on commands that needed them in #1773.

But, when the arguments were moved from app-level to subcommand-level,
the code that matches the arguments was not updated to look at
subcommand-matches instead of app-matches.

(cherry picked from commit 1d9947cd68)

Co-authored-by: steviez <steven@anza.xyz>
2024-10-15 10:41:18 -10:00
mergify[bot] 4b521aa4e7
v2.0: ledger-tool: Set initial last full snapshot slot (backport of #2314) (#2343)
ledger-tool: Set initial last full snapshot slot (#2314)

(cherry picked from commit 75a640e666)

Co-authored-by: Brooks <brooks@anza.xyz>
2024-07-31 14:16:24 -04:00
mergify[bot] 16dd41c04c
v2.0: ledger-tool: unify and colorize help (backport of #2322) (#2335)
ledger-tool: unify and colorize help (#2322)

(cherry picked from commit 38458f4a34)

Co-authored-by: Brooks <brooks@anza.xyz>
2024-07-30 13:11:56 -04:00
mergify[bot] b463b9255c
v2.0: Make unified scheduler opt-in for block verification (backport of #1668) (#1874)
Make unified scheduler opt-in for block verification (#1668)

* Enable unified scheduler for block verification

* Revert making unified scheduler enabled by default

(cherry picked from commit 40508cd03f)

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2024-06-27 16:59:05 +09:00
steviez 3ee204b5cb
ledger-tool: Get shreds from BigTable blocks (#1638)
There is often a desire to examine/replay/etc older blocks. If the
blocks are recent enough, they can be pulled from an actively running
node. Otherwise, the blocks must be pulled down from warehouse node
archives. These archives are uploaded on a per-epoch basis so they are
quite large, and can take multiple hours to download and decompress.

With the addition of Entry data to BigTable, blocks can be recreated
from BigTable data. Namely, we can recreate the Entries with proper PoH
and transaction data. We can then shred them such that they are the
same format as blocks that are produced from the cluster.

This change introduces a new command that will read BigTable data and
insert shreds into a local Blockstore. The new command is:
  $ agave-ledger-tool bigtable shreds ...

Several important notes about the change:
- Shred for some slot S will not be signed by the actual leader for
  slot S. Instead, shreds will be signed with a "dummy" keypair. The
  shred signatures does not affect the ability to replay the block.
- Entry PoH data does not go back to genesis in BigTable. This data
  could be extracted and uploaded from the existing rocksdb archives;
  however, that work is not planned as far as I know. --allow-mock-poh
  can be passed to generate filler PoH data. Blocks created with this
  flag are replayable by passing --skip-poh-verify to ledger-tool.
- A snapshot will be unpacked to determine items such as the shred
  version, tick hash rate and ticks per slot. This snapshot must be in
  the same epoch as the requested slots
2024-06-20 16:02:28 -05:00
steviez b70e6bbfd2
ledger-tool: Make verify --print-bank-hash support json (#1745)
The bank-hash command in ledger-tool was recently deprecated. However,
the command is used by some of the scripts that coordinate starting up
a fresh cluster. So, the deprecation of bank-hash broke those scripts.

This change fixes the scripts by doing the following:
- Makes --print-bank-hash support --output json
- Updates scripts to install jq on provisioned nodes
- Update remote-node.sh to parse the bank hash from json using jq
2024-06-20 00:15:18 -05:00
Alessandro Decina 2107adcf35
bpf_loader: use an explicit thread-local pool for stack and heap memory (#1370)
* Rename ComputeBudget::max_invoke_stack_height to max_instruction_stack_depth

The new name is consistent with the existing
ComputeBudget::max_instruction_trace_length.

Also expose compute_budget:MAX_INSTRUCTION_DEPTH.

* bpf_loader: use an explicit thread-local pool for stack and heap memory

Use a fixed thread-local pool to hold stack and heap memory. This
mitigates the long standing issue of jemalloc causing TLB shootdowns to
serve such frequent large allocations.

Because we need 1 stack and 1 heap region per instruction, and the
current max instruction nesting is hardcoded to 5, the pre-allocated
size is (MAX_STACK + MAX_HEAP) * 5 * NUM_THREADS. With the current
limits that's about 2.5MB per thread. Note that this is memory that
would eventually get allocated anyway, we're just pre-allocating it now.

* programs/sbf: add test for stack/heap zeroing

Add TEST_STACK_HEAP_ZEROED which tests that stack and heap regions are
zeroed across reuse from the memory pool.
2024-06-20 08:57:11 +07:00
steviez ba2ff5384d
ledger-tool: Deduplicate max-genesis-archive-unpacked-size argument (#1774)
The argument is currently declared in multiple places. So, delcare the
argument in one central place.
2024-06-18 13:15:02 -05:00
steviez 4e0afd6c8c
ledger-tool: Make joining AccountsBackgroundService optional (#1673)
AccountsBackgroundService performs several operations that can take a
long time to complete and do not check the exit flag mid-operation.
Thus, ledger-tool can get hung up for a while waiting for ABS to
finish. However, many ledger-tool command do not ABS to have finished.

So, return a handle to the ABS thread and allow the caller to decide
whether to join ABS or not. As of right now, create-snapshot is the
only command that requires ABS to have finished before continuing.
2024-06-17 23:36:49 -05:00
steviez 878ef1f89f
ledger-tool: Subfunction for snapshot args (#1773)
There are several arguments to control snapshot configuration in the
various ledger-tool commands. The inclusion of args in each command
is inconsistent, especially for commands outside of main.rs

This change consolidates the snapshot related arguments into a single
function to help create consistency and reduce duplicate code
2024-06-17 21:55:07 -05:00
Lucas Ste ebd063eb79
Add Shuttle multithreading test infrastructure (#1634) 2024-06-17 19:24:43 -03:00
steviez d056710c3e
ledger-tool: Use helper for AccountsDb args (#1749)
There are a handful of commands that specify AccountsDb config. To
create consistency between all these commands and remove repeated code,
build all of the arguments in a helper function that is passed to the
relevant commands.
2024-06-17 13:58:05 -05:00
galactus a6b2283142
ScanConfig defaults no longer sort results (#1539) 2024-06-14 08:27:33 -04:00
steviez 6d8fa7a97b
ledger-tool: Deprecate bank-hash command (#1710)
ledger-tool has a somewhat cluttered interface. The bank-hash command
offers little extra functionality compared to what the verify command
does, and could instead be an optional flag for the verify command.

This change leaves the bank-hash command present, but prints an error
message and tells the user to instead use verify --print-bank-hash. The
command will be removed altogether in the future.
2024-06-13 12:13:39 -05:00
Sean Young 873808cf21
Add option to record transactions to ledger-tool (#181) 2024-06-12 11:43:53 +01:00
Brooks d092ad862e
cli: Fixes snapshot paths (#1631) 2024-06-10 08:28:40 -04:00
Joe C 62eb4cc068
Add Compute Budget Crate (#1121) 2024-06-01 01:38:19 -05:00
Joe C 664bf100b5
program-runtime: invoke context: rename `programs_loaded_for_tx` to `program_cache_for_tx_batch` (#1390)
program-runtime: invoke context: rename `programs_loaded_for_tx`
2024-05-16 19:08:01 -05:00
steviez bd09ba69b1
ledger-tool: Remove duplicate slot output code (#1255)
This makes the slot subcommand use the same output method as bigtable
block to output blocks (-vv). Doing so creates consistency between the
two commands as well as removing a duplicate implementation in the
ledger-tool code. The shared type also supports json output which the
ledger-tool implementation did not.
2024-05-15 17:31:19 -05:00
Yihau Chen ec9bd79849
clippy: fix legacy_numeric_constants (#1314)
clippy: legacy_numeric_constants
2024-05-15 11:29:19 +08:00
Kevin Heavey 59e3eaafb5
Prefer explicitly using serde_derive instead of the "derive" feature of serde (#1329)
* use serde_derive instead of the derive feature of serde

* make serde_derive version match serde version

* remove serde dep entirely from programs/address-lookup-table

* replace remaining usage of serde "derive" feature

* update lock files

* explicitly use serde_derive in svm and frozen-abi

* add comments about serde and serde_derive versions matching
2024-05-15 01:11:31 +02:00
Andrei Silviu Dragnea 02147439bd
ledger-tool: Add --enable-extended-tx-metadata-storage arg (#1344) 2024-05-14 14:01:40 -05:00
Yihau Chen b42d8f68a4
clippy: allow dead_code for GraphVoteAccountModeError, DroppableTask and SubscriptionToken (#1246)
* clippy: allow dead_code for SubscriptionToken

* clippy: allow dead_code for GraphVoteAccountModeError

* clippy: allow dead_code for DroppableTask

* remove the string in the GraphVoteAccountModeError

* abort the joinhandle when drop

* only trace JoinHandle in the drop

* add comment
2024-05-13 10:45:37 +08:00
Brooks 11383ae0a1
Do not purge old snapshot archives in bank_to_snapshot_archive() (#1226) 2024-05-09 15:45:33 +00:00
steviez 4fc1435e5e
ledger-tool: Remove blockstore json command (#1206)
This command is basically an exact duplicate of the print command,
except print had OutputFormat::Display hard-coded whereas the json
command had OutputFormat::Json hard-coded.

Instead of hard-coded values, parse value of --output-format from CLI
2024-05-06 18:32:09 -05:00
ripatel-fd aa2f078836
Fix dupe account handling in 'ledger-tool program run' command (#1191)
The agave-ledger-tool's functionality to execute sBPF programs out-of-band
did not properly handle duplicate accounts.  This patch introduces instruction
account deduplication.

Locally tested to confirm that duplicate accounts are serialized correctly
and that non-duplicate account behavior didn't change.

Co-authored-by: Richard Patel <ripatel@jumptrading.com>
2024-05-06 15:36:10 +02:00
Joe C 34b76ac580
Program-Runtime: Add `EnvironmentConfig` to `InvokeContext` (#1059)
* program-runtime: add `EnvironmentConfig` to `InvokeContext`

* move `blockhash` to `EnvironmentConfig`

* move `lamports_per_signature` to `EnvironmentConfig`

* move `feature_set` to `EnvironmentConfig`

* move `sysvar_cache` to `EnvironmentConfig`

* add `get_feature_set` getter
2024-05-01 15:22:43 -05:00
steviez 629ae32177
ledger-tool: Make blockstore purge stop at actual highest slot (#1058)
If someone wants to purge from a given slot and on, they might pass a
sentinel value like 999999999 for --end-slot. Given the following logic
in purge, this could extra looping, even after the end of blockstore
data has been reached.

So, compute the actual highest blockstore slot, and adjust the user
supplied end_slot to min(end_slot, highest_slot)
2024-04-25 20:09:43 -05:00
Alexander Meißner e43338f3d6
Refactor - Rename `LoadedProgram` to `ProgramCacheEntry` (#880)
LoadedProgramOwner => ProgramCacheEntryOwner.
LoadedProgramType => ProgramCacheEntryType.
LoadedProgram => ProgramCacheEntry.
LoadedProgramStats => ProgramCacheStats.
LoadedProgramMatchCriteria => ProgramCacheMatchCriteria.
LoadedProgramsForTxBatch => ProgramCacheForTxBatch.
2024-04-18 21:38:08 +02:00
Alexander Meißner ac36dbbc87
Refactor - `LoadedProgramType::Loaded` and `LoadedProgramOwner` (#606)
* Adds LoadedProgram::account_owner() and LoadedProgramOwner.
Merges LoadedProgramTypes LegacyV0, LegacyV1, Typed and TestLoaded into Loaded.

* Review feedback.
2024-04-17 17:48:14 +02:00
Andrei Silviu Dragnea 590612942a
ledger-tool: Add --log-messages-bytes-limit arg (#854) 2024-04-17 16:17:07 +02:00
Justin Starry 056bfcede7
Deprecate `is_sysvar_id` function (#789)
Deprecate is_sysvar_id
2024-04-16 06:22:52 +08:00
Justin Starry 09241ae9c3
Use reserved account keys list to restrict tx write locks (#541)
* Plumb through reserved account keys set

* Plumb through tests
2024-04-13 09:37:58 +08:00
Joe C 03ef611f0c
program-runtime: hoist `RuntimeConfig` up to SVM (#630)
program-runtime: hoist `RuntimeConfig` out into SVM
2024-04-07 10:45:57 -05:00
Alexander Meißner 55c05c5ea5
Fix - `FailedVerification` and `Closed` tombstones (#419)
* Only the verifier can cause FailedVerification, everything else is Closed

* Removes the environments parameter from load_program_accounts().

* cargo fmt

* Simplify invocation of deployed program

* Attempt to invoke a program before it is deployed

* Attempt to invoke a buffer before it is used in a deployment

* Escalates Option return value of load_program_accounts() to load_program_with_pubkey().

* Review feedback
2024-04-05 13:03:18 +02:00
Alessandro Decina f36a45a971
ledger-tool: name rayon threads (#464)
ledger-tool: name and shame rayon threads
2024-03-29 06:13:01 +11:00
Pankaj Garg 91b1ee3df6
Fix: deploy program on last slot of epoch during environment change (#101)
* Fix: deploy program on last slot of epoch during environment change

* solana-runtime: deploy at last epoch slot test

* disable deployment of sol_alloc_free

* Move tx-batch-constructor to its own function

* use new_from_cache

---------

Co-authored-by: Alessandro Decina <alessandro.d@gmail.com>
2024-03-22 07:43:28 -07:00
HaoranYi 0f8f8cd970
Revert deprecate executable feature (#309)
* revert deprecate executable feature

* add native loader account transfer test

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2024-03-20 10:54:15 -05:00
Pankaj Garg cc3afa5588
Remove public visibility of program cache from bank (#279) 2024-03-17 15:29:20 -07:00
Yihau Chen 51dc7e6fb7
[anza migration]: add 'agave=info' to default log level (#223) 2024-03-14 20:35:33 +08:00
steviez 5ed30beb2a
ledger-tool: Allow compute-slot-cost to operate on dead slots (#213)
Make this command accept the --allow-dead-slots arg as well
2024-03-13 11:29:05 -05:00
Brooks 88f6a7a459
Removes holding storages in AccountsHashVerifier for fastboot (#120) 2024-03-11 17:09:26 -04:00
Dmitri Makarov ba43f74dcf
[SVM] Move RuntimeConfig to program-runtime (#96)
RuntimeConfig doesn't use anything SVM specific and logically belongs
in program runtime rather than SVM.  This change moves the definition
of RuntimeConfig struct from the SVM crate to program-runtime and
adjusts `use` statements accordingly.
2024-03-07 10:16:16 -08:00
Brooks 93f5b514fa
Adds StartingSnapshotStorages to AccountsHashVerifier (#58) 2024-03-04 16:32:51 -05:00
Yihau Chen 3f9a7a52ea [anza migration] rename crates (#10)
* rename geyser-plugin-interface

* rename cargo registry

* rename watchtower

* rename ledger tool

* rename validator

* rename install

* rename geyser plugin interface when patch
2024-03-03 12:31:24 +08:00
Tyera a7f9fe103f
Split runtime utilities out of stake_state.rs (#35386)
* Add points module

* Add rewards module

* Hide rewards doc

* Fixup ledger-tool imports
2024-03-01 09:08:55 -07:00
Sean Young 9bb59aa30f
ledger-tool: verify: add --record-slots and --verify-slots (#34246)
ledger-tool: verify: add --verify-slots and --verify-slots-details

This adds:

    --record-slots <FILENAME>
	Write the slot hashes to this file.

    --record-slots-config hash-only|accounts
	Store the bank (=accounts) json file, or not.

    --verify-slots <FILENAME>
        Verify slot hashes against this file.

The first case can be used to dump a list of (slot, hash) to a json file
during a replay. The second case can be used to check slot hashes against
previously recorded values.

This is useful for debugging consensus failures, eg:

    # on good commit/branch
    ledger-tool verify --record-slots good.json --record-slots-config=accounts

    # on bad commit or potentially consensus breaking branch
    ledger-tool verify --verify-slots good.json

On a hash mismatch an error will be logged with the expected hash vs the
computed hash.
2024-03-01 08:39:30 +00:00
Alexander Meißner 74758d9fbf
Refactor - Move recompilation out of program loading (#35297)
Moves recompilation specifics out of load_program().
2024-02-23 17:25:32 +01:00
steviez bfcd4c8656
ledger-tool: Use error handling in blockstore command code (#35157)
There are lots of operations that could fail, including lots of the
Blockstore calls. The old code matched on Ok(_) or did unwrap()'s
which clutter the code and increase indentation.

This change wraps the entire command in a function that returns a
Result. The wrapper then does a single unwrap_or_else() and prints
any error message. Everywhere else is now free to use the ? operator
2024-02-22 00:19:28 -06:00
Ryo Onodera 024d6ecc4f
Add --unified-scheduler-handler-threads (#35195)
* Add --unified-scheduler-handler-threads

* Adjust value name

* Warn if the flag was ignored

* Tweak message a bit
2024-02-22 09:05:17 +09:00