ledger-tool: Fix create-snapshot default value for output_directory (#3148)
The arguments to specify full and incremental snapshot archives paths
used to be a global argument; these were moved to only be instantiated
on commands that needed them in #1773.
But, when the arguments were moved from app-level to subcommand-level,
the code that matches the arguments was not updated to look at
subcommand-matches instead of app-matches.
(cherry picked from commit 1d9947cd68)
Co-authored-by: steviez <steven@anza.xyz>
There is often a desire to examine/replay/etc older blocks. If the
blocks are recent enough, they can be pulled from an actively running
node. Otherwise, the blocks must be pulled down from warehouse node
archives. These archives are uploaded on a per-epoch basis so they are
quite large, and can take multiple hours to download and decompress.
With the addition of Entry data to BigTable, blocks can be recreated
from BigTable data. Namely, we can recreate the Entries with proper PoH
and transaction data. We can then shred them such that they are the
same format as blocks that are produced from the cluster.
This change introduces a new command that will read BigTable data and
insert shreds into a local Blockstore. The new command is:
$ agave-ledger-tool bigtable shreds ...
Several important notes about the change:
- Shred for some slot S will not be signed by the actual leader for
slot S. Instead, shreds will be signed with a "dummy" keypair. The
shred signatures does not affect the ability to replay the block.
- Entry PoH data does not go back to genesis in BigTable. This data
could be extracted and uploaded from the existing rocksdb archives;
however, that work is not planned as far as I know. --allow-mock-poh
can be passed to generate filler PoH data. Blocks created with this
flag are replayable by passing --skip-poh-verify to ledger-tool.
- A snapshot will be unpacked to determine items such as the shred
version, tick hash rate and ticks per slot. This snapshot must be in
the same epoch as the requested slots
The bank-hash command in ledger-tool was recently deprecated. However,
the command is used by some of the scripts that coordinate starting up
a fresh cluster. So, the deprecation of bank-hash broke those scripts.
This change fixes the scripts by doing the following:
- Makes --print-bank-hash support --output json
- Updates scripts to install jq on provisioned nodes
- Update remote-node.sh to parse the bank hash from json using jq
* Rename ComputeBudget::max_invoke_stack_height to max_instruction_stack_depth
The new name is consistent with the existing
ComputeBudget::max_instruction_trace_length.
Also expose compute_budget:MAX_INSTRUCTION_DEPTH.
* bpf_loader: use an explicit thread-local pool for stack and heap memory
Use a fixed thread-local pool to hold stack and heap memory. This
mitigates the long standing issue of jemalloc causing TLB shootdowns to
serve such frequent large allocations.
Because we need 1 stack and 1 heap region per instruction, and the
current max instruction nesting is hardcoded to 5, the pre-allocated
size is (MAX_STACK + MAX_HEAP) * 5 * NUM_THREADS. With the current
limits that's about 2.5MB per thread. Note that this is memory that
would eventually get allocated anyway, we're just pre-allocating it now.
* programs/sbf: add test for stack/heap zeroing
Add TEST_STACK_HEAP_ZEROED which tests that stack and heap regions are
zeroed across reuse from the memory pool.
AccountsBackgroundService performs several operations that can take a
long time to complete and do not check the exit flag mid-operation.
Thus, ledger-tool can get hung up for a while waiting for ABS to
finish. However, many ledger-tool command do not ABS to have finished.
So, return a handle to the ABS thread and allow the caller to decide
whether to join ABS or not. As of right now, create-snapshot is the
only command that requires ABS to have finished before continuing.
There are several arguments to control snapshot configuration in the
various ledger-tool commands. The inclusion of args in each command
is inconsistent, especially for commands outside of main.rs
This change consolidates the snapshot related arguments into a single
function to help create consistency and reduce duplicate code
There are a handful of commands that specify AccountsDb config. To
create consistency between all these commands and remove repeated code,
build all of the arguments in a helper function that is passed to the
relevant commands.
ledger-tool has a somewhat cluttered interface. The bank-hash command
offers little extra functionality compared to what the verify command
does, and could instead be an optional flag for the verify command.
This change leaves the bank-hash command present, but prints an error
message and tells the user to instead use verify --print-bank-hash. The
command will be removed altogether in the future.
This makes the slot subcommand use the same output method as bigtable
block to output blocks (-vv). Doing so creates consistency between the
two commands as well as removing a duplicate implementation in the
ledger-tool code. The shared type also supports json output which the
ledger-tool implementation did not.
* use serde_derive instead of the derive feature of serde
* make serde_derive version match serde version
* remove serde dep entirely from programs/address-lookup-table
* replace remaining usage of serde "derive" feature
* update lock files
* explicitly use serde_derive in svm and frozen-abi
* add comments about serde and serde_derive versions matching
* clippy: allow dead_code for SubscriptionToken
* clippy: allow dead_code for GraphVoteAccountModeError
* clippy: allow dead_code for DroppableTask
* remove the string in the GraphVoteAccountModeError
* abort the joinhandle when drop
* only trace JoinHandle in the drop
* add comment
This command is basically an exact duplicate of the print command,
except print had OutputFormat::Display hard-coded whereas the json
command had OutputFormat::Json hard-coded.
Instead of hard-coded values, parse value of --output-format from CLI
The agave-ledger-tool's functionality to execute sBPF programs out-of-band
did not properly handle duplicate accounts. This patch introduces instruction
account deduplication.
Locally tested to confirm that duplicate accounts are serialized correctly
and that non-duplicate account behavior didn't change.
Co-authored-by: Richard Patel <ripatel@jumptrading.com>
If someone wants to purge from a given slot and on, they might pass a
sentinel value like 999999999 for --end-slot. Given the following logic
in purge, this could extra looping, even after the end of blockstore
data has been reached.
So, compute the actual highest blockstore slot, and adjust the user
supplied end_slot to min(end_slot, highest_slot)
* Only the verifier can cause FailedVerification, everything else is Closed
* Removes the environments parameter from load_program_accounts().
* cargo fmt
* Simplify invocation of deployed program
* Attempt to invoke a program before it is deployed
* Attempt to invoke a buffer before it is used in a deployment
* Escalates Option return value of load_program_accounts() to load_program_with_pubkey().
* Review feedback
* Fix: deploy program on last slot of epoch during environment change
* solana-runtime: deploy at last epoch slot test
* disable deployment of sol_alloc_free
* Move tx-batch-constructor to its own function
* use new_from_cache
---------
Co-authored-by: Alessandro Decina <alessandro.d@gmail.com>
RuntimeConfig doesn't use anything SVM specific and logically belongs
in program runtime rather than SVM. This change moves the definition
of RuntimeConfig struct from the SVM crate to program-runtime and
adjusts `use` statements accordingly.
ledger-tool: verify: add --verify-slots and --verify-slots-details
This adds:
--record-slots <FILENAME>
Write the slot hashes to this file.
--record-slots-config hash-only|accounts
Store the bank (=accounts) json file, or not.
--verify-slots <FILENAME>
Verify slot hashes against this file.
The first case can be used to dump a list of (slot, hash) to a json file
during a replay. The second case can be used to check slot hashes against
previously recorded values.
This is useful for debugging consensus failures, eg:
# on good commit/branch
ledger-tool verify --record-slots good.json --record-slots-config=accounts
# on bad commit or potentially consensus breaking branch
ledger-tool verify --verify-slots good.json
On a hash mismatch an error will be logged with the expected hash vs the
computed hash.
There are lots of operations that could fail, including lots of the
Blockstore calls. The old code matched on Ok(_) or did unwrap()'s
which clutter the code and increase indentation.
This change wraps the entire command in a function that returns a
Result. The wrapper then does a single unwrap_or_else() and prints
any error message. Everywhere else is now free to use the ? operator
mergify[bot]
steviez
Alessandro Decina
Lucas Ste
galactus
Sean Young
Brooks
Joe C
Yihau Chen
Kevin Heavey
Andrei Silviu Dragnea
ripatel-fd
Joe C
Alexander Meißner
Justin Starry
Pankaj Garg
HaoranYi
Yihau Chen
Dmitri Makarov
Tyera
Ryo Onodera