Bigtable: update google proto files and allow configuration of max_message_size (#34740)
* Update proto files with tonic-build v0.9.2
* Manually ignore invalid doc-tests
* Add new ReadRowsRequest fields
* Add LedgerStorageConfig::max_message_size and default value
* Add BigtableConnection::max_message_size and use on client creation
* Add max_message_size to RpcBigtableConfig and make const pub
* Add solana-validator cli arg
(cherry picked from commit 166be2995e)
Co-authored-by: Tyera <tyera@solana.com>
ledger-tool: support v0 transactions in blocks from bigtable (#34544)
Support v0 transactions
(cherry picked from commit 7d209a448d)
Co-authored-by: Tyera <tyera@solana.com>
Output BankHashDetails file when leader drops its' own block (#34256)
Currently, the file is generated when a node drops a block that was
produced by another node. However, it would also be beneficial to see
the account state when a node drops its' own block.
Output the file in this additional failure codepath
(cherry picked from commit 935e06f8f1)
Co-authored-by: steviez <steven@solana.com>
* Do not unnecessarily re-verify unloaded program
* clippy fixes
* new unit test for extract
* fixes after rebase
* fixes after rebase
* address review comments
* fix clippy
* allow pedantic invalid cast lint
* allow lint with false-positive triggered by `test-case` crate
* nightly `fmt` correction
* adapt to rust layout changes
* remove dubious test
* Use transmute instead of pointer cast and de/ref when check_aligned is false.
* Renames clippy::integer_arithmetic to clippy::arithmetic_side_effects.
* bump rust nightly to 2023-08-25
* Upgrades Rust to 1.72.0
---------
Co-authored-by: Trent Nelson <trent@solana.com>
In most cases, either a &Bank or an Arc<Bank> is more proper.
- &Bank is used if the function only needs a momentary reference
- Arc<Bank> is used if the function needs its' own copy
This PR leaves several instances of &Arc<Bank> around; these instances
are situations where a clone may only happen conditionally.
* Integrate program loader-v4 with bank
* fix tests
* new struct for ProgramRuntimeEnvironments
* remove environment from program_runtime_environment_v
* move find_program_in_cache() to invoke_context
* cleanup
When ledger-tool runs, it may create secondary directories for things
like accounts, accounts-index, etc as not to potentially interfere with
solana-validator's directories. These would show as multiple directories
with ".ledger-tool" appended to the typical directory name.
To more clearly group these items, make the default directories for
snapshots, accounts and accounts-index all within a common "ledger-tool"
directory.
When a consensus divergance occurs, the current workflow involves a
handful of manual steps to hone in on the offending slot and
transaction. This process isn't overly difficult to execute; however, it
is tedious and currently involves creating and parsing logs.
This change introduces functionality to output a debug file that
contains the components go into the bank hash. The file can be generated
in two ways:
- Via solana-validator when the node realizes it has diverged
- Via solana-ledger-tool verify by passing a flag
When a divergance occurs now, the steps to debug would be:
- Grab the file from the node that diverged
- Generate a file for the same slot with ledger-tool with a known good
version
- Diff the files, they are pretty-printed json
Several of the program command subcommands do not require a ledger as
they act solely on the program object. So, defer checking the ledger
path until we know we need to load. Additionally, remove genesis arg
from these commands that do not load a genesis.bin
The current desired open file descriptor limit is 1,000,000. This is
quite a large number, and not needed for every command. Namely, commands
that do not unpack a snapshot and create an AccountsDB will likely not
use this many files.
There is already an option in BlockstoreOptions to ignore errors if the
desired value cannot be set; this PR just bubbles that option up to a
CLI flag in ledger-tool.
The --encoding flag only makes sense for the genesis command if the
genesis accounts will be printed. Hence, the encoding flag used the
Arg::requires() function on --accounts argument.
However, in what appears to be a clap bug, this made the --accounts flag
required even though it should not be. So, remove the .requires()
dependency between these two arguments.
* Move CostModel and CostTracker to its own crate
* compile new crate and update imports
* update sbf Cargo.lock
* fix AbiExample
* fix cargo sort
* Fix AbiExample
If a slot is marked as optimistically confirmed, it is probable but not
guaranteed that its' ancestors will also be marked as optimistically
confirmed in the Blockstore. Given the importance of examining
optimistically confirmed slots around cluster restarts, manually walk
an AncestorIterator to avoid the chance of a slot improperly being
ignored in cluster restart scenarios.
The optional args allow reuse by ledger-tool repair roots command Also,
hold cleanup lock for duration of Blockstore::scan_and_fix_roots().
This prevents a scenario where scan_and_fix_roots() could identify a
slot as needing to be marked root, that slot getting cleaned by
LedgerCleanupService, and then scan_and_fix_roots() marking the slot as
root on the now purged slot.
The previous help message was pretty generic as it covered two potential
error scenarios. This change gives a more detailed error message to each
of those errors.
* Restrict access to Bank's HardForks
Callers could previously obtain a a lock to read/write HardForks from
any Bank. This would allow any caller to modify, and creates the
opportunity for inconsistent handling of what is considered a valid hard
fork (ie too old).
This PR adds a function to Bank so consistent sanity checks can be
applied; the caller will already have a Bank as that is where they would
have obtained the HardForks from in the first place. Additionally,
change the getter to return a copy of HardForks (simple Vec).
* Allow hard fork at bank slot if bank is not yet frozen
* Makes Bank::load_program() return correct tombstones.
* Removes early TX failure caused by closed and invalid programs.
* Adjusts the feature gate of simplify_writable_program_account_check.
mergify[bot]
yihau
Alexander Meißner
Brooks
steviez
Pankaj Garg
Alessandro Decina
Jon Cinque
HaoranYi
behzad nouri
Illia Bobyr
Andrew Fitzgerald