There are operations in bank_fork_utils that may fail; we explicitly
call std::process::exit() on several of these. Granted we may end up
exiting the process higher up the callstack, bubbling the errors up
allow a caller that could handle the error to do so.
As we develop new features or modifications, we occassionally need to
introduce new columns to the Blockstore. Adding a new column introduces
a compatibility break given that opening the database in Primary mode
(R/W access) requires opening all columns. Reverting to an old software
version that is unaware of the new column is obviously problematic.
In the past, we have addressed by backporting minimal "stub" PR's to
older versions. This is annoying, and only allow compatibility for the
single version or two that we backport to.
This PR adds a change to automatically detect all columns, and create
default column descriptors for columns we were unaware of. As a result,
older software versions can open a Blockstore that was modified by a
newer software version, even if that new version added columns that the
old version is unaware of.
* Add entries table to bt init
* Add entries to storage-proto
* Use new Blockstore method in bigtable_upload
* Add LedgerStorage::upload_confirmed_block_with_entries and use in bigtable_upload
* Upload entries to bigtable
This helper simply called std::mem::size_of<Self::Index>(). However, all
of the underlying functions that create keys manually copy fields into a
byte array. The fields are copied in end-to-end whereas size_of() might
include alignment bytes.
For example, a (u64, u32) only has 12 bytes of "data", but it would
have size 16 due to the 4 alignment padding bytes that would be
added to get the u32 (size 4) aligned with the u64 (size 8).
The function matches the access type and calls a different RocksDB
function depending on whether we have primary or secondary access. But,
lots of the code is the same for these two paths so de-duplicate the
repeated sections.
Currently, the RPC API that touch the Blockstore emit a datapoint for
each call. For an RPC node serving many requests, these datapoints
could get quite noisy, both in logs as well as traffic to the metrics
agent.
So, instead of submitting a datapoint for every call, accumulate the
number of calls in a struct and report that entire struct periodically.
The Blockstore currently maintains a RwLock<Slot> of the maximum root
it has seen inserted. The value is initialized during
Blockstore::open() and updated during calls to Blockstore::set_roots().
The max root is queried fairly often for several use cases, and caching
the value is cheaper than constructing an iterator to look it up every
time.
However, the access patterns of these RwLock match that of an atomic.
That is, there is no critical section of code that is run while the
lock is head. Rather, read/write locks are acquired in order to read/
update, respectively. So, change the RwLock<u64> to an AtomicU64.
The test_duplicate_with_pruned_ancestor test needs to get around a
limitation where the shreds with a parent older than the latest root are
discarded. The previous approach manually adjusted the root value in the
blockstore; this is not ideal in that it is fiddling with the inner
working of Blockstore.
So, use the is_trusted argument in Blockstore::insert_shreds(); setting
is_trusted=true bypasses the sanity checks (including the parent >=
latest root check).
JsonRpcRequestProcessor::check_blockstore_root() contained some logic
that performed duplicate sanity checking on a Blockstore fetch result.
The checking involved creating rocksdb iterators, which has non-trivial
overhead.
This PR removes the duplicate checking, and also adds comments to help
reason about how JsonRpcRequestProcessor interprets the Blockstore
result.
These services currently live in core/; however, they operate on the
ledger. Mores so, these two services operate on the blockstore only,
and not necessarily the entire ledger. So, it makes sense to move these
services out of core and into ledger. We've recently been doing similar
changes with breaking things out into individual crates in order to
reduce the scope of core.
So, this change moves the services from core/ to ledger/, and replaces
ledger with blockstore.
Instead of returning Result<Option<UnixTimestamp>>, return
Result<UnixTimestamp> and map None to an error. This makes the return
type similar to that of Blockstore::get_rooted_block().
* Remove RWLock from EntryNotifier because it causes perf degradation when entry notifications are enabled on geyser
* remove unused RWLock
* Remove RWLock
* Add InstalledScheduler for blockstore_processor
* Reverse if clauses
* Add more comments for process_batches()
* Elaborate comment
* Simplify schedule_transaction_executions type
upload_confirmed_blocks() states that it will return the passed in
ending_slot when there are no blocks to upload. This is enforced in one
early return but not the other. The result is that BigTableUploadService
could potentially get stuck in a loop of trying to upload the same slot.
While this case seems to be caused when an operator restarts their node
without --no-snapshot-fetch (which can cause a gap in blockstore), we
can still be friendly and allow them to break out of this loop.
* Initialize fork graph in program cache during bank_forks creation
* rename BankForks::new to BankForks::new_rw_arc
* fix compilation
* no need to set fork_graph on insert()
* fix partition tests
This macro is used a lot for tests to create a ledger path in order to
open a Blockstore. Files will be left on disk unless the test remembers
to call Blockstore::destroy() on the directory. So, instead of requiring
this, use the get_tmp_ledger_path_auto_delete!() macro that creates a
TempDir (which automatically deletes itself when it goes out of scope).
A previous change removed logic that populated the
TransactionStatusIndex entries at each of the legacy primary index keys
(0 and 1). While these entries will not be read or written in the
future, these entries are necessary for backwards compatibility. Namely,
branches <= v1.17 expect these entries to be present and .unwrap()'s
could fail if they are not.
So, add the initialization of these entries back into Blockstore logic.
We can remove initialization of these entries once our stable and beta
branches are both versions that do not expect these entries to be
present (should be v1.18).
* Convert OldestSlot to named struct
* Add clean_slot_0 to OldestSlot
* Set AtomicBool to true when all primary-index keys returning slot 0 should be purged
* Add PurgedFilter::clean_slot_0
* Use clean_slot_0 to preserve deprecated TransactionMemos
* Also set AtomicBool to true immediately on boot, if highest_primary_index_slot.is_none
* Add test
* Fixup test
* Fix typo
* Add Blockstore::highest_primary_index_slot
* Add getter
* Populate highest_primary_index_slot on boot
* Wipe highest_primary_index_slot when surpassed by oldest_slot
* Update highest_primary_index_slot in exact purge
* Return indexes early if highest_primary_index_slot has been cleared
* Limit read_transaction_status based on highest_primary_index_slot
* Limit read_transaction_memos based on highest_primary_index_slot
* Use highest_primary_index_slot to add early return to get_transaction_status_with_counter
* Fixup tests
* Use existing getter for highest_primary_index_slot
Co-authored-by: steviez <stevecz@umich.edu>
---------
Co-authored-by: steviez <stevecz@umich.edu>
* Always call initialize_transaction_status_index() at startup,
doing so will ensure dummy entries are actually cleaned
* Rename initialize_transaction_status_index()
* Stop initializing TransactionStatusIndex column entries, these
are no longer needed and old software will initialize if needed
The special columns, TransactionStatus and AddressSignatures, are only
populated if --enable-rpc-transaction-history is passed. Cleaning these
columns for a range of slots is very expensive, as the block for each
slot must be read, deserialized, and then parsed to extract all of the
transaction signatures and address pubkeys.
This change adds a simple check to see if there are any values at all in
the special columns. If there are not, then the whole process described
above can be skipped for nodes that are not storing the special columns.
clone_from_slice() would hypothetically visit each item in the slice and
clone it whereas copy_from_slice() can memcpy the whole slice in one go.
Technically, Rust does the right thing for us by making
clone_from_slice() defer to copy_from_slice() for types that implement
Copy trait. However, we should still use the more efficient method
directly to show intent.
* Adjust test_purge_transaction_status_exact to test slots that cross primary indexes
* Minimize deletes by checking the primary-index range
* Fix test_purge_special_columns_compaction_filter
These entries were found to improve compaction performance when
LedgerCleanupService was performing direct compactions on each primary
index. Cleaning by primary index has been deprecated for a while, and
as such, these dummy entries are no longer needed and can be removed.
* token: Update to 4.0.0
* token-2022: Bump and support new account and instruction types
* Update token-2022 in fetch_spl / program-test
* Fixup downstream uses
* Mint and destination were flipped in 0.9.0
* Don't use `convert_pubkey`
* Bump spl dependencies to versions which avoid recompilations
* Move vote related code to its own crate
* Update imports in code and tests
* update programs/sbf/Cargo.lock
* fix check errors
* update abi_digest
* rebase fixes
* fixes after rebase
removes outdated matches crate from the dependencies
std::matches has been stable since rust 1.42.0.
Other use-cases are covered by assert_matches crate.
* allow pedantic invalid cast lint
* allow lint with false-positive triggered by `test-case` crate
* nightly `fmt` correction
* adapt to rust layout changes
* remove dubious test
* Use transmute instead of pointer cast and de/ref when check_aligned is false.
* Renames clippy::integer_arithmetic to clippy::arithmetic_side_effects.
* bump rust nightly to 2023-08-25
* Upgrades Rust to 1.72.0
---------
Co-authored-by: Trent Nelson <trent@solana.com>
* remove unnecessary hashes around raw string literals
* remove unncessary literal `unwrap()`s
* remove panicking `unwrap()`
* remove unnecessary `unwrap()`
* use `[]` instead of `vec![]` where applicable
* remove (more) unnecessary explicit `into_iter()` calls
* remove redundant pattern matching
* don't cast to same type and constness
* do not `cfg(any(...` a single item
* remove needless pass by `&mut`
* prefer `or_default()` to `or_insert_with(T::default())`
* `filter_map()` better written as `filter()`
* incorrect `PartialOrd` impl on `Ord` type
* replace "slow zero-filled `Vec` initializations"
* remove redundant local bindings
* add required lifetime to associated constant
* sdk: Add concurrent support for rand 0.7 and 0.8
* Update rand, rand_chacha, and getrandom versions
* Run command to replace `gen_range`
Run `git grep -l gen_range | xargs sed -i'' -e 's/gen_range(\(\S*\), /gen_range(\1../'
* sdk: Fix users of older `gen_range`
* Replace `hash::new_rand` with `hash::new_with_thread_rng`
Run:
```
git grep -l hash::new_rand | xargs sed -i'' -e 's/hash::new_rand([^)]*/hash::new_with_thread_rng(/'
```
* perf: Use `Keypair::new()` instead of `generate`
* Use older rand version in zk-token-sdk
* program-runtime: Inline random key generation
* bloom: Fix clippy warnings in tests
* streamer: Scope rng usage correctly
* perf: Fix clippy warning
* accounts-db: Map to char to generate a random string
* Remove `from_secret_key_bytes`, it's just `keypair_from_seed`
* ledger: Generate keypairs by hand
* ed25519-tests: Use new rand
* runtime: Use new rand in all tests
* gossip: Clean up clippy and inline keypair generators
* core: Inline keypair generation for tests
* Push sbf lockfile change
* sdk: Sort dependencies correctly
* Remove `hash::new_with_thread_rng`, use `Hash::new_unique()`
* Use Keypair::new where chacha isn't used
* sdk: Fix build by marking rand 0.7 optional
* Hardcode secret key length, add static assertion
* Unify `getrandom` crate usage to fix linking errors
* bloom: Fix tests that require a random hash
* Remove some dependencies, try to unify others
* Remove unnecessary uses of rand and rand_core
* Update lockfiles
* Add back some dependencies to reduce rebuilds
* Increase max rebuilds from 14 to 15
* frozen-abi: Remove `getrandom`
* Bump rebuilds to 17
* Remove getrandom from zk-token-proof
In most cases, either a &Bank or an Arc<Bank> is more proper.
- &Bank is used if the function only needs a momentary reference
- Arc<Bank> is used if the function needs its' own copy
This PR leaves several instances of &Arc<Bank> around; these instances
are situations where a clone may only happen conditionally.
The current desired open file descriptor limit is 1,000,000. This is
quite a large number, and not needed for every command. Namely, commands
that do not unpack a snapshot and create an AccountsDB will likely not
use this many files.
There is already an option in BlockstoreOptions to ignore errors if the
desired value cannot be set; this PR just bubbles that option up to a
CLI flag in ledger-tool.
* Move functions that take a &Database under impl Blockstore {...}
* Replace &Database with &self in those functions and update callers
* Use LedgerColumn's from Blockstore instead of re-fetching
* Add missing roots LedgerColumn and have it report metrics like others
* Remove several redundant comments
Periodic compaction was previously disabled on all columns in #27571 in
favor of the delete_file_in_range() approach that #26651 introduced.
However, several columns still rely on periodic compaction to reclaim
storage. Namely, the TransactionStatus and AddressSignatures columns, as
these columns contain a slot in their key, but as a non-primary index.
The result of periodic compaction not running on these columns is that
no storage space is being reclaimed from columns. This is obviously bad
and would lead to a node eventually running of storage space and
crashing.
This PR reintroduces periodic compaction, but only for the columns that
need it.
The existing signature unpacked elements from a Shred and took an owned
Vec<u8>, forcing a .clone() from the caller. The Shred can be passed in
directly to simplify argument list and avoid the clone.
We have several other functions that return data shreds; however, these
other functions map shred::Error to BlockstoreError. Make this function
consistent with those and map the error.
Per rust-rocksdb docs, snappy compression will be the default if snappy
feature is enabled in that crate. We don't want compression by default
and there is seemingly a minor bug with the compression type selection
upstream, so explicitly set compression type to none in our code.
* Move CostModel and CostTracker to its own crate
* compile new crate and update imports
* update sbf Cargo.lock
* fix AbiExample
* fix cargo sort
* Fix AbiExample
If a slot is marked as optimistically confirmed, it is probable but not
guaranteed that its' ancestors will also be marked as optimistically
confirmed in the Blockstore. Given the importance of examining
optimistically confirmed slots around cluster restarts, manually walk
an AncestorIterator to avoid the chance of a slot improperly being
ignored in cluster restart scenarios.
The optional args allow reuse by ledger-tool repair roots command Also,
hold cleanup lock for duration of Blockstore::scan_and_fix_roots().
This prevents a scenario where scan_and_fix_roots() could identify a
slot as needing to be marked root, that slot getting cleaned by
LedgerCleanupService, and then scan_and_fix_roots() marking the slot as
root on the now purged slot.
Andrew Fitzgerald
Ashwin Sekar
steviez
Tyera
Brooks
Ryo Onodera
Liam Vovk
Pankaj Garg
Tao Zhu
Alexander Meißner
kirill lykov
Jon Cinque
Trent Nelson
behzad nouri
carllin
dependabot[bot]
Justin Starry
Illia Bobyr