* Add entries table to bt init
* Add entries to storage-proto
* Use new Blockstore method in bigtable_upload
* Add LedgerStorage::upload_confirmed_block_with_entries and use in bigtable_upload
* Upload entries to bigtable
This helper simply called std::mem::size_of<Self::Index>(). However, all
of the underlying functions that create keys manually copy fields into a
byte array. The fields are copied in end-to-end whereas size_of() might
include alignment bytes.
For example, a (u64, u32) only has 12 bytes of "data", but it would
have size 16 due to the 4 alignment padding bytes that would be
added to get the u32 (size 4) aligned with the u64 (size 8).
The function matches the access type and calls a different RocksDB
function depending on whether we have primary or secondary access. But,
lots of the code is the same for these two paths so de-duplicate the
repeated sections.
Currently, the RPC API that touch the Blockstore emit a datapoint for
each call. For an RPC node serving many requests, these datapoints
could get quite noisy, both in logs as well as traffic to the metrics
agent.
So, instead of submitting a datapoint for every call, accumulate the
number of calls in a struct and report that entire struct periodically.
The Blockstore currently maintains a RwLock<Slot> of the maximum root
it has seen inserted. The value is initialized during
Blockstore::open() and updated during calls to Blockstore::set_roots().
The max root is queried fairly often for several use cases, and caching
the value is cheaper than constructing an iterator to look it up every
time.
However, the access patterns of these RwLock match that of an atomic.
That is, there is no critical section of code that is run while the
lock is head. Rather, read/write locks are acquired in order to read/
update, respectively. So, change the RwLock<u64> to an AtomicU64.
The test_duplicate_with_pruned_ancestor test needs to get around a
limitation where the shreds with a parent older than the latest root are
discarded. The previous approach manually adjusted the root value in the
blockstore; this is not ideal in that it is fiddling with the inner
working of Blockstore.
So, use the is_trusted argument in Blockstore::insert_shreds(); setting
is_trusted=true bypasses the sanity checks (including the parent >=
latest root check).
JsonRpcRequestProcessor::check_blockstore_root() contained some logic
that performed duplicate sanity checking on a Blockstore fetch result.
The checking involved creating rocksdb iterators, which has non-trivial
overhead.
This PR removes the duplicate checking, and also adds comments to help
reason about how JsonRpcRequestProcessor interprets the Blockstore
result.
These services currently live in core/; however, they operate on the
ledger. Mores so, these two services operate on the blockstore only,
and not necessarily the entire ledger. So, it makes sense to move these
services out of core and into ledger. We've recently been doing similar
changes with breaking things out into individual crates in order to
reduce the scope of core.
So, this change moves the services from core/ to ledger/, and replaces
ledger with blockstore.
Instead of returning Result<Option<UnixTimestamp>>, return
Result<UnixTimestamp> and map None to an error. This makes the return
type similar to that of Blockstore::get_rooted_block().
* Remove RWLock from EntryNotifier because it causes perf degradation when entry notifications are enabled on geyser
* remove unused RWLock
* Remove RWLock
* Add InstalledScheduler for blockstore_processor
* Reverse if clauses
* Add more comments for process_batches()
* Elaborate comment
* Simplify schedule_transaction_executions type
upload_confirmed_blocks() states that it will return the passed in
ending_slot when there are no blocks to upload. This is enforced in one
early return but not the other. The result is that BigTableUploadService
could potentially get stuck in a loop of trying to upload the same slot.
While this case seems to be caused when an operator restarts their node
without --no-snapshot-fetch (which can cause a gap in blockstore), we
can still be friendly and allow them to break out of this loop.
* Initialize fork graph in program cache during bank_forks creation
* rename BankForks::new to BankForks::new_rw_arc
* fix compilation
* no need to set fork_graph on insert()
* fix partition tests
This macro is used a lot for tests to create a ledger path in order to
open a Blockstore. Files will be left on disk unless the test remembers
to call Blockstore::destroy() on the directory. So, instead of requiring
this, use the get_tmp_ledger_path_auto_delete!() macro that creates a
TempDir (which automatically deletes itself when it goes out of scope).
A previous change removed logic that populated the
TransactionStatusIndex entries at each of the legacy primary index keys
(0 and 1). While these entries will not be read or written in the
future, these entries are necessary for backwards compatibility. Namely,
branches <= v1.17 expect these entries to be present and .unwrap()'s
could fail if they are not.
So, add the initialization of these entries back into Blockstore logic.
We can remove initialization of these entries once our stable and beta
branches are both versions that do not expect these entries to be
present (should be v1.18).
* Convert OldestSlot to named struct
* Add clean_slot_0 to OldestSlot
* Set AtomicBool to true when all primary-index keys returning slot 0 should be purged
* Add PurgedFilter::clean_slot_0
* Use clean_slot_0 to preserve deprecated TransactionMemos
* Also set AtomicBool to true immediately on boot, if highest_primary_index_slot.is_none
* Add test
* Fixup test
* Fix typo
* Add Blockstore::highest_primary_index_slot
* Add getter
* Populate highest_primary_index_slot on boot
* Wipe highest_primary_index_slot when surpassed by oldest_slot
* Update highest_primary_index_slot in exact purge
* Return indexes early if highest_primary_index_slot has been cleared
* Limit read_transaction_status based on highest_primary_index_slot
* Limit read_transaction_memos based on highest_primary_index_slot
* Use highest_primary_index_slot to add early return to get_transaction_status_with_counter
* Fixup tests
* Use existing getter for highest_primary_index_slot
Co-authored-by: steviez <stevecz@umich.edu>
---------
Co-authored-by: steviez <stevecz@umich.edu>
* Always call initialize_transaction_status_index() at startup,
doing so will ensure dummy entries are actually cleaned
* Rename initialize_transaction_status_index()
* Stop initializing TransactionStatusIndex column entries, these
are no longer needed and old software will initialize if needed
The special columns, TransactionStatus and AddressSignatures, are only
populated if --enable-rpc-transaction-history is passed. Cleaning these
columns for a range of slots is very expensive, as the block for each
slot must be read, deserialized, and then parsed to extract all of the
transaction signatures and address pubkeys.
This change adds a simple check to see if there are any values at all in
the special columns. If there are not, then the whole process described
above can be skipped for nodes that are not storing the special columns.
clone_from_slice() would hypothetically visit each item in the slice and
clone it whereas copy_from_slice() can memcpy the whole slice in one go.
Technically, Rust does the right thing for us by making
clone_from_slice() defer to copy_from_slice() for types that implement
Copy trait. However, we should still use the more efficient method
directly to show intent.
* Adjust test_purge_transaction_status_exact to test slots that cross primary indexes
* Minimize deletes by checking the primary-index range
* Fix test_purge_special_columns_compaction_filter
Tyera
Brooks
steviez
Ryo Onodera
Ashwin Sekar
Liam Vovk
Pankaj Garg
Tao Zhu
Alexander Meißner
kirill lykov
Andrew Fitzgerald