* Find the bank hash of the heaviest fork, replay if necessary.
* Make it more explicit how heaviest fork slot is selected.
* Use process_single_slot instead of process_blockstore_from_root, the latter
may re-insert banks already frozen.
* Put BlockstoreProcessError into the error message.
* Check that all existing blocks link to correct parent before replay.
* Use the default number of threads instead.
* Check whether block is full and other small fixes.
* Fix root_bank and move comments to function level.
* Remove the extra parent link check.
Previously, entry verification had a dedicated threadpool used to verify
PoH hashes as well as some basic transaction verification via
Bank::verify_transaction(). It should also be noted that the entry
verification code provides logic to offload to a GPU if one is present.
Regardless of whether a GPU is present or not, some of the verification
must be done on a CPU. Moreso, the CPU verification of entries and
transaction execution are serial operations; entry verification finishes
first before moving onto transaction execution.
So, tx execution and entry verification are not competing for CPU cycles
at the same time and can use the same pool.
One exception to the above statement is that if someone is using the
feature to replay forks in parallel, then hypothetically, different
forks may end up competing for the same resources at the same time.
However, that is already true given that we had pools that were shared
between replay of multiple forks. So, this change doesn't really change
much for that case, but will reduce overhead in the single fork case
which is the vast majority of the time.
* Switch to blockstore.is_full() check because replay thread isn't active.
* Use make_chaining_slot_entries and add first_parent to the method.
Small style fixes.
* Switch to blockstore.is_full() check because replay thread isn't active.
The threadpool used to replay multiple transactions in parallel is
currently global state via a lazy_static definition. Making this pool
owned by ReplayStage will enable subsequent work to make the pool
size configurable on the CLI.
This makes `ReplayStage` create and hold the threadpool which is passed
down to blockstore_processor::confirm_slot().
blockstore_processor::process_blockstore_from_root() now creates its'
own threadpool as well; however, this pool is only alive while for
the scope of that function and does not persist the lifetime of the
process.
In a previous change, we removed the threadpool used to fetch entries
in parallel in favor of combining all fetches into a single rocksdb
multi_get() call.
This change does the same thing, except for a threadpool that was used
to fetch entries when we needed them to purge the transaction status
and address signatures columns.
RuntimeConfig doesn't use anything SVM specific and logically belongs
in program runtime rather than SVM. This change moves the definition
of RuntimeConfig struct from the SVM crate to program-runtime and
adjusts `use` statements accordingly.
* blockstore: atomize slot clearing, relax parent slot meta check
clear_unconfirmed_slot can leave blockstore in an irrecoverable state
if it panics in the middle. write batch this function, so that any
errors can be recovered after restart.
additionally relax the constraint that the parent slot meta must exist,
as it could have been cleaned up if outdated.
* pr feedback: use PurgeType, don't pass slot_meta
* pr feedback: add unit test
* pr feedback: refactor into separate function
* pr feedback: add special columns to helper, err msg, comments
* pr feedback: reword comments and write batch error message
* pr feedback: bubble write_batch error to caller
* pr feedback: reword comments
Co-authored-by: steviez <stevecz@umich.edu>
---------
Co-authored-by: steviez <stevecz@umich.edu>
ledger-tool: verify: add --verify-slots and --verify-slots-details
This adds:
--record-slots <FILENAME>
Write the slot hashes to this file.
--record-slots-config hash-only|accounts
Store the bank (=accounts) json file, or not.
--verify-slots <FILENAME>
Verify slot hashes against this file.
The first case can be used to dump a list of (slot, hash) to a json file
during a replay. The second case can be used to check slot hashes against
previously recorded values.
This is useful for debugging consensus failures, eg:
# on good commit/branch
ledger-tool verify --record-slots good.json --record-slots-config=accounts
# on bad commit or potentially consensus breaking branch
ledger-tool verify --verify-slots good.json
On a hash mismatch an error will be logged with the expected hash vs the
computed hash.
There are several cases for fetching entries from the Blockstore:
- Fetching entries for block replay
- Fetching entries for CompletedDataSetService
- Fetching entries to service RPC getBlock requests
All of these operations occur in a different calling thread. However,
the currently implementation utilizes a shared thread-pool within the
Blockstore function. There are several problems with this:
- The thread pool is shared between all of the listed cases, despite
block replay being the most critical. These other services shouldn't
be able to interfere with block replay
- The thread pool is overprovisioned for the average use; thread
utilization on both regular validators and RPC nodes shows that many
of the thread see very little activity. But, these thread existing
introduce "accounting" overhead
- rocksdb exposes an API to fetch multiple items at once, potentially
with some parallelization under the hood. Using parallelization in
our API and the underlying rocksdb is overkill and we're doing more
damage than good.
This change removes that threadpool completely, and instead fetches
all of the desired entries in a single call. This has been observed
to have a minor degradation on the time spent within the Blockstore
get_slot_entries_with_shred_info() function. Namely, some buffer
copying and deserialization that previously occurred in parallel now
occur serially.
However, the metric that tracks the amount of time spent replaying
blocks (inclusive of fetch) is unchanged. Thus, despite spending
marginally more time to fetch/copy/deserialize with only a single
thread, the gains from not thrashing everything else with the pool
keep us at parity.
Working towards adding a new Merkle shred variant with retransmitter's
signature, the commit uses struct instead of tuple to describe Merkle shred
variant.
Currently, ReplayStage sends new roots to BlockstoreCleanupService, and
BlockstoreCleanupService decides when to clean based on advancement of
the latest root. This is totally unnecessary as the latest root is
cached by the Blockstore, and this value can simply be fetched.
This change removes the channel completely, and instead just fetches
the latest root from Blockstore directly. Moreso, some logic is added
to check the latest root less frequently, based on the set purge
interval.
All in all, we went from sending > 100 slots/min across a crossbeam
channel to reading an atomic roughly 3 times/min, while also removing
the need for an additional thread that read from the channel.
Put the slot as the first field to make grep'ing for datapoints for a
specific slot in logs easier. This does not effect the datapoints
submission / presentation in metrics database
During a cluster upgrade when only half of the cluster can ingest the new shred
variant, sending shreds of the new variant can cause nodes to diverge.
The commit adds a feature to enable chained Merkle shreds explicitly.
The log statement is currently a bit misleading, and could be
interpretted as saying this routine deleted a shred.
Adjust the log statement to state that this routine is looking for the
shred but couldn't find it. Also, elevate the log to error level as
inconsistent state across columns should not be happening.
* Update genesis processing to have a fallback collector id for tests
* DCOU-ify the collector id for tests parameter (#1902)
* wrap test_collector_id in DCOU
* rename param to collector_id_for_tests
* fix program test
* fix dcou
---------
Co-authored-by: Brooks <brooks@prumo.org>
* blockstore: write only dirty erasure meta and merkle root metas
* pr feedback: use enum to distinguish clean/dirty
* pr feedback: comments, rename
* pr feedback: use AsRef
Wen
Joe C
Brooks
steviez
Ashwin Sekar
Pankaj Garg
Lucas Steuernagel
Dmitri Makarov
Ashwin Sekar
Yihau Chen
Brooks
Sean Young
behzad nouri
sakridge
Pankaj Garg
steviez
Andrew Fitzgerald
Justin Starry
Nick Frostbutter
GoodDaisy