* Switch to blockstore.is_full() check because replay thread isn't active.
* Use make_chaining_slot_entries and add first_parent to the method.
Small style fixes.
* Switch to blockstore.is_full() check because replay thread isn't active.
The threadpool used to replay multiple transactions in parallel is
currently global state via a lazy_static definition. Making this pool
owned by ReplayStage will enable subsequent work to make the pool
size configurable on the CLI.
This makes `ReplayStage` create and hold the threadpool which is passed
down to blockstore_processor::confirm_slot().
blockstore_processor::process_blockstore_from_root() now creates its'
own threadpool as well; however, this pool is only alive while for
the scope of that function and does not persist the lifetime of the
process.
In a previous change, we removed the threadpool used to fetch entries
in parallel in favor of combining all fetches into a single rocksdb
multi_get() call.
This change does the same thing, except for a threadpool that was used
to fetch entries when we needed them to purge the transaction status
and address signatures columns.
RuntimeConfig doesn't use anything SVM specific and logically belongs
in program runtime rather than SVM. This change moves the definition
of RuntimeConfig struct from the SVM crate to program-runtime and
adjusts `use` statements accordingly.
* blockstore: atomize slot clearing, relax parent slot meta check
clear_unconfirmed_slot can leave blockstore in an irrecoverable state
if it panics in the middle. write batch this function, so that any
errors can be recovered after restart.
additionally relax the constraint that the parent slot meta must exist,
as it could have been cleaned up if outdated.
* pr feedback: use PurgeType, don't pass slot_meta
* pr feedback: add unit test
* pr feedback: refactor into separate function
* pr feedback: add special columns to helper, err msg, comments
* pr feedback: reword comments and write batch error message
* pr feedback: bubble write_batch error to caller
* pr feedback: reword comments
Co-authored-by: steviez <stevecz@umich.edu>
---------
Co-authored-by: steviez <stevecz@umich.edu>
ledger-tool: verify: add --verify-slots and --verify-slots-details
This adds:
--record-slots <FILENAME>
Write the slot hashes to this file.
--record-slots-config hash-only|accounts
Store the bank (=accounts) json file, or not.
--verify-slots <FILENAME>
Verify slot hashes against this file.
The first case can be used to dump a list of (slot, hash) to a json file
during a replay. The second case can be used to check slot hashes against
previously recorded values.
This is useful for debugging consensus failures, eg:
# on good commit/branch
ledger-tool verify --record-slots good.json --record-slots-config=accounts
# on bad commit or potentially consensus breaking branch
ledger-tool verify --verify-slots good.json
On a hash mismatch an error will be logged with the expected hash vs the
computed hash.
There are several cases for fetching entries from the Blockstore:
- Fetching entries for block replay
- Fetching entries for CompletedDataSetService
- Fetching entries to service RPC getBlock requests
All of these operations occur in a different calling thread. However,
the currently implementation utilizes a shared thread-pool within the
Blockstore function. There are several problems with this:
- The thread pool is shared between all of the listed cases, despite
block replay being the most critical. These other services shouldn't
be able to interfere with block replay
- The thread pool is overprovisioned for the average use; thread
utilization on both regular validators and RPC nodes shows that many
of the thread see very little activity. But, these thread existing
introduce "accounting" overhead
- rocksdb exposes an API to fetch multiple items at once, potentially
with some parallelization under the hood. Using parallelization in
our API and the underlying rocksdb is overkill and we're doing more
damage than good.
This change removes that threadpool completely, and instead fetches
all of the desired entries in a single call. This has been observed
to have a minor degradation on the time spent within the Blockstore
get_slot_entries_with_shred_info() function. Namely, some buffer
copying and deserialization that previously occurred in parallel now
occur serially.
However, the metric that tracks the amount of time spent replaying
blocks (inclusive of fetch) is unchanged. Thus, despite spending
marginally more time to fetch/copy/deserialize with only a single
thread, the gains from not thrashing everything else with the pool
keep us at parity.
Working towards adding a new Merkle shred variant with retransmitter's
signature, the commit uses struct instead of tuple to describe Merkle shred
variant.
Currently, ReplayStage sends new roots to BlockstoreCleanupService, and
BlockstoreCleanupService decides when to clean based on advancement of
the latest root. This is totally unnecessary as the latest root is
cached by the Blockstore, and this value can simply be fetched.
This change removes the channel completely, and instead just fetches
the latest root from Blockstore directly. Moreso, some logic is added
to check the latest root less frequently, based on the set purge
interval.
All in all, we went from sending > 100 slots/min across a crossbeam
channel to reading an atomic roughly 3 times/min, while also removing
the need for an additional thread that read from the channel.
Put the slot as the first field to make grep'ing for datapoints for a
specific slot in logs easier. This does not effect the datapoints
submission / presentation in metrics database
During a cluster upgrade when only half of the cluster can ingest the new shred
variant, sending shreds of the new variant can cause nodes to diverge.
The commit adds a feature to enable chained Merkle shreds explicitly.
The log statement is currently a bit misleading, and could be
interpretted as saying this routine deleted a shred.
Adjust the log statement to state that this routine is looking for the
shred but couldn't find it. Also, elevate the log to error level as
inconsistent state across columns should not be happening.
* Update genesis processing to have a fallback collector id for tests
* DCOU-ify the collector id for tests parameter (#1902)
* wrap test_collector_id in DCOU
* rename param to collector_id_for_tests
* fix program test
* fix dcou
---------
Co-authored-by: Brooks <brooks@prumo.org>
* blockstore: write only dirty erasure meta and merkle root metas
* pr feedback: use enum to distinguish clean/dirty
* pr feedback: comments, rename
* pr feedback: use AsRef
* Finalize unified scheduler plumbing with min impl
* Fix comment
* Rename leftover type name...
* Make logging text less ambiguous
* Make PhantomData simplyer without already used S
* Make TaskHandler stateless again
* Introduce HandlerContext to simplify TaskHandler
* Add comment for coexistence of Pool::{new,new_dyn}
* Fix grammar
* Remove confusing const for upcoming changes
* Demote InstalledScheduler::context() into dcou
* Delay drop of context up to return_to_pool()-ing
* Revert "Demote InstalledScheduler::context() into dcou"
This reverts commit 049a126c905df0ba8ad975c5cb1007ae90a21050.
* Revert "Delay drop of context up to return_to_pool()-ing"
This reverts commit 60b1bd2511a714690b0b2331e49bc3d0c72e3475.
* Make context handling really type-safe
* Update comment
* Fix grammar...
* Refine type aliases for boxed traits
* Swap the tuple order for readability & semantics
* Simplify PooledScheduler::result_with_timings type
* Restore .in_sequence()
* Use where for aesthetics
* Simplify if...
* Fix typo...
* Polish ::schedule_execution() a bit
* Fix rebase conflicts..
* Make test more readable
* Fix test failures after rebase...
These entries are legacy code at this point; however, older release
branches require these entries to be present. Also, while it would be
nice to clean up these entries immediately, they only occupy a small
amount of space so having them linger a little longer isn't a big deal.
Pankaj Garg
Wen
steviez
Brooks
Lucas Steuernagel
Dmitri Makarov
Will Hickey
Ashwin Sekar
Yihau Chen
Brooks
Sean Young
behzad nouri
sakridge
Pankaj Garg
steviez
Andrew Fitzgerald
Justin Starry
Nick Frostbutter
GoodDaisy
Ryo Onodera