There are several cases for fetching entries from the Blockstore:
- Fetching entries for block replay
- Fetching entries for CompletedDataSetService
- Fetching entries to service RPC getBlock requests
All of these operations occur in a different calling thread. However,
the currently implementation utilizes a shared thread-pool within the
Blockstore function. There are several problems with this:
- The thread pool is shared between all of the listed cases, despite
block replay being the most critical. These other services shouldn't
be able to interfere with block replay
- The thread pool is overprovisioned for the average use; thread
utilization on both regular validators and RPC nodes shows that many
of the thread see very little activity. But, these thread existing
introduce "accounting" overhead
- rocksdb exposes an API to fetch multiple items at once, potentially
with some parallelization under the hood. Using parallelization in
our API and the underlying rocksdb is overkill and we're doing more
damage than good.
This change removes that threadpool completely, and instead fetches
all of the desired entries in a single call. This has been observed
to have a minor degradation on the time spent within the Blockstore
get_slot_entries_with_shred_info() function. Namely, some buffer
copying and deserialization that previously occurred in parallel now
occur serially.
However, the metric that tracks the amount of time spent replaying
blocks (inclusive of fetch) is unchanged. Thus, despite spending
marginally more time to fetch/copy/deserialize with only a single
thread, the gains from not thrashing everything else with the pool
keep us at parity.
09925a11eb