#### Problem
TieredWritableFile currently uses File instead of BufWriter.
This will introduce more syscall when doing file writes.
#### Summary of Changes
This PR makes TieredWritableFile uses BufWriter to allow the
write-call to be more optimized to reduce the number of syscalls.
#### Test Plan
Existing tiered-storage test.
Will run experiments to verify its performance improvement.
#### Dependency
https://github.com/anza-xyz/agave/pull/260
#### Problem
TieredStorageFile struct currently offers new_readonly() and new_writable()
to allow both read and write work-load to share the same struct. However,
as we need the writer to use BufWriter to improve performance as well as
enable Hasher on writes. There is a need to refactor TieredStorageFile to
split its usage for read-only and writable.
#### Summary of Changes
Refactor TieredStorageFile to TieredReadonlyFIle and TieredWritableFile.
#### Test Plan
Existing tiered-storage tests.
* Switch to blockstore.is_full() check because replay thread isn't active.
* Use make_chaining_slot_entries and add first_parent to the method.
Small style fixes.
* Switch to blockstore.is_full() check because replay thread isn't active.
* add `ParseError` in `zk-token-elgamal`
* implement `FromStr` for `ElGamalPubkey` and `ElGamalCiphertext`
* implement `FromStr` for `AeCiphertext`
* fix target
* cargo fmt
* use constants for byte length check
* make `FromStr` functions available on chain
* use macros for the `FromStr` implementations
* restrict `from_str` macro to `pub(crate)`
* decode directly into array
* cargo fmt
* Apply suggestions from code review
Co-authored-by: Jon C <me@jonc.dev>
* remove unnecessary imports
* remove the need for `ParseError` dependency
---------
Co-authored-by: Jon C <me@jonc.dev>
#### Problem
tiered_storage/writer.rs was added when we planned to support multiple
tiers in the tiered-storage (i.e., at least hot and cold). However, as we
changed our plan to handle cold accounts as state-compressed accounts,
we don't need a general purposed tiered-storage writer at this moment.
#### Summary of Changes
Remove tiered_storage/writer.rs as we currently don't have plans to develop cold storage.
#### Test Plan
Existing tiered-storage tests.
#### Problem
As we further optimize the HotStorageMeta in #146, there is a need
for a HotAccount struct that contains all the hot account information.
Meanwhile, we currently don't have plans to develop a cold account
format at this moment. As a result, this makes it desirable to repurpose
TieredReadableAccount to HotAccount.
#### Summary of Changes
Repurpose TieredReadableAccount to HotAccount.
#### Test Plan
Existing tiered-storage tests.
The default value was previously being determined down where the thread
pool is being created. Providing a default value at the CLI level is
consistent with other args, and gives an operator better visibility into
what the default will actually be
The threadpool used to replay multiple transactions in parallel is
currently global state via a lazy_static definition. Making this pool
owned by ReplayStage will enable subsequent work to make the pool
size configurable on the CLI.
This makes `ReplayStage` create and hold the threadpool which is passed
down to blockstore_processor::confirm_slot().
blockstore_processor::process_blockstore_from_root() now creates its'
own threadpool as well; however, this pool is only alive while for
the scope of that function and does not persist the lifetime of the
process.