The `add_checkpoint` method is intended to be idempotent. In the case
that we add a checkpoint at an already-checkpointed block height, we
should only raise an error in the case that the note commitment tree
position or the set of notes spent in the checkpointed block has
changed.
The `shardtree` migration is applied to a database state that was
created via linear scanning, so we have complete wallet information for
those blocks.
We only need to load frontiers into the ShardTree that are close enough
to the wallet's known chain tip to fill `PRUNING_DEPTH` checkpoints, so
that ShardTree's witness generation will be able to correctly handle
anchor depths. Loading frontiers further back than this doesn't add any
useful nodes to the ShardTree (as we don't support rollbacks beyond
`PRUNING_DEPTH`, and we won't be finding notes in earlier blocks), and
hurts performance (as frontier importing has a significant Merkle tree
hashing cost).
Closeszcash/librustzcash#877.
Previously `extended_range` only covered the extent of the leaves of
all subtrees in which notes were found during a scan. When the scanned
range was large, this was not guaranteed to be contained within the
subtree leaves, causing an assertion failure when an invalid `ScanRange`
was constructed.
The Merkle hashes used for the note commitment trees are domain
separated by level, so when pretending that the subtree roots are leaves
of the cap tree, we need to adjust for their level not being zero.
Closeszcash/librustzcash#874.
Co-authored-by: Sean Bowe <ewillbefull@gmail.com>
This implements a priority queue backed by the wallet database for scan
range ordering. The scan queue is updated on each call to `put_blocks`
or to `update_chain_tip`.
Instead of calling `put_block` for each block scanned,
`scan_cached_blocks` will now defer the block writes until the scan of a
batch is complete and will perform the block writes and note commitment
tree updates all within a single transaction.
This should ordinarily be fine in terms of memory consumption, because
the block data being saved is pruned to only that spend an output
information that is related to transactions in the wallet, which will
normally be sparse enough that the block range size that is appropriate
for a given platform to run within a batch scanner adequately bounds the
memory consumption of this pruned representation.
There are cases where we wish to return informaiton that is relevant to
a specific shielded protocol and `Transparent` is an invalid case. This
is a minor preparatory refactoring that makes this distinction
expressible.
In preparation for out-of-order range-based scanning, it is necessary
to ensure that the size of the Sapling note commitment tree is carried
along through the scan process and that stored blocks are always
persisted with the updated note commitment tree size.
Local chain validation will be performed internal to
`scan_cached_blocks`, and as handling of chain reorgs will need to
change to support out-of-order scanning, the `validate_chain` method
will be superfluous. It is removed in advance of other changes in order
to avoid updating it to reflect the forthcoming changes.
We move thes fields out into a separate BlockMetadata struct to ensure
that future additions to block metadata are structurally separated from
future additions to block data.
Kris Nuttycombe
Jack Grigg
str4d
Sean Bowe
Kris Nuttycombe