This adds the `data_api::scanning::spanning_tree` module under
a new `unstable-spanning-tree` feature flag, making it available to
other implementations who want to be able to write their own storage
backends without having to reinvent the spanning tree logic.
This fixes the following bug:
Due to complexities related to non-linear scanning, checkpoints are only
added to the wallet's commitment tree in cases where there are notes
discovered within a scanned block. At present, the `shardtree` API only
makes it possible to add multiple checkpoints of the same tree state
when adding checkpoints at the chain tip, and this functionality is not
used by `zcash_client_backend` because we perform checkpoint insertion
in batches that may contain gaps in the case that multiple blocks
contain no Sapling notes. While it would be possible to fix this by
altering the `shardtree` API to permit explicit insertion of multiple
checkpoints of the same tree state at a given note position, this fix
takes a simpler approach.
Instead of ensuring that a checkpoint exists at every block and
computing the required checkpoint depth directly from the minimum number
of confirmations required when attempting a spend, we alter the
`WalletCommitmentTrees` API to allow internal information of the note
commitment tree to be used to determine this checkpoint depth, given the
minimum number of commitments as an argument. This allows us to select a
usable checkpoint from the sparse checkpoint set that resulted from the
sparse insertion of checkpoints described above.
The intent of this API is to provide a single API which returns in a
single call:
* per-account balances, including pending values
* wallet sync progress
Fixes#865Fixes#900
This also removes the zcash_client_sqlite-specific database
initialization procedures in favor of a standardized approach using the
methods available via the data access API.
The `zcash_client_backend::proto::service::compact_tx_streamer_client`
is now the only module controlled by that feature flag, exposing the
service types for use by parsers.
Prior to the scan-before-sync changes, the wallet was able to assume
that the maximum scanned block height at the time of the spend was
within a few blocks of the chain tip. However, under linear scanning
after the spend-before-sync changes this invariant no longer holds,
resulting in a situation where in linear sync conditions the wallet
could attempt to create transactions with already-past expiry heights.
This change separates the notion of "chain tip" from "max scanned
height", relying upon the `scan_queue` table to maintain the wallet's
view of the consensus chain height and using information from the
`blocks` table only in situations where the latest and/or earliest
scanned height is required.
As part of this change, the `WalletRead` interface is also modified to
disambiguate these concepts.
If `from_height = 0` and `limit > 1`, then the conditional being used to
initialise `continuity_check_metadata` was previously running for every
block, and would therefore panic while processing the block at height 1.
Previously this was not clearly specified, and the implementations in
`zcash_client_sqlite` behaved similarly to when `from_height = None`.
Closeszcash/librustzcash#892.
In general, it is preferable to use globally relevant identifiers where
possible. This PR removes the `WalletRead::TxRef` associated type in
favor of using `TxId` directly for the transaction identifier, and
restricts the use of the `NoteRef` type to those scenarios where the
result of one query is intended to be used directly as the input to
another query.
Closes#834
This implements a priority queue backed by the wallet database for scan
range ordering. The scan queue is updated on each call to `put_blocks`
or to `update_chain_tip`.
Instead of calling `put_block` for each block scanned,
`scan_cached_blocks` will now defer the block writes until the scan of a
batch is complete and will perform the block writes and note commitment
tree updates all within a single transaction.
This should ordinarily be fine in terms of memory consumption, because
the block data being saved is pruned to only that spend an output
information that is related to transactions in the wallet, which will
normally be sparse enough that the block range size that is appropriate
for a given platform to run within a batch scanner adequately bounds the
memory consumption of this pruned representation.
There are cases where we wish to return informaiton that is relevant to
a specific shielded protocol and `Transparent` is an invalid case. This
is a minor preparatory refactoring that makes this distinction
expressible.
In preparation for out-of-order range-based scanning, it is necessary
to ensure that the size of the Sapling note commitment tree is carried
along through the scan process and that stored blocks are always
persisted with the updated note commitment tree size.
Local chain validation will be performed internal to
`scan_cached_blocks`, and as handling of chain reorgs will need to
change to support out-of-order scanning, the `validate_chain` method
will be superfluous. It is removed in advance of other changes in order
to avoid updating it to reflect the forthcoming changes.
We move thes fields out into a separate BlockMetadata struct to ensure
that future additions to block metadata are structurally separated from
future additions to block data.
Memos may be absent for both sent and received notes in cases where only
compact block information has been used to populate the wallet database.
This fixes a potential crash in the case that we attempt to decode a
SQLite `NULL` as a byte array. It does, however, introduce a slight
semantic confusion that will need to be considered in the case of future
updates where a note may not have an associated memo; at present, the
only reason we might not have the memo is that we might not have
retrieved the full transaction information from the chain, but in the
future there might be other possible reasons for this absence.
Fixes#384
This removes the `CommitmentTree`, `IncrementalWitness`, and
`MerklePath` types in favor of equivalent versions available
from the `incrementalmerkletree` crate.
This is in preparation for extraction into the `incrementalmerkletree`
crate, which is not Sapling-specific and therefore cannot hard-code
the depths of these data structures.
This better reflects the semantics of wallet behavior. Also, this
adds a `zcash_client_backend::WalletRead::get_min_unspent_height`
method that replaces the deprecated & removed (and misleadingly
named) `get_rewind_height` method.
This removes the path-based dependencies on the `zcash_note_encryption`
crate in favor of using versioned dependencies locally. This better
reflects the future state in which `zcash_note_encryption` is factored
out of the workspace and maintained in a separate repository.
The `halo2_proofs/multicore` flag must be disabled when running wasm
builds; this ensures that we do not accidentally include it as a
transitive dependency when building with `--no-default-features`.
This change makes it possible for wallets using the
`zcash_client_backend::data_api::wallet` module to perform transaction
preparation, including input selection and fee calculation, as an
independent step prior to creating proofs and signatures. This can be
used to improve user experience by making it possible to report the
proposed effects of the transaction to the wallet user (including
privacy implications) prior to authorizing the transaction.
Previously, if a caller wanted to use a block source to perform
scanning from the first available block, they would have to guess
at the block height to start from. Changing this to an optional
argument makes this explicit.
This allows callers to validate smaller intervals of the given
`BlockSourceT` shortening processing times of the function call at the
expense of obtaining a partial result on a given section of interest of
the block source.
`params: &ParamsT` has been removed from the arguments since they were
only needed to fall back to `sapling_activation_height` when `None` as
passed as the `validate_from` argument. Passing `None` as validation
start point on a pre-populated `block_source` would result in an error
`ChainError::block_height_discontinuity(sapling_activation_height - 1, current_height)`
With this new API callers must specify a concrete `validate_from`
argument and assume that `validate_chain` will not take any default
fallbacks to chain `ParamsT`.
The addition of a `limit` to the chain validation function changes the
meaning of its successful output, being now a `BlockHeight, BlockHash)`
tuple indicating the block height and block hash up to which the chain
as been validated on its continuity of heights and hashes. Callers
providing a `limit` aregument are responsible of subsequent calls to
`validate_chain()` to complete validating the remaining blocks stored on
the `block_source`.
Closeszcash/librustzcash#705
The MSRVs of the component crates are left as-is, partly because our
dependencies don't require us to bump them, and partly because those
crates have no pending changes and are relatively stable. We also plan
to split the component crates out into a separate repository, where it
will be easier to have a separate MSRV.
Closeszcash/librustzcash#759.
The change selection algorithm has the most useful information for
determining whether or not a note is dust, so this adds a new error case
to `ChangeError` that allows the change selection to report the presence
of input notes without economic value back to its caller.
This adds a set of abstractions that allow wallets to provide
independent strategies for fee estimation and note selection, and
implementations of these strategies that perform these operations in the
same fashion as the existing `spend` and `shield_transparent_funds`
functions.
This required a somewhat hefty rework of the error handling in
zcash_client_backend. It fixes an issue with the error types whereby
callees needed to have a bit too much information about the error
types produced by their callers.
Reflect the updated note selection and error handling in zcash_client_sqlite.
This adds a fee calculation strategy abstraction that can be used to
dynamically compute fees so that the total fees required may be taken
taken into account during note selection, and also removes automatic
change creation from the transaction builder.
Change outputs must now be directly created by the caller by the caller.
This is a necessary prerequisite for permitting fees to adjust based
upon the contents of the transaction being constructed.
The initial implementation of the fee strategy simply uses the current
default fee.
Previously, `shield_transparent_funds` was only shielding funds
associated with the legacy default transparent address. This meant
that transparent funds sent to unified addresses could not reliably
be shielded, as a unified address will frequently be constructed
using a diversifier index greater than zero.
This modifies the `get_transparent_receivers` method to return address
metadata containing the account ID and diversifier index used to derive
each address along with the receiver.
In 47a0d0d2b7 `WalletWriteTransparent` was
merged into `WalletWrite`, including its associated type `UtxoRef`.
However, `shield_transparent_funds` placed a bound on this associated
type that was not moved to `WalletWrite`. This left the generic
parameter `U` unconstrained, which didn't cause a local failure because
the method has no local tests, but was immediately apparent when trying
to use the method in the mobile SDKs.
This modifies `decrypt_and_store_transaction` to check for inputs
to a transaction being decrypted that correspond to utxos known
to our wallet. For each such UTXO found, it is marked spent.
These traits introduce a problem, in that constraints on a method cannot
be conditionally required based upon the presence or absence of a
feature flag. Instead, we make the methods previously introduced by
the removed traits present in all cases on the `WalletRead` and
`WalletWrite` traits, but ensure that their implementations return
an error if the caller attempts to use them in a wallet that has not
been configured with support for transparent inputs functionality.
Due to how the wallets retrieved unspent transparent outputs from the
light wallet server, the account associated with a particular UTXO may
not be known by the light wallet. Instead of requiring the caller to
perform a separate lookup and match the address of the received UTXO
with a known account, it's simpler to perform this lookup internally at
the time of insertion or update.
In order to make this operation more efficient, the `addresses_table`
migration is modified to add a column to cache the transparent receiver
so that it may be used in the joins in the UTXO insert and update
operations.
The currently deprecated implementations of `insert_sent_utxo`,
`insert_sent_note`, `put_sent_utxo` and `put_sent_note` all store to the
same `sent_notes` table internally. Since there's no immediate plan to
change this arrangement, it's better to have a single pair of internal
`insert_sent_output` and `put_sent_output` methods instead.
As of zcash/librustzcash#633, `SaplingDomain::IncomingViewingKey` now
allocates memory internally, and this memory persists as long as the
`BatchRunner` is alive. Now that we have decoupled the measurement of
heap usage for batch tasks from their internals, we can add bounds to
all of the generic parameters of `Batch` to enable correctly measuring
their actual heap usage.
We also add `DynamicUsage` impls for a bunch of `zcash_primitives` types
that will be used with `BatchRunner` (or its equivalent implementation
in `zcashd`) by callers.
This enables the heap usage measurements to be conditionally enabled by
the `BatchRunner` user. Importantly, when heap usage measurements are
not enabled, the `DynamicUsage` bound on `Batch` is not required.
This refactor also fixes a bug in the prior implementation. We were
counting the heap usage of a task when it started to run, but the item
may have been in the `rayon` work-stealing queues for a non-negligible
period before then. We now count the heap usage immediately before
spawning the task into the `rayon` thread pool.
Previously we were sending an `Option<DecryptedNote>` from each `Batch`
back to its parent `BatchRunner`. However, this requires allocating
sufficient space in the channel to handle the case where every output
can be decrypted. In general this will not be the case, and we can
instead signal "nothing decrypted" by just dropping the channel sender.
This reduces the post-batch-scanning memory usage of `BatchRunner` from
being linear in the number of on-chain outputs, to being linear in the
number of outputs for the wallet.
Ported from zcash/zcash@f7f6c2070d.
This removes the dependency on `SaplingIvk::to_repr()`, and enables us
to alter the type of `D::IncomingViewingKey` to improve the performance
of batch scanning.
For the welding rig, we already annotate the viewing keys with
`AccountId`, so we use `(AccountId, Scope)` as the tag.
This updates the data access API to provide diversified address
functionality. In order to support this change, the addresses table
is updated to store diversifier index information in big-endian order
to allow sorting by diversifier index, and account initialization
is updated to store the diversifier index accordingly. The currently
unreleased `addresses_table` migration is updated to reflect this
change.
This fixes a bug in the logic ported from the Android SDK: it was
possible to remove a transaction in the middle of a chain, which would
cause a long-spent note to become unspent and cause the wallet balance
to be over-counted. We now restrict transaction removal to unmined
transactions, which is sufficient for the Android SDK use cases.
This is to replace the database mutations in the Android SDK. It is
placed behind an `unstable` feature flag until we are satisfied that it
is suitable as a general-purpose API (or replace it).
This updates `shield_transparent_funds` to look up the internal
(change) address for the account specified, and use that as the
destination for shielding transparent funds.
Fixed#614
This modifies wallet scanning to perform per-block batched
decryption. It also alters the structure of the `ScanningKey`
trait to correctly include internal (change) keys in the scan
process.
Nullifier computation only requires the nullifier deriving key,
not the entire Sapling viewing key. This separation of concerns
will be needed for batch decryption when wallet-internal keys
will need to be considered.
Kris Nuttycombe
Kris Nuttycombe
Jack Grigg
Andrew Arnott
Hazel OHearn
Nate Wilcox
Daira Emma Hopwood
Francisco Gindre
Greg Pfeil
str4d
Sean Bowe
Aditya Kulkarni