zec
/
librustzcash
mirror of https://github.com/zcash/librustzcash.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge pull request #907 from nuttycom/sbs/wallet_birthday 

zcash_client_backend: Add account birthday management to the Data Access API
This commit is contained in:
str4d 2023-09-01 18:10:30 +01:00 committed by GitHub
parent cf4c769609 5b3f544102
commit 229f6e82ad
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
17 changed files with 851 additions and 665 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
zcash_client_backend/CHANGELOG.md
View File

@ -6,6 +6,17 @@ and this library adheres to Rust's notion of
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Notable Changes
- `zcash_client_backend` now supports out-of-order scanning of blockchain history.
- This release of `zcash_client_backend` defines the concept of an account
birthday. The account birthday is defined as the minimum height among blocks
to be scanned when recovering an account.
- Account creation now requires the caller to provide account birthday information,
including the state of the note commitment tree at the end of the block prior to
the birthday height.
### Added
- `impl Eq for zcash_client_backend::address::RecipientAddress`
- `impl Eq for zcash_client_backend::zip321::{Payment, TransactionRequest}`
@ -18,7 +29,8 @@ and this library adheres to Rust's notion of
- `ScannedBlock`
- `ShieldedProtocol`
- `WalletCommitmentTrees`
- `WalletRead::{chain_height, block_metadata, block_fully_scanned, suggest_scan_ranges}`
- `WalletRead::{chain_height, block_metadata, block_fully_scanned, suggest_scan_ranges,
get_wallet_birthday, get_account_birthday}`
- `WalletWrite::{put_blocks, update_chain_tip}`
- `chain::CommitmentTreeRoot`
- `scanning` A new module containing types required for `suggest_scan_ranges`
@ -39,6 +51,7 @@ and this library adheres to Rust's notion of
- `zcash_client_backend::data_api`:
- `WalletRead::TxRef` has been removed in favor of consistently using `TxId` instead.
- `WalletRead::get_transaction` now takes a `TxId` as its argument.
- `WalletRead::create_account` now takes an additional `birthday` argument.
- `WalletWrite::{store_decrypted_tx, store_sent_tx}` now return `Result<(), Self::Error>`
as the `WalletRead::TxRef` associated type has been removed. Use
`WalletRead::get_transaction` with the transaction's `TxId` instead.

2
zcash_client_backend/Cargo.toml
View File

@ -45,6 +45,7 @@ memuse = "0.2"
tracing = "0.1"
# - Protobuf interfaces and gRPC bindings
hex = "0.4"
prost = "0.11"
tonic = { version = "0.9", optional = true }
@ -80,7 +81,6 @@ which = "4"
[dev-dependencies]
assert_matches = "1.5"
gumdrop = "0.8"
hex = "0.4"
jubjub = "0.10"
proptest = "1.0.0"
rand_core = "0.6"

143
zcash_client_backend/src/data_api.rs
View File

@ -1,8 +1,9 @@
//! Interfaces for wallet data persistence & low-level wallet utilities.
use std::collections::HashMap;
use std::fmt::Debug;
use std::io;
use std::num::NonZeroU32;
use std::{collections::HashMap, num::TryFromIntError};
use incrementalmerkletree::{frontier::Frontier, Retention};
use secrecy::SecretVec;
@ -24,6 +25,7 @@ use crate::{
address::{AddressMetadata, UnifiedAddress},
decrypt::DecryptedOutput,
keys::{UnifiedFullViewingKey, UnifiedSpendingKey},
proto::service::TreeState,
wallet::{ReceivedSaplingNote, WalletTransparentOutput, WalletTx},
};
@ -117,6 +119,16 @@ pub trait WalletRead {
/// transaction is not in the main chain.
fn get_tx_height(&self, txid: TxId) -> Result<Option<BlockHeight>, Self::Error>;
/// Returns the birthday height for the wallet.
///
/// This returns the earliest birthday height among accounts maintained by this wallet,
/// or `Ok(None)` if the wallet has no initialized accounts.
fn get_wallet_birthday(&self) -> Result<Option<BlockHeight>, Self::Error>;
/// Returns the birthday height for the given account, or an error if the account is not known
/// to the wallet.
fn get_account_birthday(&self, account: AccountId) -> Result<BlockHeight, Self::Error>;
/// Returns the most recently generated unified address for the specified account, if the
/// account identifier specified refers to a valid account for this wallet.
///
@ -472,25 +484,77 @@ impl SentTransactionOutput {
pub struct AccountBirthday {
height: BlockHeight,
sapling_frontier: Frontier<Node, NOTE_COMMITMENT_TREE_DEPTH>,
recover_until: Option<BlockHeight>,
}
/// Errors that can occur in the construction of an [`AccountBirthday`] from a [`TreeState`].
pub enum BirthdayError {
HeightInvalid(TryFromIntError),
Decode(io::Error),
}
impl From<TryFromIntError> for BirthdayError {
fn from(value: TryFromIntError) -> Self {
Self::HeightInvalid(value)
}
}
impl From<io::Error> for BirthdayError {
fn from(value: io::Error) -> Self {
Self::Decode(value)
}
}
impl AccountBirthday {
/// Constructs a new [`AccountBirthday`] from its constituent parts.
///
/// * `height`: The birthday height of the account. This is defined as the height of the last
/// block that is known to contain no transactions sent to addresses belonging to the account.
/// * `height`: The birthday height of the account. This is defined as the height of the first
/// block to be scanned in wallet recovery.
/// * `sapling_frontier`: The Sapling note commitment tree frontier as of the end of the block
/// at `height`.
/// prior to the birthday height.
/// * `recover_until`: An optional height at which the wallet should exit "recovery mode". In
/// order to avoid confusing shifts in wallet balance and spendability that may temporarily be
/// visible to a user during the process of recovering from seed, wallets may optionally set a
/// "recover until" height. The wallet is considered to be in "recovery mode" until there
/// exist no unscanned ranges between the wallet's birthday height and the provided
/// `recover_until` height, exclusive.
///
/// This API is intended primarily to be used in testing contexts; under normal circumstances,
/// [`AccountBirthday::from_treestate`] should be used instead.
#[cfg(feature = "test-dependencies")]
pub fn from_parts(
height: BlockHeight,
sapling_frontier: Frontier<Node, NOTE_COMMITMENT_TREE_DEPTH>,
recover_until: Option<BlockHeight>,
) -> Self {
Self {
height,
sapling_frontier,
recover_until,
}
}
/// Constructs a new [`AccountBirthday`] from a [`TreeState`] returned from `lightwalletd`.
///
/// * `treestate`: The tree state corresponding to the last block prior to the wallet's
/// birthday height.
/// * `recover_until`: An optional height at which the wallet should exit "recovery mode". In
/// order to avoid confusing shifts in wallet balance and spendability that may temporarily be
/// visible to a user during the process of recovering from seed, wallets may optionally set a
/// "recover until" height. The wallet is considered to be in "recovery mode" until there
/// exist no unscanned ranges between the wallet's birthday height and the provided
/// `recover_until` height, exclusive.
pub fn from_treestate(
treestate: TreeState,
recover_until: Option<BlockHeight>,
) -> Result<Self, BirthdayError> {
Ok(Self {
height: BlockHeight::try_from(treestate.height + 1)?,
sapling_frontier: treestate.sapling_tree()?.to_frontier(),
recover_until,
})
}
/// Returns the Sapling note commitment tree frontier as of the end of the block at
/// [`Self::height`].
pub fn sapling_frontier(&self) -> &Frontier<Node, NOTE_COMMITMENT_TREE_DEPTH> {
@ -501,6 +565,30 @@ impl AccountBirthday {
pub fn height(&self) -> BlockHeight {
self.height
}
/// Returns the height at which the wallet should exit "recovery mode".
pub fn recover_until(&self) -> Option<BlockHeight> {
self.recover_until
}
#[cfg(feature = "test-dependencies")]
/// Constructs a new [`AccountBirthday`] at Sapling activation, with no
/// "recover until" height.
///
/// # Panics
///
/// Panics if the Sapling activation height is not set.
pub fn from_sapling_activation<P: zcash_primitives::consensus::Parameters>(
params: &P,
) -> AccountBirthday {
use zcash_primitives::consensus::NetworkUpgrade;
AccountBirthday::from_parts(
params.activation_height(NetworkUpgrade::Sapling).unwrap(),
Frontier::empty(),
None,
)
}
}
/// This trait encapsulates the write capabilities required to update stored
@ -509,24 +597,36 @@ pub trait WalletWrite: WalletRead {
/// The type of identifiers used to look up transparent UTXOs.
type UtxoRef;
/// Tells the wallet to track the next available account-level spend authority, given
/// the current set of [ZIP 316] account identifiers known to the wallet database.
/// Tells the wallet to track the next available account-level spend authority, given the
/// current set of [ZIP 316] account identifiers known to the wallet database.
///
/// Returns the account identifier for the newly-created wallet database entry, along
/// with the associated [`UnifiedSpendingKey`].
/// Returns the account identifier for the newly-created wallet database entry, along with the
/// associated [`UnifiedSpendingKey`].
///
/// If `seed` was imported from a backup and this method is being used to restore a
/// previous wallet state, you should use this method to add all of the desired
/// accounts before scanning the chain from the seed's birthday height.
/// If `birthday.height()` is below the current chain tip, this operation will
/// trigger a re-scan of the blocks at and above the provided height. The birthday height is
/// defined as the minimum block height that will be scanned for funds belonging to the wallet.
///
/// By convention, wallets should only allow a new account to be generated after funds
/// have been received by the currently-available account (in order to enable
/// automated account recovery).
/// For new wallets, callers should construct the [`AccountBirthday`] using
/// [`AccountBirthday::from_treestate`] for the block at height `chain_tip_height - 100`.
/// Setting the birthday height to a tree state below the pruning depth ensures that reorgs
/// cannot cause funds intended for the wallet to be missed; otherwise, if the chain tip height
/// were used for the wallet birthday, a transaction targeted at a height greater than the
/// chain tip could be mined at a height below that tip as part of a reorg.
///
/// If `seed` was imported from a backup and this method is being used to restore a previous
/// wallet state, you should use this method to add all of the desired accounts before scanning
/// the chain from the seed's birthday height.
///
/// By convention, wallets should only allow a new account to be generated after confirmed
/// funds have been received by the currently-available account (in order to enable automated
/// account recovery).
///
/// [ZIP 316]: https://zips.z.cash/zip-0316
fn create_account(
&mut self,
seed: &SecretVec<u8>,
birthday: AccountBirthday,
) -> Result<(AccountId, UnifiedSpendingKey), Self::Error>;
/// Generates and persists the next available diversified address, given the current
@ -645,9 +745,9 @@ pub mod testing {
};
use super::{
chain::CommitmentTreeRoot, scanning::ScanRange, BlockMetadata, DecryptedTransaction,
NoteId, NullifierQuery, ScannedBlock, SentTransaction, WalletCommitmentTrees, WalletRead,
WalletWrite, SAPLING_SHARD_HEIGHT,
chain::CommitmentTreeRoot, scanning::ScanRange, AccountBirthday, BlockMetadata,
DecryptedTransaction, NoteId, NullifierQuery, ScannedBlock, SentTransaction,
WalletCommitmentTrees, WalletRead, WalletWrite, SAPLING_SHARD_HEIGHT,
};
pub struct MockWalletDb {
@ -717,6 +817,14 @@ pub mod testing {
Ok(None)
}
fn get_wallet_birthday(&self) -> Result<Option<BlockHeight>, Self::Error> {
Ok(None)
}
fn get_account_birthday(&self, _account: AccountId) -> Result<BlockHeight, Self::Error> {
Err(())
}
fn get_current_address(
&self,
_account: AccountId,
@ -818,6 +926,7 @@ pub mod testing {
fn create_account(
&mut self,
seed: &SecretVec<u8>,
_birthday: AccountBirthday,
) -> Result<(AccountId, UnifiedSpendingKey), Self::Error> {
let account = AccountId::from(0);
UnifiedSpendingKey::from_seed(&self.network, seed.expose_secret(), account)

19
zcash_client_backend/src/proto.rs
View File

@ -1,9 +1,13 @@
//! Generated code for handling light client protobuf structs.
use std::io;
use incrementalmerkletree::frontier::CommitmentTree;
use zcash_primitives::{
block::{BlockHash, BlockHeader},
consensus::BlockHeight,
sapling::{note::ExtractedNoteCommitment, Nullifier},
merkle_tree::read_commitment_tree,
sapling::{note::ExtractedNoteCommitment, Node, Nullifier, NOTE_COMMITMENT_TREE_DEPTH},
transaction::{components::sapling, TxId},
};
@ -141,3 +145,16 @@ impl compact_formats::CompactSaplingSpend {
Nullifier::from_slice(&self.nf).map_err(|_| ())
}
}
impl service::TreeState {
/// Deserializes and returns the Sapling note commitment tree field of the tree state.
pub fn sapling_tree(&self) -> io::Result<CommitmentTree<Node, NOTE_COMMITMENT_TREE_DEPTH>> {
let sapling_tree_bytes = hex::decode(&self.sapling_tree).map_err(|e| {
io::Error::new(
io::ErrorKind::InvalidData,
format!("Hex decoding of Sapling tree bytes failed: {:?}", e),
)
})?;
read_commitment_tree::<Node, _, NOTE_COMMITMENT_TREE_DEPTH>(&sapling_tree_bytes[..])
}
}

10
zcash_client_sqlite/CHANGELOG.md
View File

@ -26,8 +26,10 @@ and this library adheres to Rust's notion of
wallet did not contain enough observed blocks to satisfy the `min_confirmations`
value specified; this situation is now treated as an error.
- `zcash_client_sqlite::error::SqliteClientError` has new error variants:
- `SqliteClientError::AccountUnknown`
- `SqliteClientError::BlockConflict`
- `SqliteClientError::CacheMiss`
- `SqliteClientError::ChainHeightUnknown`
- `zcash_client_backend::FsBlockDbError` has a new error variant:
- `FsBlockDbError::CacheMiss`
- `zcash_client_sqlite::FsBlockDb::write_block_metadata` now overwrites any
@ -36,8 +38,14 @@ and this library adheres to Rust's notion of
### Removed
- The empty `wallet::transact` module has been removed.
- `zcash_client_sqlite::NoteId` has been replaced with `zcash_client_sqlite::ReceivedNoteId`
as the `SentNoteId` variant of is now unused following changes to
as the `SentNoteId` variant is now unused following changes to
`zcash_client_backend::data_api::WalletRead`.
- `zcash_client_sqlite::wallet::init::{init_blocks_table, init_accounts_table}`
have been removed. `zcash_client_backend::data_api::WalletWrite::create_account`
should be used instead; the initialization of the note commitment tree
previously performed by `init_blocks_table` is now handled by passing an
`AccountBirthday` containing the note commitment tree frontier as of the
end of the birthday height block to `create_account` instead.
### Fixed
- Fixed an off-by-one error in the `BlockSource` implementation for the SQLite-backed

32
zcash_client_sqlite/src/chain.rs
View File

@ -324,8 +324,6 @@ where
mod tests {
use std::num::NonZeroU32;
use secrecy::Secret;
use zcash_primitives::{
block::BlockHash,
transaction::{components::Amount, fees::zip317::FeeRule},
@ -335,8 +333,8 @@ mod tests {
use zcash_client_backend::{
address::RecipientAddress,
data_api::{
chain::error::Error, wallet::input_selection::GreedyInputSelector, WalletRead,
WalletWrite,
chain::error::Error, wallet::input_selection::GreedyInputSelector, AccountBirthday,
WalletRead,
},
fees::{zip317::SingleOutputChangeStrategy, DustOutputPolicy},
scanning::ScanError,
@ -345,7 +343,7 @@ mod tests {
};
use crate::{
testing::{birthday_at_sapling_activation, AddressType, TestBuilder},
testing::{AddressType, TestBuilder},
wallet::{get_balance, truncate_to_height},
AccountId,
};
@ -354,7 +352,7 @@ mod tests {
fn valid_chain_states() {
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(birthday_at_sapling_activation)
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
let dfvk = st.test_account_sapling().unwrap();
@ -387,7 +385,7 @@ mod tests {
fn invalid_chain_cache_disconnected() {
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(birthday_at_sapling_activation)
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
let dfvk = st.test_account_sapling().unwrap();
@ -438,7 +436,7 @@ mod tests {
fn data_db_truncation() {
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(birthday_at_sapling_activation)
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
let dfvk = st.test_account_sapling().unwrap();
@ -498,12 +496,13 @@ mod tests {
#[test]
fn scan_cached_blocks_allows_blocks_out_of_order() {
let mut st = TestBuilder::new().with_block_cache().build();
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
// Add an account to the wallet
let seed = Secret::new([0u8; 32].to_vec());
let (_, usk) = st.wallet_mut().create_account(&seed).unwrap();
let dfvk = usk.sapling().to_diversifiable_full_viewing_key();
let (_, usk, _) = st.test_account().unwrap();
let dfvk = st.test_account_sapling().unwrap();
// Create a block with height SAPLING_ACTIVATION_HEIGHT
let value = Amount::from_u64(50000).unwrap();
@ -558,7 +557,7 @@ mod tests {
fn scan_cached_blocks_finds_received_notes() {
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(birthday_at_sapling_activation)
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
let dfvk = st.test_account_sapling().unwrap();
@ -600,9 +599,8 @@ mod tests {
fn scan_cached_blocks_finds_change_notes() {
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(birthday_at_sapling_activation)
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
let dfvk = st.test_account_sapling().unwrap();
// Account balance should be zero
@ -645,7 +643,7 @@ mod tests {
fn scan_cached_blocks_detects_spends_out_of_order() {
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(birthday_at_sapling_activation)
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
let dfvk = st.test_account_sapling().unwrap();

14
zcash_client_sqlite/src/error.rs
View File

@ -66,6 +66,9 @@ pub enum SqliteClientError {
/// The space of allocatable diversifier indices has been exhausted for the given account.
DiversifierIndexOutOfRange,
/// The account for which information was requested does not belong to the wallet.
AccountUnknown(AccountId),
/// An error occurred deriving a spending key from a seed and an account
/// identifier.
KeyDerivationError(AccountId),
@ -86,7 +89,15 @@ pub enum SqliteClientError {
/// commitment trees.
CommitmentTree(ShardTreeError<commitment_tree::Error>),
/// The block at the specified height was not available from the block cache.
CacheMiss(BlockHeight),
/// The height of the chain was not available; a call to [`WalletWrite::update_chain_tip`] is
/// required before the requested operation can succeed.
///
/// [`WalletWrite::update_chain_tip`]:
/// zcash_client_backend::data_api::WalletWrite::update_chain_tip
ChainHeightUnknown,
}
impl error::Error for SqliteClientError {
@ -124,6 +135,8 @@ impl fmt::Display for SqliteClientError {
SqliteClientError::BlockConflict(h) => write!(f, "A block hash conflict occurred at height {}; rewind required.", u32::from(*h)),
SqliteClientError::NonSequentialBlocks => write!(f, "`put_blocks` requires that the provided block range be sequential"),
SqliteClientError::DiversifierIndexOutOfRange => write!(f, "The space of available diversifier indices is exhausted"),
SqliteClientError::AccountUnknown(id) => write!(f, "Account {} does not belong to this wallet.", u32::from(*id)),
SqliteClientError::KeyDerivationError(acct_id) => write!(f, "Key derivation failed for account {:?}", acct_id),
SqliteClientError::AccountIdDiscontinuity => write!(f, "Wallet account identifiers must be sequential."),
SqliteClientError::AccountIdOutOfRange => write!(f, "Wallet account identifiers must be less than 0x7FFFFFFF."),
@ -131,6 +144,7 @@ impl fmt::Display for SqliteClientError {
SqliteClientError::AddressNotRecognized(_) => write!(f, "The address associated with a received txo is not identifiable as belonging to the wallet."),
SqliteClientError::CommitmentTree(err) => write!(f, "An error occurred accessing or updating note commitment tree data: {}.", err),
SqliteClientError::CacheMiss(height) => write!(f, "Requested height {} does not exist in the block cache.", height),
SqliteClientError::ChainHeightUnknown => write!(f, "Chain height unknown; please call `update_chain_tip`")
}
}
}

42
zcash_client_sqlite/src/lib.rs
View File

@ -64,9 +64,9 @@ use zcash_client_backend::{
self,
chain::{BlockSource, CommitmentTreeRoot},
scanning::{ScanPriority, ScanRange},
BlockMetadata, DecryptedTransaction, NoteId, NullifierQuery, PoolType, Recipient,
ScannedBlock, SentTransaction, ShieldedProtocol, WalletCommitmentTrees, WalletRead,
WalletWrite, SAPLING_SHARD_HEIGHT,
AccountBirthday, BlockMetadata, DecryptedTransaction, NoteId, NullifierQuery, PoolType,
Recipient, ScannedBlock, SentTransaction, ShieldedProtocol, WalletCommitmentTrees,
WalletRead, WalletWrite, SAPLING_SHARD_HEIGHT,
},
keys::{UnifiedFullViewingKey, UnifiedSpendingKey},
proto::compact_formats::CompactBlock,
@ -206,6 +206,14 @@ impl<C: Borrow<rusqlite::Connection>, P: consensus::Parameters> WalletRead for W
wallet::get_tx_height(self.conn.borrow(), txid).map_err(SqliteClientError::from)
}
fn get_wallet_birthday(&self) -> Result<Option<BlockHeight>, Self::Error> {
wallet::wallet_birthday(self.conn.borrow()).map_err(SqliteClientError::from)
}
fn get_account_birthday(&self, account: AccountId) -> Result<BlockHeight, Self::Error> {
wallet::account_birthday(self.conn.borrow(), account).map_err(SqliteClientError::from)
}
fn get_current_address(
&self,
account: AccountId,
@ -356,6 +364,7 @@ impl<P: consensus::Parameters> WalletWrite for WalletDb<rusqlite::Connection, P>
fn create_account(
&mut self,
seed: &SecretVec<u8>,
birthday: AccountBirthday,
) -> Result<(AccountId, UnifiedSpendingKey), Self::Error> {
self.transactionally(|wdb| {
let account = wallet::get_max_account_id(wdb.conn.0)?
@ -370,7 +379,7 @@ impl<P: consensus::Parameters> WalletWrite for WalletDb<rusqlite::Connection, P>
.map_err(|_| SqliteClientError::KeyDerivationError(account))?;
let ufvk = usk.to_unified_full_viewing_key();
wallet::add_account(wdb.conn.0, &wdb.params, account, &ufvk)?;
wallet::add_account(wdb.conn.0, &wdb.params, account, &ufvk, birthday)?;
Ok((account, usk))
})
@ -1085,26 +1094,21 @@ extern crate assert_matches;
#[cfg(test)]
mod tests {
use zcash_client_backend::data_api::{WalletRead, WalletWrite};
use zcash_client_backend::data_api::{AccountBirthday, WalletRead, WalletWrite};
use crate::{
testing::{birthday_at_sapling_activation, TestBuilder},
AccountId,
use crate::{testing::TestBuilder, AccountId};
#[cfg(feature = "unstable")]
use {
crate::testing::AddressType,
zcash_client_backend::keys::sapling,
zcash_primitives::{consensus::Parameters, transaction::components::Amount},
};
#[cfg(feature = "unstable")]
use zcash_primitives::{consensus::Parameters, transaction::components::Amount};
#[cfg(feature = "unstable")]
use zcash_client_backend::keys::sapling;
#[cfg(feature = "unstable")]
use crate::testing::AddressType;
#[test]
pub(crate) fn get_next_available_address() {
let mut st = TestBuilder::new()
.with_test_account(birthday_at_sapling_activation)
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
let account = AccountId::from(0);
@ -1125,7 +1129,7 @@ mod tests {
// Add an account to the wallet.
let st = TestBuilder::new()
.with_block_cache()
.with_test_account(birthday_at_sapling_activation)
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
let (_, usk, _) = st.test_account().unwrap();

13
zcash_client_sqlite/src/testing.rs
View File

@ -130,7 +130,7 @@ impl<Cache> TestBuilder<Cache> {
let test_account = if let Some(birthday) = self.test_account_birthday {
let seed = Secret::new(vec![0u8; 32]);
let (account, usk) = db_data.create_account(&seed).unwrap();
let (account, usk) = db_data.create_account(&seed, birthday.clone()).unwrap();
Some((account, usk, birthday))
} else {
None
@ -532,17 +532,6 @@ impl<Cache> TestState<Cache> {
}
}
#[cfg(test)]
pub(crate) fn birthday_at_sapling_activation<P: consensus::Parameters>(
params: &P,
) -> AccountBirthday {
use incrementalmerkletree::frontier::Frontier;
AccountBirthday::from_parts(
params.activation_height(NetworkUpgrade::Sapling).unwrap(),
Frontier::empty(),
)
}
#[allow(dead_code)]
pub(crate) enum AddressType {
DefaultExternal,

184
zcash_client_sqlite/src/wallet.rs
View File

@ -64,15 +64,20 @@
//! wallet.
//! - `memo` the shielded memo associated with the output, if any.
use incrementalmerkletree::Retention;
use rusqlite::{self, named_params, OptionalExtension, ToSql};
use shardtree::ShardTree;
use std::cmp;
use std::collections::HashMap;
use std::convert::TryFrom;
use std::io::{self, Cursor};
use std::num::NonZeroU32;
use zcash_client_backend::data_api::scanning::{ScanPriority, ScanRange};
use tracing::debug;
use zcash_client_backend::data_api::{NoteId, ShieldedProtocol};
use zcash_client_backend::data_api::{
scanning::{ScanPriority, ScanRange},
AccountBirthday, NoteId, ShieldedProtocol, SAPLING_SHARD_HEIGHT,
};
use zcash_primitives::transaction::TransactionData;
use zcash_primitives::{
@ -95,10 +100,11 @@ use zcash_client_backend::{
wallet::WalletTx,
};
use crate::VERIFY_LOOKAHEAD;
use crate::wallet::commitment_tree::SqliteShardStore;
use crate::{
error::SqliteClientError, SqlTransaction, WalletCommitmentTrees, WalletDb, PRUNING_DEPTH,
};
use crate::{SAPLING_TABLES_PREFIX, VERIFY_LOOKAHEAD};
use self::scanning::replace_queue_entries;
@ -157,29 +163,90 @@ pub(crate) fn add_account<P: consensus::Parameters>(
params: &P,
account: AccountId,
key: &UnifiedFullViewingKey,
birthday: AccountBirthday,
) -> Result<(), SqliteClientError> {
add_account_internal(conn, params, "accounts", account, key)
}
// Set the wallet birthday, falling back to the chain tip if not specified
let chain_tip = scan_queue_extrema(conn)?.map(|(_, max)| max);
pub(crate) fn add_account_internal<P: consensus::Parameters, E: From<rusqlite::Error>>(
conn: &rusqlite::Transaction,
network: &P,
accounts_table: &'static str,
account: AccountId,
key: &UnifiedFullViewingKey,
) -> Result<(), E> {
let ufvk_str: String = key.encode(network);
conn.execute(
&format!(
"INSERT INTO {} (account, ufvk) VALUES (:account, :ufvk)",
accounts_table
),
named_params![":account": &<u32>::from(account), ":ufvk": &ufvk_str],
"INSERT INTO accounts (account, ufvk, birthday_height, recover_until_height)
VALUES (:account, :ufvk, :birthday_height, :recover_until_height)",
named_params![
":account": u32::from(account),
":ufvk": &key.encode(params),
":birthday_height": u32::from(birthday.height()),
":recover_until_height": birthday.recover_until().map(u32::from)
],
)?;
// If a birthday frontier is available, insert it into the note commitment tree. If the
// birthday frontier is the empty frontier, we don't need to do anything.
if let Some(frontier) = birthday.sapling_frontier().value() {
debug!("Inserting frontier into ShardTree: {:?}", frontier);
let shard_store = SqliteShardStore::<
_,
zcash_primitives::sapling::Node,
SAPLING_SHARD_HEIGHT,
>::from_connection(conn, SAPLING_TABLES_PREFIX)?;
let mut shard_tree: ShardTree<
_,
{ zcash_primitives::sapling::NOTE_COMMITMENT_TREE_DEPTH },
SAPLING_SHARD_HEIGHT,
> = ShardTree::new(shard_store, PRUNING_DEPTH.try_into().unwrap());
shard_tree.insert_frontier_nodes(
frontier.clone(),
Retention::Checkpoint {
// This subtraction is safe, because all leaves in the tree appear in blocks, and
// the invariant that birthday.height() always corresponds to the block for which
// `frontier` is the tree state at the start of the block. Together, this means
// there exists a prior block for which frontier is the tree state at the end of
// the block.
id: birthday.height() - 1,
is_marked: false,
},
)?;
}
let sapling_activation_height = params
.activation_height(NetworkUpgrade::Sapling)
.expect("Sapling activation height must be available.");
// Add the ignored range up to the birthday height.
if sapling_activation_height < birthday.height() {
let ignored_range = sapling_activation_height..birthday.height();
replace_queue_entries::<SqliteClientError>(
conn,
&ignored_range,
Some(ScanRange::from_parts(
ignored_range.clone(),
ScanPriority::Ignored,
))
.into_iter(),
false,
)?;
};
// Rewrite the scan ranges starting from the birthday height so that we'll ensure we
// re-scan to find any notes that might belong to the newly added account.
if let Some(t) = chain_tip {
let rescan_range = birthday.height()..(t + 1);
replace_queue_entries::<SqliteClientError>(
conn,
&rescan_range,
Some(ScanRange::from_parts(
rescan_range.clone(),
ScanPriority::Historic,
))
.into_iter(),
true,
)?;
}
// Always derive the default Unified Address for the account.
let (address, d_idx) = key.default_address();
insert_address(conn, network, account, d_idx, &address)?;
insert_address(conn, params, account, d_idx, &address)?;
Ok(())
}
@ -235,17 +302,17 @@ pub(crate) fn insert_address<P: consensus::Parameters>(
) -> Result<(), rusqlite::Error> {
let mut stmt = conn.prepare_cached(
"INSERT INTO addresses (
account,
diversifier_index_be,
address,
cached_transparent_receiver_address
)
VALUES (
:account,
:diversifier_index_be,
:address,
:cached_transparent_receiver_address
)",
account,
diversifier_index_be,
address,
cached_transparent_receiver_address
)
VALUES (
:account,
:diversifier_index_be,
:address,
:cached_transparent_receiver_address
)",
)?;
// the diversifier index is stored in big-endian order to allow sorting
@ -612,6 +679,40 @@ pub(crate) fn get_sent_memo(
.transpose()
}
/// Returns the minimum birthday height for accounts in the wallet.
//
// TODO ORCHARD: we should consider whether we want to permit protocol-restricted accounts; if so,
// we would then want this method to take a protocol identifier to be able to learn the wallet's
// "Orchard birthday" which might be different from the overall wallet birthday.
pub(crate) fn wallet_birthday(
conn: &rusqlite::Connection,
) -> Result<Option<BlockHeight>, rusqlite::Error> {
conn.query_row(
"SELECT MIN(birthday_height) AS wallet_birthday FROM accounts",
[],
|row| {
row.get::<_, Option<u32>>(0)
.map(|opt| opt.map(BlockHeight::from))
},
)
}
pub(crate) fn account_birthday(
conn: &rusqlite::Connection,
account: AccountId,
) -> Result<BlockHeight, SqliteClientError> {
conn.query_row(
"SELECT birthday_height
FROM accounts
WHERE account = :account_id",
named_params![":account_id": u32::from(account)],
|row| row.get::<_, u32>(0).map(BlockHeight::from),
)
.optional()
.map_err(SqliteClientError::from)
.and_then(|opt| opt.ok_or(SqliteClientError::AccountUnknown(account)))
}
/// Returns the minimum and maximum heights for blocks stored in the wallet database.
pub(crate) fn block_height_extrema(
conn: &rusqlite::Connection,
@ -949,7 +1050,12 @@ pub(crate) fn truncate_to_height<P: consensus::Parameters>(
// Prioritize the range starting at the height we just rewound to for verification
let query_range = block_height..(block_height + VERIFY_LOOKAHEAD);
let scan_range = ScanRange::from_parts(query_range.clone(), ScanPriority::Verify);
replace_queue_entries(conn, &query_range, Some(scan_range).into_iter())?;
replace_queue_entries::<SqliteClientError>(
conn,
&query_range,
Some(scan_range).into_iter(),
false,
)?;
}
Ok(())
@ -1573,17 +1679,15 @@ mod tests {
use zcash_primitives::transaction::components::Amount;
use zcash_client_backend::data_api::WalletRead;
use zcash_client_backend::data_api::{AccountBirthday, WalletRead};
use crate::{
testing::{birthday_at_sapling_activation, TestBuilder},
AccountId,
};
use crate::{testing::TestBuilder, AccountId};
use super::get_balance;
#[cfg(feature = "transparent-inputs")]
use {
incrementalmerkletree::frontier::Frontier,
secrecy::Secret,
zcash_client_backend::{
data_api::WalletWrite, encoding::AddressCodec, wallet::WalletTransparentOutput,
@ -1597,7 +1701,7 @@ mod tests {
#[test]
fn empty_database_has_no_balance() {
let st = TestBuilder::new()
.with_test_account(birthday_at_sapling_activation)
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
// The account should be empty
@ -1628,11 +1732,15 @@ mod tests {
#[test]
#[cfg(feature = "transparent-inputs")]
fn put_received_transparent_utxo() {
use crate::testing::TestBuilder;
let mut st = TestBuilder::new().build();
// Add an account to the wallet
let seed = Secret::new([0u8; 32].to_vec());
let (account_id, _usk) = st.wallet_mut().create_account(&seed).unwrap();
let birthday =
AccountBirthday::from_parts(st.sapling_activation_height(), Frontier::empty(), None);
let (account_id, _usk) = st.wallet_mut().create_account(&seed, birthday).unwrap();
let uaddr = st
.wallet()
.get_current_address(account_id)

396
zcash_client_sqlite/src/wallet/init.rs
View File

@ -1,39 +1,22 @@
//! Functions for initializing the various databases.
use incrementalmerkletree::Retention;
use std::{collections::HashMap, fmt};
use tracing::debug;
use std::fmt;
use rusqlite::{self, named_params};
use rusqlite::{self};
use schemer::{Migrator, MigratorError};
use schemer_rusqlite::RusqliteAdapter;
use secrecy::SecretVec;
use shardtree::{error::ShardTreeError, ShardTree};
use shardtree::error::ShardTreeError;
use uuid::Uuid;
use zcash_primitives::{
block::BlockHash,
consensus::{self, BlockHeight, NetworkUpgrade},
merkle_tree::read_commitment_tree,
sapling,
consensus::{self},
transaction::components::amount::BalanceError,
zip32::AccountId,
};
use zcash_client_backend::{
data_api::{
scanning::{ScanPriority, ScanRange},
SAPLING_SHARD_HEIGHT,
},
keys::UnifiedFullViewingKey,
};
use crate::WalletDb;
use crate::{error::SqliteClientError, wallet, WalletDb, PRUNING_DEPTH, SAPLING_TABLES_PREFIX};
use super::{
commitment_tree::{self, SqliteShardStore},
scanning::insert_queue_entries,
};
use super::commitment_tree::{self};
mod migrations;
@ -176,234 +159,37 @@ fn init_wallet_db_internal<P: consensus::Parameters + 'static>(
Ok(())
}
/// Initialises the data database with the given set of account [`UnifiedFullViewingKey`]s.
///
/// **WARNING** This method should be used with care, and should ordinarily be unnecessary.
/// Prefer to use [`WalletWrite::create_account`] instead.
///
/// [`WalletWrite::create_account`]: zcash_client_backend::data_api::WalletWrite::create_account
///
/// The [`UnifiedFullViewingKey`]s are stored internally and used by other APIs such as
/// [`scan_cached_blocks`], and [`create_spend_to_address`]. Account identifiers in `keys` **MUST**
/// form a consecutive sequence beginning at account 0, and the [`UnifiedFullViewingKey`]
/// corresponding to a given account identifier **MUST** be derived from the wallet's mnemonic seed
/// at the BIP-44 `account` path level as described by [ZIP
/// 316](https://zips.z.cash/zip-0316)
///
/// # Examples
///
/// ```
/// # #[cfg(feature = "transparent-inputs")]
/// # {
/// use tempfile::NamedTempFile;
/// use secrecy::Secret;
/// use std::collections::HashMap;
///
/// use zcash_primitives::{
/// consensus::{Network, Parameters},
/// zip32::{AccountId, ExtendedSpendingKey}
/// };
///
/// use zcash_client_backend::{
/// keys::{
/// sapling,
/// UnifiedFullViewingKey
/// },
/// };
///
/// use zcash_client_sqlite::{
/// WalletDb,
/// wallet::init::{init_accounts_table, init_wallet_db}
/// };
///
/// let data_file = NamedTempFile::new().unwrap();
/// let mut db_data = WalletDb::for_path(data_file.path(), Network::TestNetwork).unwrap();
/// init_wallet_db(&mut db_data, Some(Secret::new(vec![]))).unwrap();
///
/// let seed = [0u8; 32]; // insecure; replace with a strong random seed
/// let account = AccountId::from(0);
/// let extsk = sapling::spending_key(&seed, Network::TestNetwork.coin_type(), account);
/// let dfvk = extsk.to_diversifiable_full_viewing_key();
/// let ufvk = UnifiedFullViewingKey::new(None, Some(dfvk), None).unwrap();
/// let ufvks = HashMap::from([(account, ufvk)]);
/// init_accounts_table(&mut db_data, &ufvks).unwrap();
/// # }
/// ```
///
/// [`get_address`]: crate::wallet::get_address
/// [`scan_cached_blocks`]: zcash_client_backend::data_api::chain::scan_cached_blocks
/// [`create_spend_to_address`]: zcash_client_backend::data_api::wallet::create_spend_to_address
pub fn init_accounts_table<P: consensus::Parameters>(
wallet_db: &mut WalletDb<rusqlite::Connection, P>,
keys: &HashMap<AccountId, UnifiedFullViewingKey>,
) -> Result<(), SqliteClientError> {
wallet_db.transactionally(|wdb| {
let mut empty_check = wdb.conn.0.prepare("SELECT * FROM accounts LIMIT 1")?;
if empty_check.exists([])? {
return Err(SqliteClientError::TableNotEmpty);
}
// Ensure that the account identifiers are sequential and begin at zero.
if let Some(account_id) = keys.keys().max() {
if usize::try_from(u32::from(*account_id)).unwrap() >= keys.len() {
return Err(SqliteClientError::AccountIdDiscontinuity);
}
}
// Insert accounts atomically
for (account, key) in keys.iter() {
wallet::add_account(wdb.conn.0, &wdb.params, *account, key)?;
}
Ok(())
})
}
/// Initialises the data database with the given block.
///
/// This enables a newly-created database to be immediately-usable, without needing to
/// synchronise historic blocks.
///
/// # Examples
///
/// ```
/// use tempfile::NamedTempFile;
/// use zcash_primitives::{
/// block::BlockHash,
/// consensus::{BlockHeight, Network},
/// };
/// use zcash_client_sqlite::{
/// WalletDb,
/// wallet::init::init_blocks_table,
/// };
///
/// // The block height.
/// let height = BlockHeight::from_u32(500_000);
/// // The hash of the block header.
/// let hash = BlockHash([0; 32]);
/// // The nTime field from the block header.
/// let time = 12_3456_7890;
/// // The serialized Sapling commitment tree as of this block.
/// // Pre-compute and hard-code, or obtain from a service.
/// let sapling_tree = &[];
///
/// let data_file = NamedTempFile::new().unwrap();
/// let mut db = WalletDb::for_path(data_file.path(), Network::TestNetwork).unwrap();
/// init_blocks_table(&mut db, height, hash, time, sapling_tree);
/// ```
pub fn init_blocks_table<P: consensus::Parameters>(
wallet_db: &mut WalletDb<rusqlite::Connection, P>,
height: BlockHeight,
hash: BlockHash,
time: u32,
sapling_tree: &[u8],
) -> Result<(), SqliteClientError> {
wallet_db.transactionally(|wdb| {
let mut empty_check = wdb.conn.0.prepare("SELECT * FROM blocks LIMIT 1")?;
if empty_check.exists([])? {
return Err(SqliteClientError::TableNotEmpty);
}
let block_end_tree =
read_commitment_tree::<sapling::Node, _, { sapling::NOTE_COMMITMENT_TREE_DEPTH }>(
sapling_tree,
)
.map_err(|e| {
rusqlite::Error::FromSqlConversionFailure(
sapling_tree.len(),
rusqlite::types::Type::Blob,
Box::new(e),
)
})?;
wdb.conn.0.execute(
"INSERT INTO blocks (height, hash, time, sapling_tree)
VALUES (:height, :hash, :time, :sapling_tree)",
named_params![
":height": u32::from(height),
":hash": hash.0,
":time": time,
":sapling_tree": sapling_tree,
],
)?;
if let Some(sapling_activation) = wdb.params.activation_height(NetworkUpgrade::Sapling) {
let scan_range_start = std::cmp::min(sapling_activation, height);
let scan_range_end = height + 1;
debug!(
"Setting ignored block range {}..{}",
scan_range_start, scan_range_end
);
insert_queue_entries(
wdb.conn.0,
Some(ScanRange::from_parts(
scan_range_start..scan_range_end,
ScanPriority::Ignored,
))
.iter(),
)?;
}
if let Some(nonempty_frontier) = block_end_tree.to_frontier().value() {
debug!("Inserting frontier into ShardTree: {:?}", nonempty_frontier);
let shard_store =
SqliteShardStore::<_, sapling::Node, SAPLING_SHARD_HEIGHT>::from_connection(
wdb.conn.0,
SAPLING_TABLES_PREFIX,
)?;
let mut shard_tree: ShardTree<
_,
{ sapling::NOTE_COMMITMENT_TREE_DEPTH },
SAPLING_SHARD_HEIGHT,
> = ShardTree::new(shard_store, PRUNING_DEPTH.try_into().unwrap());
shard_tree.insert_frontier_nodes(
nonempty_frontier.clone(),
Retention::Checkpoint {
id: height,
is_marked: false,
},
)?;
}
Ok(())
})
}
#[cfg(test)]
#[allow(deprecated)]
mod tests {
use rusqlite::{self, named_params, ToSql};
use secrecy::Secret;
use std::collections::HashMap;
use tempfile::NamedTempFile;
use zcash_client_backend::{
address::RecipientAddress,
data_api::{scanning::ScanPriority, WalletRead},
data_api::scanning::ScanPriority,
encoding::{encode_extended_full_viewing_key, encode_payment_address},
keys::{sapling, UnifiedFullViewingKey, UnifiedSpendingKey},
};
use zcash_primitives::{
block::BlockHash,
consensus::{self, BlockHeight, BranchId, Network, NetworkUpgrade, Parameters},
transaction::{TransactionData, TxVersion},
zip32::sapling::ExtendedFullViewingKey,
zip32::{sapling::ExtendedFullViewingKey, AccountId},
};
use crate::{
error::SqliteClientError, testing::TestBuilder, wallet::scanning::priority_code, AccountId,
WalletDb,
};
use crate::{testing::TestBuilder, wallet::scanning::priority_code, WalletDb};
use super::{init_accounts_table, init_blocks_table, init_wallet_db};
use super::init_wallet_db;
#[cfg(feature = "transparent-inputs")]
use {
crate::wallet::{self, pool_code, PoolType},
zcash_address::test_vectors,
zcash_client_backend::data_api::WalletWrite,
zcash_primitives::{legacy::keys as transparent, zip32::DiversifierIndex},
zcash_primitives::zip32::DiversifierIndex,
};
#[test]
@ -416,8 +202,9 @@ mod tests {
let expected_tables = vec![
"CREATE TABLE \"accounts\" (
account INTEGER PRIMARY KEY,
ufvk TEXT NOT NULL
)",
ufvk TEXT NOT NULL,
birthday_height INTEGER NOT NULL,
recover_until_height INTEGER )",
"CREATE TABLE addresses (
account INTEGER NOT NULL,
diversifier_index_be BLOB NOT NULL,
@ -581,6 +368,7 @@ mod tests {
// v_sapling_shard_unscanned_ranges
format!(
"CREATE VIEW v_sapling_shard_unscanned_ranges AS
WITH wallet_birthday AS (SELECT MIN(birthday_height) AS height FROM accounts)
SELECT
shard.shard_index,
shard.shard_index << 16 AS start_position,
@ -601,9 +389,11 @@ mod tests {
scan_queue.block_range_start <= prev_shard.subtree_end_height
AND (scan_queue.block_range_end - 1) >= shard.subtree_end_height
)
WHERE scan_queue.priority > {}",
INNER JOIN wallet_birthday
WHERE scan_queue.priority > {}
AND scan_queue.block_range_end > wallet_birthday.height",
u32::from(st.network().activation_height(NetworkUpgrade::Sapling).unwrap()),
priority_code(&ScanPriority::Scanned),
priority_code(&ScanPriority::Scanned)
),
// v_transactions
"CREATE VIEW v_transactions AS
@ -875,7 +665,10 @@ mod tests {
let extfvk = secret_key.to_extended_full_viewing_key();
init_0_3_0(&mut db_data, &extfvk, account).unwrap();
init_wallet_db(&mut db_data, Some(Secret::new(seed.to_vec()))).unwrap();
assert_matches!(
init_wallet_db(&mut db_data, Some(Secret::new(seed.to_vec()))),
Ok(_)
);
}
#[test]
@ -1043,7 +836,10 @@ mod tests {
let extfvk = secret_key.to_extended_full_viewing_key();
init_autoshielding(&mut db_data, &extfvk, account).unwrap();
init_wallet_db(&mut db_data, Some(Secret::new(seed.to_vec()))).unwrap();
assert_matches!(
init_wallet_db(&mut db_data, Some(Secret::new(seed.to_vec()))),
Ok(_)
);
}
#[test]
@ -1197,136 +993,30 @@ mod tests {
account,
)
.unwrap();
init_wallet_db(&mut db_data, Some(Secret::new(seed.to_vec()))).unwrap();
}
#[test]
fn init_accounts_table_only_works_once() {
let data_file = NamedTempFile::new().unwrap();
let mut db_data = WalletDb::for_path(data_file.path(), Network::TestNetwork).unwrap();
init_wallet_db(&mut db_data, Some(Secret::new(vec![]))).unwrap();
// We can call the function as many times as we want with no data
init_accounts_table(&mut db_data, &HashMap::new()).unwrap();
init_accounts_table(&mut db_data, &HashMap::new()).unwrap();
let seed = [0u8; 32];
let account = AccountId::from(0);
// First call with data should initialise the accounts table
let extsk = sapling::spending_key(&seed, db_data.params.coin_type(), account);
let dfvk = extsk.to_diversifiable_full_viewing_key();
#[cfg(feature = "transparent-inputs")]
let ufvk = UnifiedFullViewingKey::new(
Some(
transparent::AccountPrivKey::from_seed(&db_data.params, &seed, account)
.unwrap()
.to_account_pubkey(),
),
Some(dfvk),
None,
)
.unwrap();
#[cfg(not(feature = "transparent-inputs"))]
let ufvk = UnifiedFullViewingKey::new(Some(dfvk), None).unwrap();
let ufvks = HashMap::from([(account, ufvk)]);
init_accounts_table(&mut db_data, &ufvks).unwrap();
// Subsequent calls should return an error
init_accounts_table(&mut db_data, &HashMap::new()).unwrap_err();
init_accounts_table(&mut db_data, &ufvks).unwrap_err();
}
#[test]
fn init_accounts_table_allows_no_gaps() {
let params = Network::TestNetwork;
let data_file = NamedTempFile::new().unwrap();
let mut db_data = WalletDb::for_path(data_file.path(), params).unwrap();
init_wallet_db(&mut db_data, Some(Secret::new(vec![]))).unwrap();
// allow sequential initialization
let seed = [0u8; 32];
let ufvks = |ids: &[u32]| {
ids.iter()
.map(|a| {
let account = AccountId::from(*a);
UnifiedSpendingKey::from_seed(&params, &seed, account)
.map(|k| (account, k.to_unified_full_viewing_key()))
.unwrap()
})
.collect::<HashMap<_, _>>()
};
// should fail if we have a gap
assert_matches!(
init_accounts_table(&mut db_data, &ufvks(&[0, 2])),
Err(SqliteClientError::AccountIdDiscontinuity)
init_wallet_db(&mut db_data, Some(Secret::new(seed.to_vec()))),
Ok(_)
);
// should succeed if there are no gaps
assert!(init_accounts_table(&mut db_data, &ufvks(&[0, 1, 2])).is_ok());
}
#[test]
fn init_blocks_table_only_works_once() {
let data_file = NamedTempFile::new().unwrap();
let mut db_data = WalletDb::for_path(data_file.path(), Network::TestNetwork).unwrap();
init_wallet_db(&mut db_data, Some(Secret::new(vec![]))).unwrap();
// First call with data should initialise the blocks table
init_blocks_table(
&mut db_data,
BlockHeight::from(1u32),
BlockHash([1; 32]),
1,
&[0x0, 0x0, 0x0],
)
.unwrap();
// Subsequent calls should return an error
init_blocks_table(
&mut db_data,
BlockHeight::from(2u32),
BlockHash([2; 32]),
2,
&[0x0, 0x0, 0x0],
)
.unwrap_err();
}
#[test]
fn init_accounts_table_stores_correct_address() {
let data_file = NamedTempFile::new().unwrap();
let mut db_data = WalletDb::for_path(data_file.path(), Network::TestNetwork).unwrap();
init_wallet_db(&mut db_data, None).unwrap();
let seed = [0u8; 32];
// Add an account to the wallet
let account_id = AccountId::from(0);
let usk = UnifiedSpendingKey::from_seed(&db_data.params, &seed, account_id).unwrap();
let ufvk = usk.to_unified_full_viewing_key();
let expected_address = ufvk.sapling().unwrap().default_address().1;
let ufvks = HashMap::from([(account_id, ufvk)]);
init_accounts_table(&mut db_data, &ufvks).unwrap();
// The account's address should be in the data DB
let ua = db_data.get_current_address(AccountId::from(0)).unwrap();
assert_eq!(ua.unwrap().sapling().unwrap(), &expected_address);
}
#[test]
#[cfg(feature = "transparent-inputs")]
fn account_produces_expected_ua_sequence() {
let data_file = NamedTempFile::new().unwrap();
let mut db_data = WalletDb::for_path(data_file.path(), Network::MainNetwork).unwrap();
init_wallet_db(&mut db_data, None).unwrap();
use zcash_client_backend::data_api::AccountBirthday;
let network = Network::MainNetwork;
let data_file = NamedTempFile::new().unwrap();
let mut db_data = WalletDb::for_path(data_file.path(), network).unwrap();
let seed = test_vectors::UNIFIED[0].root_seed;
let (account, _usk) = db_data.create_account(&Secret::new(seed.to_vec())).unwrap();
assert_matches!(
init_wallet_db(&mut db_data, Some(Secret::new(seed.to_vec()))),
Ok(_)
);
let birthday = AccountBirthday::from_sapling_activation(&network);
let (account, _usk) = db_data
.create_account(&Secret::new(seed.to_vec()), birthday)
.unwrap();
assert_eq!(account, AccountId::from(0u32));
for tv in &test_vectors::UNIFIED[..3] {

6
zcash_client_sqlite/src/wallet/init/migrations.rs
View File

@ -1,3 +1,4 @@
mod add_account_birthdays;
mod add_transaction_views;
mod add_utxo_account;
mod addresses_table;
@ -38,6 +39,8 @@ pub(super) fn all_migrations<P: consensus::Parameters + 'static>(
// / | \
// shardtree_support nullifier_map sapling_memo_consistency
// |
// add_account_birthdays
// |
// v_sapling_shard_unscanned_ranges
vec![
Box::new(initial_setup::Migration {}),
@ -63,6 +66,9 @@ pub(super) fn all_migrations<P: consensus::Parameters + 'static>(
Box::new(sapling_memo_consistency::Migration {
params: params.clone(),
}),
Box::new(add_account_birthdays::Migration {
params: params.clone(),
}),
Box::new(v_sapling_shard_unscanned_ranges::Migration {
params: params.clone(),
}),

74
zcash_client_sqlite/src/wallet/init/migrations/add_account_birthdays.rs Normal file
View File

@ -0,0 +1,74 @@
//! This migration adds a birthday height to each account record.
use std::collections::HashSet;
use schemer_rusqlite::RusqliteMigration;
use uuid::Uuid;
use zcash_primitives::consensus::{self, NetworkUpgrade};
use crate::wallet::init::WalletMigrationError;
use super::shardtree_support;
pub(super) const MIGRATION_ID: Uuid = Uuid::from_u128(0xeeec0d0d_fee0_4231_8c68_5f3a7c7c2245);
pub(super) struct Migration<P> {
pub(super) params: P,
}
impl<P> schemer::Migration for Migration<P> {
fn id(&self) -> Uuid {
MIGRATION_ID
}
fn dependencies(&self) -> HashSet<Uuid> {
[shardtree_support::MIGRATION_ID].into_iter().collect()
}
fn description(&self) -> &'static str {
"Adds a birthday height for each account."
}
}
impl<P: consensus::Parameters> RusqliteMigration for Migration<P> {
type Error = WalletMigrationError;
fn up(&self, transaction: &rusqlite::Transaction) -> Result<(), Self::Error> {
transaction.execute_batch(&format!(
"ALTER TABLE accounts ADD COLUMN birthday_height INTEGER;
-- set the birthday height to the height of the first block in the blocks table
UPDATE accounts SET birthday_height = MIN(blocks.height) FROM blocks;
-- if the blocks table is empty, set the birthday height to Sapling activation - 1
UPDATE accounts SET birthday_height = {} WHERE birthday_height IS NULL;
CREATE TABLE accounts_new (
account INTEGER PRIMARY KEY,
ufvk TEXT NOT NULL,
birthday_height INTEGER NOT NULL,
recover_until_height INTEGER
);
INSERT INTO accounts_new (account, ufvk, birthday_height)
SELECT account, ufvk, birthday_height FROM accounts;
PRAGMA foreign_keys=OFF;
PRAGMA legacy_alter_table = ON;
DROP TABLE accounts;
ALTER TABLE accounts_new RENAME TO accounts;
PRAGMA legacy_alter_table = OFF;
PRAGMA foreign_keys=ON;",
u32::from(
self.params
.activation_height(NetworkUpgrade::Sapling)
.unwrap()
)
))?;
Ok(())
}
fn down(&self, _transaction: &rusqlite::Transaction) -> Result<(), Self::Error> {
panic!("This migration cannot be reverted.");
}
}

20
zcash_client_sqlite/src/wallet/init/migrations/addresses_table.rs
View File

@ -1,13 +1,13 @@
use std::collections::HashSet;
use rusqlite::Transaction;
use rusqlite::{named_params, Transaction};
use schemer;
use schemer_rusqlite::RusqliteMigration;
use uuid::Uuid;
use zcash_client_backend::{address::RecipientAddress, keys::UnifiedFullViewingKey};
use zcash_primitives::{consensus, zip32::AccountId};
use crate::wallet::{add_account_internal, init::WalletMigrationError};
use crate::wallet::{init::WalletMigrationError, insert_address};
#[cfg(feature = "transparent-inputs")]
use zcash_primitives::legacy::keys::IncomingViewingKey;
@ -152,13 +152,17 @@ impl<P: consensus::Parameters> RusqliteMigration for Migration<P> {
}
}
add_account_internal::<P, WalletMigrationError>(
transaction,
&self.params,
"accounts_new",
account,
&ufvk,
transaction.execute(
"INSERT INTO accounts_new (account, ufvk)
VALUES (:account, :ufvk)",
named_params![
":account": u32::from(account),
":ufvk": ufvk.encode(&self.params),
],
)?;
let (address, d_idx) = ufvk.default_address();
insert_address(transaction, &self.params, account, d_idx, &address)?;
}
transaction.execute_batch(

13
zcash_client_sqlite/src/wallet/init/migrations/v_sapling_shard_unscanned_ranges.rs
View File

@ -6,11 +6,11 @@ use std::collections::HashSet;
use schemer_rusqlite::RusqliteMigration;
use uuid::Uuid;
use zcash_client_backend::data_api::{scanning::ScanPriority, SAPLING_SHARD_HEIGHT};
use zcash_primitives::consensus;
use zcash_primitives::consensus::{self, NetworkUpgrade};
use crate::wallet::{init::WalletMigrationError, scanning::priority_code};
use super::shardtree_support;
use super::add_account_birthdays;
pub(super) const MIGRATION_ID: Uuid = Uuid::from_u128(0xfa934bdc_97b6_4980_8a83_b2cb1ac465fd);
@ -24,7 +24,7 @@ impl<P> schemer::Migration for Migration<P> {
}
fn dependencies(&self) -> HashSet<Uuid> {
[shardtree_support::MIGRATION_ID].into_iter().collect()
[add_account_birthdays::MIGRATION_ID].into_iter().collect()
}
fn description(&self) -> &'static str {
@ -39,6 +39,7 @@ impl<P: consensus::Parameters> RusqliteMigration for Migration<P> {
transaction.execute_batch(
&format!(
"CREATE VIEW v_sapling_shard_unscanned_ranges AS
WITH wallet_birthday AS (SELECT MIN(birthday_height) AS height FROM accounts)
SELECT
shard.shard_index,
shard.shard_index << {} AS start_position,
@ -59,10 +60,12 @@ impl<P: consensus::Parameters> RusqliteMigration for Migration<P> {
scan_queue.block_range_start <= prev_shard.subtree_end_height
AND (scan_queue.block_range_end - 1) >= shard.subtree_end_height
)
WHERE scan_queue.priority > {}",
INNER JOIN wallet_birthday
WHERE scan_queue.priority > {}
AND scan_queue.block_range_end > wallet_birthday.height;",
SAPLING_SHARD_HEIGHT,
SAPLING_SHARD_HEIGHT,
u32::from(self.params.activation_height(consensus::NetworkUpgrade::Sapling).unwrap()),
u32::from(self.params.activation_height(NetworkUpgrade::Sapling).unwrap()),
priority_code(&ScanPriority::Scanned),
)
)?;

151
zcash_client_sqlite/src/wallet/sapling.rs
View File

@ -20,7 +20,7 @@ use zcash_client_backend::{
use crate::{error::SqliteClientError, ReceivedNoteId};
use super::memo_repr;
use super::{memo_repr, wallet_birthday};
/// This trait provides a generalization over shielded output representations.
pub(crate) trait ReceivedSaplingOutput {
@ -132,6 +132,15 @@ pub(crate) fn get_spendable_sapling_notes(
anchor_height: BlockHeight,
exclude: &[ReceivedNoteId],
) -> Result<Vec<ReceivedSaplingNote<ReceivedNoteId>>, SqliteClientError> {
let birthday_height = match wallet_birthday(conn)? {
Some(birthday) => birthday,
None => {
// the wallet birthday can only be unknown if there are no accounts in the wallet; in
// such a case, the wallet has no notes to spend.
return Ok(vec![]);
}
};
let mut stmt_unscanned_tip = conn.prepare_cached(
"SELECT 1 FROM v_sapling_shard_unscanned_ranges
WHERE :anchor_height BETWEEN subtree_start_height AND IFNULL(subtree_end_height, :anchor_height)
@ -159,8 +168,10 @@ pub(crate) fn get_spendable_sapling_notes(
-- select all the unscanned ranges involving the shard containing this note
WHERE sapling_received_notes.commitment_tree_position >= unscanned.start_position
AND sapling_received_notes.commitment_tree_position < unscanned.end_position_exclusive
-- exclude unscanned ranges above the anchor height which don't affect spendability
-- exclude unscanned ranges that start above the anchor height (they don't affect spendability)
AND unscanned.block_range_start <= :anchor_height
-- exclude unscanned ranges that end below the wallet birthday
AND unscanned.block_range_end > :wallet_birthday
)",
)?;
@ -169,9 +180,10 @@ pub(crate) fn get_spendable_sapling_notes(
let notes = stmt_select_notes.query_and_then(
named_params![
":account": &u32::from(account),
":anchor_height": &u32::from(anchor_height),
":account": u32::from(account),
":anchor_height": u32::from(anchor_height),
":exclude": &excluded_ptr,
":wallet_birthday": u32::from(birthday_height)
],
to_spendable_note,
)?;
@ -186,6 +198,15 @@ pub(crate) fn select_spendable_sapling_notes(
anchor_height: BlockHeight,
exclude: &[ReceivedNoteId],
) -> Result<Vec<ReceivedSaplingNote<ReceivedNoteId>>, SqliteClientError> {
let birthday_height = match wallet_birthday(conn)? {
Some(birthday) => birthday,
None => {
// the wallet birthday can only be unknown if there are no accounts in the wallet; in
// such a case, the wallet has no notes to spend.
return Ok(vec![]);
}
};
let mut stmt_unscanned_tip = conn.prepare_cached(
"SELECT 1 FROM v_sapling_shard_unscanned_ranges
WHERE :anchor_height BETWEEN subtree_start_height AND IFNULL(subtree_end_height, :anchor_height)
@ -233,8 +254,10 @@ pub(crate) fn select_spendable_sapling_notes(
-- select all the unscanned ranges involving the shard containing this note
WHERE sapling_received_notes.commitment_tree_position >= unscanned.start_position
AND sapling_received_notes.commitment_tree_position < unscanned.end_position_exclusive
-- exclude unscanned ranges above the anchor height which don't affect spendability
-- exclude unscanned ranges that start above the anchor height (they don't affect spendability)
AND unscanned.block_range_start <= :anchor_height
-- exclude unscanned ranges that end below the wallet birthday
AND unscanned.block_range_end > :wallet_birthday
)
)
SELECT id_note, diversifier, value, rcm, commitment_tree_position
@ -252,7 +275,8 @@ pub(crate) fn select_spendable_sapling_notes(
":account": &u32::from(account),
":anchor_height": &u32::from(anchor_height),
":target_value": &i64::from(target_value),
":exclude": &excluded_ptr
":exclude": &excluded_ptr,
":wallet_birthday": u32::from(birthday_height)
],
to_spendable_note,
)?;
@ -403,8 +427,6 @@ pub(crate) fn put_received_note<T: ReceivedSaplingOutput>(
pub(crate) mod tests {
use std::{convert::Infallible, num::NonZeroU32};
use secrecy::Secret;
use zcash_proofs::prover::LocalTxProver;
use zcash_primitives::{
@ -428,7 +450,7 @@ pub(crate) mod tests {
self,
error::Error,
wallet::input_selection::{GreedyInputSelector, GreedyInputSelectorError},
ShieldedProtocol, WalletRead, WalletWrite,
AccountBirthday, ShieldedProtocol, WalletRead,
},
decrypt_transaction,
fees::{fixed, zip317, DustOutputPolicy},
@ -446,7 +468,7 @@ pub(crate) mod tests {
#[cfg(feature = "transparent-inputs")]
use {
zcash_client_backend::wallet::WalletTransparentOutput,
zcash_client_backend::{data_api::WalletWrite, wallet::WalletTransparentOutput},
zcash_primitives::transaction::components::{amount::NonNegativeAmount, OutPoint, TxOut},
};
@ -461,12 +483,13 @@ pub(crate) mod tests {
#[test]
fn send_proposed_transfer() {
let mut st = TestBuilder::new().with_block_cache().build();
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
// Add an account to the wallet
let seed = Secret::new([0u8; 32].to_vec());
let (account, usk) = st.wallet_mut().create_account(&seed).unwrap();
let dfvk = usk.sapling().to_diversifiable_full_viewing_key();
let (account, usk, _) = st.test_account().unwrap();
let dfvk = st.test_account_sapling().unwrap();
// Add funds to the wallet in a single note
let value = Amount::from_u64(60000).unwrap();
@ -609,12 +632,10 @@ pub(crate) mod tests {
#[test]
fn create_to_address_fails_on_incorrect_usk() {
let mut st = TestBuilder::new().build();
// Add an account to the wallet
let seed = Secret::new([0u8; 32].to_vec());
let (_, usk) = st.wallet_mut().create_account(&seed).unwrap();
let dfvk = usk.sapling().to_diversifiable_full_viewing_key();
let mut st = TestBuilder::new()
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
let dfvk = st.test_account_sapling().unwrap();
let to = dfvk.default_address().1.into();
// Create a USK that doesn't exist in the wallet
@ -637,12 +658,12 @@ pub(crate) mod tests {
#[test]
fn create_to_address_fails_with_no_blocks() {
let mut st = TestBuilder::new().build();
let mut st = TestBuilder::new()
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
// Add an account to the wallet
let seed = Secret::new([0u8; 32].to_vec());
let (_, usk) = st.wallet_mut().create_account(&seed).unwrap();
let dfvk = usk.sapling().to_diversifiable_full_viewing_key();
let (_, usk, _) = st.test_account().unwrap();
let dfvk = st.test_account_sapling().unwrap();
let to = dfvk.default_address().1.into();
// Account balance should be zero
@ -667,12 +688,13 @@ pub(crate) mod tests {
#[test]
fn create_to_address_fails_on_unverified_notes() {
let mut st = TestBuilder::new().with_block_cache().build();
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
// Add an account to the wallet
let seed = Secret::new([0u8; 32].to_vec());
let (_, usk) = st.wallet_mut().create_account(&seed).unwrap();
let dfvk = usk.sapling().to_diversifiable_full_viewing_key();
let (_, usk, _) = st.test_account().unwrap();
let dfvk = st.test_account_sapling().unwrap();
// Add funds to the wallet in a single note
let value = Amount::from_u64(50000).unwrap();
@ -778,12 +800,13 @@ pub(crate) mod tests {
#[test]
fn create_to_address_fails_on_locked_notes() {
let mut st = TestBuilder::new().with_block_cache().build();
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
// Add an account to the wallet
let seed = Secret::new([0u8; 32].to_vec());
let (_, usk) = st.wallet_mut().create_account(&seed).unwrap();
let dfvk = usk.sapling().to_diversifiable_full_viewing_key();
let (_, usk, _) = st.test_account().unwrap();
let dfvk = st.test_account_sapling().unwrap();
// Add funds to the wallet in a single note
let value = Amount::from_u64(50000).unwrap();
@ -876,12 +899,13 @@ pub(crate) mod tests {
#[test]
fn ovk_policy_prevents_recovery_from_chain() {
let mut st = TestBuilder::new().with_block_cache().build();
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
// Add an account to the wallet
let seed = Secret::new([0u8; 32].to_vec());
let (_, usk) = st.wallet_mut().create_account(&seed).unwrap();
let dfvk = usk.sapling().to_diversifiable_full_viewing_key();
let (_, usk, _) = st.test_account().unwrap();
let dfvk = st.test_account_sapling().unwrap();
// Add funds to the wallet in a single note
let value = Amount::from_u64(50000).unwrap();
@ -976,12 +1000,13 @@ pub(crate) mod tests {
#[test]
fn create_to_address_succeeds_to_t_addr_zero_change() {
let mut st = TestBuilder::new().with_block_cache().build();
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
// Add an account to the wallet
let seed = Secret::new([0u8; 32].to_vec());
let (_, usk) = st.wallet_mut().create_account(&seed).unwrap();
let dfvk = usk.sapling().to_diversifiable_full_viewing_key();
let (_, usk, _) = st.test_account().unwrap();
let dfvk = st.test_account_sapling().unwrap();
// Add funds to the wallet in a single note
let value = Amount::from_u64(60000).unwrap();
@ -1019,12 +1044,13 @@ pub(crate) mod tests {
#[test]
fn create_to_address_spends_a_change_note() {
let mut st = TestBuilder::new().with_block_cache().build();
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
// Add an account to the wallet
let seed = Secret::new([0u8; 32].to_vec());
let (_, usk) = st.wallet_mut().create_account(&seed).unwrap();
let dfvk = usk.sapling().to_diversifiable_full_viewing_key();
let (_, usk, _) = st.test_account().unwrap();
let dfvk = st.test_account_sapling().unwrap();
// Add funds to the wallet in a single note
let value = Amount::from_u64(60000).unwrap();
@ -1062,12 +1088,13 @@ pub(crate) mod tests {
#[test]
fn zip317_spend() {
let mut st = TestBuilder::new().with_block_cache().build();
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
// Add an account to the wallet
let seed = Secret::new([0u8; 32].to_vec());
let (_, usk) = st.wallet_mut().create_account(&seed).unwrap();
let dfvk = usk.sapling().to_diversifiable_full_viewing_key();
let (_, usk, _) = st.test_account().unwrap();
let dfvk = st.test_account_sapling().unwrap();
// Add funds to the wallet
let (h1, _, _) = st.generate_next_block(
@ -1159,12 +1186,14 @@ pub(crate) mod tests {
#[test]
#[cfg(feature = "transparent-inputs")]
fn shield_transparent() {
let mut st = TestBuilder::new().with_block_cache().build();
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
let (account_id, usk, _) = st.test_account().unwrap();
let dfvk = st.test_account_sapling().unwrap();
// Add an account to the wallet
let seed = Secret::new([0u8; 32].to_vec());
let (account_id, usk) = st.wallet_mut().create_account(&seed).unwrap();
let dfvk = usk.sapling().to_diversifiable_full_viewing_key();
let uaddr = st
.wallet()
.get_current_address(account_id)

378
zcash_client_sqlite/src/wallet/scanning.rs
View File

@ -1,34 +1,60 @@
use rusqlite::{self, named_params, types::Value, OptionalExtension};
use shardtree::error::ShardTreeError;
use std::cmp::{max, min, Ordering};
use std::collections::BTreeSet;
use std::ops::{Not, Range};
use std::rc::Rc;
use tracing::{debug, trace};
use zcash_client_backend::data_api::scanning::{ScanPriority, ScanRange};
use incrementalmerkletree::{Address, Position};
use zcash_client_backend::data_api::scanning::{ScanPriority, ScanRange};
use zcash_primitives::consensus::{self, BlockHeight, NetworkUpgrade};
use zcash_client_backend::data_api::SAPLING_SHARD_HEIGHT;
use crate::error::SqliteClientError;
use crate::{PRUNING_DEPTH, VERIFY_LOOKAHEAD};
use super::block_height_extrema;
use crate::{
error::SqliteClientError,
wallet::{block_height_extrema, commitment_tree, init::WalletMigrationError},
PRUNING_DEPTH, VERIFY_LOOKAHEAD,
};
#[derive(Debug, Clone, Copy)]
enum Insert {
enum InsertOn {
Left,
Right,
}
struct Insert {
on: InsertOn,
force_rescan: bool,
}
impl Insert {
fn left(force_rescan: bool) -> Self {
Insert {
on: InsertOn::Left,
force_rescan,
}
}
fn right(force_rescan: bool) -> Self {
Insert {
on: InsertOn::Right,
force_rescan,
}
}
}
impl Not for Insert {
type Output = Self;
fn not(self) -> Self::Output {
match self {
Insert::Left => Insert::Right,
Insert::Right => Insert::Left,
Insert {
on: match self.on {
InsertOn::Left => InsertOn::Right,
InsertOn::Right => InsertOn::Left,
},
force_rescan: self.force_rescan,
}
}
}
@ -42,9 +68,9 @@ enum Dominance {
impl From<Insert> for Dominance {
fn from(value: Insert) -> Self {
match value {
Insert::Left => Dominance::Left,
Insert::Right => Dominance::Right,
match value.on {
InsertOn::Left => Dominance::Left,
InsertOn::Right => Dominance::Right,
}
}
}
@ -115,7 +141,7 @@ fn dominance(current: &ScanPriority, inserted: &ScanPriority, insert: Insert) ->
match (current.cmp(inserted), (current, inserted)) {
(Ordering::Equal, _) => Dominance::Equal,
(_, (_, ScanPriority::Verify | ScanPriority::Scanned)) => Dominance::from(insert),
(_, (ScanPriority::Scanned, _)) => Dominance::from(!insert),
(_, (ScanPriority::Scanned, _)) if !insert.force_rescan => Dominance::from(!insert),
(Ordering::Less, _) => Dominance::from(insert),
(Ordering::Greater, _) => Dominance::from(!insert),
}
@ -197,7 +223,7 @@ fn join_nonoverlapping(left: ScanRange, right: ScanRange) -> Joined {
}
}
fn insert(current: ScanRange, to_insert: ScanRange) -> Joined {
fn insert(current: ScanRange, to_insert: ScanRange, force_rescans: bool) -> Joined {
fn join_overlapping(left: ScanRange, right: ScanRange, insert: Insert) -> Joined {
assert!(
left.block_range().start <= right.block_range().start
@ -205,9 +231,9 @@ fn insert(current: ScanRange, to_insert: ScanRange) -> Joined {
);
// recompute the range dominance based upon the queue entry priorities
let dominance = match insert {
Insert::Left => dominance(&right.priority(), &left.priority(), insert),
Insert::Right => dominance(&left.priority(), &right.priority(), insert),
let dominance = match insert.on {
InsertOn::Left => dominance(&right.priority(), &left.priority(), insert),
InsertOn::Right => dominance(&left.priority(), &right.priority(), insert),
};
match dominance {
@ -237,15 +263,23 @@ fn insert(current: ScanRange, to_insert: ScanRange) -> Joined {
use RangeOrdering::*;
match RangeOrdering::cmp(to_insert.block_range(), current.block_range()) {
LeftFirstDisjoint => join_nonoverlapping(to_insert, current),
LeftFirstOverlap | RightContained => join_overlapping(to_insert, current, Insert::Left),
LeftFirstOverlap | RightContained => {
join_overlapping(to_insert, current, Insert::left(force_rescans))
}
Equal => Joined::One(ScanRange::from_parts(
to_insert.block_range().clone(),
match dominance(&current.priority(), &to_insert.priority(), Insert::Right) {
match dominance(
&current.priority(),
&to_insert.priority(),
Insert::right(force_rescans),
) {
Dominance::Left | Dominance::Equal => current.priority(),
Dominance::Right => to_insert.priority(),
},
)),
RightFirstOverlap | LeftContained => join_overlapping(current, to_insert, Insert::Right),
RightFirstOverlap | LeftContained => {
join_overlapping(current, to_insert, Insert::right(force_rescans))
}
RightFirstDisjoint => join_nonoverlapping(current, to_insert),
}
}
@ -294,9 +328,9 @@ impl SpanningTree {
to_insert: ScanRange,
insert: Insert,
) -> Self {
let (left, right) = match insert {
Insert::Left => (Box::new(left.insert(to_insert)), right),
Insert::Right => (left, Box::new(right.insert(to_insert))),
let (left, right) = match insert.on {
InsertOn::Left => (Box::new(left.insert(to_insert, insert.force_rescan)), right),
InsertOn::Right => (left, Box::new(right.insert(to_insert, insert.force_rescan))),
};
SpanningTree::Parent {
span: left.span().start..right.span().end,
@ -305,12 +339,18 @@ impl SpanningTree {
}
}
fn from_split(left: Self, right: Self, to_insert: ScanRange, split_point: BlockHeight) -> Self {
fn from_split(
left: Self,
right: Self,
to_insert: ScanRange,
split_point: BlockHeight,
force_rescans: bool,
) -> Self {
let (l_insert, r_insert) = to_insert
.split_at(split_point)
.expect("Split point is within the range of to_insert");
let left = Box::new(left.insert(l_insert));
let right = Box::new(right.insert(r_insert));
let left = Box::new(left.insert(l_insert, force_rescans));
let right = Box::new(right.insert(r_insert, force_rescans));
SpanningTree::Parent {
span: left.span().start..right.span().end,
left,
@ -318,9 +358,9 @@ impl SpanningTree {
}
}
fn insert(self, to_insert: ScanRange) -> Self {
fn insert(self, to_insert: ScanRange, force_rescans: bool) -> Self {
match self {
SpanningTree::Leaf(cur) => Self::from_joined(insert(cur, to_insert)),
SpanningTree::Leaf(cur) => Self::from_joined(insert(cur, to_insert, force_rescans)),
SpanningTree::Parent { span, left, right } => {
// This algorithm always preserves the existing partition point, and does not do
// any rebalancing or unification of ranges within the tree. This should be okay
@ -331,15 +371,15 @@ impl SpanningTree {
match RangeOrdering::cmp(&span, to_insert.block_range()) {
LeftFirstDisjoint => {
// extend the right-hand branch
Self::from_insert(left, right, to_insert, Insert::Right)
Self::from_insert(left, right, to_insert, Insert::right(force_rescans))
}
LeftFirstOverlap => {
let split_point = left.span().end;
if split_point > to_insert.block_range().start {
Self::from_split(*left, *right, to_insert, split_point)
Self::from_split(*left, *right, to_insert, split_point, force_rescans)
} else {
// to_insert is fully contained in or equals the right child
Self::from_insert(left, right, to_insert, Insert::Right)
Self::from_insert(left, right, to_insert, Insert::right(force_rescans))
}
}
RightContained => {
@ -348,42 +388,42 @@ impl SpanningTree {
let split_point = left.span().end;
if to_insert.block_range().start >= split_point {
// to_insert is fully contained in the right
Self::from_insert(left, right, to_insert, Insert::Right)
Self::from_insert(left, right, to_insert, Insert::right(force_rescans))
} else if to_insert.block_range().end <= split_point {
// to_insert is fully contained in the left
Self::from_insert(left, right, to_insert, Insert::Left)
Self::from_insert(left, right, to_insert, Insert::left(force_rescans))
} else {
// to_insert must be split.
Self::from_split(*left, *right, to_insert, split_point)
Self::from_split(*left, *right, to_insert, split_point, force_rescans)
}
}
Equal => {
let split_point = left.span().end;
if split_point > to_insert.block_range().start {
Self::from_split(*left, *right, to_insert, split_point)
Self::from_split(*left, *right, to_insert, split_point, force_rescans)
} else {
// to_insert is fully contained in the right subtree
right.insert(to_insert)
right.insert(to_insert, force_rescans)
}
}
LeftContained => {
// the current span is fully contained within to_insert, so we will extend
// or overwrite both sides
let split_point = left.span().end;
Self::from_split(*left, *right, to_insert, split_point)
Self::from_split(*left, *right, to_insert, split_point, force_rescans)
}
RightFirstOverlap => {
let split_point = left.span().end;
if split_point < to_insert.block_range().end {
Self::from_split(*left, *right, to_insert, split_point)
Self::from_split(*left, *right, to_insert, split_point, force_rescans)
} else {
// to_insert is fully contained in or equals the left child
Self::from_insert(left, right, to_insert, Insert::Left)
Self::from_insert(left, right, to_insert, Insert::left(force_rescans))
}
}
RightFirstDisjoint => {
// extend the left-hand branch
Self::from_insert(left, right, to_insert, Insert::Left)
Self::from_insert(left, right, to_insert, Insert::left(force_rescans))
}
}
}
@ -443,14 +483,66 @@ pub(crate) fn insert_queue_entries<'a>(
Ok(())
}
pub(crate) fn replace_queue_entries(
/// A trait that abstracts over the construction of wallet errors.
///
/// In order to make it possible to use [`replace_queue_entries`] in database migrations as well as
/// in code that returns `SqliteClientError`, it is necessary for that method to be polymorphic in
/// the error type.
pub(crate) trait WalletError {
fn db_error(err: rusqlite::Error) -> Self;
fn corrupt(message: String) -> Self;
fn chain_height_unknown() -> Self;
fn commitment_tree(err: ShardTreeError<commitment_tree::Error>) -> Self;
}
impl WalletError for SqliteClientError {
fn db_error(err: rusqlite::Error) -> Self {
SqliteClientError::DbError(err)
}
fn corrupt(message: String) -> Self {
SqliteClientError::CorruptedData(message)
}
fn chain_height_unknown() -> Self {
SqliteClientError::ChainHeightUnknown
}
fn commitment_tree(err: ShardTreeError<commitment_tree::Error>) -> Self {
SqliteClientError::CommitmentTree(err)
}
}
impl WalletError for WalletMigrationError {
fn db_error(err: rusqlite::Error) -> Self {
WalletMigrationError::DbError(err)
}
fn corrupt(message: String) -> Self {
WalletMigrationError::CorruptedData(message)
}
fn chain_height_unknown() -> Self {
WalletMigrationError::CorruptedData(
"Wallet migration requires a valid account birthday.".to_owned(),
)
}
fn commitment_tree(err: ShardTreeError<commitment_tree::Error>) -> Self {
WalletMigrationError::CommitmentTree(err)
}
}
pub(crate) fn replace_queue_entries<E: WalletError>(
conn: &rusqlite::Transaction<'_>,
query_range: &Range<BlockHeight>,
entries: impl Iterator<Item = ScanRange>,
) -> Result<(), SqliteClientError> {
force_rescans: bool,
) -> Result<(), E> {
let (to_create, to_delete_ends) = {
let mut suggested_stmt = conn.prepare_cached(
"SELECT block_range_start, block_range_end, priority
let mut suggested_stmt = conn
.prepare_cached(
"SELECT block_range_start, block_range_end, priority
FROM scan_queue
WHERE (
-- the start is contained within or adjacent to the range
@ -468,12 +560,15 @@ pub(crate) fn replace_queue_entries(
AND block_range_end <= :end
)
ORDER BY block_range_end",
)?;
)
.map_err(E::db_error)?;
let mut rows = suggested_stmt.query(named_params![
":start": u32::from(query_range.start),
":end": u32::from(query_range.end),
])?;
let mut rows = suggested_stmt
.query(named_params![
":start": u32::from(query_range.start),
":end": u32::from(query_range.end),
])
.map_err(E::db_error)?;
// Iterate over the ranges in the scan queue that overlap the range that we have
// identified as needing to be fully scanned. For each such range add it to the
@ -481,25 +576,22 @@ pub(crate) fn replace_queue_entries(
// some in the process).
let mut to_create: Option<SpanningTree> = None;
let mut to_delete_ends: Vec<Value> = vec![];
while let Some(row) = rows.next()? {
while let Some(row) = rows.next().map_err(E::db_error)? {
let entry = ScanRange::from_parts(
Range {
start: BlockHeight::from(row.get::<_, u32>(0)?),
end: BlockHeight::from(row.get::<_, u32>(1)?),
start: BlockHeight::from(row.get::<_, u32>(0).map_err(E::db_error)?),
end: BlockHeight::from(row.get::<_, u32>(1).map_err(E::db_error)?),
},
{
let code = row.get::<_, i64>(2)?;
let code = row.get::<_, i64>(2).map_err(E::db_error)?;
parse_priority_code(code).ok_or_else(|| {
SqliteClientError::CorruptedData(format!(
"scan priority not recognized: {}",
code
))
E::corrupt(format!("scan priority not recognized: {}", code))
})?
},
);
to_delete_ends.push(Value::from(u32::from(entry.block_range().end)));
to_create = if let Some(cur) = to_create {
Some(cur.insert(entry))
Some(cur.insert(entry, force_rescans))
} else {
Some(SpanningTree::Leaf(entry))
};
@ -509,7 +601,7 @@ pub(crate) fn replace_queue_entries(
// start with the scanned range.
for entry in entries {
to_create = if let Some(cur) = to_create {
Some(cur.insert(entry))
Some(cur.insert(entry, force_rescans))
} else {
Some(SpanningTree::Leaf(entry))
};
@ -523,10 +615,11 @@ pub(crate) fn replace_queue_entries(
conn.execute(
"DELETE FROM scan_queue WHERE block_range_end IN rarray(:ends)",
named_params![":ends": ends_ptr],
)?;
)
.map_err(E::db_error)?;
let scan_ranges = tree.into_vec();
insert_queue_entries(conn, scan_ranges.iter())?;
insert_queue_entries(conn, scan_ranges.iter()).map_err(E::db_error)?;
}
Ok(())
@ -607,10 +700,11 @@ pub(crate) fn scan_complete<P: consensus::Parameters>(
vec![]
};
replace_queue_entries(
replace_queue_entries::<SqliteClientError>(
conn,
&query_range,
Some(scanned).into_iter().chain(extensions.into_iter()),
false,
)?;
Ok(())
@ -710,10 +804,11 @@ pub(crate) fn update_chain_tip<P: consensus::Parameters>(
};
if let Some(query_range) = query_range {
replace_queue_entries(
replace_queue_entries::<SqliteClientError>(
conn,
&query_range,
shard_entry.into_iter().chain(tip_entry.into_iter()),
false,
)?;
} else {
// If we have neither shard data nor any existing block data in the database, we should also
@ -737,13 +832,12 @@ pub(crate) fn update_chain_tip<P: consensus::Parameters>(
mod tests {
use std::ops::Range;
use incrementalmerkletree::{Hashable, Level};
use secrecy::Secret;
use incrementalmerkletree::{frontier::Frontier, Hashable, Level, Position};
use zcash_client_backend::data_api::{
chain::CommitmentTreeRoot,
scanning::{ScanPriority, ScanRange},
WalletCommitmentTrees, WalletRead, WalletWrite,
AccountBirthday, WalletCommitmentTrees, WalletRead, WalletWrite,
};
use zcash_primitives::{
block::BlockHash,
@ -753,11 +847,9 @@ mod tests {
};
use crate::{
testing::{birthday_at_sapling_activation, AddressType, TestBuilder},
wallet::{
init::init_blocks_table,
scanning::{insert_queue_entries, replace_queue_entries, suggest_scan_ranges},
},
error::SqliteClientError,
testing::{AddressType, TestBuilder},
wallet::scanning::{insert_queue_entries, replace_queue_entries, suggest_scan_ranges},
};
use super::{join_nonoverlapping, Joined, RangeOrdering, SpanningTree};
@ -904,7 +996,7 @@ mod tests {
let scan_range = scan_range(range.clone(), *priority);
match acc {
None => Some(SpanningTree::Leaf(scan_range)),
Some(t) => Some(t.insert(scan_range)),
Some(t) => Some(t.insert(scan_range, false)),
}
})
}
@ -1035,7 +1127,7 @@ mod tests {
// a `ChainTip` insertion should not overwrite a scanned range.
let mut t = spanning_tree(&[(0..3, ChainTip), (3..5, Scanned), (5..7, ChainTip)]).unwrap();
t = t.insert(scan_range(0..7, ChainTip));
t = t.insert(scan_range(0..7, ChainTip), false);
assert_eq!(
t.into_vec(),
vec![
@ -1054,7 +1146,7 @@ mod tests {
scan_range(280310..280320, Scanned)
]
);
t = t.insert(scan_range(280300..280340, ChainTip));
t = t.insert(scan_range(280300..280340, ChainTip), false);
assert_eq!(
t.into_vec(),
vec![
@ -1077,19 +1169,49 @@ mod tests {
])
.unwrap();
t = t.insert(scan_range(0..3, Scanned));
t = t.insert(scan_range(5..8, Scanned));
t = t.insert(scan_range(0..3, Scanned), false);
t = t.insert(scan_range(5..8, Scanned), false);
assert_eq!(t.into_vec(), vec![scan_range(0..10, Scanned)]);
}
#[test]
fn spanning_tree_force_rescans() {
use ScanPriority::*;
let mut t = spanning_tree(&[
(0..3, Historic),
(3..5, Scanned),
(5..7, ChainTip),
(7..10, Scanned),
])
.unwrap();
t = t.insert(scan_range(4..9, OpenAdjacent), true);
let expected = vec![
scan_range(0..3, Historic),
scan_range(3..4, Scanned),
scan_range(4..5, OpenAdjacent),
scan_range(5..7, ChainTip),
scan_range(7..9, OpenAdjacent),
scan_range(9..10, Scanned),
];
assert_eq!(t.clone().into_vec(), expected);
// An insert of an ignored range should not override a scanned range; the existing
// priority should prevail, and so the expected state of the tree is unchanged.
t = t.insert(scan_range(2..5, Ignored), true);
assert_eq!(t.into_vec(), expected);
}
#[test]
fn scan_complete() {
use ScanPriority::*;
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(birthday_at_sapling_activation)
.with_test_account(AccountBirthday::from_sapling_activation)
.build();
let dfvk = st.test_account_sapling().unwrap();
@ -1203,62 +1325,58 @@ mod tests {
fn create_account_creates_ignored_range() {
use ScanPriority::*;
let mut st = TestBuilder::new().build();
let mut st = TestBuilder::new()
.with_block_cache()
.with_test_account(|network| {
// We use Canopy activation as an arbitrary birthday height that's greater than Sapling
// activation.
let birthday_height = network.activation_height(NetworkUpgrade::Canopy).unwrap();
let frontier_position = Position::from((0x1 << 16) + 1234);
let frontier = Frontier::from_parts(
frontier_position,
Node::empty_leaf(),
vec![Node::empty_leaf(); frontier_position.past_ommer_count().into()],
)
.unwrap();
AccountBirthday::from_parts(birthday_height, frontier, None)
})
.build();
let (_, _, birthday) = st.test_account().unwrap();
let dfvk = st.test_account_sapling().unwrap();
let sap_active = st.sapling_activation_height();
// We use Canopy activation as an arbitrary birthday height that's greater than Sapling
// activation.
let birthday_height = st
.network()
.activation_height(NetworkUpgrade::Canopy)
.unwrap();
// call `init_blocks_table` to initialize the scan queue
init_blocks_table(
st.wallet_mut(),
birthday_height,
BlockHash([1; 32]),
1,
&[0x0, 0x0, 0x0],
)
.unwrap();
let seed = Secret::new(vec![0u8; 32]);
let (_, usk) = st.wallet_mut().create_account(&seed).unwrap();
let _dfvk = usk.to_unified_full_viewing_key().sapling().unwrap().clone();
let expected = vec![
// The range up to and including the wallet's birthday height is ignored.
scan_range(
u32::from(sap_active)..u32::from(birthday_height + 1),
Ignored,
),
// The range up to the wallet's birthday height is ignored.
scan_range(u32::from(sap_active)..u32::from(birthday.height()), Ignored),
];
assert_matches!(
suggest_scan_ranges(&st.wallet().conn, Ignored),
Ok(scan_ranges) if scan_ranges == expected
);
let actual = suggest_scan_ranges(&st.wallet().conn, Ignored).unwrap();
assert_eq!(actual, expected);
// Set up some shard history
// Set up some shard root history before the wallet birthday
st.wallet_mut()
.put_sapling_subtree_roots(
0,
&[
// Add the end of a commitment tree below the wallet birthday. We currently
// need to scan from this height up to the tip to make notes spendable, though
// this should not be necessary as we have added a frontier that should
// complete the left-hand side of the required shard; this can be fixed once we
// have proper account birthdays.
CommitmentTreeRoot::from_parts(
birthday_height - 1000,
// fake a hash, the value doesn't matter
Node::empty_leaf(),
),
],
&[CommitmentTreeRoot::from_parts(
birthday.height() - 1000,
// fake a hash, the value doesn't matter
Node::empty_leaf(),
)],
)
.unwrap();
st.generate_block_at(
birthday.height(),
BlockHash([0u8; 32]),
&dfvk,
AddressType::DefaultExternal,
Amount::const_from_i64(10000),
u64::from(birthday.sapling_frontier().value().unwrap().position() + 1)
.try_into()
.unwrap(),
);
st.scan_cached_blocks(birthday.height(), 1);
// Update the chain tip
let tip_height = st
.wallet()
@ -1269,28 +1387,28 @@ mod tests {
// Verify that the suggested scan ranges match what is expected
let expected = vec![
// The birthday height was "last scanned" (as the wallet birthday) so we verify 10
// blocks starting at that height.
// The birthday height is the "first to be scanned" (as the wallet birthday),
// so we verify 10 blocks starting at that height.
scan_range(
u32::from(birthday_height)..u32::from(birthday_height + 10),
u32::from(birthday.height())..u32::from(birthday.height() + 10),
Verify,
),
// The remainder of the shard after the verify segment is required in order to make
// notes spendable, so it has priority `ChainTip`
scan_range(
u32::from(birthday_height + 10)..u32::from(tip_height + 1),
u32::from(birthday.height() + 10)..u32::from(tip_height + 1),
ChainTip,
),
// The remainder of the shard prior to the birthday height must be scanned because the
// wallet doesn't know that it already has enough data from the initial frontier to
// avoid having to scan this range.
scan_range(
u32::from(birthday_height - 1000)..u32::from(birthday_height),
u32::from(birthday.height() - 1000)..u32::from(birthday.height()),
ChainTip,
),
// The range below the wallet's birthday height is ignored
scan_range(
u32::from(sap_active)..u32::from(birthday_height - 1000),
u32::from(sap_active)..u32::from(birthday.height() - 1000),
Ignored,
),
];
@ -1322,10 +1440,11 @@ mod tests {
{
let tx = st.wallet_mut().conn.transaction().unwrap();
replace_queue_entries(
replace_queue_entries::<SqliteClientError>(
&tx,
&(BlockHeight::from(150)..BlockHeight::from(160)),
vec![scan_range(150..160, Scanned)].into_iter(),
false,
)
.unwrap();
tx.commit().unwrap();
@ -1364,10 +1483,11 @@ mod tests {
{
let tx = st.wallet_mut().conn.transaction().unwrap();
replace_queue_entries(
replace_queue_entries::<SqliteClientError>(
&tx,
&(BlockHeight::from(90)..BlockHeight::from(100)),
vec![scan_range(90..100, Scanned)].into_iter(),
false,
)
.unwrap();
tx.commit().unwrap();