zebra/zebra-chain/src/subtree.rs

114 lines
3.7 KiB
Rust
Raw Normal View History

//! Struct representing Sapling/Orchard note commitment subtrees
fix(state): Use correct end heights for end of block subtrees during the full sync (#7566) * Avoid manual handling of previous sapling trees by using iterator windows instead * Avoid manual sapling subtree index handling by comparing prev and current subtree indexes instead * Simplify adding notes by using the exact number of remaining notes * Simplify by skipping the first block, because it can't complete a subtree * Re-use existing tree update code * Apply the sapling changes to orchard subtree updates * add a reverse database column family iterator function * Make skipping the lowest tree independent of iteration order * Move new subtree checks into the iterator, rename to end_height * Split subtree calculation into a new method * Split the calculate and write methods * Quickly check the first subtree before running the full upgrade * Do the quick checks every time Zebra runs, and refactor slow check error handling * Do quick checks for orchard as well * Make orchard tree upgrade match sapling upgrade code * Upgrade subtrees in reverse height order * Bump the database patch version so the upgrade runs again * Reset previous subtree upgrade data before doing this one * Add extra checks to subtree calculation to diagnose errors * Use correct heights for subtrees completed at the end of a block * Add even more checks to diagnose issues * Instrument upgrade methods to improve diagnostics * Prevent modification of re-used trees * Debug with subtree positions as well * Fix an off-by-one error with completed subtrees * Fix typos and confusing comments Co-authored-by: Marek <mail@marek.onl> * Fix mistaken previous tree handling and end tree comments * Remove unnecessary subtraction in remaining leaves calc * Log heights when assertions fail * Fix new subtree detection filter * Move new subtree check into a method, cleanup unused code * Remove redundant assertions * Wait for subtree upgrade before testing RPCs * Fix subtree search in quick check * Temporarily upgrade subtrees in forward height order * Clarify some comments * Fix missing test imports * Fix subtree logging * Add a comment about a potential hang with future upgrades * Fix zebrad var ownership * Log more info when add_subtrees.rs fails * cargo fmt --all * Fix unrelated clippy::unnecessary_unwrap * cargo clippy --fix --all-features --all-targets; cargo fmt --all * Stop the quick check depending on tree de-duplication * Refactor waiting for the upgrade into functions * Wait for state upgrades whenever the cached state is updated * Wait for the testnet upgrade in the right place * Fix unused variable * Fix a subtree detection bug and comments * Remove an early reference to reverse direction * Stop skipping subtrees completed at the end of blocks * Actually fix new subtree code --------- Co-authored-by: Marek <mail@marek.onl>
2023-09-19 07:49:36 -07:00
use std::{fmt, num::TryFromIntError};
change(state): Add note subtree indexes for new and existing blocks (#7437) * Copy the add_subtrees upgrade from the original branch * Copy the database write changes in shielded.rs from the original branch * Copy the tree API changes from the original branch * Simplify subtree APIs to avoid exposing frontiers * Fix a dead code warning by re-using existing methods * Use mpsc::Receiver<CancelFormatChange> in the subtree upgrade * Run the subtree upgrade on startup * Bump the database format version to 25.2.0 * Fix a confusing 'upgrade complete' log * Clarify some comments and error messages * Simplify prev_tree unwrap to avoid an (impossible?) concurrency bug * Use separate subtree writing functions * Use common note commitment list code * Fix subtree completion condition and add asserts * Simplify subtree API and avoid exposing Address * Fix API compatibility when Arcs are removed * Log when each subtree is added * If a format change is cancelled, don't mark the database as upgraded or do format checks * Log subtree progress about once every two minutes * Adds a state validity check for subtrees upgrade * Orchard is faster, decrease log interval * Clarify subtree index docs * Move a log to the correct location * Refactor subtree upgrade to remove duplicate inverted loop conditions * updates subtree state validity check * Add a subtree format check when there is no upgrade * Fix an off-by-one error with the final subtree check * Use error-level logs for database format checks * Skip format checks in tests that create invalid formats * fix state validity test * Add a concurrency comment to subtree by height methods * Add individual subtree state methods: reverts removing these methods in an earlier PR * fastmod "subtrees_by_index" "subtree_list_by_index_for_rpc" --------- Co-authored-by: arya2 <aryasolhi@gmail.com>
2023-09-05 09:52:06 -07:00
use serde::{Deserialize, Serialize};
use crate::block::Height;
#[cfg(any(test, feature = "proptest-impl"))]
use proptest_derive::Arbitrary;
/// Height at which Zebra tracks subtree roots
pub const TRACKED_SUBTREE_HEIGHT: u8 = 16;
/// A note commitment subtree index, used to identify a subtree in a shielded pool.
/// Also used to count subtrees.
#[derive(Copy, Clone, Debug, Eq, PartialEq, Ord, PartialOrd, Serialize, Deserialize)]
#[cfg_attr(any(test, feature = "proptest-impl"), derive(Arbitrary))]
#[serde(transparent)]
pub struct NoteCommitmentSubtreeIndex(pub u16);
fix(state): Use correct end heights for end of block subtrees during the full sync (#7566) * Avoid manual handling of previous sapling trees by using iterator windows instead * Avoid manual sapling subtree index handling by comparing prev and current subtree indexes instead * Simplify adding notes by using the exact number of remaining notes * Simplify by skipping the first block, because it can't complete a subtree * Re-use existing tree update code * Apply the sapling changes to orchard subtree updates * add a reverse database column family iterator function * Make skipping the lowest tree independent of iteration order * Move new subtree checks into the iterator, rename to end_height * Split subtree calculation into a new method * Split the calculate and write methods * Quickly check the first subtree before running the full upgrade * Do the quick checks every time Zebra runs, and refactor slow check error handling * Do quick checks for orchard as well * Make orchard tree upgrade match sapling upgrade code * Upgrade subtrees in reverse height order * Bump the database patch version so the upgrade runs again * Reset previous subtree upgrade data before doing this one * Add extra checks to subtree calculation to diagnose errors * Use correct heights for subtrees completed at the end of a block * Add even more checks to diagnose issues * Instrument upgrade methods to improve diagnostics * Prevent modification of re-used trees * Debug with subtree positions as well * Fix an off-by-one error with completed subtrees * Fix typos and confusing comments Co-authored-by: Marek <mail@marek.onl> * Fix mistaken previous tree handling and end tree comments * Remove unnecessary subtraction in remaining leaves calc * Log heights when assertions fail * Fix new subtree detection filter * Move new subtree check into a method, cleanup unused code * Remove redundant assertions * Wait for subtree upgrade before testing RPCs * Fix subtree search in quick check * Temporarily upgrade subtrees in forward height order * Clarify some comments * Fix missing test imports * Fix subtree logging * Add a comment about a potential hang with future upgrades * Fix zebrad var ownership * Log more info when add_subtrees.rs fails * cargo fmt --all * Fix unrelated clippy::unnecessary_unwrap * cargo clippy --fix --all-features --all-targets; cargo fmt --all * Stop the quick check depending on tree de-duplication * Refactor waiting for the upgrade into functions * Wait for state upgrades whenever the cached state is updated * Wait for the testnet upgrade in the right place * Fix unused variable * Fix a subtree detection bug and comments * Remove an early reference to reverse direction * Stop skipping subtrees completed at the end of blocks * Actually fix new subtree code --------- Co-authored-by: Marek <mail@marek.onl>
2023-09-19 07:49:36 -07:00
impl fmt::Display for NoteCommitmentSubtreeIndex {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str(&self.0.to_string())
}
}
impl From<u16> for NoteCommitmentSubtreeIndex {
fn from(value: u16) -> Self {
Self(value)
}
}
change(state): Add note subtree indexes for new and existing blocks (#7437) * Copy the add_subtrees upgrade from the original branch * Copy the database write changes in shielded.rs from the original branch * Copy the tree API changes from the original branch * Simplify subtree APIs to avoid exposing frontiers * Fix a dead code warning by re-using existing methods * Use mpsc::Receiver<CancelFormatChange> in the subtree upgrade * Run the subtree upgrade on startup * Bump the database format version to 25.2.0 * Fix a confusing 'upgrade complete' log * Clarify some comments and error messages * Simplify prev_tree unwrap to avoid an (impossible?) concurrency bug * Use separate subtree writing functions * Use common note commitment list code * Fix subtree completion condition and add asserts * Simplify subtree API and avoid exposing Address * Fix API compatibility when Arcs are removed * Log when each subtree is added * If a format change is cancelled, don't mark the database as upgraded or do format checks * Log subtree progress about once every two minutes * Adds a state validity check for subtrees upgrade * Orchard is faster, decrease log interval * Clarify subtree index docs * Move a log to the correct location * Refactor subtree upgrade to remove duplicate inverted loop conditions * updates subtree state validity check * Add a subtree format check when there is no upgrade * Fix an off-by-one error with the final subtree check * Use error-level logs for database format checks * Skip format checks in tests that create invalid formats * fix state validity test * Add a concurrency comment to subtree by height methods * Add individual subtree state methods: reverts removing these methods in an earlier PR * fastmod "subtrees_by_index" "subtree_list_by_index_for_rpc" --------- Co-authored-by: arya2 <aryasolhi@gmail.com>
2023-09-05 09:52:06 -07:00
impl TryFrom<u64> for NoteCommitmentSubtreeIndex {
type Error = TryFromIntError;
fn try_from(value: u64) -> Result<Self, Self::Error> {
u16::try_from(value).map(Self)
}
}
// If we want to automatically convert NoteCommitmentSubtreeIndex to the generic integer literal
// type, we can only implement conversion into u64. (Or u16, but not both.)
impl From<NoteCommitmentSubtreeIndex> for u64 {
fn from(value: NoteCommitmentSubtreeIndex) -> Self {
value.0.into()
}
}
// TODO:
// - consider defining sapling::SubtreeRoot and orchard::SubtreeRoot types or type wrappers,
// to avoid type confusion between the leaf Node and subtree root types.
/// Subtree root of Sapling or Orchard note commitment tree,
/// with its associated block height and subtree index.
#[derive(Copy, Clone, Debug, Eq, PartialEq)]
#[cfg_attr(any(test, feature = "proptest-impl"), derive(Arbitrary))]
pub struct NoteCommitmentSubtree<SubtreeRoot> {
/// Index of this subtree
pub index: NoteCommitmentSubtreeIndex,
/// Root of this subtree.
pub root: SubtreeRoot,
/// End boundary of this subtree, the block height of its last leaf.
pub end_height: Height,
}
impl<SubtreeRoot> NoteCommitmentSubtree<SubtreeRoot> {
/// Creates new [`NoteCommitmentSubtree`]
pub fn new(
index: impl Into<NoteCommitmentSubtreeIndex>,
end_height: Height,
root: SubtreeRoot,
) -> Self {
let index = index.into();
Self {
index,
end_height,
root,
}
}
/// Converts struct to [`NoteCommitmentSubtreeData`].
pub fn into_data(self) -> NoteCommitmentSubtreeData<SubtreeRoot> {
NoteCommitmentSubtreeData::new(self.end_height, self.root)
}
}
/// Subtree root of Sapling or Orchard note commitment tree, with block height, but without the subtree index.
/// Used for database key-value serialization, where the subtree index is the key, and this struct is the value.
#[derive(Copy, Clone, Debug, Eq, PartialEq, serde::Serialize)]
#[cfg_attr(any(test, feature = "proptest-impl"), derive(Arbitrary))]
pub struct NoteCommitmentSubtreeData<SubtreeRoot> {
/// Merkle root of the 2^16-leaf subtree.
pub root: SubtreeRoot,
/// Height of the block containing the note that completed this subtree.
pub end_height: Height,
}
impl<SubtreeRoot> NoteCommitmentSubtreeData<SubtreeRoot> {
/// Creates new [`NoteCommitmentSubtreeData`]
pub fn new(end_height: Height, root: SubtreeRoot) -> Self {
Self { end_height, root }
}
/// Creates new [`NoteCommitmentSubtree`] from a [`NoteCommitmentSubtreeData`] and index
pub fn with_index(
self,
index: impl Into<NoteCommitmentSubtreeIndex>,
) -> NoteCommitmentSubtree<SubtreeRoot> {
NoteCommitmentSubtree::new(index, self.end_height, self.root)
}
}