zebra/zebra-state/src/service/non_finalized_state/chain.rs

1482 lines
57 KiB
Rust
Raw Normal View History

//! [`Chain`] implements a single non-finalized blockchain,
//! starting at the finalized tip.
use std::{
cmp::Ordering,
collections::{BTreeMap, BTreeSet, HashMap, HashSet},
ops::{Deref, RangeInclusive},
sync::Arc,
};
use mset::MultiSet;
use tracing::instrument;
use zebra_chain::{
amount::{Amount, NegativeAllowed, NonNegative},
block::{self, Height},
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
history_tree::HistoryTree,
orchard,
parameters::Network,
primitives::Groth16Proof,
sapling, sprout,
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
transaction::Transaction::*,
transaction::{self, Transaction},
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
transparent,
value_balance::ValueBalance,
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
work::difficulty::PartialCumulativeWork,
};
use crate::{
service::check, ContextuallyValidBlock, HashOrHeight, OutputLocation, TransactionLocation,
ValidateContextError,
};
use self::index::TransparentTransfers;
pub mod index;
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
#[derive(Debug, Clone)]
pub struct Chain {
// The function `eq_internal_state` must be updated every time a field is added to [`Chain`].
/// The configured network for this chain.
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
network: Network,
/// The contextually valid blocks which form this non-finalized partial chain, in height order.
pub(crate) blocks: BTreeMap<block::Height, ContextuallyValidBlock>,
/// An index of block heights for each block hash in `blocks`.
pub height_by_hash: HashMap<block::Hash, block::Height>,
/// An index of [`TransactionLocation`]s for each transaction hash in `blocks`.
pub tx_by_hash: HashMap<transaction::Hash, TransactionLocation>,
/// The [`transparent::Utxo`]s created by `blocks`.
///
/// Note that these UTXOs may not be unspent.
/// Outputs can be spent by later transactions or blocks in the chain.
//
// TODO: replace OutPoint with OutputLocation?
pub(crate) created_utxos: HashMap<transparent::OutPoint, transparent::OrderedUtxo>,
/// The [`transparent::OutPoint`]s spent by `blocks`,
/// including those created by earlier transactions or blocks in the chain.
Reject UTXO double spends (#2511) * Reject transparent output double-spends Check that transparent spends use unspent outputs from: * earlier transaction in the same block, * earlier blocks in the parent non-finalized chain, or * the finalized state. * Fixup UTXOs in proptests * Add a comment * Clarify a consensus rule implementation * Fix an incorrect comment * Fix an incorrect error message * Clarify a comment * Document `unspent_utxos` * Simplify the UTXO check Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Further simplify and fix the UTXO check - split each error case into a separate check - combine `contains` and `insert` - add a missing check against the non-finalized unspent UTXOs - rename arguments and edit error strings for clarity * Share test methods between check test modules * Make some chain fields available to tests * Make error field names consistent with transparent::Input * WIP: Add tests for UTXO double-spends - accept output and spend in the same block - accept output and spend in a later block - reject output and double-spend all in the same block - reject output then double-spend in a later block - reject output, spend, then double-spend all in different blocks * Use Extend rather than multiple pushes Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Use Extend for more pushes * Limit the number of proptest cases, to speed up tests * Test rejection of UTXOs that were never in the chain * Test rejection of spends of later transactions in the same block Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-22 16:40:15 -07:00
pub(crate) spent_utxos: HashSet<transparent::OutPoint>,
/// The Sprout note commitment tree of the tip of this [`Chain`],
/// including all finalized notes, and the non-finalized notes in this chain.
pub(super) sprout_note_commitment_tree: sprout::tree::NoteCommitmentTree,
/// The Sprout note commitment tree for each anchor.
/// This is required for interstitial states.
pub(crate) sprout_trees_by_anchor:
HashMap<sprout::tree::Root, sprout::tree::NoteCommitmentTree>,
/// The Sapling note commitment tree of the tip of this [`Chain`],
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
/// including all finalized notes, and the non-finalized notes in this chain.
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
pub(super) sapling_note_commitment_tree: sapling::tree::NoteCommitmentTree,
/// The Sapling note commitment tree for each height.
pub(crate) sapling_trees_by_height: BTreeMap<block::Height, sapling::tree::NoteCommitmentTree>,
/// The Orchard note commitment tree of the tip of this [`Chain`],
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
/// including all finalized notes, and the non-finalized notes in this chain.
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
pub(super) orchard_note_commitment_tree: orchard::tree::NoteCommitmentTree,
/// The Orchard note commitment tree for each height.
pub(crate) orchard_trees_by_height: BTreeMap<block::Height, orchard::tree::NoteCommitmentTree>,
/// The ZIP-221 history tree of the tip of this [`Chain`],
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
/// including all finalized blocks, and the non-finalized `blocks` in this chain.
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
pub(crate) history_tree: HistoryTree,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
/// The Sprout anchors created by `blocks`.
pub(crate) sprout_anchors: MultiSet<sprout::tree::Root>,
/// The Sprout anchors created by each block in `blocks`.
pub(crate) sprout_anchors_by_height: BTreeMap<block::Height, sprout::tree::Root>,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
/// The Sapling anchors created by `blocks`.
pub(crate) sapling_anchors: MultiSet<sapling::tree::Root>,
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
/// The Sapling anchors created by each block in `blocks`.
2021-11-30 08:05:35 -08:00
pub(crate) sapling_anchors_by_height: BTreeMap<block::Height, sapling::tree::Root>,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
/// The Orchard anchors created by `blocks`.
pub(crate) orchard_anchors: MultiSet<orchard::tree::Root>,
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
/// The Orchard anchors created by each block in `blocks`.
2021-11-30 08:05:35 -08:00
pub(crate) orchard_anchors_by_height: BTreeMap<block::Height, orchard::tree::Root>,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
/// The Sprout nullifiers revealed by `blocks`.
pub(super) sprout_nullifiers: HashSet<sprout::Nullifier>,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
/// The Sapling nullifiers revealed by `blocks`.
pub(super) sapling_nullifiers: HashSet<sapling::Nullifier>,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
/// The Orchard nullifiers revealed by `blocks`.
pub(super) orchard_nullifiers: HashSet<orchard::Nullifier>,
/// Partial transparent address index data from `blocks`.
pub(super) partial_transparent_transfers: HashMap<transparent::Address, TransparentTransfers>,
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
/// The cumulative work represented by `blocks`.
///
/// Since the best chain is determined by the largest cumulative work,
/// the work represented by finalized blocks can be ignored,
/// because they are common to all non-finalized chains.
pub(super) partial_cumulative_work: PartialCumulativeWork,
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
/// The chain value pool balances of the tip of this [`Chain`],
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
/// including the block value pool changes from all finalized blocks,
/// and the non-finalized blocks in this chain.
///
/// When a new chain is created from the finalized tip,
/// it is initialized with the finalized tip chain value pool balances.
pub(crate) chain_value_pools: ValueBalance<NonNegative>,
}
impl Chain {
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
/// Create a new Chain with the given trees and network.
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
pub(crate) fn new(
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
network: Network,
sprout_note_commitment_tree: sprout::tree::NoteCommitmentTree,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
sapling_note_commitment_tree: sapling::tree::NoteCommitmentTree,
orchard_note_commitment_tree: orchard::tree::NoteCommitmentTree,
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
history_tree: HistoryTree,
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
finalized_tip_chain_value_pools: ValueBalance<NonNegative>,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
) -> Self {
Self {
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
network,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
blocks: Default::default(),
height_by_hash: Default::default(),
tx_by_hash: Default::default(),
created_utxos: Default::default(),
sprout_note_commitment_tree,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
sapling_note_commitment_tree,
orchard_note_commitment_tree,
spent_utxos: Default::default(),
sprout_anchors: MultiSet::new(),
sprout_anchors_by_height: Default::default(),
sprout_trees_by_anchor: Default::default(),
sapling_anchors: MultiSet::new(),
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
sapling_anchors_by_height: Default::default(),
sapling_trees_by_height: Default::default(),
orchard_anchors: MultiSet::new(),
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
orchard_anchors_by_height: Default::default(),
orchard_trees_by_height: Default::default(),
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
sprout_nullifiers: Default::default(),
sapling_nullifiers: Default::default(),
orchard_nullifiers: Default::default(),
partial_transparent_transfers: Default::default(),
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
partial_cumulative_work: Default::default(),
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
history_tree,
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
chain_value_pools: finalized_tip_chain_value_pools,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
}
}
/// Is the internal state of `self` the same as `other`?
///
/// [`Chain`] has custom [`Eq`] and [`Ord`] implementations based on proof of work,
/// which are used to select the best chain. So we can't derive [`Eq`] for [`Chain`].
///
/// Unlike the custom trait impls, this method returns `true` if the entire internal state
/// of two chains is equal.
///
/// If the internal states are different, it returns `false`,
/// even if the blocks in the two chains are equal.
#[cfg(test)]
pub(crate) fn eq_internal_state(&self, other: &Chain) -> bool {
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
use zebra_chain::history_tree::NonEmptyHistoryTree;
// blocks, heights, hashes
self.blocks == other.blocks &&
self.height_by_hash == other.height_by_hash &&
self.tx_by_hash == other.tx_by_hash &&
// transparent UTXOs
self.created_utxos == other.created_utxos &&
self.spent_utxos == other.spent_utxos &&
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
// note commitment trees
self.sprout_note_commitment_tree.root() == other.sprout_note_commitment_tree.root() &&
self.sprout_trees_by_anchor == other.sprout_trees_by_anchor &&
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
self.sapling_note_commitment_tree.root() == other.sapling_note_commitment_tree.root() &&
self.sapling_trees_by_height== other.sapling_trees_by_height &&
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
self.orchard_note_commitment_tree.root() == other.orchard_note_commitment_tree.root() &&
self.orchard_trees_by_height== other.orchard_trees_by_height &&
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
// history tree
self.history_tree.as_ref().map(NonEmptyHistoryTree::hash) == other.history_tree.as_ref().map(NonEmptyHistoryTree::hash) &&
// anchors
self.sprout_anchors == other.sprout_anchors &&
self.sprout_anchors_by_height == other.sprout_anchors_by_height &&
self.sapling_anchors == other.sapling_anchors &&
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
self.sapling_anchors_by_height == other.sapling_anchors_by_height &&
self.orchard_anchors == other.orchard_anchors &&
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
self.orchard_anchors_by_height == other.orchard_anchors_by_height &&
// nullifiers
self.sprout_nullifiers == other.sprout_nullifiers &&
self.sapling_nullifiers == other.sapling_nullifiers &&
self.orchard_nullifiers == other.orchard_nullifiers &&
// transparent address indexes
self.partial_transparent_transfers == other.partial_transparent_transfers &&
// proof of work
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
self.partial_cumulative_work == other.partial_cumulative_work &&
// chain value pool balances
self.chain_value_pools == other.chain_value_pools
}
/// Push a contextually valid non-finalized block into this chain as the new tip.
///
/// If the block is invalid, drops this chain, and returns an error.
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
///
/// Note: a [`ContextuallyValidBlock`] isn't actually contextually valid until
/// [`Self::update_chain_tip_with`] returns success.
#[instrument(level = "debug", skip(self, block), fields(block = %block.block))]
pub fn push(mut self, block: ContextuallyValidBlock) -> Result<Chain, ValidateContextError> {
// update cumulative data members
self.update_chain_tip_with(&block)?;
tracing::debug!(block = %block.block, "adding block to chain");
self.blocks.insert(block.height, block);
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
Ok(self)
}
/// Remove the lowest height block of the non-finalized portion of a chain.
#[instrument(level = "debug", skip(self))]
pub(crate) fn pop_root(&mut self) -> ContextuallyValidBlock {
let block_height = self.non_finalized_root_height();
// remove the lowest height block from self.blocks
let block = self
.blocks
.remove(&block_height)
.expect("only called while blocks is populated");
// update cumulative data members
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
self.revert_chain_with(&block, RevertPosition::Root);
// return the prepared block
block
}
/// Returns the height of the chain root.
pub fn non_finalized_root_height(&self) -> block::Height {
self.blocks
.keys()
.next()
.cloned()
.expect("only called while blocks is populated")
}
/// Fork a chain at the block with the given hash, if it is part of this
/// chain.
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
///
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
/// The trees must match the trees of the finalized tip and are used
/// to rebuild them after the fork.
#[allow(clippy::unwrap_in_result)]
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
pub fn fork(
&self,
fork_tip: block::Hash,
sprout_note_commitment_tree: sprout::tree::NoteCommitmentTree,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
sapling_note_commitment_tree: sapling::tree::NoteCommitmentTree,
orchard_note_commitment_tree: orchard::tree::NoteCommitmentTree,
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
history_tree: HistoryTree,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
) -> Result<Option<Self>, ValidateContextError> {
if !self.height_by_hash.contains_key(&fork_tip) {
2021-06-29 20:52:31 -07:00
return Ok(None);
}
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
let mut forked = self.with_trees(
sprout_note_commitment_tree,
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
sapling_note_commitment_tree,
orchard_note_commitment_tree,
history_tree,
);
while forked.non_finalized_tip_hash() != fork_tip {
forked.pop_tip();
}
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
// Rebuild the note commitment trees, starting from the finalized tip tree.
// TODO: change to a more efficient approach by removing nodes
// from the tree of the original chain (in [`Self::pop_tip`]).
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
// See https://github.com/ZcashFoundation/zebra/issues/2378
for block in forked.blocks.values() {
for transaction in block.block.transactions.iter() {
for sprout_note_commitment in transaction.sprout_note_commitments() {
forked
.sprout_note_commitment_tree
.append(*sprout_note_commitment)
.expect("must work since it was already appended before the fork");
}
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
for sapling_note_commitment in transaction.sapling_note_commitments() {
forked
.sapling_note_commitment_tree
.append(*sapling_note_commitment)
.expect("must work since it was already appended before the fork");
}
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
for orchard_note_commitment in transaction.orchard_note_commitments() {
forked
.orchard_note_commitment_tree
.append(*orchard_note_commitment)
.expect("must work since it was already appended before the fork");
}
}
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
// Note that anchors don't need to be recreated since they are already
// handled in revert_chain_state_with.
let sapling_root = forked
.sapling_anchors_by_height
.get(&block.height)
.expect("Sapling anchors must exist for pre-fork blocks");
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
let orchard_root = forked
.orchard_anchors_by_height
.get(&block.height)
.expect("Orchard anchors must exist for pre-fork blocks");
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
forked.history_tree.push(
self.network,
block.block.clone(),
*sapling_root,
*orchard_root,
)?;
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
}
2021-06-29 20:52:31 -07:00
Ok(Some(forked))
}
/// Returns the [`ContextuallyValidBlock`] with [`block::Hash`] or
/// [`Height`](zebra_chain::block::Height), if it exists in this chain.
pub fn block(&self, hash_or_height: HashOrHeight) -> Option<&ContextuallyValidBlock> {
let height =
hash_or_height.height_or_else(|hash| self.height_by_hash.get(&hash).cloned())?;
self.blocks.get(&height)
}
/// Returns the [`Transaction`] with [`transaction::Hash`], if it exists in this chain.
pub fn transaction(
&self,
hash: transaction::Hash,
) -> Option<(&Arc<Transaction>, block::Height)> {
self.tx_by_hash.get(&hash).map(|tx_loc| {
(
&self.blocks[&tx_loc.height].block.transactions[tx_loc.index.as_usize()],
tx_loc.height,
)
})
}
/// Returns the [`Transaction`] at [`TransactionLocation`], if it exists in this chain.
#[allow(dead_code)]
pub fn transaction_by_loc(&self, tx_loc: TransactionLocation) -> Option<&Arc<Transaction>> {
self.blocks
.get(&tx_loc.height)?
.block
.transactions
.get(tx_loc.index.as_usize())
}
/// Returns the [`transaction::Hash`] for the transaction at [`TransactionLocation`],
/// if it exists in this chain.
#[allow(dead_code)]
pub fn transaction_hash_by_loc(
&self,
tx_loc: TransactionLocation,
) -> Option<&transaction::Hash> {
self.blocks
.get(&tx_loc.height)?
.transaction_hashes
.get(tx_loc.index.as_usize())
}
/// Returns the non-finalized tip block hash and height.
#[allow(dead_code)]
pub fn non_finalized_tip(&self) -> (block::Hash, block::Height) {
(
self.non_finalized_tip_hash(),
self.non_finalized_tip_height(),
)
}
feat(rpc): Implement `z_gettreestate` RPC (#3990) * Impl the elementary structure of the `z_gettreestate` RPC * Fix merging bugs * Fix a merge bug * Fix a merge bug * Move a derive attribute Co-authored-by: teor <teor@riseup.net> * Clarify the support of negative heights * Add Orchard note commitment trees to the response * Add the time to the response * Finalize the `z_gettreestate` RPC * Add a note that verified blocks have coinbase height * Refactor `from_str` for `HashOrHeight` * Fix a mistake in the docs Co-authored-by: teor <teor@riseup.net> * Clarify request types Co-authored-by: teor <teor@riseup.net> * Simplify `hash_or_height` conversion to height Co-authored-by: teor <teor@riseup.net> * Add a TODO about optimization Co-authored-by: teor <teor@riseup.net> * Add a doc comment * Make sure Sapling & Orchard trees don't get mixed up * Serialize Sapling commitment trees * Refactor some comments * Serialize Orchard commitment trees * Serialize block heights * Simplify the serialization of commitment trees * Remove the block time from the RPC response * Simplify the serialization of block heights * Put Sapling & Orchard requests together * Remove a redundant TODO * Add block times to the RPC response * Derive `Clone, Debug, Eq, PartialEq` for `GetTreestate` Co-authored-by: teor <teor@riseup.net> * Derive `Clone`, `Debug`, `Eq` and `PartialEq` for `SerializedTree` * Document the fields of `GetTreestate` * Skip the serialization of empty trees This ensures compatibility with `zcashd` in the `z_gettreestate` RPC. * Document the `impl` of `merkle_tree::Hashable` for nodes * Make the structure of the JSON response consistent with `zcashd` * Derive `Eq` for nodes Co-authored-by: teor <teor@riseup.net> * Convert Sapling commitment trees to a format compatible with zcashd * Refactor the conversion of Sapling commitment trees * Refactor some comments * Refactor comments * Add a description of the conversion Co-authored-by: Conrado Gouvea <conrado@zfnd.org> * Fix comment indenting * Document the conversion between the dense and sparse formats Co-authored-by: teor <teor@riseup.net> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2022-05-12 00:00:12 -07:00
/// Returns the Sapling
/// [`NoteCommitmentTree`](sapling::tree::NoteCommitmentTree) specified by a
/// hash or height, if it exists in the non-finalized `chain`.
pub fn sapling_tree(
&self,
hash_or_height: HashOrHeight,
) -> Option<&sapling::tree::NoteCommitmentTree> {
let height =
hash_or_height.height_or_else(|hash| self.height_by_hash.get(&hash).cloned())?;
self.sapling_trees_by_height.get(&height)
}
/// Returns the Orchard
/// [`NoteCommitmentTree`](orchard::tree::NoteCommitmentTree) specified by a
/// hash or height, if it exists in the non-finalized `chain`.
pub fn orchard_tree(
&self,
hash_or_height: HashOrHeight,
) -> Option<&orchard::tree::NoteCommitmentTree> {
let height =
hash_or_height.height_or_else(|hash| self.height_by_hash.get(&hash).cloned())?;
self.orchard_trees_by_height.get(&height)
}
/// Returns the block hash of the tip block.
pub fn non_finalized_tip_hash(&self) -> block::Hash {
self.blocks
.values()
.next_back()
.expect("only called while blocks is populated")
.hash
}
/// Returns the non-finalized root block hash and height.
#[allow(dead_code)]
pub fn non_finalized_root(&self) -> (block::Hash, block::Height) {
(
self.non_finalized_root_hash(),
self.non_finalized_root_height(),
)
}
/// Returns the block hash of the non-finalized root block.
pub fn non_finalized_root_hash(&self) -> block::Hash {
self.blocks
.values()
.next()
.expect("only called while blocks is populated")
.hash
}
/// Returns the block hash of the `n`th block from the non-finalized root.
///
/// This is the block at `non_finalized_root_height() + n`.
#[allow(dead_code)]
pub fn non_finalized_nth_hash(&self, n: usize) -> Option<block::Hash> {
self.blocks.values().nth(n).map(|block| block.hash)
}
/// Remove the highest height block of the non-finalized portion of a chain.
fn pop_tip(&mut self) {
let block_height = self.non_finalized_tip_height();
let block = self
.blocks
.remove(&block_height)
.expect("only called while blocks is populated");
assert!(
!self.blocks.is_empty(),
"Non-finalized chains must have at least one block to be valid"
);
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
self.revert_chain_with(&block, RevertPosition::Tip);
}
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
/// Return the non-finalized tip height for this chain.
///
/// # Panics
///
/// Panics if called while the chain is empty,
/// or while the chain is updating its internal state with the first block.
pub fn non_finalized_tip_height(&self) -> block::Height {
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
self.max_block_height()
.expect("only called while blocks is populated")
}
2020-11-06 14:31:25 -08:00
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
/// Return the non-finalized tip height for this chain,
/// or `None` if `self.blocks` is empty.
fn max_block_height(&self) -> Option<block::Height> {
self.blocks.keys().next_back().cloned()
}
/// Return the non-finalized tip block for this chain,
/// or `None` if `self.blocks` is empty.
pub fn tip_block(&self) -> Option<&ContextuallyValidBlock> {
self.blocks.values().next_back()
}
/// Returns true if the non-finalized part of this chain is empty.
2020-11-06 14:31:25 -08:00
pub fn is_empty(&self) -> bool {
self.blocks.is_empty()
}
Reject UTXO double spends (#2511) * Reject transparent output double-spends Check that transparent spends use unspent outputs from: * earlier transaction in the same block, * earlier blocks in the parent non-finalized chain, or * the finalized state. * Fixup UTXOs in proptests * Add a comment * Clarify a consensus rule implementation * Fix an incorrect comment * Fix an incorrect error message * Clarify a comment * Document `unspent_utxos` * Simplify the UTXO check Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Further simplify and fix the UTXO check - split each error case into a separate check - combine `contains` and `insert` - add a missing check against the non-finalized unspent UTXOs - rename arguments and edit error strings for clarity * Share test methods between check test modules * Make some chain fields available to tests * Make error field names consistent with transparent::Input * WIP: Add tests for UTXO double-spends - accept output and spend in the same block - accept output and spend in a later block - reject output and double-spend all in the same block - reject output then double-spend in a later block - reject output, spend, then double-spend all in different blocks * Use Extend rather than multiple pushes Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Use Extend for more pushes * Limit the number of proptest cases, to speed up tests * Test rejection of UTXOs that were never in the chain * Test rejection of spends of later transactions in the same block Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-22 16:40:15 -07:00
/// Returns the non-finalized length of this chain.
#[allow(dead_code)]
pub fn len(&self) -> usize {
self.blocks.len()
}
Reject UTXO double spends (#2511) * Reject transparent output double-spends Check that transparent spends use unspent outputs from: * earlier transaction in the same block, * earlier blocks in the parent non-finalized chain, or * the finalized state. * Fixup UTXOs in proptests * Add a comment * Clarify a consensus rule implementation * Fix an incorrect comment * Fix an incorrect error message * Clarify a comment * Document `unspent_utxos` * Simplify the UTXO check Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Further simplify and fix the UTXO check - split each error case into a separate check - combine `contains` and `insert` - add a missing check against the non-finalized unspent UTXOs - rename arguments and edit error strings for clarity * Share test methods between check test modules * Make some chain fields available to tests * Make error field names consistent with transparent::Input * WIP: Add tests for UTXO double-spends - accept output and spend in the same block - accept output and spend in a later block - reject output and double-spend all in the same block - reject output then double-spend in a later block - reject output, spend, then double-spend all in different blocks * Use Extend rather than multiple pushes Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Use Extend for more pushes * Limit the number of proptest cases, to speed up tests * Test rejection of UTXOs that were never in the chain * Test rejection of spends of later transactions in the same block Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-22 16:40:15 -07:00
/// Returns the unspent transaction outputs (UTXOs) in this non-finalized chain.
///
/// Callers should also check the finalized state for available UTXOs.
/// If UTXOs remain unspent when a block is finalized, they are stored in the finalized state,
/// and removed from the relevant chain(s).
pub fn unspent_utxos(&self) -> HashMap<transparent::OutPoint, transparent::OrderedUtxo> {
Reject UTXO double spends (#2511) * Reject transparent output double-spends Check that transparent spends use unspent outputs from: * earlier transaction in the same block, * earlier blocks in the parent non-finalized chain, or * the finalized state. * Fixup UTXOs in proptests * Add a comment * Clarify a consensus rule implementation * Fix an incorrect comment * Fix an incorrect error message * Clarify a comment * Document `unspent_utxos` * Simplify the UTXO check Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Further simplify and fix the UTXO check - split each error case into a separate check - combine `contains` and `insert` - add a missing check against the non-finalized unspent UTXOs - rename arguments and edit error strings for clarity * Share test methods between check test modules * Make some chain fields available to tests * Make error field names consistent with transparent::Input * WIP: Add tests for UTXO double-spends - accept output and spend in the same block - accept output and spend in a later block - reject output and double-spend all in the same block - reject output then double-spend in a later block - reject output, spend, then double-spend all in different blocks * Use Extend rather than multiple pushes Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Use Extend for more pushes * Limit the number of proptest cases, to speed up tests * Test rejection of UTXOs that were never in the chain * Test rejection of spends of later transactions in the same block Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-22 16:40:15 -07:00
let mut unspent_utxos = self.created_utxos.clone();
unspent_utxos.retain(|out_point, _utxo| !self.spent_utxos.contains(out_point));
Reject UTXO double spends (#2511) * Reject transparent output double-spends Check that transparent spends use unspent outputs from: * earlier transaction in the same block, * earlier blocks in the parent non-finalized chain, or * the finalized state. * Fixup UTXOs in proptests * Add a comment * Clarify a consensus rule implementation * Fix an incorrect comment * Fix an incorrect error message * Clarify a comment * Document `unspent_utxos` * Simplify the UTXO check Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Further simplify and fix the UTXO check - split each error case into a separate check - combine `contains` and `insert` - add a missing check against the non-finalized unspent UTXOs - rename arguments and edit error strings for clarity * Share test methods between check test modules * Make some chain fields available to tests * Make error field names consistent with transparent::Input * WIP: Add tests for UTXO double-spends - accept output and spend in the same block - accept output and spend in a later block - reject output and double-spend all in the same block - reject output then double-spend in a later block - reject output, spend, then double-spend all in different blocks * Use Extend rather than multiple pushes Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Use Extend for more pushes * Limit the number of proptest cases, to speed up tests * Test rejection of UTXOs that were never in the chain * Test rejection of spends of later transactions in the same block Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-22 16:40:15 -07:00
unspent_utxos
}
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
// Address index queries
/// Returns the transparent transfers for `addresses` in this non-finalized chain.
///
/// If none of the addresses have an address index, returns an empty iterator.
///
/// # Correctness
///
/// Callers should apply the returned indexes to the corresponding finalized state indexes.
///
/// The combined result will only be correct if the chains match.
/// The exact type of match varies by query.
pub fn partial_transparent_indexes<'a>(
&'a self,
addresses: &'a HashSet<transparent::Address>,
) -> impl Iterator<Item = &TransparentTransfers> {
addresses
.iter()
.copied()
.flat_map(|address| self.partial_transparent_transfers.get(&address))
}
/// Returns the transparent balance change for `addresses` in this non-finalized chain.
///
/// If the balance doesn't change for any of the addresses, returns zero.
///
/// # Correctness
///
/// Callers should apply this balance change to the finalized state balance for `addresses`.
///
/// The total balance will only be correct if this partial chain matches the finalized state.
/// Specifically, the root of this partial chain must be a child block of the finalized tip.
pub fn partial_transparent_balance_change(
&self,
addresses: &HashSet<transparent::Address>,
) -> Amount<NegativeAllowed> {
let balance_change: Result<Amount<NegativeAllowed>, _> = self
.partial_transparent_indexes(addresses)
.map(|transfers| transfers.balance())
.sum();
balance_change.expect(
"unexpected amount overflow: value balances are valid, so partial sum should be valid",
)
}
/// Returns the transparent UTXO changes for `addresses` in this non-finalized chain.
///
/// If the UTXOs don't change for any of the addresses, returns empty lists.
///
/// # Correctness
///
/// Callers should apply these non-finalized UTXO changes to the finalized state UTXOs.
///
/// The UTXOs will only be correct if the non-finalized chain matches or overlaps with
/// the finalized state.
///
/// Specifically, a block in the partial chain must be a child block of the finalized tip.
/// (But the child block does not have to be the partial chain root.)
pub fn partial_transparent_utxo_changes(
&self,
addresses: &HashSet<transparent::Address>,
) -> (
BTreeMap<OutputLocation, transparent::Output>,
BTreeSet<OutputLocation>,
) {
let created_utxos = self
.partial_transparent_indexes(addresses)
.flat_map(|transfers| transfers.created_utxos())
.map(|(out_loc, output)| (*out_loc, output.clone()))
.collect();
let spent_utxos = self
.partial_transparent_indexes(addresses)
.flat_map(|transfers| transfers.spent_utxos())
.cloned()
.collect();
(created_utxos, spent_utxos)
}
/// Returns the [`transaction::Hash`]es used by `addresses` to receive or spend funds,
/// in the non-finalized chain, filtered using the `query_height_range`.
///
/// If none of the addresses receive or spend funds in this partial chain, returns an empty list.
///
/// # Correctness
///
/// Callers should combine these non-finalized transactions with the finalized state transactions.
///
/// The transaction IDs will only be correct if the non-finalized chain matches or overlaps with
/// the finalized state.
///
/// Specifically, a block in the partial chain must be a child block of the finalized tip.
/// (But the child block does not have to be the partial chain root.)
///
/// This condition does not apply if there is only one address.
/// Since address transactions are only appended by blocks,
/// and the finalized state query reads them in order,
/// it is impossible to get inconsistent transactions for a single address.
pub fn partial_transparent_tx_ids(
&self,
addresses: &HashSet<transparent::Address>,
query_height_range: RangeInclusive<Height>,
) -> BTreeMap<TransactionLocation, transaction::Hash> {
self.partial_transparent_indexes(addresses)
.flat_map(|transfers| transfers.tx_ids(&self.tx_by_hash, query_height_range.clone()))
.collect()
}
// Cloning
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
/// Clone the Chain but not the history and note commitment trees, using
/// the specified trees instead.
///
/// Useful when forking, where the trees are rebuilt anyway.
fn with_trees(
&self,
sprout_note_commitment_tree: sprout::tree::NoteCommitmentTree,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
sapling_note_commitment_tree: sapling::tree::NoteCommitmentTree,
orchard_note_commitment_tree: orchard::tree::NoteCommitmentTree,
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
history_tree: HistoryTree,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
) -> Self {
Chain {
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
network: self.network,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
blocks: self.blocks.clone(),
height_by_hash: self.height_by_hash.clone(),
tx_by_hash: self.tx_by_hash.clone(),
created_utxos: self.created_utxos.clone(),
spent_utxos: self.spent_utxos.clone(),
sprout_note_commitment_tree,
sprout_trees_by_anchor: self.sprout_trees_by_anchor.clone(),
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
sapling_note_commitment_tree,
sapling_trees_by_height: self.sapling_trees_by_height.clone(),
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
orchard_note_commitment_tree,
orchard_trees_by_height: self.orchard_trees_by_height.clone(),
sprout_anchors: self.sprout_anchors.clone(),
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
sapling_anchors: self.sapling_anchors.clone(),
orchard_anchors: self.orchard_anchors.clone(),
sprout_anchors_by_height: self.sprout_anchors_by_height.clone(),
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
sapling_anchors_by_height: self.sapling_anchors_by_height.clone(),
orchard_anchors_by_height: self.orchard_anchors_by_height.clone(),
sprout_nullifiers: self.sprout_nullifiers.clone(),
sapling_nullifiers: self.sapling_nullifiers.clone(),
orchard_nullifiers: self.orchard_nullifiers.clone(),
partial_transparent_transfers: self.partial_transparent_transfers.clone(),
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
partial_cumulative_work: self.partial_cumulative_work,
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
history_tree,
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
chain_value_pools: self.chain_value_pools,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
}
}
}
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
/// The revert position being performed on a chain.
#[derive(Copy, Clone, Debug, PartialEq, Eq, Hash)]
enum RevertPosition {
/// The chain root is being reverted via [`Chain::pop_root`], when a block
/// is finalized.
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
Root,
/// The chain tip is being reverted via [`Chain::pop_tip`],
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
/// when a chain is forked.
Tip,
}
/// Helper trait to organize inverse operations done on the [`Chain`] type.
///
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
/// Used to overload update and revert methods, based on the type of the argument,
/// and the position of the removed block in the chain.
///
/// This trait was motivated by the length of the `push`, [`Chain::pop_root`],
/// and [`Chain::pop_tip`] functions, and fear that it would be easy to
/// introduce bugs when updating them, unless the code was reorganized to keep
/// related operations adjacent to each other.
trait UpdateWith<T> {
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
/// When `T` is added to the chain tip,
/// update [`Chain`] cumulative data members to add data that are derived from `T`.
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
fn update_chain_tip_with(&mut self, _: &T) -> Result<(), ValidateContextError>;
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
/// When `T` is removed from `position` in the chain,
/// revert [`Chain`] cumulative data members to remove data that are derived from `T`.
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
fn revert_chain_with(&mut self, _: &T, position: RevertPosition);
}
impl UpdateWith<ContextuallyValidBlock> for Chain {
#[instrument(skip(self, contextually_valid), fields(block = %contextually_valid.block))]
#[allow(clippy::unwrap_in_result)]
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
fn update_chain_tip_with(
2021-06-29 20:52:31 -07:00
&mut self,
contextually_valid: &ContextuallyValidBlock,
2021-06-29 20:52:31 -07:00
) -> Result<(), ValidateContextError> {
let (
block,
hash,
height,
new_outputs,
spent_outputs,
transaction_hashes,
chain_value_pool_change,
) = (
contextually_valid.block.as_ref(),
contextually_valid.hash,
contextually_valid.height,
&contextually_valid.new_outputs,
&contextually_valid.spent_outputs,
&contextually_valid.transaction_hashes,
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
&contextually_valid.chain_value_pool_change,
);
// add hash to height_by_hash
let prior_height = self.height_by_hash.insert(hash, height);
assert!(
prior_height.is_none(),
"block heights must be unique within a single chain"
);
// add work to partial cumulative work
let block_work = block
.header
.difficulty_threshold
.to_work()
.expect("work has already been validated");
self.partial_cumulative_work += block_work;
// for each transaction in block
for (transaction_index, (transaction, transaction_hash)) in block
.transactions
.iter()
.zip(transaction_hashes.iter().cloned())
.enumerate()
{
let (
inputs,
outputs,
joinsplit_data,
sapling_shielded_data_per_spend_anchor,
sapling_shielded_data_shared_anchor,
orchard_shielded_data,
) = match transaction.deref() {
V4 {
inputs,
outputs,
joinsplit_data,
sapling_shielded_data,
..
} => (inputs, outputs, joinsplit_data, sapling_shielded_data, &None, &None),
V5 {
inputs,
outputs,
sapling_shielded_data,
orchard_shielded_data,
..
} => (
inputs,
outputs,
&None,
&None,
sapling_shielded_data,
orchard_shielded_data,
),
V1 { .. } | V2 { .. } | V3 { .. } => unreachable!(
"older transaction versions only exist in finalized blocks, because of the mandatory canopy checkpoint",
),
};
// add key `transaction.hash` and value `(height, tx_index)` to `tx_by_hash`
let transaction_location = TransactionLocation::from_usize(height, transaction_index);
let prior_pair = self
.tx_by_hash
.insert(transaction_hash, transaction_location);
assert_eq!(
prior_pair, None,
"transactions must be unique within a single chain"
);
2. fix(state): index spending transaction IDs for each address (#4355) * Make jobs that use cached state wait for state rebuilds * Run jobs that need cached state even if the rebuild was skipped * Fix missing dependencies And update a TODO * Split writing transaction indexes into transparent and shielded * Split writing transparent indexes into created and spent * Correctly populate spending address transaction ID indexes * Increment the database format to rebuild address tx ID indexes * Update non-finalized docs to prevent similar bugs * Fix a comment * Make jobs that use cached state wait for state rebuilds * Run jobs that need cached state even if the rebuild was skipped * Fix missing dependencies And update a TODO * refactor(ci): look for available disks instead of files changed This ensure that if the constants.rs file was changed, we search for disks available in the whole repository with the same state. If there's no disk available a rebuild is triggered depending the missing disk. And if there's a disk available, tests are run with this one. * fix(ci): lwd syncs needs to wait for zebra disk rebuild * docs(ci): use better comments on integration tests * fix(ci): we must authenticate to GCP to find disks * fix(ci): add needed permissions for google auth * fix(ci): the output needs to be echoed * imp(ci): reduce diff with main * fix(ci): remove redundant dependency Co-authored-by: teor <teor@riseup.net> * fix(ci): also add `false` to the JSON object output * fix(ci): hasty copy/paste * force a push event * fix(ci): standardize comments * fix(ci): run disk rebuilds if no disk was found * fix(ci): do not restrict on push * fix(ci): build on any event if a cached disk is not found * fix(ci): sync .patch file with changes on the workflow Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-05-19 19:22:01 -07:00
// add the utxos this produced
self.update_chain_tip_with(&(outputs, &transaction_hash, new_outputs))?;
2. fix(state): index spending transaction IDs for each address (#4355) * Make jobs that use cached state wait for state rebuilds * Run jobs that need cached state even if the rebuild was skipped * Fix missing dependencies And update a TODO * Split writing transaction indexes into transparent and shielded * Split writing transparent indexes into created and spent * Correctly populate spending address transaction ID indexes * Increment the database format to rebuild address tx ID indexes * Update non-finalized docs to prevent similar bugs * Fix a comment * Make jobs that use cached state wait for state rebuilds * Run jobs that need cached state even if the rebuild was skipped * Fix missing dependencies And update a TODO * refactor(ci): look for available disks instead of files changed This ensure that if the constants.rs file was changed, we search for disks available in the whole repository with the same state. If there's no disk available a rebuild is triggered depending the missing disk. And if there's a disk available, tests are run with this one. * fix(ci): lwd syncs needs to wait for zebra disk rebuild * docs(ci): use better comments on integration tests * fix(ci): we must authenticate to GCP to find disks * fix(ci): add needed permissions for google auth * fix(ci): the output needs to be echoed * imp(ci): reduce diff with main * fix(ci): remove redundant dependency Co-authored-by: teor <teor@riseup.net> * fix(ci): also add `false` to the JSON object output * fix(ci): hasty copy/paste * force a push event * fix(ci): standardize comments * fix(ci): run disk rebuilds if no disk was found * fix(ci): do not restrict on push * fix(ci): build on any event if a cached disk is not found * fix(ci): sync .patch file with changes on the workflow Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-05-19 19:22:01 -07:00
// delete the utxos this consumed
self.update_chain_tip_with(&(inputs, &transaction_hash, spent_outputs))?;
// add the shielded data
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
self.update_chain_tip_with(joinsplit_data)?;
self.update_chain_tip_with(sapling_shielded_data_per_spend_anchor)?;
self.update_chain_tip_with(sapling_shielded_data_shared_anchor)?;
self.update_chain_tip_with(orchard_shielded_data)?;
}
2021-06-29 20:52:31 -07:00
// Update the note commitment trees indexed by height.
self.sapling_trees_by_height
.insert(height, self.sapling_note_commitment_tree.clone());
self.orchard_trees_by_height
.insert(height, self.orchard_note_commitment_tree.clone());
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
// Having updated all the note commitment trees and nullifier sets in
// this block, the roots of the note commitment trees as of the last
// transaction are the treestates of this block.
let sprout_root = self.sprout_note_commitment_tree.root();
self.sprout_anchors.insert(sprout_root);
self.sprout_anchors_by_height.insert(height, sprout_root);
self.sprout_trees_by_anchor
.insert(sprout_root, self.sprout_note_commitment_tree.clone());
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
let sapling_root = self.sapling_note_commitment_tree.root();
self.sapling_anchors.insert(sapling_root);
self.sapling_anchors_by_height.insert(height, sapling_root);
2021-11-30 08:05:35 -08:00
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
let orchard_root = self.orchard_note_commitment_tree.root();
self.orchard_anchors.insert(orchard_root);
self.orchard_anchors_by_height.insert(height, orchard_root);
self.history_tree.push(
self.network,
contextually_valid.block.clone(),
sapling_root,
orchard_root,
)?;
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
// update the chain value pool balances
self.update_chain_tip_with(chain_value_pool_change)?;
2021-06-29 20:52:31 -07:00
Ok(())
}
#[instrument(skip(self, contextually_valid), fields(block = %contextually_valid.block))]
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
fn revert_chain_with(
&mut self,
contextually_valid: &ContextuallyValidBlock,
position: RevertPosition,
) {
let (
block,
hash,
height,
new_outputs,
spent_outputs,
transaction_hashes,
chain_value_pool_change,
) = (
contextually_valid.block.as_ref(),
contextually_valid.hash,
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
contextually_valid.height,
&contextually_valid.new_outputs,
&contextually_valid.spent_outputs,
&contextually_valid.transaction_hashes,
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
&contextually_valid.chain_value_pool_change,
);
// remove the blocks hash from `height_by_hash`
assert!(
self.height_by_hash.remove(&hash).is_some(),
"hash must be present if block was added to chain"
);
// remove work from partial_cumulative_work
let block_work = block
.header
.difficulty_threshold
.to_work()
.expect("work has already been validated");
self.partial_cumulative_work -= block_work;
Add ZIP-221 history tree to non-finalized state (#2583) * Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree * HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq * ZIP-221: Validate chain history commitments in the non-finalized state (#2301) * sketch of implementation * refined implementation; still incomplete * update librustzcash, change zcash_history to work with it * simplified code per review; renamed MMR to HistoryTree * expand HistoryTree implementation * handle and propagate errors * simplify check.rs tracing * add suggested TODO * add HistoryTree::prune * fix bug in pruning * fix compilation of tests; still need to make them pass * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * improvements from code review * improve check.rs comments and variable names * fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests * fix finalized_state proptest * fix non_finalized_state tests by setting the correct commitments * renamed mmr.rs to history_tree.rs * Add HistoryTree struct * expand non_finalized_state protest * fix typo * Add HistoryTree struct * Update zebra-chain/src/primitives/zcash_history.rs Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * fix formatting * Apply suggestions from code review Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * history_tree.rs: fixes from code review * fixes to work with updated HistoryTree * Improvements from code review * Add Debug implementations to allow comparing Chains with proptest_assert_eq Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Improvements from code review * Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees Co-authored-by: teor <teor@riseup.net> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 06:42:40 -07:00
// Note: the history tree is not modified in this method.
// This method is called on two scenarios:
// - When popping the root: the history tree does not change.
// - When popping the tip: the history tree is rebuilt in fork().
// for each transaction in block
for (transaction, transaction_hash) in
block.transactions.iter().zip(transaction_hashes.iter())
{
let (
inputs,
outputs,
joinsplit_data,
sapling_shielded_data_per_spend_anchor,
sapling_shielded_data_shared_anchor,
orchard_shielded_data,
) = match transaction.deref() {
V4 {
inputs,
outputs,
joinsplit_data,
sapling_shielded_data,
..
} => (inputs, outputs, joinsplit_data, sapling_shielded_data, &None, &None),
V5 {
inputs,
outputs,
sapling_shielded_data,
orchard_shielded_data,
..
} => (
inputs,
outputs,
&None,
&None,
sapling_shielded_data,
orchard_shielded_data,
),
V1 { .. } | V2 { .. } | V3 { .. } => unreachable!(
"older transaction versions only exist in finalized blocks, because of the mandatory canopy checkpoint",
),
};
// remove the utxos this produced
self.revert_chain_with(&(outputs, transaction_hash, new_outputs), position);
2. fix(state): index spending transaction IDs for each address (#4355) * Make jobs that use cached state wait for state rebuilds * Run jobs that need cached state even if the rebuild was skipped * Fix missing dependencies And update a TODO * Split writing transaction indexes into transparent and shielded * Split writing transparent indexes into created and spent * Correctly populate spending address transaction ID indexes * Increment the database format to rebuild address tx ID indexes * Update non-finalized docs to prevent similar bugs * Fix a comment * Make jobs that use cached state wait for state rebuilds * Run jobs that need cached state even if the rebuild was skipped * Fix missing dependencies And update a TODO * refactor(ci): look for available disks instead of files changed This ensure that if the constants.rs file was changed, we search for disks available in the whole repository with the same state. If there's no disk available a rebuild is triggered depending the missing disk. And if there's a disk available, tests are run with this one. * fix(ci): lwd syncs needs to wait for zebra disk rebuild * docs(ci): use better comments on integration tests * fix(ci): we must authenticate to GCP to find disks * fix(ci): add needed permissions for google auth * fix(ci): the output needs to be echoed * imp(ci): reduce diff with main * fix(ci): remove redundant dependency Co-authored-by: teor <teor@riseup.net> * fix(ci): also add `false` to the JSON object output * fix(ci): hasty copy/paste * force a push event * fix(ci): standardize comments * fix(ci): run disk rebuilds if no disk was found * fix(ci): do not restrict on push * fix(ci): build on any event if a cached disk is not found * fix(ci): sync .patch file with changes on the workflow Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-05-19 19:22:01 -07:00
// reset the utxos this consumed
self.revert_chain_with(&(inputs, transaction_hash, spent_outputs), position);
// remove `transaction.hash` from `tx_by_hash`
assert!(
self.tx_by_hash.remove(transaction_hash).is_some(),
"transactions must be present if block was added to chain"
);
// remove the shielded data
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
self.revert_chain_with(joinsplit_data, position);
self.revert_chain_with(sapling_shielded_data_per_spend_anchor, position);
self.revert_chain_with(sapling_shielded_data_shared_anchor, position);
self.revert_chain_with(orchard_shielded_data, position);
}
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
let anchor = self
.sprout_anchors_by_height
.remove(&height)
.expect("Sprout anchor must be present if block was added to chain");
assert!(
self.sprout_anchors.remove(&anchor),
"Sprout anchor must be present if block was added to chain"
);
if !self.sprout_anchors.contains(&anchor) {
self.sprout_trees_by_anchor.remove(&anchor);
}
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
let anchor = self
.sapling_anchors_by_height
.remove(&height)
.expect("Sapling anchor must be present if block was added to chain");
assert!(
self.sapling_anchors.remove(&anchor),
"Sapling anchor must be present if block was added to chain"
);
self.sapling_trees_by_height
.remove(&height)
.expect("Sapling note commitment tree must be present if block was added to chain");
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
let anchor = self
.orchard_anchors_by_height
.remove(&height)
.expect("Orchard anchor must be present if block was added to chain");
assert!(
self.orchard_anchors.remove(&anchor),
"Orchard anchor must be present if block was added to chain"
);
self.orchard_trees_by_height
.remove(&height)
.expect("Orchard note commitment tree must be present if block was added to chain");
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
// revert the chain value pool balances, if needed
self.revert_chain_with(chain_value_pool_change, position);
}
}
// Created UTXOs
//
// TODO: replace arguments with a struct
impl
UpdateWith<(
// The outputs from a transaction in this block
&Vec<transparent::Output>,
// The hash of the transaction that the outputs are from
&transaction::Hash,
// The UTXOs for all outputs created by this transaction (or block)
&HashMap<transparent::OutPoint, transparent::OrderedUtxo>,
)> for Chain
{
#[allow(clippy::unwrap_in_result)]
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
fn update_chain_tip_with(
2021-06-29 20:52:31 -07:00
&mut self,
&(created_outputs, creating_tx_hash, block_created_outputs): &(
&Vec<transparent::Output>,
&transaction::Hash,
&HashMap<transparent::OutPoint, transparent::OrderedUtxo>,
),
2021-06-29 20:52:31 -07:00
) -> Result<(), ValidateContextError> {
for output_index in 0..created_outputs.len() {
let outpoint = transparent::OutPoint {
hash: *creating_tx_hash,
index: output_index.try_into().expect("valid indexes fit in u32"),
};
let created_utxo = block_created_outputs
.get(&outpoint)
.expect("new_outputs contains all created UTXOs");
// Update the chain's created UTXOs
let previous_entry = self.created_utxos.insert(outpoint, created_utxo.clone());
assert_eq!(
previous_entry, None,
"unexpected created output: duplicate update or duplicate UTXO",
);
// Update the address index with this UTXO
if let Some(receiving_address) = created_utxo.utxo.output.address(self.network) {
let address_transfers = self
.partial_transparent_transfers
.entry(receiving_address)
.or_default();
address_transfers.update_chain_tip_with(&(&outpoint, created_utxo))?;
}
}
2021-06-29 20:52:31 -07:00
Ok(())
}
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
fn revert_chain_with(
&mut self,
&(created_outputs, creating_tx_hash, block_created_outputs): &(
&Vec<transparent::Output>,
&transaction::Hash,
&HashMap<transparent::OutPoint, transparent::OrderedUtxo>,
),
position: RevertPosition,
) {
for output_index in 0..created_outputs.len() {
let outpoint = transparent::OutPoint {
hash: *creating_tx_hash,
index: output_index.try_into().expect("valid indexes fit in u32"),
};
let created_utxo = block_created_outputs
.get(&outpoint)
.expect("new_outputs contains all created UTXOs");
// Revert the chain's created UTXOs
let removed_entry = self.created_utxos.remove(&outpoint);
assert!(
removed_entry.is_some(),
"unexpected revert of created output: duplicate revert or duplicate UTXO",
);
// Revert the address index for this UTXO
if let Some(receiving_address) = created_utxo.utxo.output.address(self.network) {
let address_transfers = self
.partial_transparent_transfers
.get_mut(&receiving_address)
.expect("block has previously been applied to the chain");
address_transfers.revert_chain_with(&(&outpoint, created_utxo), position);
// Remove this transfer if it is now empty
if address_transfers.is_empty() {
self.partial_transparent_transfers
.remove(&receiving_address);
}
}
}
}
}
// Transparent inputs
//
// TODO: replace arguments with a struct
impl
UpdateWith<(
// The inputs from a transaction in this block
&Vec<transparent::Input>,
// The hash of the transaction that the inputs are from
2. fix(state): index spending transaction IDs for each address (#4355) * Make jobs that use cached state wait for state rebuilds * Run jobs that need cached state even if the rebuild was skipped * Fix missing dependencies And update a TODO * Split writing transaction indexes into transparent and shielded * Split writing transparent indexes into created and spent * Correctly populate spending address transaction ID indexes * Increment the database format to rebuild address tx ID indexes * Update non-finalized docs to prevent similar bugs * Fix a comment * Make jobs that use cached state wait for state rebuilds * Run jobs that need cached state even if the rebuild was skipped * Fix missing dependencies And update a TODO * refactor(ci): look for available disks instead of files changed This ensure that if the constants.rs file was changed, we search for disks available in the whole repository with the same state. If there's no disk available a rebuild is triggered depending the missing disk. And if there's a disk available, tests are run with this one. * fix(ci): lwd syncs needs to wait for zebra disk rebuild * docs(ci): use better comments on integration tests * fix(ci): we must authenticate to GCP to find disks * fix(ci): add needed permissions for google auth * fix(ci): the output needs to be echoed * imp(ci): reduce diff with main * fix(ci): remove redundant dependency Co-authored-by: teor <teor@riseup.net> * fix(ci): also add `false` to the JSON object output * fix(ci): hasty copy/paste * force a push event * fix(ci): standardize comments * fix(ci): run disk rebuilds if no disk was found * fix(ci): do not restrict on push * fix(ci): build on any event if a cached disk is not found * fix(ci): sync .patch file with changes on the workflow Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-05-19 19:22:01 -07:00
// (not the transaction the spent output was created by)
&transaction::Hash,
// The outputs for all inputs spent in this transaction (or block)
&HashMap<transparent::OutPoint, transparent::OrderedUtxo>,
)> for Chain
{
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
fn update_chain_tip_with(
2021-06-29 20:52:31 -07:00
&mut self,
&(spending_inputs, spending_tx_hash, spent_outputs): &(
&Vec<transparent::Input>,
&transaction::Hash,
&HashMap<transparent::OutPoint, transparent::OrderedUtxo>,
),
2021-06-29 20:52:31 -07:00
) -> Result<(), ValidateContextError> {
for spending_input in spending_inputs.iter() {
let spent_outpoint = if let Some(spent_outpoint) = spending_input.outpoint() {
spent_outpoint
} else {
continue;
};
// Index the spent outpoint in the chain
let first_spend = self.spent_utxos.insert(spent_outpoint);
assert!(
first_spend,
"unexpected duplicate spent output: should be checked earlier"
);
// TODO: fix tests to supply correct spent outputs, then turn this into an expect()
let spent_output = if let Some(spent_output) = spent_outputs.get(&spent_outpoint) {
spent_output
} else if !cfg!(test) {
panic!("unexpected missing spent output: all spent outputs must be indexed");
} else {
continue;
};
// Index the spent output for the address
if let Some(spending_address) = spent_output.utxo.output.address(self.network) {
let address_transfers = self
.partial_transparent_transfers
.entry(spending_address)
.or_default();
address_transfers.update_chain_tip_with(&(
spending_input,
spending_tx_hash,
spent_output,
))?;
}
}
2021-06-29 20:52:31 -07:00
Ok(())
}
fn revert_chain_with(
&mut self,
&(spending_inputs, spending_tx_hash, spent_outputs): &(
&Vec<transparent::Input>,
&transaction::Hash,
&HashMap<transparent::OutPoint, transparent::OrderedUtxo>,
),
position: RevertPosition,
) {
for spending_input in spending_inputs.iter() {
let spent_outpoint = if let Some(spent_outpoint) = spending_input.outpoint() {
spent_outpoint
} else {
continue;
};
// Revert the spent outpoint in the chain
let spent_outpoint_was_removed = self.spent_utxos.remove(&spent_outpoint);
assert!(
spent_outpoint_was_removed,
"spent_utxos must be present if block was added to chain"
);
// TODO: fix tests to supply correct spent outputs, then turn this into an expect()
let spent_output = if let Some(spent_output) = spent_outputs.get(&spent_outpoint) {
spent_output
} else if !cfg!(test) {
panic!(
"unexpected missing reverted spent output: all spent outputs must be indexed"
);
} else {
continue;
};
// Revert the spent output for the address
if let Some(receiving_address) = spent_output.utxo.output.address(self.network) {
let address_transfers = self
.partial_transparent_transfers
.get_mut(&receiving_address)
.expect("block has previously been applied to the chain");
address_transfers
.revert_chain_with(&(spending_input, spending_tx_hash, spent_output), position);
// Remove this transfer if it is now empty
if address_transfers.is_empty() {
self.partial_transparent_transfers
.remove(&receiving_address);
}
}
}
}
}
impl UpdateWith<Option<transaction::JoinSplitData<Groth16Proof>>> for Chain {
#[instrument(skip(self, joinsplit_data))]
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
fn update_chain_tip_with(
&mut self,
joinsplit_data: &Option<transaction::JoinSplitData<Groth16Proof>>,
2021-06-29 20:52:31 -07:00
) -> Result<(), ValidateContextError> {
if let Some(joinsplit_data) = joinsplit_data {
for cm in joinsplit_data.note_commitments() {
self.sprout_note_commitment_tree.append(*cm)?;
}
check::nullifier::add_to_non_finalized_chain_unique(
&mut self.sprout_nullifiers,
joinsplit_data.nullifiers(),
)?;
}
2021-06-29 20:52:31 -07:00
Ok(())
}
/// # Panics
///
/// Panics if any nullifier is missing from the chain when we try to remove it.
///
/// See [`check::nullifier::remove_from_non_finalized_chain`] for details.
#[instrument(skip(self, joinsplit_data))]
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
fn revert_chain_with(
&mut self,
joinsplit_data: &Option<transaction::JoinSplitData<Groth16Proof>>,
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
_position: RevertPosition,
) {
if let Some(joinsplit_data) = joinsplit_data {
check::nullifier::remove_from_non_finalized_chain(
&mut self.sprout_nullifiers,
joinsplit_data.nullifiers(),
);
}
}
}
impl<AnchorV> UpdateWith<Option<sapling::ShieldedData<AnchorV>>> for Chain
where
AnchorV: sapling::AnchorVariant + Clone,
{
Reject duplicate Sapling and Orchard nullifiers (#2497) * Add sapling and orchard duplicate nullifier errors * Reject duplicate finalized sapling and orchard nullifiers Reject duplicate sapling and orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is already in the finalized state. * Reject duplicate non-finalized sapling and orchard nullifiers Reject duplicate sapling and orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, or * an earlier block in the non-finalized chain. * Refactor sprout nullifier tests to remove common code * Add sapling nullifier tests Test that the state rejects duplicate sapling nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, * an earlier block in the non-finalized chain, or * the finalized state. * Add orchard nullifier tests Test that the state rejects duplicate orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, * an earlier block in the non-finalized chain, or * the finalized state. * Check for specific nullifiers in the state in tests * Replace slices with vectors in arguments * Remove redundant code and variables * Simplify sapling TransferData tests Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Remove an extra : * Remove redundant vec! Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-19 17:39:05 -07:00
#[instrument(skip(self, sapling_shielded_data))]
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
fn update_chain_tip_with(
&mut self,
sapling_shielded_data: &Option<sapling::ShieldedData<AnchorV>>,
2021-06-29 20:52:31 -07:00
) -> Result<(), ValidateContextError> {
if let Some(sapling_shielded_data) = sapling_shielded_data {
// The `_u` here indicates that the Sapling note commitment is
// specified only by the `u`-coordinate of the Jubjub curve
// point `(u, v)`.
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
for cm_u in sapling_shielded_data.note_commitments() {
self.sapling_note_commitment_tree.append(*cm_u)?;
}
Reject duplicate Sapling and Orchard nullifiers (#2497) * Add sapling and orchard duplicate nullifier errors * Reject duplicate finalized sapling and orchard nullifiers Reject duplicate sapling and orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is already in the finalized state. * Reject duplicate non-finalized sapling and orchard nullifiers Reject duplicate sapling and orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, or * an earlier block in the non-finalized chain. * Refactor sprout nullifier tests to remove common code * Add sapling nullifier tests Test that the state rejects duplicate sapling nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, * an earlier block in the non-finalized chain, or * the finalized state. * Add orchard nullifier tests Test that the state rejects duplicate orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, * an earlier block in the non-finalized chain, or * the finalized state. * Check for specific nullifiers in the state in tests * Replace slices with vectors in arguments * Remove redundant code and variables * Simplify sapling TransferData tests Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Remove an extra : * Remove redundant vec! Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-19 17:39:05 -07:00
check::nullifier::add_to_non_finalized_chain_unique(
&mut self.sapling_nullifiers,
sapling_shielded_data.nullifiers(),
)?;
}
2021-06-29 20:52:31 -07:00
Ok(())
}
Reject duplicate Sapling and Orchard nullifiers (#2497) * Add sapling and orchard duplicate nullifier errors * Reject duplicate finalized sapling and orchard nullifiers Reject duplicate sapling and orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is already in the finalized state. * Reject duplicate non-finalized sapling and orchard nullifiers Reject duplicate sapling and orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, or * an earlier block in the non-finalized chain. * Refactor sprout nullifier tests to remove common code * Add sapling nullifier tests Test that the state rejects duplicate sapling nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, * an earlier block in the non-finalized chain, or * the finalized state. * Add orchard nullifier tests Test that the state rejects duplicate orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, * an earlier block in the non-finalized chain, or * the finalized state. * Check for specific nullifiers in the state in tests * Replace slices with vectors in arguments * Remove redundant code and variables * Simplify sapling TransferData tests Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Remove an extra : * Remove redundant vec! Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-19 17:39:05 -07:00
/// # Panics
///
/// Panics if any nullifier is missing from the chain when we try to remove it.
///
/// See [`check::nullifier::remove_from_non_finalized_chain`] for details.
#[instrument(skip(self, sapling_shielded_data))]
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
fn revert_chain_with(
&mut self,
sapling_shielded_data: &Option<sapling::ShieldedData<AnchorV>>,
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
_position: RevertPosition,
) {
if let Some(sapling_shielded_data) = sapling_shielded_data {
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
// Note commitments are not removed from the tree here because we
// don't support that operation yet. Instead, we recreate the tree
// from the finalized tip in `NonFinalizedState`.
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
Reject duplicate Sapling and Orchard nullifiers (#2497) * Add sapling and orchard duplicate nullifier errors * Reject duplicate finalized sapling and orchard nullifiers Reject duplicate sapling and orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is already in the finalized state. * Reject duplicate non-finalized sapling and orchard nullifiers Reject duplicate sapling and orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, or * an earlier block in the non-finalized chain. * Refactor sprout nullifier tests to remove common code * Add sapling nullifier tests Test that the state rejects duplicate sapling nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, * an earlier block in the non-finalized chain, or * the finalized state. * Add orchard nullifier tests Test that the state rejects duplicate orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, * an earlier block in the non-finalized chain, or * the finalized state. * Check for specific nullifiers in the state in tests * Replace slices with vectors in arguments * Remove redundant code and variables * Simplify sapling TransferData tests Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Remove an extra : * Remove redundant vec! Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-19 17:39:05 -07:00
check::nullifier::remove_from_non_finalized_chain(
&mut self.sapling_nullifiers,
sapling_shielded_data.nullifiers(),
);
}
}
}
impl UpdateWith<Option<orchard::ShieldedData>> for Chain {
Reject duplicate Sapling and Orchard nullifiers (#2497) * Add sapling and orchard duplicate nullifier errors * Reject duplicate finalized sapling and orchard nullifiers Reject duplicate sapling and orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is already in the finalized state. * Reject duplicate non-finalized sapling and orchard nullifiers Reject duplicate sapling and orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, or * an earlier block in the non-finalized chain. * Refactor sprout nullifier tests to remove common code * Add sapling nullifier tests Test that the state rejects duplicate sapling nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, * an earlier block in the non-finalized chain, or * the finalized state. * Add orchard nullifier tests Test that the state rejects duplicate orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, * an earlier block in the non-finalized chain, or * the finalized state. * Check for specific nullifiers in the state in tests * Replace slices with vectors in arguments * Remove redundant code and variables * Simplify sapling TransferData tests Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Remove an extra : * Remove redundant vec! Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-19 17:39:05 -07:00
#[instrument(skip(self, orchard_shielded_data))]
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
fn update_chain_tip_with(
2021-06-29 20:52:31 -07:00
&mut self,
orchard_shielded_data: &Option<orchard::ShieldedData>,
) -> Result<(), ValidateContextError> {
if let Some(orchard_shielded_data) = orchard_shielded_data {
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
for cm_x in orchard_shielded_data.note_commitments() {
self.orchard_note_commitment_tree.append(*cm_x)?;
}
Reject duplicate Sapling and Orchard nullifiers (#2497) * Add sapling and orchard duplicate nullifier errors * Reject duplicate finalized sapling and orchard nullifiers Reject duplicate sapling and orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is already in the finalized state. * Reject duplicate non-finalized sapling and orchard nullifiers Reject duplicate sapling and orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, or * an earlier block in the non-finalized chain. * Refactor sprout nullifier tests to remove common code * Add sapling nullifier tests Test that the state rejects duplicate sapling nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, * an earlier block in the non-finalized chain, or * the finalized state. * Add orchard nullifier tests Test that the state rejects duplicate orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, * an earlier block in the non-finalized chain, or * the finalized state. * Check for specific nullifiers in the state in tests * Replace slices with vectors in arguments * Remove redundant code and variables * Simplify sapling TransferData tests Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Remove an extra : * Remove redundant vec! Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-19 17:39:05 -07:00
check::nullifier::add_to_non_finalized_chain_unique(
&mut self.orchard_nullifiers,
orchard_shielded_data.nullifiers(),
)?;
}
2021-06-29 20:52:31 -07:00
Ok(())
}
Reject duplicate Sapling and Orchard nullifiers (#2497) * Add sapling and orchard duplicate nullifier errors * Reject duplicate finalized sapling and orchard nullifiers Reject duplicate sapling and orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is already in the finalized state. * Reject duplicate non-finalized sapling and orchard nullifiers Reject duplicate sapling and orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, or * an earlier block in the non-finalized chain. * Refactor sprout nullifier tests to remove common code * Add sapling nullifier tests Test that the state rejects duplicate sapling nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, * an earlier block in the non-finalized chain, or * the finalized state. * Add orchard nullifier tests Test that the state rejects duplicate orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, * an earlier block in the non-finalized chain, or * the finalized state. * Check for specific nullifiers in the state in tests * Replace slices with vectors in arguments * Remove redundant code and variables * Simplify sapling TransferData tests Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Remove an extra : * Remove redundant vec! Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-19 17:39:05 -07:00
/// # Panics
///
/// Panics if any nullifier is missing from the chain when we try to remove it.
///
/// See [`check::nullifier::remove_from_non_finalized_chain`] for details.
#[instrument(skip(self, orchard_shielded_data))]
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
fn revert_chain_with(
&mut self,
orchard_shielded_data: &Option<orchard::ShieldedData>,
_position: RevertPosition,
) {
if let Some(orchard_shielded_data) = orchard_shielded_data {
Track anchors and note commitment trees in zebra-state (#2458) * Tidy chain Cargo.toml * Organize imports * Add method to get note commitments from all Actions in Orchard shielded data * Add method to get note commitments from all JoinSplits in Sprout JoinSplitData * Add Request and Response variants for awaiting anchors * Add anchors and note commitment trees to finalized state db * Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees * Track anchors and note commitment trees in Chain Append note commitments to their trees when doing update_chain_state_with, then use the resulting Sapling and Orchard roots to pass to history_tree, and add new roots to the anchor sets. * Handle errors when appending to note commitment trees * Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with * Implementing note commitments in finalized state * Finish serialization of Orchard tree; remove old tree when updating finalize state * Add serialization and finalized state updates for Sprout and Sapling trees * Partially handle trees in non-finalized state. Use Option for trees in Chain * Rebuild trees when forking; change finalized state tree getters to not require height * Pass empty trees to tests; use empty trees by default in Chain * Also rebuild anchor sets when forking * Use empty tree as default in finalized state tree getters (for now) * Use HashMultiSet for anchors in order to make pop_root() work correctly * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS * Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more * Apply suggestions from code review * Add comments about order of note commitments and related methods/fields * Don't use Option for trees * Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS * Remove unneeded anchor set rebuilding in fork() * Improve proptest formatting * Add missing comparisons to eq_internal_state * Renamed sprout::tree::NoteCommitmentTree::hash() to root() * Improve comments * Add asserts, add issues to TODOs * Remove impl Default for Chain since it was only used by tests * Improve documentation and assertions; add tree serialization tests * Remove Sprout code, which will be moved to another branch * Add todo! in Sprout tree append() * Remove stub request, response *Anchor* handling for now * Add test for validating Sapling note commitment tree using test blocks * Increase database version (new columns added for note commitment trees and anchors) * Update test to make sure the order of sapling_note_commitments() is being tested * Improve comments and structure of the test * Improve variable names again * Rustfmt Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org> Co-authored-by: teor <teor@riseup.net>
2021-07-29 06:37:18 -07:00
// Note commitments are not removed from the tree here because we
// don't support that operation yet. Instead, we recreate the tree
// from the finalized tip in NonFinalizedState.
Reject duplicate Sapling and Orchard nullifiers (#2497) * Add sapling and orchard duplicate nullifier errors * Reject duplicate finalized sapling and orchard nullifiers Reject duplicate sapling and orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is already in the finalized state. * Reject duplicate non-finalized sapling and orchard nullifiers Reject duplicate sapling and orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, or * an earlier block in the non-finalized chain. * Refactor sprout nullifier tests to remove common code * Add sapling nullifier tests Test that the state rejects duplicate sapling nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, * an earlier block in the non-finalized chain, or * the finalized state. * Add orchard nullifier tests Test that the state rejects duplicate orchard nullifiers in a new block, when the block is added to a non-finalized chain, and the duplicate nullifier is in: * the same shielded data, * the same transaction, * the same block, * an earlier block in the non-finalized chain, or * the finalized state. * Check for specific nullifiers in the state in tests * Replace slices with vectors in arguments * Remove redundant code and variables * Simplify sapling TransferData tests Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Remove an extra : * Remove redundant vec! Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-19 17:39:05 -07:00
check::nullifier::remove_from_non_finalized_chain(
&mut self.orchard_nullifiers,
orchard_shielded_data.nullifiers(),
);
}
}
}
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
impl UpdateWith<ValueBalance<NegativeAllowed>> for Chain {
fn update_chain_tip_with(
&mut self,
block_value_pool_change: &ValueBalance<NegativeAllowed>,
) -> Result<(), ValidateContextError> {
match self
.chain_value_pools
.add_chain_value_pool_change(*block_value_pool_change)
{
Ok(chain_value_pools) => self.chain_value_pools = chain_value_pools,
Err(value_balance_error) => Err(ValidateContextError::AddValuePool {
value_balance_error,
chain_value_pools: self.chain_value_pools,
block_value_pool_change: *block_value_pool_change,
// assume that the current block is added to `blocks` after `update_chain_tip_with`
height: self.max_block_height().and_then(|height| height + 1),
})?,
};
Ok(())
}
/// Revert the chain state using a block chain value pool change.
///
/// When forking from the tip, subtract the block's chain value pool change.
///
/// When finalizing the root, leave the chain value pool balances unchanged.
/// [`Self::chain_value_pools`] tracks the chain value pools for all
/// finalized blocks, and the non-finalized blocks in this chain. So
/// finalizing the root doesn't change the set of blocks it tracks.
Add value pool/balances to non-finalized state (#2656) * add value balances to non finalized state * fix 2 tests * fix remaining constrain issues in tests * extend value pool test to non finalized * WIP: fix tests after adding value pools to non-finalized state (#2647) * Update Chain::eq_internal_state with Chain.value_balance Also increase the number of cases in its tests, because they didn't detect this bug. * Calculate the chain value pool change before `Chain::push` Code - store the chain value pool change in `ContextuallyValidBlock` - convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos` (rather than `from` or `into`) - replace `block_utxos` with `new_outputs` in `PreparedBlock` - replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock` Tests - create test methods for `PreparedBlock` and `ContextuallyValidBlock` - use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos` to make tests pass * fix conflicts * build `set_current_value_pool()` only for tests * remove redundant cfgs * change cfg of set_current_value_pool() * Clarify some chain field documentation * Fix bugs in the non-finalized chain value pool calculations 1. Only revert the chain value pool balances when the tip is popped. Don't modify them when the root is finalized. 2. Only update or revert the chain value pool balances once per block. (Previously, the block changes were multiplied by the number of *transactions*.) And make corresponding changes to method names and documentation. * Add extra proptests to try to identify value balance failures * Simplify some transaction generation code * Add extra debugging info to value balance errors * Actually update non-finalized chain value pools in `UpdateWith` Previously, we were dropping the updated value pools in the `Ok` result. So the initial (finalized) chain value pool balances were never modified. * Rename and document value balance add methods The names and documentation of these methods were confusing. * Create genesis-based proptests that check chain value pools * Increase coverage for some test vectors * Test each chain value balance calculation for blocks 0-10 * Make continuous blockchain test errors easier to debug * Test the exact transparent pool values for the first few blocks Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 06:57:07 -07:00
///
/// # Panics
///
/// Panics if the chain pool value balance is invalid
/// after we subtract the block value pool change.
fn revert_chain_with(
&mut self,
block_value_pool_change: &ValueBalance<NegativeAllowed>,
position: RevertPosition,
) {
use std::ops::Neg;
if position == RevertPosition::Tip {
self.chain_value_pools = self
.chain_value_pools
.add_chain_value_pool_change(block_value_pool_change.neg())
.expect("reverting the tip will leave the pools in a previously valid state");
}
}
}
impl Ord for Chain {
/// Chain order for the [`NonFinalizedState`][1]'s `chain_set`.
///
/// Chains with higher cumulative Proof of Work are [`Ordering::Greater`],
/// breaking ties using the tip block hash.
///
/// Despite the consensus rules, Zebra uses the tip block hash as a
/// tie-breaker. Zebra blocks are downloaded in parallel, so download
/// timestamps may not be unique. (And Zebra currently doesn't track
/// download times, because [`Block`](block::Block)s are immutable.)
///
/// This departure from the consensus rules may delay network convergence,
/// for as long as the greater hash belongs to the later mined block.
/// But Zebra nodes should converge as soon as the tied work is broken.
///
/// "At a given point in time, each full validator is aware of a set of candidate blocks.
/// These form a tree rooted at the genesis block, where each node in the tree
/// refers to its parent via the hashPrevBlock block header field.
///
/// A path from the root toward the leaves of the tree consisting of a sequence
/// of one or more valid blocks consistent with consensus rules,
/// is called a valid block chain.
///
/// In order to choose the best valid block chain in its view of the overall block tree,
/// a node sums the work ... of all blocks in each valid block chain,
/// and considers the valid block chain with greatest total work to be best.
///
/// To break ties between leaf blocks, a node will prefer the block that it received first.
///
/// The consensus protocol is designed to ensure that for any given block height,
/// the vast majority of nodes should eventually agree on their best valid block chain
/// up to that height."
///
/// <https://zips.z.cash/protocol/protocol.pdf#blockchain>
///
/// # Correctness
///
/// `Chain::cmp` is used in a `BTreeSet`, so the fields accessed by `cmp` must not have
/// interior mutability.
///
/// # Panics
///
/// If two chains compare equal.
///
/// This panic enforces the [`NonFinalizedState::chain_set`][2] unique chain invariant.
///
/// If the chain set contains duplicate chains, the non-finalized state might
/// handle new blocks or block finalization incorrectly.
///
/// [1]: super::NonFinalizedState
/// [2]: super::NonFinalizedState::chain_set
fn cmp(&self, other: &Self) -> Ordering {
if self.partial_cumulative_work != other.partial_cumulative_work {
self.partial_cumulative_work
.cmp(&other.partial_cumulative_work)
} else {
let self_hash = self
.blocks
.values()
.last()
.expect("always at least 1 element")
.hash;
let other_hash = other
.blocks
.values()
.last()
.expect("always at least 1 element")
.hash;
// This comparison is a tie-breaker within the local node, so it does not need to
// be consistent with the ordering on `ExpandedDifficulty` and `block::Hash`.
match self_hash.0.cmp(&other_hash.0) {
Ordering::Equal => unreachable!("Chain tip block hashes are always unique"),
ordering => ordering,
}
}
}
}
impl PartialOrd for Chain {
fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
Some(self.cmp(other))
}
}
impl PartialEq for Chain {
/// Chain equality for [`NonFinalizedState::chain_set`][1], using proof of
/// work, then the tip block hash as a tie-breaker.
///
/// # Panics
///
/// If two chains compare equal.
///
/// See [`Chain::cmp`] for details.
///
/// [1]: super::NonFinalizedState::chain_set
fn eq(&self, other: &Self) -> bool {
self.partial_cmp(other) == Some(Ordering::Equal)
}
}
impl Eq for Chain {}