* Do prelim checking of Sprout anchors in non-finalized state
Does not check intra-transaction interstitial states yet
* Populate sprout anchors to allow other state tests to pass
* Preliminary interstitial sprout note commitment tree anchor checks implementation
* Make sure only prior anchors are checked in the same transaction
* Add tests
* Refactor a comment
* Refactor rustdoc
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Use the first `JoinSplit`s from mainnet
* Print debug messages
* Use correct blocks for the tests
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add Transaction::sprout_joinsplits()
* Add Anchor variants to ValidateContextError
* Make Chain anchor collections pub(crate)
* tracing::instrument several methods in state
* Add contains_*_anchors methods to FinalizedState
* Add check::anchors module and function
* Verify that anchors_refer_to_earlier_treestates in when updating chains in non-finalized state
* Update zebra-state/src/service/check/anchors.rs
Co-authored-by: teor <teor@riseup.net>
* Add anchors() to sapling::ShieldedData
* Add sapling_anchors() to Transaction
* Use Transaction::sapling_anchors() in the anchors_refer_to_earlier_treestates() check
* Whoops, itertools
* Add a comment for improvement
Co-authored-by: teor <teor@riseup.net>
* Add & use a cfg(test) method on FinalizedState to prep test state with anchors to allow other tests to pass contextual checks
* Allow test nullifier checks to pass by populating anchor sets, allowing test anchor checks to pass
* Add mainnet block 419202 and its sapling note commitment tree root to test vectors
* Test sapling anchor verification using the first few Sapling blocks data
* Correct comment
* assert_eq instead of assert(bool)
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Update zebra-state/src/service/non_finalized_state.rs
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* add value balances to non finalized state
* fix 2 tests
* fix remaining constrain issues in tests
* extend value pool test to non finalized
* WIP: fix tests after adding value pools to non-finalized state (#2647)
* Update Chain::eq_internal_state with Chain.value_balance
Also increase the number of cases in its tests,
because they didn't detect this bug.
* Calculate the chain value pool change before `Chain::push`
Code
- store the chain value pool change in `ContextuallyValidBlock`
- convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos`
(rather than `from` or `into`)
- replace `block_utxos` with `new_outputs` in `PreparedBlock`
- replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock`
Tests
- create test methods for `PreparedBlock` and `ContextuallyValidBlock`
- use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos`
to make tests pass
* fix conflicts
* build `set_current_value_pool()` only for tests
* remove redundant cfgs
* change cfg of set_current_value_pool()
* Clarify some chain field documentation
* Fix bugs in the non-finalized chain value pool calculations
1. Only revert the chain value pool balances when the tip is popped.
Don't modify them when the root is finalized.
2. Only update or revert the chain value pool balances once per block.
(Previously, the block changes were multiplied by the number of *transactions*.)
And make corresponding changes to method names and documentation.
* Add extra proptests to try to identify value balance failures
* Simplify some transaction generation code
* Add extra debugging info to value balance errors
* Actually update non-finalized chain value pools in `UpdateWith`
Previously, we were dropping the updated value pools in the `Ok` result.
So the initial (finalized) chain value pool balances were never modified.
* Rename and document value balance add methods
The names and documentation of these methods were confusing.
* Create genesis-based proptests that check chain value pools
* Increase coverage for some test vectors
* Test each chain value balance calculation for blocks 0-10
* Make continuous blockchain test errors easier to debug
* Test the exact transparent pool values for the first few blocks
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree
* HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq
* ZIP-221: Validate chain history commitments in the non-finalized state (#2301)
* sketch of implementation
* refined implementation; still incomplete
* update librustzcash, change zcash_history to work with it
* simplified code per review; renamed MMR to HistoryTree
* expand HistoryTree implementation
* handle and propagate errors
* simplify check.rs tracing
* add suggested TODO
* add HistoryTree::prune
* fix bug in pruning
* fix compilation of tests; still need to make them pass
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* improvements from code review
* improve check.rs comments and variable names
* fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests
* fix finalized_state proptest
* fix non_finalized_state tests by setting the correct commitments
* renamed mmr.rs to history_tree.rs
* Add HistoryTree struct
* expand non_finalized_state protest
* fix typo
* Add HistoryTree struct
* Update zebra-chain/src/primitives/zcash_history.rs
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* fix formatting
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* history_tree.rs: fixes from code review
* fixes to work with updated HistoryTree
* Improvements from code review
* Add Debug implementations to allow comparing Chains with proptest_assert_eq
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Improvements from code review
* Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Tidy chain Cargo.toml
* Organize imports
* Add method to get note commitments from all Actions in Orchard shielded data
* Add method to get note commitments from all JoinSplits in Sprout JoinSplitData
* Add Request and Response variants for awaiting anchors
* Add anchors and note commitment trees to finalized state db
* Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees
* Track anchors and note commitment trees in Chain
Append note commitments to their trees when doing update_chain_state_with,
then use the resulting Sapling and Orchard roots to pass to history_tree, and add
new roots to the anchor sets.
* Handle errors when appending to note commitment trees
* Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with
* Implementing note commitments in finalized state
* Finish serialization of Orchard tree; remove old tree when updating finalize state
* Add serialization and finalized state updates for Sprout and Sapling trees
* Partially handle trees in non-finalized state. Use Option for trees in Chain
* Rebuild trees when forking; change finalized state tree getters to not require height
* Pass empty trees to tests; use empty trees by default in Chain
* Also rebuild anchor sets when forking
* Use empty tree as default in finalized state tree getters (for now)
* Use HashMultiSet for anchors in order to make pop_root() work correctly
* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS
* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more
* Apply suggestions from code review
* Add comments about order of note commitments and related methods/fields
* Don't use Option for trees
* Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS
* Remove unneeded anchor set rebuilding in fork()
* Improve proptest formatting
* Add missing comparisons to eq_internal_state
* Renamed sprout::tree::NoteCommitmentTree::hash() to root()
* Improve comments
* Add asserts, add issues to TODOs
* Remove impl Default for Chain since it was only used by tests
* Improve documentation and assertions; add tree serialization tests
* Remove Sprout code, which will be moved to another branch
* Add todo! in Sprout tree append()
* Remove stub request, response *Anchor* handling for now
* Add test for validating Sapling note commitment tree using test blocks
* Increase database version (new columns added for note commitment trees and anchors)
* Update test to make sure the order of sapling_note_commitments() is being tested
* Improve comments and structure of the test
* Improve variable names again
* Rustfmt
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
* Reject transparent output double-spends
Check that transparent spends use unspent outputs from:
* earlier transaction in the same block,
* earlier blocks in the parent non-finalized chain, or
* the finalized state.
* Fixup UTXOs in proptests
* Add a comment
* Clarify a consensus rule implementation
* Fix an incorrect comment
* Fix an incorrect error message
* Clarify a comment
* Document `unspent_utxos`
* Simplify the UTXO check
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Further simplify and fix the UTXO check
- split each error case into a separate check
- combine `contains` and `insert`
- add a missing check against the non-finalized unspent UTXOs
- rename arguments and edit error strings for clarity
* Share test methods between check test modules
* Make some chain fields available to tests
* Make error field names consistent with transparent::Input
* WIP: Add tests for UTXO double-spends
- accept output and spend in the same block
- accept output and spend in a later block
- reject output and double-spend all in the same block
- reject output then double-spend in a later block
- reject output, spend, then double-spend all in different blocks
* Use Extend rather than multiple pushes
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Use Extend for more pushes
* Limit the number of proptest cases, to speed up tests
* Test rejection of UTXOs that were never in the chain
* Test rejection of spends of later transactions in the same block
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Add sapling and orchard duplicate nullifier errors
* Reject duplicate finalized sapling and orchard nullifiers
Reject duplicate sapling and orchard nullifiers in a new block,
when the block is added to a non-finalized chain,
and the duplicate nullifier is already in the finalized state.
* Reject duplicate non-finalized sapling and orchard nullifiers
Reject duplicate sapling and orchard nullifiers in a new block,
when the block is added to a non-finalized chain,
and the duplicate nullifier is in:
* the same shielded data,
* the same transaction,
* the same block, or
* an earlier block in the non-finalized chain.
* Refactor sprout nullifier tests to remove common code
* Add sapling nullifier tests
Test that the state rejects duplicate sapling nullifiers in a new block,
when the block is added to a non-finalized chain,
and the duplicate nullifier is in:
* the same shielded data,
* the same transaction,
* the same block,
* an earlier block in the non-finalized chain, or
* the finalized state.
* Add orchard nullifier tests
Test that the state rejects duplicate orchard nullifiers in a new block,
when the block is added to a non-finalized chain,
and the duplicate nullifier is in:
* the same shielded data,
* the same transaction,
* the same block,
* an earlier block in the non-finalized chain, or
* the finalized state.
* Check for specific nullifiers in the state in tests
* Replace slices with vectors in arguments
* Remove redundant code and variables
* Simplify sapling TransferData tests
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Remove an extra :
* Remove redundant vec!
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Add an OrderedUtxo type for validation of spends within a block
This change allows us to check that transparent spends use outputs from
earlier in their block. (But we don't actually do that check yet.)
We need to keep the order of UTXOs when we're contextually verifying
each new block that is added to a chain. But the block order is
irrelevant for UTXOs stored in the state.
* Take ownership in utxos_from_ordered_utxos
* Delete a confusing comment
* Document Ord for Chain and Proof of Work
* Create a NonFinalizedState::new method
And add some debug and clone derives.
* Test that block rejection restores internal non-finalized chain states
As part of this change, add `eq_internal_state` methods,
and proptests for them.
* Check that the chain's nullifiers are not modified on error
* Clarify a test description
* Refactor loop for readability
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Fix a variable name typo
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Reject duplicate sprout nullifiers in the state
* Improve docs and error messages
* Clarify "must be present" assert logs
* Move nullifier checks to their own module
Also:
* make non-finalized nullifier checks and errors generic over
sprout, sapling, and orchard
* create and update module and function documentation
* Fix a block type name in docs
* Move state assertions or skip them during tests
These changes enable state testing, while still asserting in production.
* Add sprout duplicate nullifier tests
* Improve comments
* Set value balance to 0 to pass future chain value pool checks
* Test finalized state in sprout nullifier accept test
* Replace assert with expect
* Improve assertion messages
* add nullifier methods to orchard
* store orchard nullifiers
* bump database version
* update `IntoDisk`
* support V5 in `UpdateWith`
* add a test for finalized state
* Use the latest network upgrade in state proptests
* Speedup proptests for Chain struct in zebra-state
* Add teor2345 requested changes
* Fix type for DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES
* More costs for PROPTEST_CASES
* start refactoring transaction v4 for transaction v5
- move ShieldedData to sapling
- add AnchorVariant
- rename shielded_data to sapling_shielded data in V4
- move value_balance into ShieldedData
- update prop tests for new structure
* add AnchorVariant to Spend
- make anchor types available from sapling crate
- update serialize
* change shielded_balances_match() arguments
* change variable name anchor to shared_anchor in ShieldedData
* fix empty value balance serialization
* use AnchorV in shielded spends
* Rename anchor to per_spend_anchor
* Use nullifiers function directly in non-finalized state
* Use self.value_balance instead of passing it as an argument
* Add missing fields to ShieldedData PartialEq
* Derive Copy for tag types
* Add doc comments for ShieldedData refactor
* Implement a per-spend anchor compatibility iterator
Co-authored-by: teor <teor@riseup.net>
* add transaction V5 stub
* add v5_strategy
* deduplicate version group ids
* Update comment for V5 transactions
* Add V5 transactions to non_finalized_state
Currently these are all `unimplemented!(...)`
* Fix struct matches
* Apply trivial panic message changes
* add zcash_deserialize for V5
* make all tx versions explicit in sprout and sapling nullifier functions
* match exhaustively in sprout and sapling nullifier functions
* fix matches in zebra-consensus
* fix NU5 strategy
* We're still deciding if v5 transactions support Sprout
Co-authored-by: teor <teor@riseup.net>
As a side effect of computing Merkle roots, we build a list of
transaction hashes. Instead of discarding these, add them to
PreparedBlock and FinalizedBlock so that they can be reused rather than
recomputed.
This commit adds Merkle root validation to:
1. the block verifier;
2. the checkpoint verifier.
In the first case, Bitcoin Merkle tree malleability has no effect,
because only a single Merkle tree in each malleablity set is valid (the
others have duplicate transactions).
In the second case, we need to check that the Merkle tree does not contain any
duplicate transactions.
Closes#1385Closes#906
* implement inbound `FindBlocks`
* Handle inbound peer FindHeaders requests
* handle request before having any chain tip
* Split `find_chain_hashes` into smaller functions
Add a `max_len` argument to support `FindHeaders` requests.
Rewrite the hash collection code to use heights, so we can handle the
`stop` hash and "no intersection" cases correctly.
* Split state height functions into "any chain" and "best chain"
* Rename the best chain block method to `best_block`
* Move fmt utilities to zebra_chain::fmt
* Summarise Debug for some Message variants
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
Temporary fix so that Zebra's default logs support a typical workflow:
1. Developer or user runs Zebra with the default config
2. They send the logs to a terminal
3. When they see a bug, they copy-paste the last few log lines into a
bug report
This is the same change that was merged in #1373 and reverted in #1375.
We'll create a consistent logging design for Zebra in ticket #1381.
This commit changes the state system and database format to track the
provenance of UTXOs, in addition to the outputs themselves.
Specifically, it tracks the following additional metadata:
- the height at which the UTXO was created;
- whether or not the UTXO was created from a coinbase transaction or
not.
This metadata will allow us to:
- check the coinbase maturity consensus rule;
- check the coinbase inputs => no transparent outputs rule;
- implement lookup of transactions by utxo (using the height to find the
block and then scanning the block) for a future RPC mechanism.
Closes#1342
This provides useful and not too noisy output at INFO level. We do an
info-level message on every block commit instead of trying to do one
message every N blocks, because this is useful both for initial block
sync as well as continuous state updates on new blocks.
This change introduces two new types:
- `PreparedBlock`, representing a block which has undergone semantic
validation and has been prepared for contextual validation;
- `FinalizedBlock`, representing a block which is ready to be finalized
immediately;
and changes the `Request::CommitBlock`,`Request::CommitFinalizedBlock`
variants to use these types instead of their previous fields.
This change solves the problem of passing data between semantic
validation and contextual validation, and cleans up the state code by
allowing it to pass around a bundle of data. Previously, the state code
just passed around an `Arc<Block>`, which forced it to needlessly
recompute block hashes and other data, and was incompatible with the
already-known but not-yet-implemented data transfer requirements, namely
passing in the Sprout and Sapling anchors computed during contextual
validation.
This commit propagates the `PreparedBlock` and `FinalizedBlock` types
through the state code but only uses their data opportunistically, e.g.,
changing .hash() computations to use the precomputed hash. In the
future, these structures can be extended to pass data through the
verification pipeline for reuse as appropriate. For instance, these
changes allow the sprout and sapling anchors to be propagated through
the state.
Make tracing messages more concise by omitting information already
contained in a parent span and by shortening messages. This makes them
easier to read.