* Add Orchard support to HistoryTree
* Handle network upgrades in HistoryTree
* Add additional methods to save/load HistoryTree
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Clarification of Entry documentation
* Improvements from code review
* Add HistoryTree tests
* Improved test comments and variable names based on feedback from #2458 on similar test
* Update zebra-chain/src/history_tree.rs
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Use type aliases for V1 and V2 history trees
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
* Design: Fix Transparent Value Sign and Definition
* make the transparent value pool the sum of unspent output values, like other pools
* swap the sign of the transparent value balance
* make the remaining transaction value balance the sum of unspent input values
* update the consensus rules based on recent spec changes
* explain why the transparent chain value pool balance can't be negative
* clarify the guide and consensus rules sections
* Use consistent notation for sprout value balances
* Fix typos
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Make legacy chain limit clearer
That way, it doesn't get confused with the coinbase maturity limit.
* Allow 1-5 transactions in each generated block, not always 5
* rustfmt
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Tidy chain Cargo.toml
* Organize imports
* Add method to get note commitments from all Actions in Orchard shielded data
* Add method to get note commitments from all JoinSplits in Sprout JoinSplitData
* Add Request and Response variants for awaiting anchors
* Add anchors and note commitment trees to finalized state db
* Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees
* Track anchors and note commitment trees in Chain
Append note commitments to their trees when doing update_chain_state_with,
then use the resulting Sapling and Orchard roots to pass to history_tree, and add
new roots to the anchor sets.
* Handle errors when appending to note commitment trees
* Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with
* Implementing note commitments in finalized state
* Finish serialization of Orchard tree; remove old tree when updating finalize state
* Add serialization and finalized state updates for Sprout and Sapling trees
* Partially handle trees in non-finalized state. Use Option for trees in Chain
* Rebuild trees when forking; change finalized state tree getters to not require height
* Pass empty trees to tests; use empty trees by default in Chain
* Also rebuild anchor sets when forking
* Use empty tree as default in finalized state tree getters (for now)
* Use HashMultiSet for anchors in order to make pop_root() work correctly
* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS
* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more
* Apply suggestions from code review
* Add comments about order of note commitments and related methods/fields
* Don't use Option for trees
* Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS
* Remove unneeded anchor set rebuilding in fork()
* Improve proptest formatting
* Add missing comparisons to eq_internal_state
* Renamed sprout::tree::NoteCommitmentTree::hash() to root()
* Improve comments
* Add asserts, add issues to TODOs
* Remove impl Default for Chain since it was only used by tests
* Improve documentation and assertions; add tree serialization tests
* Remove Sprout code, which will be moved to another branch
* Add todo! in Sprout tree append()
* Remove stub request, response *Anchor* handling for now
* Add test for validating Sapling note commitment tree using test blocks
* Increase database version (new columns added for note commitment trees and anchors)
* Update test to make sure the order of sapling_note_commitments() is being tested
* Improve comments and structure of the test
* Improve variable names again
* Rustfmt
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
* Validate transparent coinbase output maturity and shielding
- Add a CoinbaseSpendRestriction enum and Transaction method
- Validate transparent coinbase spends in non-finalized chains
* Don't use genesis created UTXOs for spends in generated block chains
* Refactor out a new_transaction_ordered_outputs function
* Add Transaction::outputs_mut for tests
* Generate valid transparent spends in arbitrary block chains
* When generating blocks, fixup the block contents, then the block hash
* Test that generated chains contain at least one transparent spend
* Make generated chains long enough for reliable tests
* Add transparent and shielded input and output methods to Transaction
* Split chain generation into 3 functions
* Test that unshielded and immature transparent coinbase spends fail
* Comment punctuation
* Clarify a comment
* Clarify probability calculation
* Test that shielded mature coinbase output spends succeed
* add value_balance methods to transparent and shielded
* add value_balance() to transaction
* check the remaining value consensus rule
* change error name
* fix doc and nitpick
* refactor value_balance() method for joinsplit
* changes to value_balance() of Inputs
* implement joinsplits() method(not working)
* remove created methods
* remove special case
* change return error in utilities
* move utils functions to transaction methods
* fix the docs
* simplify some code
* add constrains explicitly
* remove turbofish
* refactor some transaction methods
* fix value balance signs, add docs
* simplify some code
* avoid panic in consensus check
* add missing doc
* move remaining value balance check to the state
* make changes from the last review
Co-authored-by: teor <teor@riseup.net>
* Create a `zebra_state::init_test` helper function
This function will be used as a replacement for `zebra_state::init`
inside tests. It's a simpler alternative because it can ignore any
details that aren't relevant for tests.
* Use `init_test` inside `zebra-state` tests
Update usages of `init` to use `init_test` instead, which simplifies
most cases.
* Use `zebra_state::init_test` in `zebra-consensus`
Replace usages of `zebra_state::init` with the new helper function. This
simplifies the code a bit.
* Add `proptest-impl` feature to `zebra-state`
This prepares the `zebra-state` crate to be able to export some
test-specific helper types and functions.
* Add `arbitrary` module to `zebra-state` root
A separate module to contain the `Prepare` trait, since it's required by
some prop-test strategies and therefore can't be in the `tests` module.
* Replace usages of `tests::Prepare`
Use the same trait but placed in a new module that's accessible based on
the feature flag.
* Remove old `Prepare` trait
It was obsoleted by the new copy in the `arbitrary` module.
* Make `StateService` crate-accessible
Prepare for it to be accessible in some test modules.
* Refactor strategy function import
Import the function directly, instead of just its containing module.
* Move some strategy functions to `tests::setup`
Create a new module for the strategy functions that are only used
internally.
Co-authored-by: teor <teor@riseup.net>
* Make some NonFinalizedState methods test-only
* Rename nullifier tests for clarity
* Reduce test times by reducing default proptest cases
The state tests should be about 4x faster after these changes.
They reduce total state test "user CPU" time to 20-30 seconds on my
machine. Previously it was around 2 minutes.
* Replace multiple pushes with extend
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Reject transparent output double-spends
Check that transparent spends use unspent outputs from:
* earlier transaction in the same block,
* earlier blocks in the parent non-finalized chain, or
* the finalized state.
* Fixup UTXOs in proptests
* Add a comment
* Clarify a consensus rule implementation
* Fix an incorrect comment
* Fix an incorrect error message
* Clarify a comment
* Document `unspent_utxos`
* Simplify the UTXO check
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Further simplify and fix the UTXO check
- split each error case into a separate check
- combine `contains` and `insert`
- add a missing check against the non-finalized unspent UTXOs
- rename arguments and edit error strings for clarity
* Share test methods between check test modules
* Make some chain fields available to tests
* Make error field names consistent with transparent::Input
* WIP: Add tests for UTXO double-spends
- accept output and spend in the same block
- accept output and spend in a later block
- reject output and double-spend all in the same block
- reject output then double-spend in a later block
- reject output, spend, then double-spend all in different blocks
* Use Extend rather than multiple pushes
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Use Extend for more pushes
* Limit the number of proptest cases, to speed up tests
* Test rejection of UTXOs that were never in the chain
* Test rejection of spends of later transactions in the same block
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* add a zero() method to Amount
* add a value balance type
* change some docs
* rename methods
* Doc changes
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* add getters and setters for `ValueBalance`
* remove commented out code
* impl Add for ValueBalance
* split the tests
* change tests
* fix derives
* change default() to zero()
* remove default constraint
* use matches!
* separate testing code into submodules
* change mod struct
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Add sapling and orchard duplicate nullifier errors
* Reject duplicate finalized sapling and orchard nullifiers
Reject duplicate sapling and orchard nullifiers in a new block,
when the block is added to a non-finalized chain,
and the duplicate nullifier is already in the finalized state.
* Reject duplicate non-finalized sapling and orchard nullifiers
Reject duplicate sapling and orchard nullifiers in a new block,
when the block is added to a non-finalized chain,
and the duplicate nullifier is in:
* the same shielded data,
* the same transaction,
* the same block, or
* an earlier block in the non-finalized chain.
* Refactor sprout nullifier tests to remove common code
* Add sapling nullifier tests
Test that the state rejects duplicate sapling nullifiers in a new block,
when the block is added to a non-finalized chain,
and the duplicate nullifier is in:
* the same shielded data,
* the same transaction,
* the same block,
* an earlier block in the non-finalized chain, or
* the finalized state.
* Add orchard nullifier tests
Test that the state rejects duplicate orchard nullifiers in a new block,
when the block is added to a non-finalized chain,
and the duplicate nullifier is in:
* the same shielded data,
* the same transaction,
* the same block,
* an earlier block in the non-finalized chain, or
* the finalized state.
* Check for specific nullifiers in the state in tests
* Replace slices with vectors in arguments
* Remove redundant code and variables
* Simplify sapling TransferData tests
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Remove an extra :
* Remove redundant vec!
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* implement Sum for Amount
* check overflows
* add a `zero()` method to `Amount`
* impl iter::Sum<&Amount<C>> for Result<Amount<C>>
And modify the tests so they test both reference and value based sums.
* use `try_fold()`
* change error doc
* use iter::repeat()
* fix test
Co-authored-by: teor <teor@riseup.net>
* Add an OrderedUtxo type for validation of spends within a block
This change allows us to check that transparent spends use outputs from
earlier in their block. (But we don't actually do that check yet.)
We need to keep the order of UTXOs when we're contextually verifying
each new block that is added to a chain. But the block order is
irrelevant for UTXOs stored in the state.
* Take ownership in utxos_from_ordered_utxos
* Delete a confusing comment
* Support incomplete Pallas addition, all the way down
* Check Orchard key derivation against test vectors (currently not passing)
* Fix up Orchard keys, notes, nullifiers, address, and their tests
* Fix Incoming Viewing Key generation
* Move around test vectors, impl From<SpendingKey> for FullViewingKey
* Add orchard test vectors module
* Pull in and use the Sinsemilla test vectors
* Test Pallas group hashes for Orchard with test vectors
* Move Orchard Arbitrary implementations to arbitrary.rs
* Improvements from code review
* Derive Eq instead of implementing it for SpendAuthorizingKey
* Dedupe Orchard NoteCommitment::extract_x; fix documentation
* Update zebra-chain/src/orchard/note.rs
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add the constraint name to the Amount debug format
* Test consensus-critical serialization for Amount
Previously we were testing `serde` and `bincode` serialization,
which uses a completely different code path.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* First pass at categorising v1.0.0-alpha.13 PRs
* Add recently merged PRs to the changelog
I also rebased the changelog branch to make it easier to see what's
included.
* Reword a few changelog entries for readability
Expand the PR titles a bit so that it's easier to understand what they
refer to.
* Remove some minor changelog entries
They aren't really relevant to external developers.
* Move and reword test coverage entry
Make it clearer what the change means to external developers.
* Add more detail to `MSG_WTX` entry
Make it clear that this has a small impact and that ZIP-239 isn't
actually ready yet.
* Add a short summary for the release
A single paragraph giving a broad overview of what's new.
* Add release date to changelog
Final step before releasing.
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Document Ord for Chain and Proof of Work
* Create a NonFinalizedState::new method
And add some debug and clone derives.
* Test that block rejection restores internal non-finalized chain states
As part of this change, add `eq_internal_state` methods,
and proptests for them.
* Check that the chain's nullifiers are not modified on error
* Clarify a test description
* Refactor loop for readability
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Fix a variable name typo
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Reject duplicate sprout nullifiers in the state
* Improve docs and error messages
* Clarify "must be present" assert logs
* Move nullifier checks to their own module
Also:
* make non-finalized nullifier checks and errors generic over
sprout, sapling, and orchard
* create and update module and function documentation
* Fix a block type name in docs
* Move state assertions or skip them during tests
These changes enable state testing, while still asserting in production.
* Add sprout duplicate nullifier tests
* Improve comments
* Set value balance to 0 to pass future chain value pool checks
* Test finalized state in sprout nullifier accept test
* Replace assert with expect
* Improve assertion messages
* Document the new genesis transaction consensus rule
Zebra previously implemented this rule, but we documented it as a bug in
`zcashd`.
* Document the actual behaviour of zs_insert
* Update state RFC for double-spends and other recent designs
* Update the value pool column family name
* Mark incremental note commitment trees as tentative
* Change history tree type
* Apply suggestions from code review
* Use the block verifier and non-finalized state in the cached state tests
This substantially increases test coverage.
Previously, the cached state tests were configured with
`checkpoint_sync = true`, which only uses the checkpoint
verifier and the finalized state.
* Log the source of blocks in commit_finalized_direct
This lets us check that we're actually testing the non-finalized state
and block verifier in the cached state tests.
It also improves diagnostics for state errors.
* Fail cached state tests if they're using incorrect heights or configs
This makes sure that the cached state tests actually test the transition
from checkpoint to block verification, and the non-finalized state.
* Add methods for getting block nullifiers
These methods will be used in a future PR to check for double-spends.
* Add doc links
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Add a TypeNameToDebug formatter to zebra_chain
This formatter makes it much easier to diagnose proptest errors.
It will be used in a future PR.
Implement Arbitrary and DerefMut for all the formatters.
Also make the formatter type bounds consistent,
to produce better compiler errors.
* Clarify how TypeNameToDebug actually works
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Add panic message to `unimplemented!`
So that it is clear why the panic happened upon initial inspection. Also
include a reference to the mempool epic, so that it's easier to find the
issue that tracks the implementation of the missing code.
* Add panic message that references a tracking issue
Make it easy to find the relevant issue if the panic occurs.
* Remove incomplete and currently unnecessary code
The current implementation works, the commented out code was just a
previous improvement idea, which is now tracked by issue #2473.
* Always use librustzcash for sighash and remove old sighash code
Also added ZIP-143 test vectors
* Remove librustzcash_sighash test that is no longer needed
Conrado Gouvea
teor
Alfredo Garcia
Pili Guerra
Deirdre Connolly
Janito Vaqueiro Ferreira Filho