* Download and load Sprout parameters using zcash_proofs
Also update some librustzcash dependencies, to avoid duplicate dependencies.
* Update upstream orchard to avoid a compilation error
* Skip librustzcash batch refactor for now, to avoid compilation errors
* Change the cache ID, so we actually cache Sprout
* Move existing file checks into zcash_proofs
* Add a 1 hour timeout to parameter file downloads
* Give other tasks priority, before spawning the download task
* Update to the latest version of our modified librustzcash fork
* Change the cache key for Sprout
* Add 40 minutes to CI timeouts for occasional sprout downloads
* Update to zcash_proofs with split downloads
* Check file sizes to help debug parameter load failures in zcash_proofs
* Start the second download once the first has finished in zcash_proofs
* Document the parameter download task
* Stop hashing existing files twice
* validate consensus rule: negative fee not allowed
* fix a test TODO
* fix imports
* move import back
* fix panic text
* join consensus rule check code
* match assertion better in tests
* fix test
* fix consensus rule validation
* remove panics
* Delete a TODO
Co-authored-by: teor <teor@riseup.net>
* Create a `LockTime::unlocked` helper constructor
Returns a `LockTime` that is unlocked at the genesis block.
* Return `Option<LockTime>` from `lock_time` method
Prepare to return `None` for when a transaction has its lock time
disabled.
* Return `None` instead of zero `LockTime`
Because a zero lock time means that the transaction was unlocked at the
genesis block, so it was never actually locked.
* Rephrase zero lock time check comment
Clarify that the check is not redundant, and is necessary for the
genesis transaction.
Co-authored-by: teor <teor@riseup.net>
* Add a `transparent::Input::sequence` getter method
Retrieve a transparent input's sequence number.
* Check if lock time is enabled by a sequence number
Validate the consensus rule that the lock time is only enabled if at
least one transparent input has a value different from `u32::MAX` as its
sequence number.
* Add more Zcash specific details to comment
Explain the Zcash specific lock time behaviors.
Co-authored-by: teor <teor@riseup.net>
* Add `time` field to `Request::Block` variant
The block time to use to check if the transaction was unlocked and
allowed to be included in the block.
* Add `Request::block_time` getter
Returns the block time for the block that owns the transaction being
validated or the current time plus a tolerance for mempool transactions.
* Validate transaction lock times
If they are enabled by a transaction's transparent input sequence
numbers, make sure that they are in the past.
* Add comments with consensus rule parts
Make it easier to map what part of the consensus rule each match arm is
responsible for.
Co-authored-by: teor <teor@riseup.net>
* add testnet test blocks around nu5
* validate coinbase expiration height
* change const name and doc
Co-authored-by: teor <teor@riseup.net>
* change commit location
Co-authored-by: teor <teor@riseup.net>
* use pre Nu5 rules when there is no activation height
* add sapling final root to nu5 test vectors
* fix tests
Co-authored-by: teor <teor@riseup.net>
* Implement incremental note commitment Merkle tree for Sprout
* Add tests for Sprout note commitment tree
* Remove the `Arbitrary` attribute
* Reverse the vector of empty roots
* Add more tests
* Refactor rustdoc
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Refactor rustdoc
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* rustdoc
* Rustdoc
* rustdoc links
* Oops, need the trait in scope to use it
* Avoid accessing the wrapped hash directly
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* rustfmt
* Add typing
* Avoid accessing the wrapped hash directly
* Implement incremental note commitment Merkle tree for Sprout
* Add tests for Sprout note commitment tree
* Remove the `Arbitrary` attribute
* Reverse the vector of empty roots
* Add more tests
* Refactor rustdoc
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Refactor rustdoc
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* rustdoc
* Rustdoc
* rustdoc links
* Oops, need the trait in scope to use it
* Avoid accessing the wrapped hash directly
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* rustfmt
* Add typing
* Avoid accessing the wrapped hash directly
* Add Overwinter final roots (test vectors)
* Test sprout note commitments trees on Overwinter blocks
* Add new test vectors
* Finish the tests for the note commitment trees
* Make the wrapped hash in `Root` private
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* First pass at async Halo2 verification service
Stubs out a batch verifier for the future.
The dependencies for orchard, halo2, librustzcash, zcash_primitives, have
not been resolved.
* Halo2 verifier service and test
* Remove redundant conversion
* Test async halo2 verifier service with pre-computed Orchard shielded data test vectors
* Fix typo
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Assert future result is_ok() in Halo2 verifier test
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Shorten tower::Service trait constraints for Halo2 verifier tests
* Remove commented out trait constraints
* .expect() vs .unwrap() to parse orchard::redpallas::VerificationKey
* Use .to_vec() for some test vectors
* Fix self-referential Display impl
* Fix deps
* Distinguish orchard vs zebra_chain::orchard imports
* Add test that halo2 verifier fails with malformed proof inputs
* Use thiserror for Halo2Error
* Use ZcashFoundation/orchard instead of dconnolly/orchard
* Add a link to the issue to remove the zfnd fork of orchard crate
* Update zebra-consensus/Cargo.toml
Co-authored-by: teor <teor@riseup.net>
* Add note
* Move artificial Orchard shielded data test vectors to zebra-test
* Align brackets
* Tidy some trait constraints and debug statements
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* Upgrade aes and fpe
* Upgrade bellman, bls12_381, jubjub to latest
* Upgrade x25519-dalek to 1.2.0 and curve25519-dalek to 3.2.0 in the Cargo.lock
* Skip outdated hdrhistogram rather than its dependencies
Co-authored-by: teor <teor@riseup.net>
* Cleanup a function that calls zcash_script
* Remove zebra_test::prelude macros that conflict with the Rust prelude
* Add sigops count support to zebra-script
* Check MAX_BLOCK_SIGOPS in the block verifier
* Test MAX_BLOCK_SIGOPS on generated and historic blocks
* Add SAFETY comments for all unsafe zebra-script code
* Explain where the consensus rule comes from
* Remove unused pretty_assertions dependency
* Allow large test block generation functions with the proptest-impl feature
* Replace `as` with `try_into` for integer conversions in unsafe code
* Expand SAFETY comments
* Revert "Remove commented-out code"
This reverts commit 9e69777925f103ee11e5940bba95b896c828839b.
* Implement deserialization for `addrv2` messages
* Limit addr and addrv2 messages to MAX_ADDRS_IN_MESSAGE
* Clarify address version comments
* Minor cleanups and fixes
* Add preallocation tests for AddrV2
* Add serialization tests for AddrV2
* Use prop_assert in AddrV2 proptests
* Use a generic utility method for deserializing IP addresses in `addrv2`
* Document the purpose of a conversion to MetaAddr
* Fix a comment typo, and clarify that comment
* Clarify the unsupported AddrV2 network ID error and enum variant names
```sh
fastmod AddrV2UnimplementedError UnsupportedAddrV2NetworkIdError zebra-network
fastmod Unimplemented Unsupported zebra-network
```
* Fix and clarify unsupported AddrV2 comments
* Replace `panic!` with `unreachable!`
* Clarify a comment about skipping a length check in a test
* Remove a redundant test
* Basic addr (v1) and addrv2 deserialization tests
* Test deserialized IPv4 and IPv6 values in addr messages
* Remove redundant io::Cursor
* Add comments with expected values of address test vectors
* Add a `Duration32::from_days` constructor
Make it simpler to construct a `Duration32` representing a certain
number of days.
* Add `MetaAddr::was_not_recently_seen` method
A helper method to check if a peer was never seen before or if it was
last seen a long time ago. This will be one of the conditions to
consider a peer as unreachable.
* Add `MetaAddr::is_probably_unreachable` method
A helper method to check if a peer should be considered unreachable. It
is considered unreachable if recent connection attempts have failed and
it was not recently seen.
If a peer is considered unreachable, Zebra shouldn't attempt to connect
to it again.
* Do not keep trying to connect to unreachable peer
A peer is probably unreachable if it was last seen a long time ago and
if it's last connection attempt failed.
* Test `was_not_recently_seen`
Redo the calculation on arbitrary `MetaAddr`s.
* Test `is_probably_unreachable`
Redo the calculation on arbitrary `MetaAddr`s.
* Test if probably unreachable peers are ignored
Given an `AddressBook` with a list of arbitrary `MetaAddr`s, check that
none of the peers listed for a reconnection is probably unreachable.
* Rename unit test to improve clarity
Remove the double negative from the name.
Co-authored-by: teor <teor@riseup.net>
* Rename constant to `MAX_RECENT_PEER_AGE`
Make the purpose of the constant clearer.
Co-authored-by: teor <teor@riseup.net>
* Rename method to `last_seen_is_recent`
Remove the double negative from the name.
* Rename method to `is_probably_reachable`
Avoid having to negate the result of the method in security critical
filter.
* Move check into `is_ready_for_connection_attempt`
Make sure the check is used in any place that requires a peer that's
ready for a connection attempt.
* Improve test documention
Describe the goal of the test better.
Co-authored-by: teor <teor@riseup.net>
* Improve `is_probably_reachable` documentation
List the conditions as bullet points.
Co-authored-by: teor <teor@riseup.net>
* Document what happens when peers have no last seen time
Co-authored-by: teor <teor@riseup.net>
* Implement addr v1 serialization using a separate AddrV1 type
* Remove commented-out code
* Split the address serialization code into modules
* Reorder v1 and in_version fields in serialization order
* Fix a missed search-and-replace
* Explain conversion to MetaAddr
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Add CompactSize64 and CompactSizeMessage types
But don't remove read_compactsize and write_compactsize yet.
* Fix CompactSize capitalisation
```sh
fastmod compactSize CompactSize zebra* book
fastmod compactsize CompactSize zebra* book
```
* Make CompactSize patterns consistent with integer lengths
* Replace unwrap_err with asserting is_err
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Replace a panic with an assertion
* Make generic serialization use CompactSizeMessage
* Fix type inference and borrow-checker errors
* Doctest fixes
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* ZIP-401 weighted random mempool eviction
* rename zcash.mempool.total_cost.bytes to zcash.mempool.cost.bytes
Co-authored-by: teor <teor@riseup.net>
* Remove duplicated lines
* Add cost() method to UnminedTx
Update serialization failure messages
* More docs quoting ZIP-401 rules
* Change mempool::Storage::new() to handle Copy-less HashMap, HashSet
* mempool: tidy cost types and evict_one()
* More consensus rule docs
* Refactor calculating mempool costs for Unmined transactions
* Add a note on asympotic performance of calculating weights of txs in mempool
* Bump test mempool / storage config to avoid weighted random cost limits
* Use mempool tx_cost_limit = u64::MAX for some tests
* Remove failing tests for now
* Allow(clippy::field-reassign-with-default) because of a move on a type that doesn't impl Copy
* Fix mistaken doctest formatting
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Increase test timeout for Windows builds
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Limit tx size
Zebra now limits the transaction size in the `zcash_deserialize()` method for
`Transaction`.
* Remove unused error variants (#2941)
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Limit tx size
Zebra now limits the transaction size in the `zcash_deserialize()` method for
`Transaction`.
* Test the tx deserialization limit
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Increment the crates that have new commits since the last version
* Increment the crates that depend on crates that have changed
* Increment the version of `zebra-script`
* Use the `zebrad` version in the `zebra-network` user agent string
* Use the `v1.0.0-alpha.19` git tag in `README.md`
* Copy the draft changelog into `CHANGELOG.md`
* Delete bumps
* Update CHANGELOG.md
Co-authored-by: teor <teor@riseup.net>
* Add newly merged PRs
Co-authored-by: teor <teor@riseup.net>
* Create a new VerifiedUnminedTx containing the miner fee
* Use VerifiedUnminedTx in mempool verification responses
And do a bunch of other cleanups.
* Use VerifiedUnminedTx in mempool download and verifier
* Use VerifiedUnminedTx in mempool storage and verified set
* Impl Display for VerifiedUnminedTx, and some convenience methods
* Use VerifiedUnminedTx in existing tests
* Limit the size of rejection lists when there is a spend conflict
Previously, `insert` would return early with an error,
and skip limiting the rejection list sizes.
* Use prop_assert macros in proptests, rather than assert
* Add `HashSet`s to help spend conflict detection
Keep track of the spent transparent outpoints and the revealed
nullifiers.
Clippy complained that the `ActiveState` had variants with large size
differences, but that was expected, so I disabled that lint on that
`enum`.
* Clear the `HashSet`s when clearing the mempool
Clear them so that they remain consistent with the set of verified
transactions.
* Use `HashSet`s to check for spend conflicts
Store new outputs into its respective `HashSet`, and abort if a
duplicate output is found.
* Remove inserted outputs when aborting
Restore the `HashSet` to its previous state.
* Remove tracked outputs when removing a transaction
Keep the mempool storage in a consistent state when a transaction is
removed.
* Remove tracked outputs when evicting from mempool
Ensure eviction also keeps the tracked outputs consistent with the
verified transactions.
* Refactor to create a `VerifiedSet` helper type
Move the code to handle the output caches into the new type. Also move
the eviction code to make things a little simpler.
* Refactor to have a single `remove` method
Centralize the code that handles the removal of a transaction to avoid
mistakes.
* Move mempool size limiting back to `Storage`
Because the evicted transactions must be added to the rejected list.
* Remove leftover `dbg!` statement
Leftover from some temporary testing code.
Co-authored-by: teor <teor@riseup.net>
* Remove unnecessary `TODO`
It is more speculation than planning, so it doesn't add much value.
Co-authored-by: teor <teor@riseup.net>
* Fix typo in documentation
The verb should match the subject "transactions" which is plural.
Co-authored-by: teor <teor@riseup.net>
* Add a comment to warn about correctness
There's a subtle but important detail in the implementation that should
be made more visible to avoid mistakes in the future.
Co-authored-by: teor <teor@riseup.net>
* Remove outdated comment
Left-over from the attempt to move the eviction into the `VerifiedSet`.
* Improve comment explaining lint removal
Rewrite the comment explaining why the Clippy lint was ignored.
* Check for spend conflicts in `VerifiedSet`
Refactor to avoid API misuse.
* Test rejected transaction rollback
Using two transactions, perform the same test adding a conflict to both
of them to check if the second inserted transaction is properly
rejected. Then remove any conflicts from the second transaction and add
it again. That should work, because if it doesn't it means that when the
second transaction was rejected it left things it shouldn't in the
cache.
* Test removal of multiple transactions
When removing multiple transactions from the mempool storage, all of the
ones requested should be removed and any other transaction should be
still be there afterwards.
* Increase mempool size to 4, so that spend conflict tests work
If the mempool size is smaller than 4,
these tests don't fail on a trivial removal bug.
Because we need a minimum number of transactions in the mempool
to trigger the bug.
Also commit a proptest seed that fails on a trivial removal bug.
(This seed fails if we remove indexes in order,
because every index past the first removes the wrong transaction.)
* Summarise transaction data in proptest error output
* Summarise spend conflict field data in proptest error output
* Summarise multiple removal field data in proptest error output
And replace the very large proptest debug output with the new summary.
Co-authored-by: teor <teor@riseup.net>
* Update versions for zebra v1.0.0-alpha.18 release
* WIP: Initial PR list
* Remove uninteresting version bumps from CHANGELOG
* Categorise and group PRs in CHANGELOG, removing uninteresting PRs
* Further refine and categorise changelog entries
* Fix tag url
* Final changes to CHANGELOG
* Add a changelog description
* Spacing
* Clarify and fix changelog PR descriptions
* Add PRs that are about to be merged
* More slight clarifications
* Spacing
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Check return value of zcash_script_new_precomputed_tx
* Set the NU5 testnet activation height to 1_590_000
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Update Nu5 constants to new values
* Update ZIP-244 test vectors for new branch ID
* Squashed commit of the following:
commit bdb120a249
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 11:54:01 2021 -0400
Use pallas::Base::from_str_vartime() in sinsemilla tests
commit e99fa49258
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 11:45:24 2021 -0400
Compiles
commit a520018114
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 10:15:17 2021 -0400
Incomplete upgrade of deps
* Squashed commit of the following:
commit 8d1b76ec5626517817c3a4d9f3950acc90a359df
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 04:02:26 2021 +0000
Update `zcash_script` to support V5 transactions
Use a newer version of `zcash_script` that has been updated to support
V5 transactions.
commit 371233628ae61e0c25d6ba8f31d9dba42823becb
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 03:06:20 2021 +0000
Update Zcash dependencies
Update some Zcash crates:
- `halo2`
- `incrementalmerkletree' (patch version)
- `orchard` (patch version)
- `zcash_history` (patch version)
- `zcash_note_encryption` (patch version)
- `zcash_primitives` (patch version)
And also update the `group` dependency so that the code remains
compatible.
commit de5cf1ec40c3fc08670fc971cdf3e65e13d9f4c7
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 03:04:13 2021 +0000
Update error message assertion
Use the updated message for the expected error variant.
* Update `zcash_script` to support V5 transactions
Use a newer version of `zcash_script` that has been updated to support
V5 transactions.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
* Rename type parameter to be more explicit
Replace the single letter with a proper name.
* Remove imports for `Request` and `Response`
The type names will conflict with the ones for the mempool service.
* Attach `Mempool` service to the `Crawler`
Add a field to the `Crawler` type to store a way to access the `Mempool`
service.
* Forward crawled transactions to downloader
The crawled transactions are now sent to the transaction downloader and
verifier, to be included in the mempool.
* Derive `Eq` and `PartialEq` for `mempool::Request`
Make it simpler to use the `MockService::expect_request` method.
* Test if crawled transactions are downloaded
Create some dummy crawled transactions, and let the crawler discover
them. Then check if they are forwarded to the mempool to be downloaded
and verified.
* Don't send empty transaction ID list to downloader
Ignore response from peers that don't provide any crawled transactions.
* Log errors when forwarding crawled transaction IDs
Calling the Mempool service should not fail, so if an error happens it
should be visible. However, errors when downloading individual
transactions can happen from time to time, so there's no need for them
to be very visible.
* Document existing `mempool::Crawler` test
Provide some depth as to what the test expect from the crawler's
behavior.
* Refactor to create `setup_crawler` helper function
Make it easier to reuse the common test setup code.
* Simplify code to expect requests
Now that `zebra_network::Request` implement `Eq`, the call can be
simplified into `expect_request`.
* Refactor to create `respond_with_transaction_ids`
A helper function that checks for a network crawl request and responds
with the given list of crawled transaction IDs.
* Refactor to create `crawler_iterator` helper
A function to intercept and respond to the fanned-out requests sent
during a single crawl iteration.
* Refactor to create `respond_to_queue_request`
Reduce the repeated code necessary to intercept and reply to a request
for queuing transactions to be downloaded.
* Add `respond_to_queue_request_with_error` helper
Intercepts a mempool request to queue transactions to be downloaded, and
responds with an error, simulating an internal problem in the mempool
service implementation.
* Derive `Arbitrary` for `NetworkUpgrade`
This is required for deriving `Arbitrary` for some error types.
* Derive `Arbitrary` for `TransactionError`
Allow random transaction errors to be generated for property tests.
* Derive `Arbitrary` for `MempoolError`
Allow random Mempool errors to be generated for property tests.
* Test if errors don't stop the mempool crawler
The crawler should be robust enough to continue operating even if the
mempool service fails to download transactions or even fails to handle
requests to enqueue transactions.
* Reduce the log level for download errors
They should happen regularly, so there's no need to have them with a
high visibility level.
Co-authored-by: teor <teor@riseup.net>
* Stop crawler if service stops
If `Mempool::poll_ready` returns an error, it's because the mempool
service has stopped and can't handle any requests, so the crawler should
stop as well.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Change assertion condition to match message
Refactor the condition so that the expression is easier to compare to
the panic message.
* Change comparison order to match panic message
Change the order of the equality so that it matches the panic message.
* mempool - support transaction expiration
* use `LatestChainTip` instead of state call
* clippy
* remove spawn task
* remove non needed async from function
* remove return value
* add a `expiry_height_mut()` method to `Transaction` for testing purposes
* fix `remove_expired_transactions()`
* add a `mempool_transaction_expiration()` test
* tidy cleanup to `expiry_height()`
* improve docs
* fix the build
* try fix macos build
* extend tests
* add doc to function
* clippy
* fix build
* start tests at block two
* Add `Transaction::spent_outpoints` getter method
Returns an iterator over the UTXO `OutPoint`s spent by the transaction.
* Add `mempool::Error::Conflict` variant
An error representing that a transaction was rejected because it
conflicts with another transaction that's already in the mempool.
* Reject conflicting mempool transactions
Reject including a transaction in the mempool if it spends outputs
already spent by, or reveals nullifiers already revealed by another
transaction in the mempool.
* Fix typo in documentation
Remove the `r` that was incorrectly added.
Co-authored-by: teor <teor@riseup.net>
* Specify that the conflict is a spend conflict
Make the situation clearer, because there are other types of conflict.
Co-authored-by: teor <teor@riseup.net>
* Clarify that the outpoints are from inputs
Because otherwise it could lead to confusion because it could also mean
the outputs of the transaction represented as `OutPoint` references.
Co-authored-by: teor <teor@riseup.net>
* Create `storage::tests::vectors` module
Refactor to follow the convention used for other tests.
* Add an `AtLeastOne::first_mut` method
A getter to allow changing the first element.
* Add an `AtLeastOne::push` method
Allow appending elements to the collection.
* Derive `Arbitrary` for `FieldNotPresent`
This is just to make the code that generates arbitrary anchors a bit
simpler.
* Test if conflicting transactions are rejected
Generate two transactions (either V4 or V5) and insert a conflicting
spend, which can be either a transparent UTXO, or a nullifier for one of
the shielded pools. Check that any attempt to insert both transactions
causes one to be accepted and the other to be rejected.
* Delete a TODO comment that we decided not to do
Co-authored-by: teor <teor@riseup.net>
* Fix an incorrect assertion when the block locator is at the tip
This might have been triggered by receiving block hash gossips
from the new Zebra code.
* Add missing tests for zebra-state requests and responses
Specifically:
* `BlockLocator` (populated state only)
* `FindBlockHashes`
* `FindBlockHeaders`
* Test `FindBlock*` before and after the current block
* Add a specific test for bug #2789
* Refactor collect_best_chain_hashes to avoid manual index calculations
* Reword a comment
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Use `TipAction::Reset` for initialization and out-of-order blocks
Needs tests for:
- cloning a `ChainTipChange` resets the cloned instance
- skipped updates reset the cloned instance
- changing forks resets the cloned instance
* Use `TipAction::Reset` for network upgrade activation blocks
* Use an `if` expression
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Another if expression
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Rename ChainTipReceiver to CurrentChainTip
`fastmod ChainTipReceiver CurrentChainTip zebra*`
* Update chain tip documentation and variable names
* Basic chain tip change implementation, without resets
Also includes the following name changes:
```
fastmod CurrentChainTip LatestChainTip zebra*
fastmod chain_tip_receiver latest_chain_tip zebra*
```
* Clarify the difference between `LatestChainTip` and `ChainTipChange`
* Store precalculated transactions in an `Arc`
Transaction `Hash`es are 32 bytes,
and the minimun transparent transaction size is 54 bytes.
So a full 2MB block can create 1.1MB of transaction hashes.
We use an `Arc` to avoid repeatedly cloning that much data.
* Remove the unused `Block` from `ChainTipBlock`
This drops the block as soon as it isn't needed any more.
Previously, it would stick around until every `ChainTipReceiver`
dropped their `ChainTipBlock`, even if they didn't use the `Block`
at all.
* Re-use finalized blocks for chain tip updates
This avoids serializing and deserializing blocks from the finalized state.
* Optimise tip sender equality checks
* Re-use precalculated block hashes and heights for chain tip updates
* Add chain tip mined transaction IDs
* Doc comment typo
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Always prefer the non-finalized tip in ChainTipSender
This significantly simplifies the internal implementation of ChainTipSender.
Also make the methods and types a bit more generic.
* Update ChainTipSender with blocks, not heights
Also fix a bug where queued non-finalized blocks would clear the chain tip.
* Provide a best tip hash in ChainTip receivers
* Skip finalized blocks once the non-finalized state is active
* Add tip hash and NoChainTip tests
* Remove a redundant finalized tip update
* Skip `None` updates to the finalized tip
The finalized and non-finalized tips never update to `None`
once they have added at least one block.
* Stop committing finalized queued blocks if there is an error
Also return the highest committed queued block.
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Rename BestTipHeight so it can be generalised to ChainTipSender
`fastmod BestTipHeight ChainTipSender zebra*`
For senders:
`fastmod best_tip_height chain_tip_sender zebra*`
For receivers:
`fastmod best_tip_height chain_tip_receiver zebra*`
* Rename best_tip_height module to chain_tip
* Wrap the chain tip watch channel in a ChainTipReceiver type
* Create a ChainTip trait to avoid tricky crate dependencies
And add convenience impls for optional and empty chain tips.
* Use the ChainTip trait in zebra-network
* Replace `Option<ChainTip>` with `NoChainTip`
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Add `Arc<Transaction>` conversions for Transaction IDs
* Use UnminedTxId as the transaction verifier response type
* Use UnminedTx in transaction verifier mempool requests
* Refactor is_mempool into a transaction verifier request method
* Order derives in alphabetical order
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* add value balances to non finalized state
* fix 2 tests
* fix remaining constrain issues in tests
* extend value pool test to non finalized
* WIP: fix tests after adding value pools to non-finalized state (#2647)
* Update Chain::eq_internal_state with Chain.value_balance
Also increase the number of cases in its tests,
because they didn't detect this bug.
* Calculate the chain value pool change before `Chain::push`
Code
- store the chain value pool change in `ContextuallyValidBlock`
- convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos`
(rather than `from` or `into`)
- replace `block_utxos` with `new_outputs` in `PreparedBlock`
- replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock`
Tests
- create test methods for `PreparedBlock` and `ContextuallyValidBlock`
- use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos`
to make tests pass
* fix conflicts
* build `set_current_value_pool()` only for tests
* remove redundant cfgs
* change cfg of set_current_value_pool()
* Clarify some chain field documentation
* Fix bugs in the non-finalized chain value pool calculations
1. Only revert the chain value pool balances when the tip is popped.
Don't modify them when the root is finalized.
2. Only update or revert the chain value pool balances once per block.
(Previously, the block changes were multiplied by the number of *transactions*.)
And make corresponding changes to method names and documentation.
* Add extra proptests to try to identify value balance failures
* Simplify some transaction generation code
* Add extra debugging info to value balance errors
* Actually update non-finalized chain value pools in `UpdateWith`
Previously, we were dropping the updated value pools in the `Ok` result.
So the initial (finalized) chain value pool balances were never modified.
* Rename and document value balance add methods
The names and documentation of these methods were confusing.
* Create genesis-based proptests that check chain value pools
* Increase coverage for some test vectors
* Test each chain value balance calculation for blocks 0-10
* Make continuous blockchain test errors easier to debug
* Test the exact transparent pool values for the first few blocks
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Add validation of ZIP-221 and ZIP-244 commitments
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Add auth commitment check in the finalized state
* Reset the verifier when comitting to state fails
* Add explanation comment
* Add test with fake activation heights
* Add generate_valid_commitments flag
* Enable fake activation heights using env var instead of feature
* Also update initial_tip_hash; refactor into progress_from_tip()
* Improve comments
* Add fake activation heights test to CI
* Fix bug that caused commitment trees to not match when generating partial arbitrary chains
* Add ChainHistoryBlockTxAuthCommitmentHash::from_commitments to organize and deduplicate code
* Remove stale comment, improve readability
* Allow overriding with PROPTEST_CASES
* partial_chain_strategy(): don't update note commitment trees when not needed; add comment
Co-authored-by: teor <teor@riseup.net>
* Improve error logging in rejection_restores_internal_state
* Summarise history tree peaks in debug output
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* add value pools to the database
* remove redundant genesis block check
* use update_with_chain_value_pool_change()
* remove constrains
* remove height from the database
* remove calls to chain_value_pool_change
* clippy
* use the "correct" value balances
* bump the database format
* remove everything that is not finalized state
* clippy
* rustfmt
* use all spent utxos
* add new_outputs utxos to all_utxos_spent_by_block
* remove panic
* add finalized state value pool test
* clippy
* clippy 2
* move import
* fix import
* rustfmt
Co-authored-by: teor <teor@riseup.net>
* Rename internal network requests for wide transaction IDs
fastmod TransactionsByHash TransactionsById zebra*
fastmod AdvertiseTransactions AdvertiseTransactionIds zebra*
fastmod MempoolTransactions MempoolTransactionIds zebra*
fastmod TransactionHashes TransactionIds zebra*
* Update network transaction request/response comments
* Rename a transaction hash method for wide transaction IDs
fastmod transaction_hashes transaction_ids zebra-network
* Add UnminedTxId methods and conversions for InventoryHash
* Map WtxIds to unmined transaction network messages
Also, use UnminedTxId and UnminedTx in:
* Zebra's internal request and response format, and
* external Zcash network protocol messages.
* Enable WtxId mempool inventory tracking for peers
* Further clarify transaction IDs
* Use Witnessed rather than Wide for transaction IDs
And rename narrow to legacy when it only applies to v1-v4 transactions.
Otherwise, rename it to mined ID.
* Rename a missed binding
* Remove an incorrectly named binding
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Make the `AuthDigest` display order match transaction IDs
And derive `Hash`, just like transaction IDs.
Don't derive `serde` for now, because it's not needed.
* Move transaction::Hash test to tests module
* Add a simple AuthDigest display order test
* Add a WtxId type for wide transaction IDs
* Add conversions between transaction IDs and bytes
* Use the WtxId type in external network protocol messages
* Generate chains with valid chain value pool balances
* Move MAX_PARTIAL_CHAIN_BLOCKS to zebra-chain
* Fix generated value overflow based on the maximum number of values
And split it into its own method.
* Split fix_remaining_value into smaller methods
* Remove unused methods
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree
* HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq
* ZIP-221: Validate chain history commitments in the non-finalized state (#2301)
* sketch of implementation
* refined implementation; still incomplete
* update librustzcash, change zcash_history to work with it
* simplified code per review; renamed MMR to HistoryTree
* expand HistoryTree implementation
* handle and propagate errors
* simplify check.rs tracing
* add suggested TODO
* add HistoryTree::prune
* fix bug in pruning
* fix compilation of tests; still need to make them pass
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* improvements from code review
* improve check.rs comments and variable names
* fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests
* fix finalized_state proptest
* fix non_finalized_state tests by setting the correct commitments
* renamed mmr.rs to history_tree.rs
* Add HistoryTree struct
* expand non_finalized_state protest
* fix typo
* Add HistoryTree struct
* Update zebra-chain/src/primitives/zcash_history.rs
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* fix formatting
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* history_tree.rs: fixes from code review
* fixes to work with updated HistoryTree
* Improvements from code review
* Add Debug implementations to allow comparing Chains with proptest_assert_eq
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Improvements from code review
* Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Make Amount arithmetic more generic
To modify generated amounts, we need some extra operations on `Amount`.
We also need to extend existing operations to both `NonNegative` and
`NegativeAllowed` amounts.
* Add a constrain method for ValueBalance
* Derive Eq for ValueBalance
* impl Neg for ValueBalance
* Make some Amount arithmetic expectations explicit
* Explain why we use i128 for multiplication
And expand the overflow error details.
* Expand Amount::sum error details
* Make amount::Error field order consistent
* Rename an amount::Error variant to Constraint, so it's clearer
* Add specific pool variants to ValueBalanceError
* Update coinbase remaining value consensus rule comment
This consensus rule was updated recently to include coinbase transactions,
but Zebra doesn't check block subsidy or miner fees yet.
* Add test methods for modifying transparent values and shielded value balances
* Temporarily set values and value balances to zero in proptests
In both generated chains and proptests that construct their own transactions.
Using zero values reduces value calculation and value check test coverage.
A future change will use non-zero values, and fix them so the check passes.
* Add extra fields to remaining transaction value errors
* Swap the transparent value balance sign to match shielded value balances
This makes the signs of all the chain value pools consistent.
* Use a NonNegative constraint for transparent values
This fix:
* makes the type signature match the consensus rules
* avoids having to write code to handle negative values
* Allocate total generated transaction input value to outputs
If there isn't enough input value for an output, set it to zero.
Temporarily reduce all generated values to avoid overflow.
(We'll remove this workaround when we calculate chain value balances.)
* Consistently use ValueBalanceError for ValueBalances
* Make the value balance signs match the spec
And rename and document methods so their signs are clearer.
* Convert amount::Errors to specific pool ValueBalanceErrors
* Move some error changes to the next PR
* Add extra info to remaining transaction value errors (#2585)
* Distinguish between overflow and negative remaining transaction value errors
And make some error types cloneable.
* Add methods for updating chain value pools (#2586)
* Move amount::test to amount::tests:vectors
* Make ValueBalance traits more consistent with Amount
- implement Add and Sub variants with Result and Assign
- derive Hash
* Clarify some comments and expects
* Create ValueBalance update methods for blocks and transactions
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Make Amount arithmetic more generic
To modify generated amounts, we need some extra operations on `Amount`.
We also need to extend existing operations to both `NonNegative` and
`NegativeAllowed` amounts.
* Add a constrain method for ValueBalance
* Derive Eq for ValueBalance
* impl Neg for ValueBalance
* Make some Amount arithmetic expectations explicit
* Explain why we use i128 for multiplication
And expand the overflow error details.
* Expand Amount::sum error details
* Make amount::Error field order consistent
* Rename an amount::Error variant to Constraint, so it's clearer
* Add specific pool variants to ValueBalanceError
* add serialization for value balances
* change test names
* change panic messages
* add a deserialization test
* return the errors from `from_bytes()` methods
* add prop test for serialize/deserialize Amount
* Move amount proptests to amount::tests::prop
Co-authored-by: teor <teor@riseup.net>
This improves proptest results in CI and locally.
Proptests should be faster, because they are not discarding 1/16 results.
Failures should be minimised more often, improving failure logs,
and generating proptest seeds locally and in CI.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add ZIP-221 history tree to finalized state
* Improve error / panic handling; improve documentation
* Return error again when preparing batch, fix expect messages
* Fix bug when pushing the Heartwood actiation block to the history tree
* Re-increase database version since it was increased in main
Co-authored-by: teor <teor@riseup.net>
* Add 1 to 3 transactions to generated blocks, rather than always 2
This change improves test coverage.
As a side-effect, it reduces the average number of generated
transactions, which should improve performance.
* Add 1 to max_size generated transparent inputs, rather than always max_size
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Return an error if genesis transparent coinbase data is invalid
This error prevents cryptic errors during genesis coinbase deserialization.
And fix and improve documentation.
* Use the fixed Zcash constant for generated genesis coinbase data
This change is required, because genesis transactions do not have a
coinbase height in their coinbase data.
* Add Orchard support to HistoryTree
* Handle network upgrades in HistoryTree
* Add additional methods to save/load HistoryTree
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Clarification of Entry documentation
* Improvements from code review
* Add HistoryTree tests
* Improved test comments and variable names based on feedback from #2458 on similar test
* Update zebra-chain/src/history_tree.rs
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Use type aliases for V1 and V2 history trees
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
* Make legacy chain limit clearer
That way, it doesn't get confused with the coinbase maturity limit.
* Allow 1-5 transactions in each generated block, not always 5
* rustfmt
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Tidy chain Cargo.toml
* Organize imports
* Add method to get note commitments from all Actions in Orchard shielded data
* Add method to get note commitments from all JoinSplits in Sprout JoinSplitData
* Add Request and Response variants for awaiting anchors
* Add anchors and note commitment trees to finalized state db
* Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees
* Track anchors and note commitment trees in Chain
Append note commitments to their trees when doing update_chain_state_with,
then use the resulting Sapling and Orchard roots to pass to history_tree, and add
new roots to the anchor sets.
* Handle errors when appending to note commitment trees
* Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with
* Implementing note commitments in finalized state
* Finish serialization of Orchard tree; remove old tree when updating finalize state
* Add serialization and finalized state updates for Sprout and Sapling trees
* Partially handle trees in non-finalized state. Use Option for trees in Chain
* Rebuild trees when forking; change finalized state tree getters to not require height
* Pass empty trees to tests; use empty trees by default in Chain
* Also rebuild anchor sets when forking
* Use empty tree as default in finalized state tree getters (for now)
* Use HashMultiSet for anchors in order to make pop_root() work correctly
* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS
* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more
* Apply suggestions from code review
* Add comments about order of note commitments and related methods/fields
* Don't use Option for trees
* Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS
* Remove unneeded anchor set rebuilding in fork()
* Improve proptest formatting
* Add missing comparisons to eq_internal_state
* Renamed sprout::tree::NoteCommitmentTree::hash() to root()
* Improve comments
* Add asserts, add issues to TODOs
* Remove impl Default for Chain since it was only used by tests
* Improve documentation and assertions; add tree serialization tests
* Remove Sprout code, which will be moved to another branch
* Add todo! in Sprout tree append()
* Remove stub request, response *Anchor* handling for now
* Add test for validating Sapling note commitment tree using test blocks
* Increase database version (new columns added for note commitment trees and anchors)
* Update test to make sure the order of sapling_note_commitments() is being tested
* Improve comments and structure of the test
* Improve variable names again
* Rustfmt
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
* Validate transparent coinbase output maturity and shielding
- Add a CoinbaseSpendRestriction enum and Transaction method
- Validate transparent coinbase spends in non-finalized chains
* Don't use genesis created UTXOs for spends in generated block chains
* Refactor out a new_transaction_ordered_outputs function
* Add Transaction::outputs_mut for tests
* Generate valid transparent spends in arbitrary block chains
* When generating blocks, fixup the block contents, then the block hash
* Test that generated chains contain at least one transparent spend
* Make generated chains long enough for reliable tests
* Add transparent and shielded input and output methods to Transaction
* Split chain generation into 3 functions
* Test that unshielded and immature transparent coinbase spends fail
* Comment punctuation
* Clarify a comment
* Clarify probability calculation
* Test that shielded mature coinbase output spends succeed
* add value_balance methods to transparent and shielded
* add value_balance() to transaction
* check the remaining value consensus rule
* change error name
* fix doc and nitpick
* refactor value_balance() method for joinsplit
* changes to value_balance() of Inputs
* implement joinsplits() method(not working)
* remove created methods
* remove special case
* change return error in utilities
* move utils functions to transaction methods
* fix the docs
* simplify some code
* add constrains explicitly
* remove turbofish
* refactor some transaction methods
* fix value balance signs, add docs
* simplify some code
* avoid panic in consensus check
* add missing doc
* move remaining value balance check to the state
* make changes from the last review
Co-authored-by: teor <teor@riseup.net>
* Reject transparent output double-spends
Check that transparent spends use unspent outputs from:
* earlier transaction in the same block,
* earlier blocks in the parent non-finalized chain, or
* the finalized state.
* Fixup UTXOs in proptests
* Add a comment
* Clarify a consensus rule implementation
* Fix an incorrect comment
* Fix an incorrect error message
* Clarify a comment
* Document `unspent_utxos`
* Simplify the UTXO check
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Further simplify and fix the UTXO check
- split each error case into a separate check
- combine `contains` and `insert`
- add a missing check against the non-finalized unspent UTXOs
- rename arguments and edit error strings for clarity
* Share test methods between check test modules
* Make some chain fields available to tests
* Make error field names consistent with transparent::Input
* WIP: Add tests for UTXO double-spends
- accept output and spend in the same block
- accept output and spend in a later block
- reject output and double-spend all in the same block
- reject output then double-spend in a later block
- reject output, spend, then double-spend all in different blocks
* Use Extend rather than multiple pushes
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Use Extend for more pushes
* Limit the number of proptest cases, to speed up tests
* Test rejection of UTXOs that were never in the chain
* Test rejection of spends of later transactions in the same block
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* add a zero() method to Amount
* add a value balance type
* change some docs
* rename methods
* Doc changes
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* add getters and setters for `ValueBalance`
* remove commented out code
* impl Add for ValueBalance
* split the tests
* change tests
* fix derives
* change default() to zero()
* remove default constraint
* use matches!
* separate testing code into submodules
* change mod struct
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* implement Sum for Amount
* check overflows
* add a `zero()` method to `Amount`
* impl iter::Sum<&Amount<C>> for Result<Amount<C>>
And modify the tests so they test both reference and value based sums.
* use `try_fold()`
* change error doc
* use iter::repeat()
* fix test
Co-authored-by: teor <teor@riseup.net>
* Add an OrderedUtxo type for validation of spends within a block
This change allows us to check that transparent spends use outputs from
earlier in their block. (But we don't actually do that check yet.)
We need to keep the order of UTXOs when we're contextually verifying
each new block that is added to a chain. But the block order is
irrelevant for UTXOs stored in the state.
* Take ownership in utxos_from_ordered_utxos
* Delete a confusing comment
* Support incomplete Pallas addition, all the way down
* Check Orchard key derivation against test vectors (currently not passing)
* Fix up Orchard keys, notes, nullifiers, address, and their tests
* Fix Incoming Viewing Key generation
* Move around test vectors, impl From<SpendingKey> for FullViewingKey
* Add orchard test vectors module
* Pull in and use the Sinsemilla test vectors
* Test Pallas group hashes for Orchard with test vectors
* Move Orchard Arbitrary implementations to arbitrary.rs
* Improvements from code review
* Derive Eq instead of implementing it for SpendAuthorizingKey
* Dedupe Orchard NoteCommitment::extract_x; fix documentation
* Update zebra-chain/src/orchard/note.rs
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add the constraint name to the Amount debug format
* Test consensus-critical serialization for Amount
Previously we were testing `serde` and `bincode` serialization,
which uses a completely different code path.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>