Commit Graph

1021 Commits

Author SHA1 Message Date
teor 0ef4629232
Automatically download and load sprout parameters (#3085)
* Download and load Sprout parameters using zcash_proofs

Also update some librustzcash dependencies, to avoid duplicate dependencies.

* Update upstream orchard to avoid a compilation error

* Skip librustzcash batch refactor for now, to avoid compilation errors

* Change the cache ID, so we actually cache Sprout

* Move existing file checks into zcash_proofs

* Add a 1 hour timeout to parameter file downloads

* Give other tasks priority, before spawning the download task

* Update to the latest version of our modified librustzcash fork

* Change the cache key for Sprout

* Add 40 minutes to CI timeouts for occasional sprout downloads

* Update to zcash_proofs with split downloads

* Check file sizes to help debug parameter load failures in zcash_proofs

* Start the second download once the first has finished in zcash_proofs

* Document the parameter download task

* Stop hashing existing files twice
2021-11-25 13:26:32 -03:00
Alfredo Garcia a61eae0065
Validate miner transaction fees (#3067)
* validate consensus rule: negative fee not allowed

* fix a test TODO

* fix imports

* move import back

* fix panic text

* join consensus rule check code

* match assertion better in tests

* fix test

* fix consensus rule validation

* remove panics

* Delete a TODO

Co-authored-by: teor <teor@riseup.net>
2021-11-24 03:36:17 +00:00
Janito Vaqueiro Ferreira Filho ec2c980bb1
Validate transaction lock times (#3060)
* Create a `LockTime::unlocked` helper constructor

Returns a `LockTime` that is unlocked at the genesis block.

* Return `Option<LockTime>` from `lock_time` method

Prepare to return `None` for when a transaction has its lock time
disabled.

* Return `None` instead of zero `LockTime`

Because a zero lock time means that the transaction was unlocked at the
genesis block, so it was never actually locked.

* Rephrase zero lock time check comment

Clarify that the check is not redundant, and is necessary for the
genesis transaction.

Co-authored-by: teor <teor@riseup.net>

* Add a `transparent::Input::sequence` getter method

Retrieve a transparent input's sequence number.

* Check if lock time is enabled by a sequence number

Validate the consensus rule that the lock time is only enabled if at
least one transparent input has a value different from `u32::MAX` as its
sequence number.

* Add more Zcash specific details to comment

Explain the Zcash specific lock time behaviors.

Co-authored-by: teor <teor@riseup.net>

* Add `time` field to `Request::Block` variant

The block time to use to check if the transaction was unlocked and
allowed to be included in the block.

* Add `Request::block_time` getter

Returns the block time for the block that owns the transaction being
validated or the current time plus a tolerance for mempool transactions.

* Validate transaction lock times

If they are enabled by a transaction's transparent input sequence
numbers, make sure that they are in the past.

* Add comments with consensus rule parts

Make it easier to map what part of the consensus rule each match arm is
responsible for.

Co-authored-by: teor <teor@riseup.net>
2021-11-23 05:53:53 +00:00
Alfredo Garcia dbd49a3f00
Validate coinbase expiration height (#3082)
* add testnet test blocks around nu5

* validate coinbase expiration height

* change const name and doc

Co-authored-by: teor <teor@riseup.net>

* change commit location

Co-authored-by: teor <teor@riseup.net>

* use pre Nu5 rules when there is no activation height

* add sapling final root to nu5 test vectors

* fix tests

Co-authored-by: teor <teor@riseup.net>
2021-11-23 05:17:05 +00:00
Pili Guerra 26b3a50e01
Updates for zebra v1.0.0-beta.1 release (#3073)
* Update versions for zebra v1.0.0-beta.1 release

* Adding original PR list for comparison and tracking as PRs merge

* First pass at categorising changes

* Merge and clarify description of related changes

* Remove or merge trivial changes

* Improve change descriptions

* Add new PRs merged

* CHANGELOG: Improve release summary

* CHANGELOG: categorise changes further

* README: Remove resolved issues and items

* Update CHANGELOG.md

Co-authored-by: teor <teor@riseup.net>

* CHANGELOG: Add new PRs merged

* CHANGELOG: Move change category

* CHANGELOG: Update release date ready for tagging

Co-authored-by: teor <teor@riseup.net>
2021-11-19 13:05:11 +01:00
Conrado Gouvea 9b17bbce3d
Add missing nSpendsSapling limit check (#3076) 2021-11-19 00:36:50 +00:00
dependabot[bot] 1643d394bf
Bump fpe from 0.5.0 to 0.5.1 (#3066)
Bumps [fpe](https://github.com/str4d/fpe) from 0.5.0 to 0.5.1.
- [Release notes](https://github.com/str4d/fpe/releases)
- [Changelog](https://github.com/str4d/fpe/blob/main/CHANGELOG.md)
- [Commits](https://github.com/str4d/fpe/commits)

---
updated-dependencies:
- dependency-name: fpe
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
2021-11-18 23:50:00 +00:00
Marek 8963007397
Sprout note commitment trees (#3051)
* Implement incremental note commitment Merkle tree for Sprout

* Add tests for Sprout note commitment tree

* Remove the `Arbitrary` attribute

* Reverse the vector of empty roots

* Add more tests

* Refactor rustdoc

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Refactor rustdoc

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* rustdoc

* Rustdoc

* rustdoc links

* Oops, need the trait in scope to use it

* Avoid accessing the wrapped hash directly

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* rustfmt

* Add typing

* Avoid accessing the wrapped hash directly

* Implement incremental note commitment Merkle tree for Sprout

* Add tests for Sprout note commitment tree

* Remove the `Arbitrary` attribute

* Reverse the vector of empty roots

* Add more tests

* Refactor rustdoc

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Refactor rustdoc

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* rustdoc

* Rustdoc

* rustdoc links

* Oops, need the trait in scope to use it

* Avoid accessing the wrapped hash directly

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* rustfmt

* Add typing

* Avoid accessing the wrapped hash directly

* Add Overwinter final roots (test vectors)

* Test sprout note commitments trees on Overwinter blocks

* Add new test vectors

* Finish the tests for the note commitment trees

* Make the wrapped hash in `Root` private

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
2021-11-18 23:05:52 +00:00
Conrado Gouvea 88b09c812a
Check nSpendsSapling, nOutputsSapling, and nActionsOrchard 2^16 limit (#3069)
* Check nSpendsSapling, nOutputsSapling, and nActionsOrchard 2^16 limit

* Apply suggestions from code review

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Removed not required #[macro_use]

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-18 15:06:07 -03:00
Deirdre Connolly eda83ebe0e
Async Halo2 verifier service (#2645)
* First pass at async Halo2 verification service

Stubs out a batch verifier for the future.
The dependencies for orchard, halo2, librustzcash, zcash_primitives, have
not been resolved.

* Halo2 verifier service and test

* Remove redundant conversion

* Test async halo2 verifier service with pre-computed Orchard shielded data test vectors

* Fix typo

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Assert future result is_ok() in Halo2 verifier test

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Shorten tower::Service trait constraints for Halo2 verifier tests

* Remove commented out trait constraints

* .expect() vs .unwrap() to parse orchard::redpallas::VerificationKey

* Use .to_vec() for some test vectors

* Fix self-referential Display impl

* Fix deps

* Distinguish orchard vs zebra_chain::orchard imports

* Add test that halo2 verifier fails with malformed proof inputs

* Use thiserror for Halo2Error

* Use ZcashFoundation/orchard instead of dconnolly/orchard

* Add a link to the issue to remove the zfnd fork of orchard crate

* Update zebra-consensus/Cargo.toml

Co-authored-by: teor <teor@riseup.net>

* Add note

* Move artificial Orchard shielded data test vectors to zebra-test

* Align brackets

* Tidy some trait constraints and debug statements

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
2021-11-17 13:26:15 +10:00
Deirdre Connolly 90da94fff3
Upgrade crypto deps (#3059)
* Upgrade aes and fpe

* Upgrade bellman, bls12_381, jubjub to latest

* Upgrade x25519-dalek to 1.2.0 and curve25519-dalek to 3.2.0 in the Cargo.lock

* Skip outdated hdrhistogram rather than its dependencies

Co-authored-by: teor <teor@riseup.net>
2021-11-16 19:47:54 +00:00
teor 1df3bdb089
Check MAX_BLOCK_SIGOPS in the block verifier (#3049)
* Cleanup a function that calls zcash_script

* Remove zebra_test::prelude macros that conflict with the Rust prelude

* Add sigops count support to zebra-script

* Check MAX_BLOCK_SIGOPS in the block verifier

* Test MAX_BLOCK_SIGOPS on generated and historic blocks

* Add SAFETY comments for all unsafe zebra-script code

* Explain where the consensus rule comes from

* Remove unused pretty_assertions dependency

* Allow large test block generation functions with the proptest-impl feature

* Replace `as` with `try_into` for integer conversions in unsafe code

* Expand SAFETY comments
2021-11-15 20:55:32 +00:00
Dimitris Apostolou afb8b3d477
Fix typos (#3055)
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-12 19:30:22 +00:00
teor d6f3b3dc9a
Parse received addrv2 messages (#3022)
* Revert "Remove commented-out code"

This reverts commit 9e69777925f103ee11e5940bba95b896c828839b.

* Implement deserialization for `addrv2` messages

* Limit addr and addrv2 messages to MAX_ADDRS_IN_MESSAGE

* Clarify address version comments

* Minor cleanups and fixes

* Add preallocation tests for AddrV2

* Add serialization tests for AddrV2

* Use prop_assert in AddrV2 proptests

* Use a generic utility method for deserializing IP addresses in `addrv2`

* Document the purpose of a conversion to MetaAddr

* Fix a comment typo, and clarify that comment

* Clarify the unsupported AddrV2 network ID error and enum variant names

```sh
fastmod AddrV2UnimplementedError UnsupportedAddrV2NetworkIdError zebra-network
fastmod Unimplemented Unsupported zebra-network
```

* Fix and clarify unsupported AddrV2 comments

* Replace `panic!` with `unreachable!`

* Clarify a comment about skipping a length check in a test

* Remove a redundant test

* Basic addr (v1) and addrv2 deserialization tests

* Test deserialized IPv4 and IPv6 values in addr messages

* Remove redundant io::Cursor

* Add comments with expected values of address test vectors
2021-11-12 00:25:23 +00:00
Conrado Gouvea 6570ebeeb8
ZIP 212: validate Sapling and Orchard output of coinbase transactions (#3029)
* Part of ZIP 212: validate Sapling and Orchard output of coinbase transactions

* Add Orchard test vector

* Revert accidentally deleted link

* Apply suggestions from code review

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Use height from loop

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Fix formatting

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-11-11 22:18:37 +00:00
Janito Vaqueiro Ferreira Filho 11b5a33651
Security: Avoid reconnecting to peers that are likely unreachable (#3030)
* Add a `Duration32::from_days` constructor

Make it simpler to construct a `Duration32` representing a certain
number of days.

* Add `MetaAddr::was_not_recently_seen` method

A helper method to check if a peer was never seen before or if it was
last seen a long time ago. This will be one of the conditions to
consider a peer as unreachable.

* Add `MetaAddr::is_probably_unreachable` method

A helper method to check if a peer should be considered unreachable. It
is considered unreachable if recent connection attempts have failed and
it was not recently seen.

If a peer is considered unreachable, Zebra shouldn't attempt to connect
to it again.

* Do not keep trying to connect to unreachable peer

A peer is probably unreachable if it was last seen a long time ago and
if it's last connection attempt failed.

* Test `was_not_recently_seen`

Redo the calculation on arbitrary `MetaAddr`s.

* Test `is_probably_unreachable`

Redo the calculation on arbitrary `MetaAddr`s.

* Test if probably unreachable peers are ignored

Given an `AddressBook` with a list of arbitrary `MetaAddr`s, check that
none of the peers listed for a reconnection is probably unreachable.

* Rename unit test to improve clarity

Remove the double negative from the name.

Co-authored-by: teor <teor@riseup.net>

* Rename constant to `MAX_RECENT_PEER_AGE`

Make the purpose of the constant clearer.

Co-authored-by: teor <teor@riseup.net>

* Rename method to `last_seen_is_recent`

Remove the double negative from the name.

* Rename method to `is_probably_reachable`

Avoid having to negate the result of the method in security critical
filter.

* Move check into `is_ready_for_connection_attempt`

Make sure the check is used in any place that requires a peer that's
ready for a connection attempt.

* Improve test documention

Describe the goal of the test better.

Co-authored-by: teor <teor@riseup.net>

* Improve `is_probably_reachable` documentation

List the conditions as bullet points.

Co-authored-by: teor <teor@riseup.net>

* Document what happens when peers have no last seen time

Co-authored-by: teor <teor@riseup.net>
2021-11-10 23:51:22 +00:00
teor c0c00b3f0d
Simplify preallocate tests (#3032)
* Simplify preallocation tests using a test function

* Use prop_assert in proptests
2021-11-11 07:53:21 +10:00
teor 85b016756d
Refactor addr v1 serialization using a separate AddrV1 type (#3021)
* Implement addr v1 serialization using a separate AddrV1 type

* Remove commented-out code

* Split the address serialization code into modules

* Reorder v1 and in_version fields in serialization order

* Fix a missed search-and-replace

* Explain conversion to MetaAddr

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-10 06:47:50 +10:00
Alfredo Garcia f7c1907fb6
validate orchard flags in v5 (#3035) 2021-11-08 21:45:54 +00:00
teor b1303ab8d7
Replace read_compactsize and write_compactsize with CompactSizeMessage (#3014)
* Replace read_compactsize and write_compactsize with CompactSizeMessage

* Add tests for CompactSize64

* Add compact size range and conversion tests
2021-11-05 15:24:24 -03:00
teor 01e63da3fe
Create CompactSize64 for non-message-length fields (#3008)
* Add CompactSize64 and CompactSizeMessage types

But don't remove read_compactsize and write_compactsize yet.

* Fix CompactSize capitalisation

```sh
fastmod compactSize CompactSize zebra* book
fastmod compactsize CompactSize zebra* book
```

* Make CompactSize patterns consistent with integer lengths

* Replace unwrap_err with asserting is_err

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Replace a panic with an assertion

* Make generic serialization use CompactSizeMessage

* Fix type inference and borrow-checker errors

* Doctest fixes

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-04 15:57:30 +00:00
Conrado Gouvea e54917ae7c
V1.0.0-beta.0 (#2973)
* V1.0.0-beta.0

* Bump version in install.md
2021-10-29 20:21:26 +00:00
Deirdre Connolly 0381c2347b
ZIP-401: weighted random mempool eviction (#2889)
* ZIP-401 weighted random mempool eviction

* rename zcash.mempool.total_cost.bytes to zcash.mempool.cost.bytes

Co-authored-by: teor <teor@riseup.net>

* Remove duplicated lines

* Add cost() method to UnminedTx

Update serialization failure messages

* More docs quoting ZIP-401 rules

* Change mempool::Storage::new() to handle Copy-less HashMap, HashSet

* mempool: tidy cost types and evict_one()

* More consensus rule docs

* Refactor calculating mempool costs for Unmined transactions

* Add a note on asympotic performance of calculating weights of txs in mempool

* Bump test mempool / storage config to avoid weighted random cost limits

* Use mempool tx_cost_limit = u64::MAX for some tests

* Remove failing tests for now

* Allow(clippy::field-reassign-with-default) because of a move on a type that doesn't impl Copy

* Fix mistaken doctest formatting

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Increase test timeout for Windows builds

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-27 00:21:19 +00:00
Marek 2920d838ff
Limit transaction size in the mempool (#2917)
* Limit tx size

Zebra now limits the transaction size in the `zcash_deserialize()` method for
`Transaction`.

* Remove unused error variants (#2941)

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Limit tx size

Zebra now limits the transaction size in the `zcash_deserialize()` method for
`Transaction`.

* Test the tx deserialization limit

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-25 15:25:28 -03:00
Marek d2a5af0ea5
V1.0.0 alpha.19 (#2907)
* Increment the crates that have new commits since the last version

* Increment the crates that depend on crates that have changed

* Increment the version of `zebra-script`

* Use the `zebrad` version in the `zebra-network` user agent string

* Use the `v1.0.0-alpha.19` git tag in `README.md`

* Copy the draft changelog into `CHANGELOG.md`

* Delete bumps

* Update CHANGELOG.md

Co-authored-by: teor <teor@riseup.net>

* Add newly merged PRs

Co-authored-by: teor <teor@riseup.net>
2021-10-21 12:33:35 +02:00
teor 2d129414e0
Store the transaction fee in the mempool storage (#2885)
* Create a new VerifiedUnminedTx containing the miner fee

* Use VerifiedUnminedTx in mempool verification responses

And do a bunch of other cleanups.

* Use VerifiedUnminedTx in mempool download and verifier

* Use VerifiedUnminedTx in mempool storage and verified set

* Impl Display for VerifiedUnminedTx, and some convenience methods

* Use VerifiedUnminedTx in existing tests
2021-10-18 11:24:37 +10:00
Deirdre Connolly e36ec346e8
Upgrade to ed25519-zebra 3.0.0 (#2864)
* Upgrade to ed25519-zebra 3.0.0

* Revert unrelated extra space

Co-authored-by: teor <teor@riseup.net>
2021-10-12 23:42:44 +00:00
teor 2f0926a8e4
Stop ignoring the mempool conflicting transaction reject list size limit (#2855)
* Limit the size of rejection lists when there is a spend conflict

Previously, `insert` would return early with an error,
and skip limiting the rejection list sizes.

* Use prop_assert macros in proptests, rather than assert
2021-10-12 10:35:50 +10:00
Janito Vaqueiro Ferreira Filho 9e78a8af40
Refactor mempool spend conflict checks to increase performance (#2826)
* Add `HashSet`s to help spend conflict detection

Keep track of the spent transparent outpoints and the revealed
nullifiers.

Clippy complained that the `ActiveState` had variants with large size
differences, but that was expected, so I disabled that lint on that
`enum`.

* Clear the `HashSet`s when clearing the mempool

Clear them so that they remain consistent with the set of verified
transactions.

* Use `HashSet`s to check for spend conflicts

Store new outputs into its respective `HashSet`, and abort if a
duplicate output is found.

* Remove inserted outputs when aborting

Restore the `HashSet` to its previous state.

* Remove tracked outputs when removing a transaction

Keep the mempool storage in a consistent state when a transaction is
removed.

* Remove tracked outputs when evicting from mempool

Ensure eviction also keeps the tracked outputs consistent with the
verified transactions.

* Refactor to create a `VerifiedSet` helper type

Move the code to handle the output caches into the new type. Also move
the eviction code to make things a little simpler.

* Refactor to have a single `remove` method

Centralize the code that handles the removal of a transaction to avoid
mistakes.

* Move mempool size limiting back to `Storage`

Because the evicted transactions must be added to the rejected list.

* Remove leftover `dbg!` statement

Leftover from some temporary testing code.

Co-authored-by: teor <teor@riseup.net>

* Remove unnecessary `TODO`

It is more speculation than planning, so it doesn't add much value.

Co-authored-by: teor <teor@riseup.net>

* Fix typo in documentation

The verb should match the subject "transactions" which is plural.

Co-authored-by: teor <teor@riseup.net>

* Add a comment to warn about correctness

There's a subtle but important detail in the implementation that should
be made more visible to avoid mistakes in the future.

Co-authored-by: teor <teor@riseup.net>

* Remove outdated comment

Left-over from the attempt to move the eviction into the `VerifiedSet`.

* Improve comment explaining lint removal

Rewrite the comment explaining why the Clippy lint was ignored.

* Check for spend conflicts in `VerifiedSet`

Refactor to avoid API misuse.

* Test rejected transaction rollback

Using two transactions, perform the same test adding a conflict to both
of them to check if the second inserted transaction is properly
rejected. Then remove any conflicts from the second transaction and add
it again. That should work, because if it doesn't it means that when the
second transaction was rejected it left things it shouldn't in the
cache.

* Test removal of multiple transactions

When removing multiple transactions from the mempool storage, all of the
ones requested should be removed and any other transaction should be
still be there afterwards.

* Increase mempool size to 4, so that spend conflict tests work

If the mempool size is smaller than 4,
these tests don't fail on a trivial removal bug.
Because we need a minimum number of transactions in the mempool
to trigger the bug.

Also commit a proptest seed that fails on a trivial removal bug.
(This seed fails if we remove indexes in order,
because every index past the first removes the wrong transaction.)

* Summarise transaction data in proptest error output

* Summarise spend conflict field data in proptest error output

* Summarise multiple removal field data in proptest error output

And replace the very large proptest debug output with the new summary.

Co-authored-by: teor <teor@riseup.net>
2021-10-10 23:54:46 +00:00
Alfredo Garcia f1718f5c92
Add `zcash_serialized_size()` to `ZcashSerialize` trait (#2824)
* add a zcash_serialized_size()

* add a size field to `UnminedTx`

* refactor zcash_serialized_size() to don't allocate RAM

* improve performance

Co-authored-by: teor <teor@riseup.net>

* clippy

Co-authored-by: teor <teor@riseup.net>
2021-10-06 22:40:11 +00:00
dependabot[bot] c8af72cd30
Bump tracing from 0.1.28 to 0.1.29 (#2836)
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.28 to 0.1.29.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.28...tracing-0.1.29)

---
updated-dependencies:
- dependency-name: tracing
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-10-06 21:27:04 +00:00
Pili Guerra a85e80a0ec
Update versions for zebra v1.0.0-alpha.18 release (#2828)
* Update versions for zebra v1.0.0-alpha.18 release

* WIP: Initial PR list

* Remove uninteresting version bumps from CHANGELOG

* Categorise and group PRs in CHANGELOG, removing uninteresting PRs

* Further refine and categorise changelog entries

* Fix tag url

* Final changes to CHANGELOG

* Add a changelog description

* Spacing

* Clarify and fix changelog PR descriptions

* Add PRs that are about to be merged

* More slight clarifications

* Spacing

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-10-05 23:22:31 -03:00
Janito Vaqueiro Ferreira Filho 339fefb6e6
Update shared NU5 dependencies, set the NU5 testnet activation network upgrade parameters (#2825)
* Check return value of zcash_script_new_precomputed_tx

* Set the NU5 testnet activation height to 1_590_000

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Update Nu5 constants to new values

* Update ZIP-244 test vectors for new branch ID

* Squashed commit of the following:

commit bdb120a249
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date:   Tue Oct 5 11:54:01 2021 -0400

    Use pallas::Base::from_str_vartime() in sinsemilla tests

commit e99fa49258
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date:   Tue Oct 5 11:45:24 2021 -0400

    Compiles

commit a520018114
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date:   Tue Oct 5 10:15:17 2021 -0400

    Incomplete upgrade of deps

* Squashed commit of the following:

commit 8d1b76ec5626517817c3a4d9f3950acc90a359df
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date:   Tue Oct 5 04:02:26 2021 +0000

    Update `zcash_script` to support V5 transactions

    Use a newer version of `zcash_script` that has been updated to support
    V5 transactions.

commit 371233628ae61e0c25d6ba8f31d9dba42823becb
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date:   Tue Oct 5 03:06:20 2021 +0000

    Update Zcash dependencies

    Update some Zcash crates:

    - `halo2`
    - `incrementalmerkletree' (patch version)
    - `orchard` (patch version)
    - `zcash_history` (patch version)
    - `zcash_note_encryption` (patch version)
    - `zcash_primitives` (patch version)

    And also update the `group` dependency so that the code remains
    compatible.

commit de5cf1ec40c3fc08670fc971cdf3e65e13d9f4c7
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date:   Tue Oct 5 03:04:13 2021 +0000

    Update error message assertion

    Use the updated message for the expected error variant.

* Update `zcash_script` to support V5 transactions

Use a newer version of `zcash_script` that has been updated to support
V5 transactions.

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
2021-10-06 11:08:41 +10:00
Janito Vaqueiro Ferreira Filho 5d9893cf31
Send crawled transaction IDs to downloader (#2801)
* Rename type parameter to be more explicit

Replace the single letter with a proper name.

* Remove imports for `Request` and `Response`

The type names will conflict with the ones for the mempool service.

* Attach `Mempool` service to the `Crawler`

Add a field to the `Crawler` type to store a way to access the `Mempool`
service.

* Forward crawled transactions to downloader

The crawled transactions are now sent to the transaction downloader and
verifier, to be included in the mempool.

* Derive `Eq` and `PartialEq` for `mempool::Request`

Make it simpler to use the `MockService::expect_request` method.

* Test if crawled transactions are downloaded

Create some dummy crawled transactions, and let the crawler discover
them. Then check if they are forwarded to the mempool to be downloaded
and verified.

* Don't send empty transaction ID list to downloader

Ignore response from peers that don't provide any crawled transactions.

* Log errors when forwarding crawled transaction IDs

Calling the Mempool service should not fail, so if an error happens it
should be visible. However, errors when downloading individual
transactions can happen from time to time, so there's no need for them
to be very visible.

* Document existing `mempool::Crawler` test

Provide some depth as to what the test expect from the crawler's
behavior.

* Refactor to create `setup_crawler` helper function

Make it easier to reuse the common test setup code.

* Simplify code to expect requests

Now that `zebra_network::Request` implement `Eq`, the call can be
simplified into `expect_request`.

* Refactor to create `respond_with_transaction_ids`

A helper function that checks for a network crawl request and responds
with the given list of crawled transaction IDs.

* Refactor to create `crawler_iterator` helper

A function to intercept and respond to the fanned-out requests sent
during a single crawl iteration.

* Refactor to create `respond_to_queue_request`

Reduce the repeated code necessary to intercept and reply to a request
for queuing transactions to be downloaded.

* Add `respond_to_queue_request_with_error` helper

Intercepts a mempool request to queue transactions to be downloaded, and
responds with an error, simulating an internal problem in the mempool
service implementation.

* Derive `Arbitrary` for `NetworkUpgrade`

This is required for deriving `Arbitrary` for some error types.

* Derive `Arbitrary` for `TransactionError`

Allow random transaction errors to be generated for property tests.

* Derive `Arbitrary` for `MempoolError`

Allow random Mempool errors to be generated for property tests.

* Test if errors don't stop the mempool crawler

The crawler should be robust enough to continue operating even if the
mempool service fails to download transactions or even fails to handle
requests to enqueue transactions.

* Reduce the log level for download errors

They should happen regularly, so there's no need to have them with a
high visibility level.

Co-authored-by: teor <teor@riseup.net>

* Stop crawler if service stops

If `Mempool::poll_ready` returns an error, it's because the mempool
service has stopped and can't handle any requests, so the crawler should
stop as well.

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-05 10:55:42 +10:00
Janito Vaqueiro Ferreira Filho 800141446c
Tweak some assertions to better match their error messages (#2822)
* Change assertion condition to match message

Refactor the condition so that the expression is easier to compare to
the panic message.

* Change comparison order to match panic message

Change the order of the equality so that it matches the panic message.
2021-10-04 10:31:56 -03:00
teor e5f5ac9ce8
Fix or disable recent nightly clippy lints (#2817)
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-01 15:26:06 +00:00
Deirdre Connolly 679b4768df
Use complete addition instead of incomplete addition inside sinsemilla_commit (#2807) 2021-09-29 13:00:25 -04:00
Alfredo Garcia 37595c4b32
Mempool support for transaction expiration (#2774)
* mempool - support transaction expiration

* use `LatestChainTip` instead of state call

* clippy

* remove spawn task

* remove non needed async from function

* remove return value

* add a `expiry_height_mut()` method to `Transaction` for testing purposes

* fix `remove_expired_transactions()`

* add a `mempool_transaction_expiration()` test

* tidy cleanup to `expiry_height()`

* improve docs

* fix the build

* try fix macos build

* extend tests

* add doc to function

* clippy

* fix build

* start tests at block two
2021-09-29 16:52:44 +00:00
Janito Vaqueiro Ferreira Filho a0d45c38f3
Reject conflicting mempool transactions (#2765)
* Add `Transaction::spent_outpoints` getter method

Returns an iterator over the UTXO `OutPoint`s spent by the transaction.

* Add `mempool::Error::Conflict` variant

An error representing that a transaction was rejected because it
conflicts with another transaction that's already in the mempool.

* Reject conflicting mempool transactions

Reject including a transaction in the mempool if it spends outputs
already spent by, or reveals nullifiers already revealed by another
transaction in the mempool.

* Fix typo in documentation

Remove the `r` that was incorrectly added.

Co-authored-by: teor <teor@riseup.net>

* Specify that the conflict is a spend conflict

Make the situation clearer, because there are other types of conflict.

Co-authored-by: teor <teor@riseup.net>

* Clarify that the outpoints are from inputs

Because otherwise it could lead to confusion because it could also mean
the outputs of the transaction represented as `OutPoint` references.

Co-authored-by: teor <teor@riseup.net>

* Create `storage::tests::vectors` module

Refactor to follow the convention used for other tests.

* Add an `AtLeastOne::first_mut` method

A getter to allow changing the first element.

* Add an `AtLeastOne::push` method

Allow appending elements to the collection.

* Derive `Arbitrary` for `FieldNotPresent`

This is just to make the code that generates arbitrary anchors a bit
simpler.

* Test if conflicting transactions are rejected

Generate two transactions (either V4 or V5) and insert a conflicting
spend, which can be either a transparent UTXO, or a nullifier for one of
the shielded pools. Check that any attempt to insert both transactions
causes one to be accepted and the other to be rejected.

* Delete a TODO comment that we decided not to do

Co-authored-by: teor <teor@riseup.net>
2021-09-28 01:03:08 +00:00
teor 4567701933
Fix an incorrect assertion when the block locator is at the tip (#2789)
* Fix an incorrect assertion when the block locator is at the tip

This might have been triggered by receiving block hash gossips
from the new Zebra code.

* Add missing tests for zebra-state requests and responses

Specifically:
* `BlockLocator` (populated state only)
* `FindBlockHashes`
* `FindBlockHeaders`

* Test `FindBlock*` before and after the current block

* Add a specific test for bug #2789

* Refactor collect_best_chain_hashes to avoid manual index calculations

* Reword a comment

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-09-27 22:43:05 +00:00
Alfredo Garcia 825da448c0
Handle zero transaction expirations (#2782)
* handle zero transaction expirations

* add consensus rule doc

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2021-09-22 13:44:52 +10:00
dependabot[bot] d178e08986 Bump tracing from 0.1.27 to 0.1.28
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.27 to 0.1.28.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.27...tracing-0.1.28)

---
updated-dependencies:
- dependency-name: tracing
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-20 16:33:13 -04:00
dependabot[bot] 21af477a37 Bump tracing from 0.1.26 to 0.1.27
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.26 to 0.1.27.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.26...tracing-0.1.27)

---
updated-dependencies:
- dependency-name: tracing
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-14 21:01:01 -04:00
dependabot[bot] 9b3f6b3105
Bump sha2 from 0.9.6 to 0.9.8 (#2746)
Bumps [sha2](https://github.com/RustCrypto/hashes) from 0.9.6 to 0.9.8.
- [Release notes](https://github.com/RustCrypto/hashes/releases)
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.9.6...sha2-v0.9.8)

---
updated-dependencies:
- dependency-name: sha2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-09-14 17:57:58 -04:00
Conrado Gouvea 8971d62727
Update version strings for v1.0.0 alpha.17 release (#2750)
* Change versions for v1.0.0-alpha.17 release
2021-09-14 17:41:50 +00:00
teor 44ac06775b
Reset `ChainTipChange`s on chain fork and network upgrade activation (#2721)
* Use `TipAction::Reset` for initialization and out-of-order blocks

Needs tests for:
- cloning a `ChainTipChange` resets the cloned instance
- skipped updates reset the cloned instance
- changing forks resets the cloned instance

* Use `TipAction::Reset` for network upgrade activation blocks

* Use an `if` expression

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Another if expression

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-09-02 12:25:42 +10:00
Conrado Gouvea 1ccb2de7c7
Add transaction downloader and verifier (#2679)
* Add transaction downloader

* Changed mempool downloader to be like inbound

* Verifier working (logs result)

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Fix coinbase check for mempool, improve is_coinbase() docs

* Change other downloads.rs docs to reflect the mempool downloads.rs changes

* Change TIMEOUTs to downloads.rs; add docs

* Renamed is_coinbase() to has_valid_coinbase_transaction_inputs() and contains_coinbase_input() to has_any_coinbase_inputs(); reorder checks

* Validate network upgrade for V4 transactions; check before computing sighash (for V5 too)

* Add block_ prefix to downloads and verifier

* Update zebra-consensus/src/transaction.rs

Co-authored-by: teor <teor@riseup.net>

* Add consensus doc; add more Block prefixes

Co-authored-by: teor <teor@riseup.net>
2021-09-02 00:06:20 +00:00
teor b6fe816473
Add a `ChainTipChange` type to `await` chain tip changes (#2715)
* Rename ChainTipReceiver to CurrentChainTip

`fastmod ChainTipReceiver CurrentChainTip zebra*`

* Update chain tip documentation and variable names

* Basic chain tip change implementation, without resets

Also includes the following name changes:
```
fastmod CurrentChainTip LatestChainTip zebra*
fastmod chain_tip_receiver latest_chain_tip zebra*
```

* Clarify the difference between `LatestChainTip` and `ChainTipChange`
2021-09-01 22:31:16 +00:00
teor dcc0dcd26c
Document consensus rules for version group IDs (#2719)
* Document consensus rules for version group IDs

* Remove spaces at end of lines

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-09-01 17:44:46 +00:00
teor a66ecbc16d
Make chain tip updates and access more efficient (#2695)
* Store precalculated transactions in an `Arc`

Transaction `Hash`es are 32 bytes,
and the minimun transparent transaction size is 54 bytes.
So a full 2MB block can create 1.1MB of transaction hashes.

We use an `Arc` to avoid repeatedly cloning that much data.

* Remove the unused `Block` from `ChainTipBlock`

This drops the block as soon as it isn't needed any more.

Previously, it would stick around until every `ChainTipReceiver`
dropped their `ChainTipBlock`, even if they didn't use the `Block`
at all.
2021-08-30 15:42:07 -03:00
teor 2e1d857b27
Add transaction IDs to the chain tip channel (#2686)
* Re-use finalized blocks for chain tip updates

This avoids serializing and deserializing blocks from the finalized state.

* Optimise tip sender equality checks

* Re-use precalculated block hashes and heights for chain tip updates

* Add chain tip mined transaction IDs

* Doc comment typo

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-08-30 12:38:41 +10:00
dependabot[bot] 424095096a Bump sha2 from 0.9.5 to 0.9.6
Bumps [sha2](https://github.com/RustCrypto/hashes) from 0.9.5 to 0.9.6.
- [Release notes](https://github.com/RustCrypto/hashes/releases)
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.9.5...sha2-v0.9.6)

---
updated-dependencies:
- dependency-name: sha2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-08-29 21:01:15 -04:00
Alfredo Garcia 968f20d423
Update versions for zebra v1.0.0-alpha.16 release (#2670)
* bump crate versions

* update zebra-script

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-27 22:13:54 +00:00
teor f9c90b3d86
Implement best tip block hashes for ChainTip receivers (#2677)
* Always prefer the non-finalized tip in ChainTipSender

This significantly simplifies the internal implementation of ChainTipSender.

Also make the methods and types a bit more generic.

* Update ChainTipSender with blocks, not heights

Also fix a bug where queued non-finalized blocks would clear the chain tip.

* Provide a best tip hash in ChainTip receivers

* Skip finalized blocks once the non-finalized state is active

* Add tip hash and NoChainTip tests

* Remove a redundant finalized tip update

* Skip `None` updates to the finalized tip

The finalized and non-finalized tips never update to `None`
once they have added at least one block.

* Stop committing finalized queued blocks if there is an error

Also return the highest committed queued block.

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-08-27 19:18:47 +00:00
teor d2e14b22f9
Refactor BestTipHeight into a generic ChainTip sender and receiver (#2676)
* Rename BestTipHeight so it can be generalised to ChainTipSender

`fastmod BestTipHeight ChainTipSender zebra*`

For senders:
`fastmod best_tip_height chain_tip_sender zebra*`

For receivers:
`fastmod best_tip_height chain_tip_receiver zebra*`

* Rename best_tip_height module to chain_tip

* Wrap the chain tip watch channel in a ChainTipReceiver type

* Create a ChainTip trait to avoid tricky crate dependencies

And add convenience impls for optional and empty chain tips.

* Use the ChainTip trait in zebra-network

* Replace `Option<ChainTip>` with `NoChainTip`

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-08-27 11:34:33 +10:00
teor 2ed6679069
Use unmined types for transaction verifier mempool requests and responses (#2666)
* Add `Arc<Transaction>` conversions for Transaction IDs

* Use UnminedTxId as the transaction verifier response type

* Use UnminedTx in transaction verifier mempool requests

* Refactor is_mempool into a transaction verifier request method

* Order derives in alphabetical order

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-25 21:02:47 +00:00
teor d7eb01d7f0
Add value pool/balances to non-finalized state (#2656)
* add value balances to non finalized state

* fix 2 tests

* fix remaining constrain issues in tests

* extend value pool test to non finalized

* WIP: fix tests after adding value pools to non-finalized state (#2647)

* Update Chain::eq_internal_state with Chain.value_balance

Also increase the number of cases in its tests,
because they didn't detect this bug.

* Calculate the chain value pool change before `Chain::push`

Code
- store the chain value pool change in `ContextuallyValidBlock`
- convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos`
  (rather than `from` or `into`)
- replace `block_utxos` with `new_outputs` in `PreparedBlock`
- replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock`

Tests
- create test methods for `PreparedBlock` and `ContextuallyValidBlock`
- use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos`
  to make tests pass

* fix conflicts

* build `set_current_value_pool()` only for tests

* remove redundant cfgs

* change cfg of set_current_value_pool()

* Clarify some chain field documentation

* Fix bugs in the non-finalized chain value pool calculations

1. Only revert the chain value pool balances when the tip is popped.
   Don't modify them when the root is finalized.

2. Only update or revert the chain value pool balances once per block.
   (Previously, the block changes were multiplied by the number of *transactions*.)

And make corresponding changes to method names and documentation.

* Add extra proptests to try to identify value balance failures

* Simplify some transaction generation code

* Add extra debugging info to value balance errors

* Actually update non-finalized chain value pools in `UpdateWith`

Previously, we were dropping the updated value pools in the `Ok` result.
So the initial (finalized) chain value pool balances were never modified.

* Rename and document value balance add methods

The names and documentation of these methods were confusing.

* Create genesis-based proptests that check chain value pools

* Increase coverage for some test vectors

* Test each chain value balance calculation for blocks 0-10

* Make continuous blockchain test errors easier to debug

* Test the exact transparent pool values for the first few blocks

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 10:57:07 -03:00
Conrado Gouvea bc4194fcb9
ZIP-221/244 auth data commitment validation in checkpoint verifier (#2633)
* Add validation of ZIP-221 and ZIP-244 commitments

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Add auth commitment check in the finalized state

* Reset the verifier when comitting to state fails

* Add explanation comment

* Add test with fake activation heights

* Add generate_valid_commitments flag

* Enable fake activation heights using env var instead of feature

* Also update initial_tip_hash; refactor into progress_from_tip()

* Improve comments

* Add fake activation heights test to CI

* Fix bug that caused commitment trees to not match when generating partial arbitrary chains

* Add ChainHistoryBlockTxAuthCommitmentHash::from_commitments to organize and deduplicate code

* Remove stale comment, improve readability

* Allow overriding with PROPTEST_CASES

* partial_chain_strategy(): don't update note commitment trees when not needed; add comment

Co-authored-by: teor <teor@riseup.net>
2021-08-23 14:17:33 +00:00
teor 4691a871bf
Split ValueBalance methods into NegativeAllowed and NonNegative (#2649)
This helps with type inference when passing values to some methods.

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-20 13:30:38 +00:00
teor 2ec6ae364f
Improve some test errors (#2643)
* Improve error logging in rejection_restores_internal_state

* Summarise history tree peaks in debug output

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-19 19:44:02 +00:00
Alfredo Garcia d2e417cf48
Add value pools to FinalizedState (#2599)
* add value pools to the database

* remove redundant genesis block check

* use update_with_chain_value_pool_change()

* remove constrains

* remove height from the database

* remove calls to chain_value_pool_change

* clippy

* use the "correct" value balances

* bump the database format

* remove everything that is not finalized state

* clippy

* rustfmt

* use all spent utxos

* add new_outputs utxos to all_utxos_spent_by_block

* remove panic

* add finalized state value pool test

* clippy

* clippy 2

* move import

* fix import

* rustfmt

Co-authored-by: teor <teor@riseup.net>
2021-08-19 13:55:36 -03:00
teor c608260256
Support witnessed transaction IDs in zebra-network requests and responses (#2638)
* Rename internal network requests for wide transaction IDs

fastmod TransactionsByHash TransactionsById zebra*
fastmod AdvertiseTransactions AdvertiseTransactionIds zebra*
fastmod MempoolTransactions MempoolTransactionIds zebra*
fastmod TransactionHashes TransactionIds zebra*

* Update network transaction request/response comments

* Rename a transaction hash method for wide transaction IDs

fastmod transaction_hashes transaction_ids zebra-network

* Add UnminedTxId methods and conversions for InventoryHash

* Map WtxIds to unmined transaction network messages

Also, use UnminedTxId and UnminedTx in:
* Zebra's internal request and response format, and
* external Zcash network protocol messages.

* Enable WtxId mempool inventory tracking for peers

* Further clarify transaction IDs

* Use Witnessed rather than Wide for transaction IDs

And rename narrow to legacy when it only applies to v1-v4 transactions.
Otherwise, rename it to mined ID.

* Rename a missed binding
* Remove an incorrectly named binding

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-08-18 22:55:24 +00:00
teor 84c5f6189d
Create types for unmined transactions and their IDs (#2634)
* Create new types for unmined transactions and their IDs

* Add accessor methods for the parts of an unmined transaction ID
2021-08-18 15:52:42 +10:00
Conrado Gouvea 5c5abf6171
ZIP-221 and ZIP-244 commitment validation in non-finalized state (#2609)
* Add validation of ZIP-221 and ZIP-244 commitments

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2021-08-17 11:49:27 -03:00
teor 6c86c8dd0d
Implement a WtxId struct, and use it in Zebra's external network protocol (#2618)
* Make the `AuthDigest` display order match transaction IDs

And derive `Hash`, just like transaction IDs.

Don't derive `serde` for now, because it's not needed.

* Move transaction::Hash test to tests module

* Add a simple AuthDigest display order test

* Add a WtxId type for wide transaction IDs

* Add conversions between transaction IDs and bytes

* Use the WtxId type in external network protocol messages
2021-08-16 21:26:08 +00:00
Alfredo Garcia c3c302309a
Add orchard binding_verification_key (#2441)
* add orchard binding_verification_key

* Merge branch 'main' into issue2102

* Merge branch 'main' into issue2102

* fix the build

* Merge branch 'main' into issue2102

* Merge branch 'main' into issue2102

* Fix docs

Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>

* Merge branch 'main' into issue2102

* rustfmt

* readd binding validation

* remove #2103 from the TODO list

* Merge branch 'main' into issue2102
2021-08-16 15:16:25 -04:00
Pili Guerra 234953e620
Update versions for zebra v1.0.0-alpha.15 release (#2612) 2021-08-16 10:06:26 +00:00
Conrado Gouvea eadca72e75
Implement ZIP-244 authorizing data commitment (auth_digest) (#2547)
* Implement ZIP-244 authorizing data commitment (auth_digest)

* s/Merke/Merkle/

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Fix AuthDataRoot computation to use padded leaves; add tests

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-13 12:58:04 -04:00
teor 76591ceeed
Generate test chains with valid chain value pools (#2597)
* Generate chains with valid chain value pool balances

* Move MAX_PARTIAL_CHAIN_BLOCKS to zebra-chain

* Fix generated value overflow based on the maximum number of values

And split it into its own method.

* Split fix_remaining_value into smaller methods

* Remove unused methods

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-12 12:38:16 +00:00
Conrado Gouvea 94175c6955
Add ZIP-221 history tree to non-finalized state (#2583)
* Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree

* HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq

* ZIP-221: Validate chain history commitments in the non-finalized state (#2301)

* sketch of implementation

* refined implementation; still incomplete

* update librustzcash, change zcash_history to work with it

* simplified code per review; renamed MMR to HistoryTree

* expand HistoryTree implementation

* handle and propagate errors

* simplify check.rs tracing

* add suggested TODO

* add HistoryTree::prune

* fix bug in pruning

* fix compilation of tests; still need to make them pass

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* improvements from code review

* improve check.rs comments and variable names

* fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests

* fix finalized_state proptest

* fix non_finalized_state tests by setting the correct commitments

* renamed mmr.rs to history_tree.rs

* Add HistoryTree struct

* expand non_finalized_state protest

* fix typo

* Add HistoryTree struct

* Update zebra-chain/src/primitives/zcash_history.rs

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* fix formatting

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* history_tree.rs: fixes from code review

* fixes to work with updated HistoryTree

* Improvements from code review

* Add Debug implementations to allow comparing Chains with proptest_assert_eq

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Improvements from code review

* Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 10:42:40 -03:00
teor 298ececabe
Refactor value balances to support generated data modification (#2596)
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-10 14:42:02 +00:00
Conrado Gouvea eac95bdf10
Cache note commitment tree roots (#2584)
* Cache note commitment tree roots

* Add comments to cached root fields

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2021-08-10 10:33:34 -03:00
Conrado Gouvea 9fc49827d6
Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree (#2582)
* Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree

* HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq
2021-08-10 09:51:50 -03:00
teor f09f2a9022
Check remaining transaction value & make value balance signs match the spec (#2566)
* Make Amount arithmetic more generic

To modify generated amounts, we need some extra operations on `Amount`.

We also need to extend existing operations to both `NonNegative` and
`NegativeAllowed` amounts.

* Add a constrain method for ValueBalance

* Derive Eq for ValueBalance

* impl Neg for ValueBalance

* Make some Amount arithmetic expectations explicit

* Explain why we use i128 for multiplication

And expand the overflow error details.

* Expand Amount::sum error details

* Make amount::Error field order consistent

* Rename an amount::Error variant to Constraint, so it's clearer

* Add specific pool variants to ValueBalanceError

* Update coinbase remaining value consensus rule comment

This consensus rule was updated recently to include coinbase transactions,
but Zebra doesn't check block subsidy or miner fees yet.

* Add test methods for modifying transparent values and shielded value balances

* Temporarily set values and value balances to zero in proptests

In both generated chains and proptests that construct their own transactions.

Using zero values reduces value calculation and value check test coverage.
A future change will use non-zero values, and fix them so the check passes.

* Add extra fields to remaining transaction value errors

* Swap the transparent value balance sign to match shielded value balances

This makes the signs of all the chain value pools consistent.

* Use a NonNegative constraint for transparent values

This fix:
* makes the type signature match the consensus rules
* avoids having to write code to handle negative values

* Allocate total generated transaction input value to outputs

If there isn't enough input value for an output, set it to zero.

Temporarily reduce all generated values to avoid overflow.
(We'll remove this workaround when we calculate chain value balances.)

* Consistently use ValueBalanceError for ValueBalances

* Make the value balance signs match the spec

And rename and document methods so their signs are clearer.

* Convert amount::Errors to specific pool ValueBalanceErrors

* Move some error changes to the next PR

* Add extra info to remaining transaction value errors (#2585)

* Distinguish between overflow and negative remaining transaction value errors

And make some error types cloneable.

* Add methods for updating chain value pools (#2586)

* Move amount::test to amount::tests:vectors

* Make ValueBalance traits more consistent with Amount

- implement Add and Sub variants with Result and Assign
- derive Hash

* Clarify some comments and expects

* Create ValueBalance update methods for blocks and transactions

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-09 14:22:26 -03:00
teor fc68240fa0
Add extra arithmetic operations and error details to Amount and ValueBalance (#2577)
* Make Amount arithmetic more generic

To modify generated amounts, we need some extra operations on `Amount`.

We also need to extend existing operations to both `NonNegative` and
`NegativeAllowed` amounts.

* Add a constrain method for ValueBalance

* Derive Eq for ValueBalance

* impl Neg for ValueBalance

* Make some Amount arithmetic expectations explicit

* Explain why we use i128 for multiplication

And expand the overflow error details.

* Expand Amount::sum error details

* Make amount::Error field order consistent

* Rename an amount::Error variant to Constraint, so it's clearer

* Add specific pool variants to ValueBalanceError
2021-08-09 10:13:27 -03:00
Alfredo Garcia 910f0ff5dc
Add Chain ValueBalance serialization (#2554)
* add serialization for value balances

* change test names

* change panic messages

* add a deserialization test

* return the errors from `from_bytes()` methods

* add prop test for serialize/deserialize Amount

* Move amount proptests to amount::tests::prop

Co-authored-by: teor <teor@riseup.net>
2021-08-09 11:22:27 +10:00
Alfredo Garcia 751185d4ec
Use the same value balance sign for transparent and shielded outputs (#2569)
* change signs

* make impl of Neg generic

* change implementation of Neg for Amount

Co-authored-by: teor <teor@riseup.net>
2021-08-08 22:41:34 +00:00
teor 8f4c3b09ea
Add test methods for modifying orchard shielded data and joinsplits (#2580)
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-07 13:23:32 +00:00
teor faae1c08bf
Stop using prop_filter_map to produce valid sapling shielded data (#2579)
This improves proptest results in CI and locally.

Proptests should be faster, because they are not discarding 1/16 results.

Failures should be minimised more often, improving failure logs,
and generating proptest seeds locally and in CI.

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-08-06 17:32:35 +00:00
teor 4eb0344f01
impl TryFrom for vector references for AtLeastOne (#2578)
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-08-06 16:39:32 +00:00
Conrado Gouvea bf713bec91
Add ZIP-221 (history tree) to finalized state (#2553)
* Add ZIP-221 history tree to finalized state

* Improve error / panic handling; improve documentation

* Return error again when preparing batch, fix expect messages

* Fix bug when pushing the Heartwood actiation block to the history tree

* Re-increase database version since it was increased in main

Co-authored-by: teor <teor@riseup.net>
2021-08-05 10:02:37 -03:00
teor 1a18f841f7
Add a random number of transactions to generated blocks, rather than always 2 (#2567)
* Add 1 to 3 transactions to generated blocks, rather than always 2

This change improves test coverage.

As a side-effect, it reduces the average number of generated
transactions, which should improve performance.

* Add 1 to max_size generated transparent inputs, rather than always max_size

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-04 20:48:15 +00:00
teor 7ad5f1cd0a
Use fixed genesis coinbase data in generated genesis blocks (#2568)
* Return an error if genesis transparent coinbase data is invalid

This error prevents cryptic errors during genesis coinbase deserialization.

And fix and improve documentation.

* Use the fixed Zcash constant for generated genesis coinbase data

This change is required, because genesis transactions do not have a
coinbase height in their coinbase data.
2021-08-04 08:43:46 -03:00
Conrado Gouvea fe989e0758
ZIP-221: Add Orchard support to history tree (#2531)
* Add Orchard support to HistoryTree

* Handle network upgrades in HistoryTree

* Add additional methods to save/load HistoryTree

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Clarification of Entry documentation

* Improvements from code review

* Add HistoryTree tests

* Improved test comments and variable names based on feedback from #2458 on similar test

* Update zebra-chain/src/history_tree.rs

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Use type aliases for V1 and V2 history trees

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
2021-08-03 18:33:51 +00:00
teor f9c0620576
Remove unreliable generated chain prevouts tests (#2548)
And adjust the chain lengths for better coverage.
2021-07-30 14:49:25 -03:00
Alfredo Garcia 0325aa2517
Add value_balance() method to Block (#2546)
* add value_balance() method to Block

* Fix a comment typo

Co-authored-by: teor <teor@riseup.net>
2021-07-29 23:46:52 +00:00
teor 4f96a4bb40
Increase coverage for generated chains and proptests (#2540)
* Make legacy chain limit clearer

That way, it doesn't get confused with the coinbase maturity limit.

* Allow 1-5 transactions in each generated block, not always 5

* rustfmt

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-07-30 09:22:10 +10:00
Pili Guerra f59d552721
Update versions for zebra v1.0.0-alpha.14 release (#2537)
Co-authored-by: teor <teor@riseup.net>
2021-07-29 19:42:21 +00:00
Deirdre Connolly e719c46b1b
Track anchors and note commitment trees in zebra-state (#2458)
* Tidy chain Cargo.toml

* Organize imports

* Add method to get note commitments from all Actions in Orchard shielded data

* Add method to get note commitments from all JoinSplits in Sprout JoinSplitData

* Add Request and Response variants for awaiting anchors

* Add anchors and note commitment trees to finalized state db

* Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees

* Track anchors and note commitment trees in Chain

Append note commitments to their trees when doing update_chain_state_with,
then use the resulting Sapling and Orchard roots to pass to history_tree, and add
new roots to the anchor sets.

* Handle errors when appending to note commitment trees

* Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with

* Implementing note commitments in finalized state

* Finish serialization of Orchard tree; remove old tree when updating finalize state

* Add serialization and finalized state updates for Sprout and Sapling trees

* Partially handle trees in non-finalized state. Use Option for trees in Chain

* Rebuild trees when forking; change finalized state tree getters to not require height

* Pass empty trees to tests; use empty trees by default in Chain

* Also rebuild anchor sets when forking

* Use empty tree as default in finalized state tree getters (for now)

* Use HashMultiSet for anchors in order to make pop_root() work correctly

* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS

* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more

* Apply suggestions from code review

* Add comments about order of note commitments and related methods/fields

* Don't use Option for trees

* Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS

* Remove unneeded anchor set rebuilding in fork()

* Improve proptest formatting

* Add missing comparisons to eq_internal_state

* Renamed sprout::tree::NoteCommitmentTree::hash() to root()

* Improve comments

* Add asserts, add issues to TODOs

* Remove impl Default for Chain since it was only used by tests

* Improve documentation and assertions; add tree serialization tests

* Remove Sprout code, which will be moved to another branch

* Add todo! in Sprout tree append()

* Remove stub request, response *Anchor* handling for now

* Add test for validating Sapling note commitment tree using test blocks

* Increase database version (new columns added for note commitment trees and anchors)

* Update test to make sure the order of sapling_note_commitments() is being tested

* Improve comments and structure of the test

* Improve variable names again

* Rustfmt

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
2021-07-29 09:37:18 -04:00
teor 3d792f7195
Validate spends of transparent coinbase outputs (#2525)
* Validate transparent coinbase output maturity and shielding

- Add a CoinbaseSpendRestriction enum and Transaction method
- Validate transparent coinbase spends in non-finalized chains

* Don't use genesis created UTXOs for spends in generated block chains

* Refactor out a new_transaction_ordered_outputs function

* Add Transaction::outputs_mut for tests

* Generate valid transparent spends in arbitrary block chains

* When generating blocks, fixup the block contents, then the block hash

* Test that generated chains contain at least one transparent spend

* Make generated chains long enough for reliable tests

* Add transparent and shielded input and output methods to Transaction

* Split chain generation into 3 functions

* Test that unshielded and immature transparent coinbase spends fail

* Comment punctuation

* Clarify a comment

* Clarify probability calculation

* Test that shielded mature coinbase output spends succeed
2021-07-29 14:23:50 +10:00
Alfredo Garcia ee3c992ca6
Calculate the remaining value in the transparent transaction value pool (#2486)
* add value_balance methods to transparent and shielded

* add value_balance() to transaction

* check the remaining value consensus rule

* change error name

* fix doc and nitpick

* refactor value_balance() method for joinsplit

* changes to value_balance() of Inputs

* implement joinsplits() method(not working)

* remove created methods

* remove special case

* change return error in utilities

* move utils functions to transaction methods

* fix the docs

* simplify some code

* add constrains explicitly

* remove turbofish

* refactor some transaction methods

* fix value balance signs, add docs

* simplify some code

* avoid panic in consensus check

* add missing doc

* move remaining value balance check to the state

* make changes from the last review

Co-authored-by: teor <teor@riseup.net>
2021-07-29 13:49:36 +10:00
teor e6e03247ba
Reject UTXO double spends (#2511)
* Reject transparent output double-spends

Check that transparent spends use unspent outputs from:
* earlier transaction in the same block,
* earlier blocks in the parent non-finalized chain, or
* the finalized state.

* Fixup UTXOs in proptests

* Add a comment

* Clarify a consensus rule implementation

* Fix an incorrect comment

* Fix an incorrect error message

* Clarify a comment

* Document `unspent_utxos`

* Simplify the UTXO check

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Further simplify and fix the UTXO check

- split each error case into a separate check
- combine `contains` and `insert`
- add a missing check against the non-finalized unspent UTXOs
- rename arguments and edit error strings for clarity

* Share test methods between check test modules

* Make some chain fields available to tests

* Make error field names consistent with transparent::Input

* WIP: Add tests for UTXO double-spends

- accept output and spend in the same block
- accept output and spend in a later block
- reject output and double-spend all in the same block
- reject output then double-spend in a later block
- reject output, spend, then double-spend all in different blocks

* Use Extend rather than multiple pushes

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Use Extend for more pushes

* Limit the number of proptest cases, to speed up tests

* Test rejection of UTXOs that were never in the chain

* Test rejection of spends of later transactions in the same block

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-22 23:40:15 +00:00
Alfredo Garcia 429ccf7f79
Add a `ValueBalance` type (#2505)
* add a zero() method to Amount

* add a value balance type

* change some docs

* rename methods

* Doc changes

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* add getters and setters for `ValueBalance`

* remove commented out code

* impl Add for ValueBalance

* split the tests

* change tests

* fix derives

* change default() to zero()

* remove default constraint

* use matches!

* separate testing code into submodules

* change mod struct

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-22 12:49:18 +00:00
teor 6df17ff78c
Clarify indexes in `new_ordered_outputs` (#2510) 2021-07-21 10:53:32 +10:00
Alfredo Garcia 1dae1f49df
implement Sum for Amount (#2500)
* implement Sum for Amount

* check overflows

* add a `zero()` method to `Amount`

* impl iter::Sum<&Amount<C>> for Result<Amount<C>>

And modify the tests so they test both reference and value based sums.

* use `try_fold()`

* change error doc

* use iter::repeat()

* fix test

Co-authored-by: teor <teor@riseup.net>
2021-07-19 23:05:36 +00:00
teor bfc3e4a46c
Add an OrderedUtxo type for transparent spend validation (#2502)
* Add an OrderedUtxo type for validation of spends within a block

This change allows us to check that transparent spends use outputs from
earlier in their block. (But we don't actually do that check yet.)

We need to keep the order of UTXOs when we're contextually verifying
each new block that is added to a chain. But the block order is
irrelevant for UTXOs stored in the state.

* Take ownership in utxos_from_ordered_utxos

* Delete a confusing comment
2021-07-19 10:52:32 -03:00
Deirdre Connolly 8a4add55f1
Fix Orchard implementation, refactor, and add more test vectors (#2445)
* Support incomplete Pallas addition, all the way down

* Check Orchard key derivation against test vectors (currently not passing)

* Fix up Orchard keys, notes, nullifiers, address, and their tests

* Fix Incoming Viewing Key generation

* Move around test vectors, impl From<SpendingKey> for FullViewingKey

* Add orchard test vectors module

* Pull in and use the Sinsemilla test vectors

* Test Pallas group hashes for Orchard with test vectors

* Move Orchard Arbitrary implementations to arbitrary.rs

* Improvements from code review

* Derive Eq instead of implementing it for SpendAuthorizingKey

* Dedupe Orchard NoteCommitment::extract_x; fix documentation

* Update zebra-chain/src/orchard/note.rs

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-07-16 13:52:35 +00:00
teor 049c92374e
clippy: remove a needless borrow (#2501) 2021-07-16 10:29:22 -03:00
teor cd78241d67
Test consensus-critical `Amount` deserialization (#2487)
* Add the constraint name to the Amount debug format

* Test consensus-critical serialization for Amount

Previously we were testing `serde` and `bincode` serialization,
which uses a completely different code path.

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-07-15 21:34:22 +00:00
Deirdre Connolly 48cf52735c
Calculate incremental note commitment trees (#2407)
* Make sapling::tree::NoteCommitmentTree incrementally updateable

* Make orchard::tree::NoteCommitmentTree incrementally updateable

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Changed to incrementalmerkletree implementation; update MerkleCRH^Orchard

* Improvements from review; organize file names (vectors.rs -> tree.rs)

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com>
2021-07-15 13:58:36 +00:00