Commit Graph

529 Commits

Author SHA1 Message Date
Janito Vaqueiro Ferreira Filho f33923f12f
Create a shared Tokio runtime for tests (#2397)
* Add a `zebra_test::RUNTIME` shared runtime

Create a lazily instantiated Tokio runtime that can be shared by tests.

* Split tests that require a shared runtime

Split two tests that were previously in one because of the need to share
a single Tokio runtime. With the `zebra_test::RUNTIME`, they can now
share the runtime without having to be a single test.
2021-07-01 10:30:31 +10:00
teor 9d2d8c8215
Fix intermittent errors in the groth16 verifier tests (#2412) 2021-06-29 12:48:50 +10:00
teor 7c44ee2ebe
Release Blocker: Stop trying to verify coinbase inputs using the script verifier (#2404)
* Stop trying to verify coinbase inputs using the script verifier

And create tests to catch similar bugs earier.

* Use Testnet in NU5 tests that temporarily should_panic

We've marked these tests as should_panic until there is a NU5 activation
height. But Testnet will have an activation height first, so we should
prefer it in the tests. (Or use both networks.)
2021-06-29 10:49:40 +10:00
Alfredo Garcia c06cd19239
Update `has_inputs_and_outputs()` for new consensus rules (#2398)
* update the has_inputs_and_outputs() to new rules

* apply clippy suggestions

* add some TODOs
2021-06-29 08:28:49 +10:00
teor 19fa36049f
Provide a height in each transaction verification request (#2400)
Block transactions already had a height, but mempool transactions didn't.

This PR adds a height to mempool transactions, and deletes redundant and
unused fields. It also adds an accessor method for that height.
2021-06-28 13:28:48 -03:00
dependabot[bot] df426cba9e build(deps): bump tracing-subscriber from 0.2.18 to 0.2.19
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.2.18 to 0.2.19.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.2.18...tracing-subscriber-0.2.19)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-06-28 11:25:55 -04:00
Janito Vaqueiro Ferreira Filho fdeb6d5ec8
Refactor Sprout Join Split validation by transaction verifier (#2371)
* Refactor to create `verify_sprout_shielded_data`

Move the join split verification code into a new
`verify_sprout_shielded_data` helper method that returns an
`AsyncChecks` set.

* Test if signed V4 tx. join splits are accepted

Create a fake V4 transaction with a dummy join split, and sign it
appropriately. Check if the transaction verifier accepts the
transaction.

* Test if unsigned V4 tx. joinsplit data is rejected

Create a fake V4 transaction with a dummy join split. Do NOT sign this
transaction's join split data, and check that the verifier rejects the
transaction.

* Join tests to share Tokio runtime

Otherwise one of the tests might fail incorrectly because of a
limitation in the test environment. `Batch` services spawn a task in the
Tokio runtime, but separate tests can have separate runtimes, so sharing
a `Batch` service can lead to the worker task only being available for
one of the tests.
2021-06-25 00:47:39 +00:00
teor df7075e962
ZIP-213: Explain how Zebra validates shielded coinbase outputs like other shielded outputs (#2382)
* Describe how a ZIP-213 rule is implemented in the transaction verifier

* Move the only coinbase-specific check outside the ZIP-213 block

This change isn't required to implement the ZIP-213 rule, but it makes
it easier to identify the specific checks for coinbase transactions.

* Add a note about coinbase in the mempool

Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
2021-06-24 23:53:32 +00:00
teor 5bce941e70
Enable V5 transaction test vectors in the groth16 tests (#2383)
This also simplifies the code.
2021-06-24 16:28:23 +10:00
Conrado Gouvea 9688811e35
Update multiple crates to ensure bitvec 0.22.3 is being used (#2351)
* Update multiple crates to ensure bitvec 0.22.3 is being used and avoid package conflicts

* Add documentation to zebra-chain::sapling to indicate that ZIP-216 rules are enforced by jubjub

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
2021-06-23 13:16:22 +10:00
Janito Vaqueiro Ferreira Filho 8ed50e578d
Validate transparent inputs and outputs in V5 transactions (#2302)
* Add missing documentation

Document methods to describe what they do and why.

* Create an `AsyncChecks` type alias

Make it simpler to write the `FuturesUnordered` type with boxed futures.
This will also end up being used more when refactoring to return the
checks so that the `call` method can wait on them.

* Create `verify_transparent_inputs_and_outputs`

Refactors the verification of the transparent inputs and outputs into a
separate method.

* Refactor transparent checks to use `call_all`

Instead of pushing the verifications into a stream of unordered futures,
use the `ServiceExt::call_all` method to build an equivalent stream
after building a stream of requests.

* Replace `CallAll` with `FuturesUnordered`

Make it more consistent with the rest of the code, and make sure that
the `len()` method is available to use for tracing.

Co-authored-by: teor <teor@riseup.net>

* Refactor to move wait for checks into a new method

Allow the code snipped to be reused by other transaction
version-specific check methods.

* Verify transparent inputs in V5 transactions

Use the script verifier to check the transparent inputs in a V5
transaction.

* Check `has_inputs_and_outputs` for all versions

Check if a transaction has inputs and outputs, independently of the
transaction version.

* Wait for checks in `call` method

Refactor to move the repeated code into the `call` method. Now the
validation methods return the set of asynchronous checks to wait for.

* Add helper function to mock transparent transfers

Creates a fake source UTXO, and then the input and output that represent
spending that UTXO. The initial UTXO can be configured to have a script
that either accepts or rejects any spend attempt.

* Test if transparent V4 transaction is accepted

Create a fake V4 transaction that includes a fake transparent transfer
of funds. The transfer uses a script to allow any UTXO to spend it.

* Test transaction V4 rejection based on script

Create a fake transparent transfer where the source UTXO has a script
that rejects spending. The script verifier should not accept this
transaction.

* Test if transparent V5 transaction is accepted

Create a mock V5 transaction that includes a transparent transfer of
funds. The transaction should be accepted by the verifier.

* Test transaction V5 rejection based on script

Create a fake transparent transfer where the source UTXO has a script
that rejects spending. The script verifier should not accept this
transaction.

* Update `Request::upgrade` getter documentation

Simplify it so that it won't become updated when #1683 is fixed.

Co-authored-by: teor <teor@riseup.net>
2021-06-23 11:54:00 +10:00
Alfredo Garcia 7638c43a7c
Move network_upgrade check into zebra-chain (#2354)
* move network_upgrade check into zebra-chain

* fix the errors

* rename function

* typo fix

* rename the check function

* make changes from last code review
2021-06-22 12:06:52 +10:00
teor 7d1c1fb84e
Document required request timeouts due to data dependencies (#2337)
* Document required request timeouts due to data dependencies

* Update AwaitUTXO docs
2021-06-18 14:43:05 -03:00
Pili Guerra 6396ac27d8
Update versions for zebra v1.0.0-alpha.11 release (#2334)
* Update versions for zebra v1.0.0-alpha.11 release

* Update Cargo.lock
2021-06-18 10:37:58 +01:00
Alfredo Garcia 544d182d25
Add and use a function for mandatory checkpoint (#2314)
* add `mandatory_checkpoint_height()` function

* use mandatory checkpoint instead of canopy in acceptance tests
2021-06-18 10:05:28 +10:00
Deirdre Connolly 928191bc92 Remove unused dep 'pairing' in zebra-consensus
It was once needed for an early iteration of the Groth16Verifier, not anymore
2021-06-15 17:44:00 -04:00
Janito Vaqueiro Ferreira Filho 0e89236405
Reject V5 transactions before NU5 activation (#2285)
* Add a `Transaction::version` getter

Returns the version of the transaction as a `u32`.

* Add `Transaction::is_overwintered` helper method

Returns if the `fOverwintered` flag should be set for the transaction's
version.

* Use new helpers to serialize transaction version

Reduce the repeated code and make it less error-prone with future
changes.

* Add getter methods to `transaction::Request` type

Refactor to move the type deconstruction code into the `Request` type.
The main objective is to make it easier to split the call handler into
methods that receive the request directly.

* Refactor to create `verify_v4_transaction` helper

Split the code specific to V4 transactions into a separate helper
method.

* Create `verify_v5_transaction` helper method

Prepare a separate method to have the validation code.

* Add `UnsupportedByNetworkUpgrade` error variant

An error for when a transaction's version isn't supported by the network
upgrade of the block it's included or for the current network upgrade if
the transaction is for the mempool.

* Verify a V5 transaction's network upgrade

For now, only NU5 supports V5 transactions.

* Test that V5 transaction is rejected on Canopy

Create a fake V5 transaction and try to verify it using a block height
from Canopy's activation. The verifier should reject the transaction
with an error saying that the network upgrade does not support that
transaction version.

* Test if V5 tx. is accepted after NU5 activation

Create a fake V5 transaction and pretend it is placed in a block that
has a height after the NU5 activation. The test should succeed, but
since the NU5 activation height has not been specified yet (neither for
the testnet nor the mainnet), for now this test is marked as
`should_panic`.

* Add `TODO` comment to the code

Add more detail to what's left to do, and link to the appropriate PRs.

* Use `u32` to store transaction version

Use a type consistent with how the version is specified.

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2021-06-15 10:15:59 +10:00
dependabot[bot] b4f6251c9d build(deps): bump once_cell from 1.7.2 to 1.8.0
Bumps [once_cell](https://github.com/matklad/once_cell) from 1.7.2 to 1.8.0.
- [Release notes](https://github.com/matklad/once_cell/releases)
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.7.2...v1.8.0)

---
updated-dependencies:
- dependency-name: once_cell
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-06-14 17:06:52 -04:00
Pili Guerra 9aafa79fa3
Update versions for zebra v1.0.0-alpha.10 release (#2245)
* Update versions for zebra v1.0.0-alpha.10 release

* Update Cargo.lock
2021-06-09 12:56:36 +02:00
teor 8ebb415e7c Clippy: remove needless borrows 2021-06-07 18:33:58 -04:00
Janito Vaqueiro Ferreira Filho 2e0318878a
Further test new transaction consensus rules (#2246)
* Add a `at_least_one!` macro for testing

Similar to the `vec!` macro, but doesn't allow creating an empty list.

* Test if `has_inputs_and_outputs` considers actions

Create a dummy transaction with no inputs and no outputs, and add a
dummy Orchard action to it. The `check::has_inputs_and_outputs`
should succeed, because the consensus rule considers having Orchard
actions as having inputs and/or outputs.

* Refactor to create helper function

Move the code to create a fake Orchard shielded data instance to a
helper function in `zebra_chain::transaction::arbitrary`, so that other
tests can also use it.

* Test coinbase V5 transaction with enable spends

A V5 coinbase transaction that has Orchard shielded data MUST NOT have
the enable spends flag set.

* Test if coinbase without enable spends is valid

A coinbase transaction with Orchard shielded data and without the enable
spends flag set should be valid.

* Add a security comment about the `at_least_one!` macro

This macro must not be used outside tests, because it allows memory denial
of service.

Co-authored-by: teor <teor@riseup.net>
2021-06-07 12:02:18 +10:00
Janito Vaqueiro Ferreira Filho b44d81669f
Move the check in `transaction::check::sapling_balances_match` to `V4` deserialization (#2234)
* Implement `PartialEq<i64>` for `Amount`

Allows to compare an `Amount` instance directly to an integer.

* Add `SerializationError::BadTransactionBalance`

Error variant representing deserialization of a transaction that doesn't
conform to the Sapling consensus rule where the balance MUST be zero if
there aren't any shielded spends and outputs.

* Validate consensus rule when deserializing

Return an error if the deserialized V4 transaction has a non-zero value
balance but doesn't have any Sapling shielded spends nor outputs.

* Add consensus rule link to field documentation

Describe how the consensus rule is validated structurally by
`ShieldedData`.

* Clarify that `value_balance` is zero

Make the description more concise and objective.

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Update field documentation

Include information about how the consensus rule is guaranteed during
serialization.

Co-authored-by: teor <teor@riseup.net>

* Remove `check::sapling_balances_match` function

The check is redundant because the respective consensus rule is
validated structurally by `ShieldedData`.

* Test deserialization of invalid V4 transaction

A transaction with no Sapling shielded spends and no outputs but with a
non-zero balance value should fail to deserialize.

* Change least-significant byte of the value balance

State how the byte index is calculated, and change the least
significant-byte to be non-zero.

Co-authored-by: teor <teor@riseup.net>
2021-06-04 08:53:00 +10:00
teor 2f0f379a9e
Standardise clippy lints and require docs (#2238)
* Standardise lints across Zebra crates, and add missing docs

The only remaining module with missing docs is `zebra_test::command`

* Todo -> TODO

* Clarify what a transcript ErrorChecker does

Also change `Error` -> `BoxError`

* TransError -> ExpectedTranscriptError

* Output Descriptions -> Output descriptions
2021-06-04 08:48:40 +10:00
Janito Vaqueiro Ferreira Filho 9416b5d5cd
Update `transaction::check::coinbase_tx_no_joinsplit_or_spend` to validate V5 coinbase transactions with Orchard shielded data (#2236)
* Add a `Transaction::orchard_shielded_data` getter

Allows accessing the Orchard shielded data if it is present in the
transaction, regardless of the transaction version.

* Refactor `orchard_nullifiers` to use new getter

Allows making the method more concise.

* Add `CoinbaseHasEnableSpendsOrchard` error variant

Used when the validation rule is not met.

* Implement `enableSpendsOrchard` in coinbase check

The flag must not be set for the coinbase transaction.

* Refactor `Transaction::orchard_*` getters

Use the fact that `Option<T>` implements `Iterator<T>` to simplify the
code and remove the need for boxing the iterators.

Co-authored-by: teor <teor@riseup.net>
2021-06-03 01:54:08 +00:00
Alfredo Garcia a9fe0d9d3e
Make sure the mandatory checkpoint includes Canopy activation (#2235)
* Make sure the Canopy activation block is a finalized checkpoint block

This enables ZIP-221 chain history from Canopy activation onwards.

* Clarify that the mandatory checkpoint test includes Canopy activation

The test was correct, but the docs and assertion message did not include activation.

* Document that the mandatory checkpoint includes Canopy activation

Co-authored-by: teor <teor@riseup.net>
2021-06-03 10:24:08 +10:00
Janito Vaqueiro Ferreira Filho db0cdb74ff
Update `has_inputs_and_outputs` to check V5 transactions (#2229)
* Fix documentation comment

Was missing a slash to become documentation.

* Add documentation link to type reference

Just to help navigation a bit.

* Implement `Transaction::orchard_actions()` getter

Returns an iterator to iterator over the actions in the Orchard shielded
data (if there is one, otherwise it returns an empty iterator).

* Add V5 support for `has_inputs_and_outputs`

Checks if the transaction has Orchard actions. If it does, it is
considered to have inputs and outputs.

* Refactor transaction test vectors

Make it easier to reuse the fake V5 transaction converter in other test
vectors.

* Move helper function to `zebra-chain` crate

Place it together with some other helper functions, including the one
that actually creates the fake V5 transaction.

* Test transaction with no inputs

`check::has_inputs_and_outputs` should return an error indicating that
the transaction has no inputs.

* Test transaction with no outputs

`check::has_inputs_and_outputs` should return an error indicating that
the transaction has no outputs.

* Note that transaction is fake in `expect` message

Should make the message easier to find, and also gives emphasis to the
fact that the transaction is a fake conversion to V5.

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2021-06-02 11:32:52 +10:00
Pili Guerra e3d2ae0a8a
Update versions for zebra v1.0.0-alpha.9 release (#2196)
* Update versions for zebra v1.0.0-alpha.9 release

* Update Cargo.lock
2021-05-26 13:01:39 +02:00
dependabot[bot] f13831eaab
build(deps): bump thiserror from 1.0.24 to 1.0.25 (#2190)
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.24 to 1.0.25.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.24...1.0.25)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-05-24 20:41:21 +10:00
dependabot[bot] 55b2127ddc
build(deps): bump futures from 0.3.14 to 0.3.15 (#2144)
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.14 to 0.3.15.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.14...0.3.15)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-05-13 07:56:47 +10:00
Pili Guerra 500dc2e511
Update version strings for Zebra v1.0.0-alpha.8 release (#2136)
* Update versions for zebra v1.0.0-alpha.8 release

* Update tower-batch and tower-fallback version strings

* Update Cargo.lock
2021-05-12 14:27:36 +02:00
Alfredo Garcia 29893f2b9b
Validate nConsensusBranchId (#2100)
* validate nConsensusBranchId
* add tests

* fix bug in transaction_to_fake_v5

Co-authored-by: teor <teor@riseup.net>
2021-05-10 01:31:45 +00:00
dependabot[bot] 5632d8cbb8
build(deps): bump tracing-subscriber from 0.2.17 to 0.2.18 (#2097)
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.2.17 to 0.2.18.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.2.17...tracing-subscriber-0.2.18)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-05-04 09:13:14 +10:00
dependabot[bot] 84aca7e78b build(deps): bump tracing from 0.1.25 to 0.1.26
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.25 to 0.1.26.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.25...tracing-0.1.26)

Signed-off-by: dependabot[bot] <support@github.com>
2021-05-03 17:05:49 -04:00
Alfredo Garcia 75d29aca24
Add V5 transparent and sapling to transaction::check, add missing coinbase PrevOut check (#2070)
* validate sapling v5 tx

* Make itertools dependency optional

We only need itertools when the `proptest-impl` feature is enabled.

* Check if V4 and V5 coinbase transactions contain PrevOut transparent inputs

This is a bugfix on V4 transaction validation. The PrevOut consensus
rule was not explicitly stated in the Zcash spec until April 2021.
(But it was implied by Bitcoin, and partially implemented by Zebra.)

Also do the shielded sapling input check for V5 transactions.

* Add spec and orchard TODOs to has_inputs_and_outputs

Also make the variable names match the spec.

* Sort transaction functions to match v5 data order

* Simplify transaction input and output checks

Move counts or iterators into `Transaction` methods, so we can remove
duplicate code, and make the consensus rule logic clearer.

* Update sapling_balances_match for Transaction v5

- Quote from the spec
- Explain why the function is redunant for v5
- Rename the function so it's clear that it is sapling-specific

Co-authored-by: teor <teor@riseup.net>
2021-04-28 10:43:00 +10:00
teor 247620320e
Rename shielded_data to sapling_shielded_data as needed (#2072)
This change help avoid bugs that confuse sapling with orchard (or sprout).

```sh
fastmod shielded_data sapling_shielded_data
```
2021-04-27 10:37:53 -03:00
Alfredo Garcia a49b9d44f6
Enable more Transaction v5 tests (#2063)
* Use NU5 and Transaction v5 in most proptests

* Stop skipping post-Canopy blocks in the block subsidy tests

Co-authored-by: teor <teor@riseup.net>
2021-04-26 08:32:21 +10:00
Pili Guerra ea1446ee92
Update version strings for Zebra v1.0.0-alpha.7 release (#2056)
* Update version strings for Zebra v1.0.0-alpha.7 release
2021-04-23 12:56:25 +00:00
dependabot[bot] 7d36a5e2c3 build(deps): bump color-eyre from 0.5.10 to 0.5.11
Bumps [color-eyre](https://github.com/yaahc/color-eyre) from 0.5.10 to 0.5.11.
- [Release notes](https://github.com/yaahc/color-eyre/releases)
- [Changelog](https://github.com/yaahc/color-eyre/blob/v0.5.11/CHANGELOG.md)
- [Commits](https://github.com/yaahc/color-eyre/compare/v0.5.10...v0.5.11)

Signed-off-by: dependabot[bot] <support@github.com>
2021-04-14 09:22:27 -04:00
dependabot[bot] e084d0f1dc build(deps): bump futures from 0.3.13 to 0.3.14
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.13 to 0.3.14.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.13...0.3.14)

Signed-off-by: dependabot[bot] <support@github.com>
2021-04-12 20:45:35 -04:00
Alfredo Garcia 5ec05e91e1 update version strings for v1.0.0-alpha.6 2021-04-08 18:48:34 -04:00
Alfredo Garcia 48a8a7b851
Refactor Sapling data and use it in V4 (#1946)
* start refactoring transaction v4 for transaction v5
- move ShieldedData to sapling
- add AnchorVariant
- rename shielded_data to sapling_shielded data in V4
- move value_balance into ShieldedData
- update prop tests for new structure

* add AnchorVariant to Spend
- make anchor types available from sapling crate
- update serialize

* change shielded_balances_match() arguments
* change variable name anchor to shared_anchor in ShieldedData
* fix empty value balance serialization
* use AnchorV in shielded spends
* Rename anchor to per_spend_anchor

* Use nullifiers function directly in non-finalized state
* Use self.value_balance instead of passing it as an argument

* Add missing fields to ShieldedData PartialEq
* Derive Copy for tag types

* Add doc comments for ShieldedData refactor
* Implement a per-spend anchor compatibility iterator

Co-authored-by: teor <teor@riseup.net>
2021-04-01 07:34:25 +10:00
Deirdre Connolly 0ffab6d589
Ed25519 async batch verification for JoinSplit signatures (#1952)
* Ed25519 async batch verification for JoinSplit signatures

We've been verifying JoinSplitSigs one-by-one pre-ZIP-215. Now as we're post-ZIP-215,
we can take advantage of the batch math to validate this signatures.

I would have pumped all the joinsplits in our MAINNET_BLOCKS test vectors but these
signatures are over the sighash, which needs the NU code to compute, and once we're
doing all that set up, we're basically doing transaction validation, so.

Resolves #1944

* Repoint to latest ed25519-zebra commit with note to point at 3.0 when released

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
2021-03-31 09:08:19 +10:00
dependabot[bot] 562c141ee1
build(deps): bump displaydoc from 0.2.0 to 0.2.1 (#1959)
Bumps [displaydoc](https://github.com/yaahc/displaydoc) from 0.2.0 to 0.2.1.
- [Release notes](https://github.com/yaahc/displaydoc/releases)
- [Changelog](https://github.com/yaahc/displaydoc/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yaahc/displaydoc/compare/v0.2.0...v0.2.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-03-30 09:27:00 +10:00
teor 306fa88214 Document the correctness of Poll::Pending wakeups 2021-03-27 08:55:49 -04:00
Deirdre Connolly 354396f525 Add some comments and tracing for groth16::Verifier tests 2021-03-25 18:34:47 -04:00
Deirdre Connolly c95716e246 Add link to Ed25519 async batch verification ticket 2021-03-25 16:34:16 -04:00
teor 16c073c4e0 Comment: transparent sighash in script verifier 2021-03-25 16:34:16 -04:00
teor 83bdc1a9f3 Add a metric for binding sig errors
But keep ignoring those errors until we fix binding sigs in #1939
2021-03-25 16:34:16 -04:00
teor 0bad6bc615 Add a TODO for ed25519 batch verification 2021-03-25 16:34:16 -04:00
teor e4f58f018a Re-enable joinsplit and sapling spend auth verification 2021-03-25 16:34:16 -04:00
teor f8dc989e99 Move a quick check earlier in transaction validation 2021-03-25 16:34:16 -04:00
Deirdre Connolly 7efc700aca
Merge pull request #1713 from ZcashFoundation/use-groth16-batch-math
Use batch optimizations, load params in groth16::Verifier, verify Spend & Output descriptions in transaction verifier
2021-03-24 12:28:25 -04:00
teor b396bb9be5 Document that the mandatory checkpoint can change
We don't want users relying on the exact height of Zebra's
mandatory checkpoint.
2021-03-22 22:17:44 -04:00
Deirdre Connolly ca1d2de87d
Bump versions for v1.0.0-alpha.5 (#1932)
Zebra's latest alpha checkpoints on Canopy activation, continues our work on NU5, and fixes a security issue.

Some notable changes include:

## Added
- Log address book metrics when PeerSet or CandidateSet don't have many peers (#1906)
- Document test coverage workflow (#1919)
- Add a final job to CI, so we can easily require all the CI jobs to pass (#1927)

## Changed
- Zebra has moved its mandatory checkpoint from Sapling to Canopy (#1898, #1926)
  - This is a breaking change for users that depend on the exact height of the mandatory checkpoint.

## Fixed
- tower-batch: wake waiting workers on close to avoid hangs (#1908)
- Assert that pre-Canopy blocks use checkpointing (#1909)
- Fix CI disk space usage by disabling incremental compilation in coverage builds (#1923)

## Security
- Stop relying on unchecked length fields when preallocating vectors (#1925)
2021-03-22 22:05:01 -04:00
teor f19f0d0949 clippy: fix inconsistent struct ordering 2021-03-22 11:51:54 -04:00
teor 38585b6116
Fix whitespace for rustfmt 2021-03-22 11:18:37 +10:00
teor a6f916671a
Add a TODO comment for a missing no-coinbase check 2021-03-22 08:59:51 +10:00
teor d19585cfca Fix some missed Sapling to Canopy checkpoint code changes 2021-03-18 10:13:47 +10:00
teor 9da220517b Update docs for Sapling to Canopy checkpoint change 2021-03-18 10:13:47 +10:00
teor a649bd10e3 Also update testnet checkpoints 2021-03-18 10:13:47 +10:00
teor 6711d14bc4 Summarise how to rebuild checkpoints in list.rs 2021-03-18 10:13:47 +10:00
Alfredo Garcia a292cbe611 change the mandatory checkpoint to Canopy 2021-03-18 10:13:47 +10:00
dependabot[bot] 97cae4b2da build(deps): bump displaydoc from 0.1.7 to 0.2.0
Bumps [displaydoc](https://github.com/yaahc/displaydoc) from 0.1.7 to 0.2.0.
- [Release notes](https://github.com/yaahc/displaydoc/releases)
- [Changelog](https://github.com/yaahc/displaydoc/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yaahc/displaydoc/commits/v0.2.0)

Signed-off-by: dependabot[bot] <support@github.com>
2021-03-17 15:50:40 -04:00
Alfredo Garcia d49eaab68e
Bump versions for zebrad 1.0.0-alpha.4 (#1913)
* Bump versions for zebrad 1.0.0-alpha.4

* add Cargo.lock
2021-03-16 21:12:37 -03:00
Jack Grigg e51f33a4b9 Use interoperable names for common metrics
These names match the equivalent metrics in zcashd, enabling common
metrics to be collected across both node types.
2021-03-17 09:38:07 +10:00
teor 873127aac1 Replace smart quotes with ascii quotes
Some tools don't deal well with unicode text. And we're not using it
consistently in Zebra anyway.
2021-03-15 03:18:10 -04:00
dependabot[bot] b618f5b522 build(deps): bump tracing-subscriber from 0.2.16 to 0.2.17
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.2.16 to 0.2.17.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.2.16...tracing-subscriber-0.2.17)

Signed-off-by: dependabot[bot] <support@github.com>
2021-03-14 19:37:24 -04:00
teor 8103e89739 Check proof-of-work in the CheckpointVerifier
These checks make some resource exhaustion attacks harder to carry out,
because the malicious blocks need a valid proof-of-work.
2021-03-12 08:55:29 +10:00
teor c1a910942d Re-use some block checks in the CheckpointVerifier 2021-03-12 08:55:29 +10:00
teor 252affdf84 Add network to CheckpointVerifier
This refactor allows us to re-use some block checks in the checkpoint
verifier.
2021-03-12 08:55:29 +10:00
teor 2e64409d78 Make checkpoint restart test automatically use extra blocks
Also use block 10. (The previous version only really used blocks 0-9.)
2021-03-04 08:06:31 +10:00
teor e1002ce2ce Test checkpoints on mainnet and testnet 2021-03-04 08:06:31 +10:00
teor b8cc3bfb23 Delete duplicate test code for chain verifier
This code was commented-out, so the tests weren't actually running.
2021-03-04 08:06:31 +10:00
teor 436a4e9a5b Add better diagnostics for the checkpoint restart tests 2021-03-04 08:06:31 +10:00
teor db23258216 Re-enable the checkpoint restart tests 2021-03-04 08:06:31 +10:00
Alfredo Garcia 8883543a85
Add transaction version 5 stubs (#1824)
* add transaction V5 stub
* add v5_strategy
* deduplicate version group ids
* Update comment for V5 transactions
* Add V5 transactions to non_finalized_state

Currently these are all `unimplemented!(...)`

* Fix struct matches
* Apply trivial panic message changes
* add zcash_deserialize for V5
* make all tx versions explicit in sprout and sapling nullifier functions
* match exhaustively in sprout and sapling nullifier functions
* fix matches in zebra-consensus
* fix NU5 strategy
* We're still deciding if v5 transactions support Sprout

Co-authored-by: teor <teor@riseup.net>
2021-03-04 07:56:41 +10:00
Alfredo Garcia 65fa1c6bd9 replace canopy.pdf with protocol.pdf 2021-03-02 16:07:04 -05:00
dependabot[bot] 70327dc9f5 build(deps): bump once_cell from 1.6.0 to 1.7.0
Bumps [once_cell](https://github.com/matklad/once_cell) from 1.6.0 to 1.7.0.
- [Release notes](https://github.com/matklad/once_cell/releases)
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.6.0...v1.7.0)

Signed-off-by: dependabot[bot] <support@github.com>
2021-02-25 15:32:03 -05:00
dependabot[bot] bf1e2bd3b8
build(deps): bump tracing from 0.1.24 to 0.1.25 (#1815)
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.24 to 0.1.25.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.24...tracing-0.1.25)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-02-25 09:28:26 +10:00
dependabot[bot] 7b675bbd73 build(deps): bump futures from 0.3.12 to 0.3.13
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.12 to 0.3.13.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.12...0.3.13)

Signed-off-by: dependabot[bot] <support@github.com>
2021-02-23 16:09:53 -05:00
dependabot[bot] dab65b33eb build(deps): bump once_cell from 1.5.2 to 1.6.0
Bumps [once_cell](https://github.com/matklad/once_cell) from 1.5.2 to 1.6.0.
- [Release notes](https://github.com/matklad/once_cell/releases)
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.5.2...v1.6.0)

Signed-off-by: dependabot[bot] <support@github.com>
2021-02-23 10:59:40 -05:00
teor 7558f74c78 Bump versions for zebrad 1.0.0-alpha.3 2021-02-23 10:39:13 -05:00
dependabot[bot] 304d7682f5 build(deps): bump tracing-subscriber from 0.2.15 to 0.2.16
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.2.15 to 0.2.16.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.2.15...tracing-subscriber-0.2.16)

Signed-off-by: dependabot[bot] <support@github.com>
2021-02-22 01:38:42 -05:00
teor 3af57ece7a
Partly revert "Fix poll_ready usage in ChainVerifier" (#1735)
* Revert "Fix poll_ready usage in ChainVerifier (#1700)"

This reverts commit 0723ac5be1.

* Keep the VERIFIER_BUFFER_BOUND change
* Correctly implement multiple readiness
2021-02-20 10:43:38 +10:00
dependabot[bot] 7bba67d97b build(deps): bump thiserror from 1.0.23 to 1.0.24
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.23 to 1.0.24.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.23...1.0.24)

Signed-off-by: dependabot[bot] <support@github.com>
2021-02-19 14:54:40 -05:00
dependabot[bot] f19ef44493 build(deps): bump tracing from 0.1.23 to 0.1.24
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.23 to 0.1.24.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.23...tracing-0.1.24)

Signed-off-by: dependabot[bot] <support@github.com>
2021-02-18 22:21:49 -05:00
dependabot[bot] 76e4b8f693 build(deps): bump tracing-futures from 0.2.4 to 0.2.5
Bumps [tracing-futures](https://github.com/tokio-rs/tracing) from 0.2.4 to 0.2.5.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-futures-0.2.4...tracing-futures-0.2.5)

Signed-off-by: dependabot[bot] <support@github.com>
2021-02-17 08:55:16 -05:00
teor 9056435942
Make the state usage clearer in script::Verifier (#1733) 2021-02-16 05:53:12 +10:00
Deirdre Connolly 0c5daa8410 Bump versions for zebrad 1.0.0-alpha.2
Including tower-batch bump to 0.2.0, tower-fallback to 0.2.0, zebra-script to 1.0.0-alpha.3
2021-02-09 16:14:29 -05:00
Alfredo Garcia 0723ac5be1
Fix poll_ready usage in ChainVerifier (#1700)
* change `poll_ready()` and `call()` of `ChainVerifier`
* add bound, move max_checkpoint_height
* add buffers to the checkpoint and block verifiers

And rename the chain verifier fields so `block` means `Arc<Block>`, and
`block_verifier` means `Buffer<BlockVerifier, ...>`.

* Fix the error types
* Use `ServiceExt::oneshot` in `ChainVerifier::call`

And:
* make the code look like the `main` branch as much as possible
* document the `poll_ready`/`call` invariant

* Use `ServiceExt::oneshot` in `chain::init`

Co-authored-by: teor <teor@riseup.net>
2021-02-09 08:13:55 +10:00
teor a91006afa7
Actually use `VerifyCheckpointError::CommitFinalized` (#1706)
Also:
* use `ServiceExt::oneshot` in the checkpoint verifier
* make some error messages more specific
* clean up the `std::future`/`futures` imports
2021-02-09 07:16:36 +10:00
teor 132c7fe4a5 Add a comment about a mempool request bug
Transaction verifier mempool requests should take the next block height,
because some consensus rules depend on the exact height.

See #1683.
2021-02-05 16:40:34 -05:00
Jane Lusby 0ac259430a Implement Async Batch verification API for groth16
This PR is the first step in getting a groth16 proving system fully
integrated with the rest of zebra. This PR implements the initial async
API, but none of the actual batching logic necessary for our eventual
verifier design.

Once the batch verification API from bellman has been implemented we
will need to swap out the "Batch" type defined in this crate with the
new `batch::Verifier` defined in bellman.
2021-02-05 14:52:48 -05:00
dependabot[bot] b3a3b8f7c7 build(deps): bump tracing from 0.1.22 to 0.1.23
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.22 to 0.1.23.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.22...tracing-0.1.23)

Signed-off-by: dependabot[bot] <support@github.com>
2021-02-05 12:26:52 -05:00
teor bd03a51482
Remove an outdated comment in the script verifier (#1680) 2021-02-04 09:48:00 +10:00
teor 919dc0307f Rewrite the comment to be more explicit 2021-02-03 17:56:21 -05:00
teor 99fe588896 Document the impact of the redjubjub channel bound 2021-02-03 17:56:21 -05:00
Alfredo Garcia d7c40af2a8
Fix shutdown panics (#1637)
* add a shutdown flag in zebra_chain::shutdown
* fix network panic on shutdown
* fix checkpoint panic on shutdown
2021-02-03 19:03:28 +10:00
dependabot[bot] c52310ef91 build(deps): bump jubjub from 0.5.1 to 0.6.0
Bumps [jubjub](https://github.com/zkcrypto/jubjub) from 0.5.1 to 0.6.0.
- [Release notes](https://github.com/zkcrypto/jubjub/releases)
- [Changelog](https://github.com/zkcrypto/jubjub/blob/main/RELEASES.md)
- [Commits](https://github.com/zkcrypto/jubjub/commits)

Signed-off-by: dependabot[bot] <support@github.com>
2021-01-30 01:54:06 -05:00
Deirdre Connolly 1b09538277
Bump versions for zebrad 1.0.0-alpha.1 (#1646)
* Bump versions where appropriate

Tested with cargo install --locked --path etc

* Remove fixed panics from 'Known Issues'

* Change to alpha release series in the README

Co-authored-by: teor <teor@riseup.net>
2021-01-27 20:31:39 -05:00
teor 494a5130c1 Fix clippy "unnecessary Vec::push" lints 2021-01-22 11:51:30 -08:00
teor 00c3ad5d8e Fix clippy "unnecessary Ok" lints 2021-01-22 11:51:30 -08:00
teor 258789ed9b Use the rustc unknown lints attribute
The clippy unknown lints attribute was deprecated in
nightly in rust-lang/rust#80524. The old lint name now produces a
warning.

Since we're using `allow(unknown_lints)` to suppress warnings, we need to
add the canonical name, so we can continue to build without warnings on
nightly.

But we also need to keep the old name, so we can continue to build
without warnings on stable.

And therefore, we also need to disable the "removed lints" warning,
otherwise we'll get warnings about the old name on nightly.

We'll need to keep this transitional clippy config until rustc 1.51 is
stable.
2021-01-19 11:02:20 -05:00
dependabot[bot] 75b510a9f6 build(deps): bump futures from 0.3.11 to 0.3.12
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.11 to 0.3.12.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.11...0.3.12)

Signed-off-by: dependabot[bot] <support@github.com>
2021-01-17 15:42:13 -05:00
dependabot[bot] 68e2942f83 build(deps): bump futures from 0.3.9 to 0.3.11
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.9 to 0.3.11.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.9...0.3.11)

Signed-off-by: dependabot[bot] <support@github.com>
2021-01-15 08:46:33 -05:00
teor 3699bbdae6 Add some additional sync correctness constraints
And adjust the sync restart delay as a consequence.
2021-01-13 20:46:25 -05:00
teor 730910cd99 Upgrade to tokio 0.3.6 from crates.io
And remove the tokio git dependency patch
2021-01-12 15:37:27 -05:00
dependabot[bot] 21e92f6392 build(deps): bump futures from 0.3.8 to 0.3.9
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.8 to 0.3.9.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.8...0.3.9)

Signed-off-by: dependabot[bot] <support@github.com>
2021-01-12 02:13:02 -05:00
Jane Lusby 15698245e1
Deduplicate metrics dependencies (#1561)
## Motivation

This PR is motivated by the regression identified in https://github.com/ZcashFoundation/zebra/issues/1349. That PR notes that the metrics stopped working for most of the crates other than `zebrad`.

## Solution

This PR resolves the regression by deduplicating the `metrics` crate dependency. During a recent change we upgraded the metrics version in `zebrad` and a couple other of our crates, but we never updated the dependencies in `zebra-state`, `zebra-consensus`, or `zebra-network`. This caused the metrics macros to attempt to retrieve the current metrics exporter through the wrong function. We would install the metrics exporter in `0.13`, but then attempt to look it up through the `0.12` crate, which contains a different instance of the metrics exporter static variable which is unset. Doing this causes the metrics macros to return `None` for the current exporter after which they just silently give up.

## Related Issues

closes https://github.com/ZcashFoundation/zebra/issues/1349

## Follow Up Work

I noticed we have quite a few duplicate dependencies in our tree. We might be able to save some compilation time by auditing those and deduplicating them as much as possible.

- https://github.com/ZcashFoundation/zebra/issues/1582
Co-authored-by: teor <teor@riseup.net>
2021-01-12 12:28:56 +10:00
dependabot[bot] f2a21ac67c build(deps): bump thiserror from 1.0.22 to 1.0.23
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.22 to 1.0.23.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.22...1.0.23)

Signed-off-by: dependabot[bot] <support@github.com>
2020-12-28 16:17:00 -05:00
Jane Lusby cfc339e8ea
export new precompute api in zebra-script (#1493)
* export new precompute api in zebra-script
* remove old API in favor of precompute API
* add multi use test cases and bump version
* update implementation to actually match henry's design
* Add safety comment for zebra-script

Co-authored-by: teor <teor@riseup.net>
2020-12-18 13:18:28 +10:00
Alfredo Garcia d725eeb4d6
Add documentation to zebra-utils and checkpoint generation (#1491)
* create readme for utils and checkpoints
* add link to checkpoint usage to book

Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: teor <teor@riseup.net>
2020-12-14 11:34:22 +10:00
Henry de Valence 0842eb2dab
zebra: move to 1.x-based versioning. (#1476)
Previously we set the crate versions to 3.x, so that the major version was
aligned with the NU version.  But we want to be able to make API changes
independently of the NU schedule.
2020-12-08 08:53:07 +10:00
teor 5b687f878a Remove unused dependencies found using cargo-udeps 2020-12-05 14:53:28 -05:00
teor b4a50fd99f
Downgrade tokio to 0.3.4 to avoid a time wheel panic (#1453)
See tokio-rs/tokio#2789 for details. We were seeing this panic during
normal operation, not just at shutdown.
2020-12-04 13:52:37 +10:00
dependabot[bot] 8c052cc39a build(deps): bump color-eyre from 0.5.9 to 0.5.10
Bumps [color-eyre](https://github.com/yaahc/color-eyre) from 0.5.9 to 0.5.10.
- [Release notes](https://github.com/yaahc/color-eyre/releases)
- [Changelog](https://github.com/yaahc/color-eyre/blob/v0.5.10/CHANGELOG.md)
- [Commits](https://github.com/yaahc/color-eyre/compare/v0.5.9...v0.5.10)

Signed-off-by: dependabot[bot] <support@github.com>
2020-12-03 10:55:16 -05:00
Jane Lusby d7bef1c155
bump color-eyre version to avoid a panic when printing spantraces (#1438) 2020-12-02 14:16:18 -08:00
teor 4109534c9b
Run `cargo fmt --all` (#1427) 2020-12-02 14:43:33 +10:00
teor 92eb92d1dd
Disable the nightly clippy unnecessary_wraps lint (#1403)
It seems to be a bit broken - some of our functions return `Result` for
consistency with similar functions. But the lint picks them up anyway.
2020-12-01 12:20:57 +10:00
Henry de Valence 4fd9203785 consensus: check Merkle root immediately after difficulty 2020-12-01 10:14:44 +10:00
Henry de Valence 4906a191f9 consensus: check for duplicate transactions in blocks
Change the Merkle root validation logic to also check that a block does not
contain duplicate transactions.  This check is redundant with later
double-spend checks, but is a useful defense-in-depth.
2020-12-01 10:14:44 +10:00
Henry de Valence 706f1fff81 consensus: tidy merkle validation in checkpointer
This moves it in with the existing `check_block` method and expands that
method's contract to cover general block validation checks.
2020-12-01 10:14:44 +10:00
Henry de Valence 7c08c0c315 consensus: check Merkle roots
As a side effect of computing Merkle roots, we build a list of
transaction hashes.  Instead of discarding these, add them to
PreparedBlock and FinalizedBlock so that they can be reused rather than
recomputed.

This commit adds Merkle root validation to:

1. the block verifier;
2. the checkpoint verifier.

In the first case, Bitcoin Merkle tree malleability has no effect,
because only a single Merkle tree in each malleablity set is valid (the
others have duplicate transactions).

In the second case, we need to check that the Merkle tree does not contain any
duplicate transactions.

Closes #1385
Closes #906
2020-12-01 10:14:44 +10:00
teor 440e183d32
Decrease the UTXO request timeout (#1407)
UTXO requests during transaction input verification can time out because:

1. The block that creates the UTXO is queued for download or verify, but
   it hasn't been committed yet. The creating block might spend UTXOs
   that come from other recent blocks, so UTXO verification can depend on
   a (non-contiguous) sequence of block verifications.

   In this case, Zebra should wait for additional block download and
   verify tasks to complete.

2. The block that creates the UTXO isn't queued for download. This can
   happen because the block is gossiped block that's much higher than the
   current tip, or because a peer sent the syncer a bad list of block
   hashes.

   In this case, Zebra should discard the timed out block, and restart
   the sync.

We need to choose a timeout that balances these two cases, so we time
out after 180 seconds.

Assuming Zebra can download at least 1 MB per second, 180 seconds is
enough time to download a few hundred blocks. So Zebra should be able to
download and verify the next block before the UTXOs that it creates time
out. (Since Zebra has already verified all the blocks before the next
block, its UTXO requests should return immediately.)

Even if some peers time out downloads, a block can only be pending
download for 80 seconds (4 retries * 20 second timeout) before the
download fails. So the UTXO timeout doesn't need to be much larger than
this overall download timeout - because the download timeout will happen
first on slow networks.

Alternately, if the download for the creating block was never queued,
Zebra should timeout as soon as possible - so it can restart the sync
and download the creating block.

As a side-effect, a lower UTXO timeout also makes it slightly easier to
debug UTXO issues, because unsatisfiable queries fail faster.
2020-12-01 10:12:25 +10:00
Henry de Valence 5e48acf3a4
consensus: add timeout to UTXO queries (#1391)
The state service API says explicitly that AwaitUTXO requests should be coupled
with a timeout layer. I didn't add this when I was testing and fixing the UTXO
lookup code (#1348, #1358) because causing zebrad to hang on a failed
dependency was useful for identifying cases where the code wasn't useful (and
then inspecting execution traces).

As a side effect, this ticket resolves most of the hangs in #1389, because
far-future gossiped blocks will have their UTXO lookups time out, though we
may wish to do other work as part of debugging the combined sync+gossip logic.
2020-11-27 09:55:37 +10:00
Deirdre Connolly 2a21c86b91 I before E except after C (or uh, not-english) 2020-11-24 22:23:57 -05:00
Henry de Valence b9347e01e4 consensus: ignore sighash-derived checks until #1377 2020-11-24 16:28:22 -08:00
Henry de Valence f798aef843 consensus: increase buffer for batch redjubjub results 2020-11-25 08:53:07 +10:00
Henry de Valence 61b3286085 consensus: add NU plumbing to block,tx,script verifiers
Closes #1367 by propagating the network upgrade through the service
requests.
2020-11-25 08:53:07 +10:00
dependabot[bot] e832f70c2c build(deps): bump tracing from 0.1.21 to 0.1.22
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.21 to 0.1.22.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.21...tracing-0.1.22)

Signed-off-by: dependabot[bot] <support@github.com>
2020-11-24 10:55:53 -05:00
teor 133ebd078a Remove a post-Canopy panic in funding stream block subsidy validation
Funding stream block subsidy validation will be implemented as part of
block subsidy validation in #801.
2020-11-24 10:01:24 -05:00
Henry de Valence f1155297a3
consensus: temporarily skip scripts on big transactions (#1360)
Temporary workaround for #1350 until that issue is properly fixed.
2020-11-24 19:31:03 +10:00
teor 2d60c00fb0
Avoid a panic when downcasting to redjubjub::Error fails (#1363)
Instead, format the original error as a string, to provide better
diagnostics.

Temporary fix for #1357, the permanent fix ticket is #1186.
2020-11-24 16:46:02 +10:00
Henry de Valence 342eb166ff state: track UTXO provenance
This commit changes the state system and database format to track the
provenance of UTXOs, in addition to the outputs themselves.
Specifically, it tracks the following additional metadata:

- the height at which the UTXO was created;
- whether or not the UTXO was created from a coinbase transaction or
  not.

This metadata will allow us to:

- check the coinbase maturity consensus rule;
- check the coinbase inputs => no transparent outputs rule;
- implement lookup of transactions by utxo (using the height to find the
  block and then scanning the block) for a future RPC mechanism.

Closes #1342
2020-11-23 22:18:43 -08:00
Henry de Valence e0817d1747 state: introduce PreparedBlock, FinalizedBlock
This change introduces two new types:

- `PreparedBlock`, representing a block which has undergone semantic
  validation and has been prepared for contextual validation;
- `FinalizedBlock`, representing a block which is ready to be finalized
  immediately;

and changes the `Request::CommitBlock`,`Request::CommitFinalizedBlock`
variants to use these types instead of their previous fields.

This change solves the problem of passing data between semantic
validation and contextual validation, and cleans up the state code by
allowing it to pass around a bundle of data.  Previously, the state code
just passed around an `Arc<Block>`, which forced it to needlessly
recompute block hashes and other data, and was incompatible with the
already-known but not-yet-implemented data transfer requirements, namely
passing in the Sprout and Sapling anchors computed during contextual
validation.

This commit propagates the `PreparedBlock` and `FinalizedBlock` types
through the state code but only uses their data opportunistically, e.g.,
changing .hash() computations to use the precomputed hash.  In the
future, these structures can be extended to pass data through the
verification pipeline for reuse as appropriate.  For instance, these
changes allow the sprout and sapling anchors to be propagated through
the state.
2020-11-23 14:16:39 +10:00
Henry de Valence e4f38582fd consensus: add remaining checks trace counter to tx verifier 2020-11-23 14:16:39 +10:00
Henry de Valence a2a70e1fc1 consensus: fix same-block utxo lookups
The UTXO query system assumes that a transaction will only request
information about UTXOs created in prior blocks.  But transactions are
allowed to spend UTXOs created by prior transactions in the same block.

This doesn't fit with the existing query model, so instead of trying to
change it, allow the script verifier to take an additional set of known
UTXOs, and propagate this set from the block.
2020-11-23 14:16:39 +10:00
Henry de Valence f0810b028d state,consensus,sync: shorten span lengths
These changes help reduce the size of the resulting spans, making the
output more compact.  Together they save about 30-40 characters.
2020-11-23 14:16:39 +10:00
Henry de Valence aa45bf2b58 consensus: add traces to block verifier 2020-11-23 14:16:39 +10:00
teor 196dc6369c Delete outdated transaction comments 2020-11-22 23:11:00 -05:00
Henry de Valence 2eceff421f consensus: remove incorrect check
This consensus rule is supposed to apply to transactions whose
transparent inputs are the *outputs* of previous coinbase
transactions, not to transactions with coinbase inputs.  Because that
logic is different enough from this logic, and requires different data
flow, it's cleaner to just remove this check for now.
2020-11-21 14:09:15 -05:00
Henry de Valence ace1103462 consensus: fix bug in tx input/output presence check
Making this check's match statement exhaustive revealed a bug similar to
the previous commit.  The logic in the spec is written in terms of
numbers, but our data is internally represented in terms of enums
(ADTs).  This kind of cross-representation rule translation is a bug
surface, which we can avoid by converting to counts and summing up.  (We
should use one style at a time).
2020-11-21 14:09:15 -05:00
Henry de Valence 96ee32e5d2 consensus: fix bug in coinbase joinsplit/spend check
This function caused spurious "WrongVersion" errors, because the match
pattern in the first arm was non-exhaustive, but the fallthrough match
arm was present and assumed it would only be reached if the version was
incorrect.

This commit cleans up the implemenation, splits out the error variants,
and renames the check to be more precise.

To avoid this kind of bug in the future, two guidelines are useful:

1. Avoid fallthrough cases that circumvent non-exhaustive match checks;
2. Avoid nested conditionals, preferring a "straight-line" sequence of
   match arm => result pairs rather than nested matches or matches with
   conditionals inside.
2020-11-21 14:09:15 -05:00
Henry de Valence b116cfcd76 consensus: add debug event on wrong version check
Adding this check reveals that the WrongVersion errors aren't coming
from the correct WrongVersion check.
2020-11-21 14:09:15 -05:00
Henry de Valence d1ee7f263a consensus: add debug span to TransactionVerifier 2020-11-21 14:09:15 -05:00
Henry de Valence 2e4f4d8e87 consensus: fix span handling in BlockVerifier
The BlockVerifier constructed a tracing span and manually entered it
inside of an async block.  Manually entering spans inside async blocks
can cause problems where the span might not be entered and exited
correctly as the resulting future is polled.  Instead, using the
.instrument creates a wrapper future that handles the bookkeeping.

I changed the span name and contents to be consistent with the spans in
the checkpoint verifier.
2020-11-21 14:09:15 -05:00
Deirdre Connolly 558661a531 Remove test attributes and allow(dead_code) for test code that tests currently unimplemented functionality 2020-11-21 05:40:25 -05:00
Henry de Valence 04acc9da6c consensus: instrument script verification 2020-11-20 17:38:21 -05:00
Henry de Valence add94c1c45 deps: move to tokio 0.3, tower 0.4
This change is mostly mechanical, with the exception of the changes to the
`tower-batch` middleware.  This middleware was adapted from `tower::buffer`,
and the `tower::buffer` code was changed to implement its own bounded queue,
because Tokio 0.3 removed the `mpsc::Sender::poll_send` method.  See

ddc64e8d4d

for more context on the Tower changes.  To match Tower as closely as possible
in order to be able to upstream `tower-batch`, those changes are copied from
`tower::Buffer` to `tower-batch`.
2020-11-20 10:08:16 -08:00
Henry de Valence aa8d95bd23 consensus: improve checkpoint request replacement diagnostics 2020-11-17 14:56:27 -08:00
Henry de Valence a3ab589d89 consensus,state: document cancellation contracts for services
This change explicitly documents cancellation contracts for our Tower services,
and tries to correct a bug in the implementation of the CheckpointVerifier,
which duplicates information from the state service but did not ensure that it
would be kept in sync.
2020-11-17 14:56:27 -08:00
Henry de Valence d5d17a9a71 consensus: remove incorrect comment
The ZcashDeserialize implementation for Block doesn't check that blocks
have a coinbase height.
2020-11-17 14:56:27 -08:00
dependabot[bot] 8c5f6d0177 build(deps): bump once_cell from 1.5.1 to 1.5.2
Bumps [once_cell](https://github.com/matklad/once_cell) from 1.5.1 to 1.5.2.
- [Release notes](https://github.com/matklad/once_cell/releases)
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.5.1...v1.5.2)

Signed-off-by: dependabot[bot] <support@github.com>
2020-11-13 14:48:11 -05:00
teor af4797130b
Replace Target<block::Height> with TargetHeight (#1289)
We don't use this generic, so let's just remove it.
2020-11-12 16:11:25 -08:00
teor 96b7572bb5 Make the checkpoint metrics more accurate 2020-11-13 09:41:14 +10:00
teor 76a9f5b8c4 Refactor block target spacing into NetworkUpgrade methods
And add a method for the minimum difficulty time gap threshold.
2020-11-12 12:30:54 +10:00
teor 405c0644f9 Add a comment explaining the issues in ZIPs 205 and 208
And add the network to the difficulty filter error.
2020-11-12 12:30:54 +10:00
Alfredo Garcia 128643d81e
Call `zebra_test::init` where needed. (#1227)
* Add missing `zebra_test::init()` to zebra-chain
* Add missing `zebra_test::init()` to zebra-consensus
* Add missing `zebra_test::init()` to zebra-network
* Add missing `zebra_test::init()` to zebra-state
* Add missing `zebra_test::init()` to zebra-test
* Add missing `zebra_test::init()` to zebrad
2020-11-10 10:29:25 +10:00
teor f90a749910
Clarify CheckpointVerifier errors (#1260)
And make an unreachable error into a panic.
2020-11-06 11:07:30 -08:00
dependabot[bot] 1e3cf6dc5c build(deps): bump tracing-subscriber from 0.2.14 to 0.2.15
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.2.14 to 0.2.15.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.2.14...tracing-subscriber-0.2.15)

Signed-off-by: dependabot[bot] <support@github.com>
2020-11-04 20:37:40 -05:00
dependabot[bot] e87340594a build(deps): bump thiserror from 1.0.21 to 1.0.22
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.21 to 1.0.22.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.21...1.0.22)

Signed-off-by: dependabot[bot] <support@github.com>
2020-11-04 19:28:42 -05:00
teor 1c31225aac
Implement Expanded to Compact Difficulty Conversion (#1196)
* Implement Expanded to Compact Difficulty
* Implement Arbitrary for CompactDifficulty
Remove the derive, and generate values from random block
hashes.
* Implement Arbitrary for ExpandedDifficulty and Work
* Use Arbitrary for CompactDifficulty in Arbitrary for Block
* Test difficulty on all block test vectors
And cleanup some duplicate test code
* Round-trip tests for compact test cases
* Round-trip tests for compact difficulty in block test vectors
* Make Add for Work return PartialCumulativeWork
Remove AddAssign for Work
Rewrite a proptest using Sub for PartialCumulativeWork
Use Arbitrary for Work
* Add roundtrip work sum tests
* Add roundtrip comparison difficulty tests
* Add failing proptest cases due to test bugs
* Use Some(_) rather than _.into()
* Reduce visibility of difficulty type inner values
* Split work and other difficulty proptests
This change makes sure that rejected work values don't disable property
tests on other types.
2020-10-30 11:36:59 +10:00
Deirdre Connolly 0cb8010ae7 Remove allow(dead_code) on transaction module] 2020-10-28 21:44:13 -04:00
Deirdre Connolly 8cac287aa2 Tidy TransactionError 2020-10-28 21:44:13 -04:00
Deirdre Connolly b2df84fc59 Dedupe VerifyTransactionError into TransactionError 2020-10-28 21:44:13 -04:00
Deirdre Connolly 1d646e6a27 Make Clippy happy 2020-10-28 21:44:13 -04:00
Deirdre Connolly 1ce2eea35f Add coinbase shielded descriptions check 2020-10-28 21:44:13 -04:00
Deirdre Connolly 1653aca570 Add shielded_balances_match check 2020-10-28 21:44:13 -04:00
Deirdre Connolly 612148fbdd consensus: add transaction::check module 2020-10-28 21:44:13 -04:00
teor 3748623d92 Remove a redundant block header test vector 2020-10-28 21:24:28 -04:00
Alfredo Garcia bcb027ebc5 change canopy.pdf to stable protocol.pdf 2020-10-28 11:34:53 -04:00
dependabot[bot] 83c844abb5 build(deps): bump futures from 0.3.6 to 0.3.7
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.6 to 0.3.7.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.6...0.3.7)

Signed-off-by: dependabot[bot] <support@github.com>
2020-10-27 02:28:48 -04:00
Henry de Valence 65e0c22fbe state: don't pre-buffer the service
There's no reason to return a pre-Buffer'd service (there's no need for
internal access to the state service, as in zebra-network), but wrapping
it internally removes control of the buffer size from the caller.
2020-10-26 12:05:35 -07:00
Henry de Valence a1a3e4db5a consensus: simplify block verify tracing output
The previous debug output printed a message that the chain verifier had
recieved a block.  But this provides no additional information compared
to printing no message in chain::Verifier and a message in whichever
verifier the block was sent to, since the resulting spans indicate where
the block was dispatched.

This commit also removes the "unexpected high block" detection; this was
an artefact of the original sync algorithm failing to handle block
advertisements, but we don't have that problem any more, so we can
simplify the code by eliminating that logic.
2020-10-26 12:05:35 -07:00
teor a141c336ab Actually fix whitespace 2020-10-26 13:49:48 -04:00
teor bbe4aa47ea Fix whitespace for rustfmt 2020-10-26 13:49:48 -04:00
teor 2fa3d8a8f4 Add a comment explaining why block metrics follow validation 2020-10-26 13:49:48 -04:00
teor f5a53d9dae Update block metrics after async transaction verification 2020-10-26 13:49:48 -04:00
teor 60322c3d48 Test that the checkpoint list gap is correct
If we change the gap, but don't rebuild the lists, `zebrad` hangs with
weird errors.
2020-10-26 20:59:40 +10:00
teor f9dc481934 Rebuild the checkpoint lists with smaller checkpoints 2020-10-26 20:59:40 +10:00
teor 20dfd04463 Reduce maximum checkpoint size in the Zebra code
The new limits are 400 blocks and 32 MB.
2020-10-26 20:59:40 +10:00
dependabot[bot] ff51c2e0c0 build(deps): bump tracing-subscriber from 0.2.13 to 0.2.14
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.2.13 to 0.2.14.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.2.13...tracing-subscriber-0.2.14)

Signed-off-by: dependabot[bot] <support@github.com>
2020-10-23 15:02:02 -04:00
teor 6dc95b1d6d
Revise the checkpoint verifier metrics (#1195)
* update continuous and processing.next metrics correctly
* remove duplicate metrics
* rename ambiguous metrics
2020-10-21 20:06:26 -07:00
Alfredo Garcia 21ad6ffc47
Reverse displayed endianness of transaction and block hashes (#1171)
* Reverse displayed endianness of transaction and block hashes
* fix zebra-checkpoints utility for new hash order
* Stop using "zebrad revhex" in zebrad-hash-lookup
* Rebuild checkpoint lists in new hash order
This change also adds additional checkpoints to the end of each list.

* Replace TransactionHash with transaction::Hash
This change should have been made in #905, but we missed Debug impls
and some docs.

Co-authored-by: Ramana Venkata <vramana@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
2020-10-22 07:54:02 +10:00
Deirdre Connolly 9549e180c0 Allow dead_code on parameters for now 2020-10-20 11:16:22 -04:00
Deirdre Connolly a7ef6f6a40 Allow dead_code for checkpoint::Verifier for now 2020-10-20 11:16:22 -04:00
Deirdre Connolly e796132057 Allow dead_code for the transaction::Request for now (mempool) 2020-10-20 11:16:22 -04:00
Henry de Valence eb43893de0 consensus: minimize API, clean docs
This reduces the API surface to the minimum required for functionality,
and cleans up module documentation.  The stub mempool module is deleted
entirely, since it will need to be redone later anyways.
2020-10-20 11:16:22 -04:00
Henry de Valence d4ce3eb054 consensus: improve docs
- remove no longer accurate documentation about transaction verifier;
- add description of the role of the crate.
2020-10-20 11:16:22 -04:00
Jane Lusby c0aa1b477e consensus: add #[source] attributes to chain errors 2020-10-20 11:16:22 -04:00
Jane Lusby 8a64c056fb consensus: integrate block, transaction Verifiers 2020-10-20 11:16:22 -04:00
Jane Lusby 4a36c4485a consensus: integrate script, transaction Verifiers 2020-10-20 11:16:22 -04:00
Henry de Valence 248c318906 consensus: sketch of transaction::Verifier
This squashes the previous sequence of commits to let us separate out
the structural skeleton (which unblocks other work and is not
consensus-critical) from the actual checks (which don't block other work
and are consensus-critical).

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2020-10-20 11:16:22 -04:00
Henry de Valence 9dc8d76d68 consensus: add stub groth16::Verifier
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2020-10-20 11:16:22 -04:00
Deirdre Connolly 97b6ba10cc Use redjubjub crate exported from zebra-chain in the redjubjub::Verifier 2020-10-20 11:16:22 -04:00
teor 3da4fdb8db Reword more errors 2020-10-15 08:31:18 +10:00
teor c74a205ecf Reword error message
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
2020-10-15 08:31:18 +10:00
teor e46df7ed5b Tidy up a use list 2020-10-15 08:31:18 +10:00
teor d09c310bbc Add failure tests for difficulty is valid 2020-10-15 08:31:18 +10:00
teor 4076402536 Add failure tests for coinbase is valid 2020-10-15 08:31:18 +10:00