* Make Amount arithmetic more generic
To modify generated amounts, we need some extra operations on `Amount`.
We also need to extend existing operations to both `NonNegative` and
`NegativeAllowed` amounts.
* Add a constrain method for ValueBalance
* Derive Eq for ValueBalance
* impl Neg for ValueBalance
* Make some Amount arithmetic expectations explicit
* Explain why we use i128 for multiplication
And expand the overflow error details.
* Expand Amount::sum error details
* Make amount::Error field order consistent
* Rename an amount::Error variant to Constraint, so it's clearer
* Add specific pool variants to ValueBalanceError
* Update coinbase remaining value consensus rule comment
This consensus rule was updated recently to include coinbase transactions,
but Zebra doesn't check block subsidy or miner fees yet.
* Add test methods for modifying transparent values and shielded value balances
* Temporarily set values and value balances to zero in proptests
In both generated chains and proptests that construct their own transactions.
Using zero values reduces value calculation and value check test coverage.
A future change will use non-zero values, and fix them so the check passes.
* Add extra fields to remaining transaction value errors
* Swap the transparent value balance sign to match shielded value balances
This makes the signs of all the chain value pools consistent.
* Use a NonNegative constraint for transparent values
This fix:
* makes the type signature match the consensus rules
* avoids having to write code to handle negative values
* Allocate total generated transaction input value to outputs
If there isn't enough input value for an output, set it to zero.
Temporarily reduce all generated values to avoid overflow.
(We'll remove this workaround when we calculate chain value balances.)
* Consistently use ValueBalanceError for ValueBalances
* Make the value balance signs match the spec
And rename and document methods so their signs are clearer.
* Convert amount::Errors to specific pool ValueBalanceErrors
* Move some error changes to the next PR
* Add extra info to remaining transaction value errors (#2585)
* Distinguish between overflow and negative remaining transaction value errors
And make some error types cloneable.
* Add methods for updating chain value pools (#2586)
* Move amount::test to amount::tests:vectors
* Make ValueBalance traits more consistent with Amount
- implement Add and Sub variants with Result and Assign
- derive Hash
* Clarify some comments and expects
* Create ValueBalance update methods for blocks and transactions
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Validate transparent coinbase output maturity and shielding
- Add a CoinbaseSpendRestriction enum and Transaction method
- Validate transparent coinbase spends in non-finalized chains
* Don't use genesis created UTXOs for spends in generated block chains
* Refactor out a new_transaction_ordered_outputs function
* Add Transaction::outputs_mut for tests
* Generate valid transparent spends in arbitrary block chains
* When generating blocks, fixup the block contents, then the block hash
* Test that generated chains contain at least one transparent spend
* Make generated chains long enough for reliable tests
* Add transparent and shielded input and output methods to Transaction
* Split chain generation into 3 functions
* Test that unshielded and immature transparent coinbase spends fail
* Comment punctuation
* Clarify a comment
* Clarify probability calculation
* Test that shielded mature coinbase output spends succeed
* Create a `zebra_state::init_test` helper function
This function will be used as a replacement for `zebra_state::init`
inside tests. It's a simpler alternative because it can ignore any
details that aren't relevant for tests.
* Use `init_test` inside `zebra-state` tests
Update usages of `init` to use `init_test` instead, which simplifies
most cases.
* Use `zebra_state::init_test` in `zebra-consensus`
Replace usages of `zebra_state::init` with the new helper function. This
simplifies the code a bit.
* Add an OrderedUtxo type for validation of spends within a block
This change allows us to check that transparent spends use outputs from
earlier in their block. (But we don't actually do that check yet.)
We need to keep the order of UTXOs when we're contextually verifying
each new block that is added to a chain. But the block order is
irrelevant for UTXOs stored in the state.
* Take ownership in utxos_from_ordered_utxos
* Delete a confusing comment
* Add panic message to `unimplemented!`
So that it is clear why the panic happened upon initial inspection. Also
include a reference to the mempool epic, so that it's easier to find the
issue that tracks the implementation of the missing code.
* Add panic message that references a tracking issue
Make it easy to find the relevant issue if the panic occurs.
* Remove incomplete and currently unnecessary code
The current implementation works, the commented out code was just a
previous improvement idea, which is now tracked by issue #2473.
* Always use librustzcash for sighash and remove old sighash code
Also added ZIP-143 test vectors
* Remove librustzcash_sighash test that is no longer needed
* Make `verify_sapling_shielded_data` more generic
Prepare to support V5 transactions which have a shared anchor.
* Verify Sapling shielded data in V5 transactions
Call the `verify_sapling_shielded_data` method and add the respective
asynchronous checks to the set of V5 checks.
* Fix expect message in V4 transaction test
It was using the same message as the previous test, even though the test
searches with different criteria.
* Test V5 transaction with Sapling spends
Create a fake V5 transaction that has Sapling spends and check that the
verifier accepts the transaction.
* Ignore rejected V5 transaction test for now
Because now it needs the `sighash` implementation for V5 to be ready.
* Reference V5 `sighash` PR in comment
So that it is easier to check if it's possible to remove the
`should_panic` or not.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove `sapling shielded pool` TODO
V5 transactions now have Sapling shielded pool properly validated.
* Link to some extra issues in TODO comment
Some other issues are also necessary for full V5 validation.
* Add a TODO in the main code to fix the tests
Some tests are blocked due to missing features required for full V5
validation. Once those features are implemented, they should be updated
to remove the `#[should_panic]` attribute so that they actually run and
check the code correctly.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Refactor `AsyncChecks` into a proper type
Add some helper methods to it so that checks can be added by
daisy-chaining calls. Also move the code to wait for the checks to
finish into the new type.
* Refactor inclusion of individual Sapling checks
Use `oneshot` instead of `ready_and` so that the method becomes
synchronous.
* Make V4 and V5 verification methods synchronous
There is no longer a need to wait for any internal service to be ready,
since now that's always done as part of an asynchronous check included
in the returned set of checks.
* add disabled sprout pool check
* change method name
* change error name
* fix typo
* make the success test case in other tx than the coinbase
* use new `height` method instead of deriving `PartialOrd` in `NetworkUpgrade`
* move check of network upgrade into function, rename, docs
* increase test coverage
* fix comment
* Update versions for zebra v1.0.0-alpha.12 release
* Update Cargo.lock
* Update release checklist with latest version changes to help keep track for future releases
* Remove reference to the fact that tower-fallback was not updated
* Refactor to create `verify_sapling_shielded_data`
Move the code to verify Sapling shielded data into a new helper method
that returns `AsyncChecks`.
* Test verifying a Sapling transaction with spends
Use the test vectors to find a transaction that has Sapling spends and
test if it the verifier considers it valid.
* Create a helper method to list test transactions
Transforms the block test vectors into a list of transactions and block
heights for each transaction.
* Use new helper function in V4 Sapling spend test
Also use the block height for that transaction as specified in the test
vector.
* Test V4 tx. with Sapling outputs but no spends
Find a transaction V4 vector that has Sapling outputs but no spends, and
check that the verifier accepts it.
* Add a `zebra_test::RUNTIME` shared runtime
Create a lazily instantiated Tokio runtime that can be shared by tests.
* Split tests that require a shared runtime
Split two tests that were previously in one because of the need to share
a single Tokio runtime. With the `zebra_test::RUNTIME`, they can now
share the runtime without having to be a single test.
* Stop trying to verify coinbase inputs using the script verifier
And create tests to catch similar bugs earier.
* Use Testnet in NU5 tests that temporarily should_panic
We've marked these tests as should_panic until there is a NU5 activation
height. But Testnet will have an activation height first, so we should
prefer it in the tests. (Or use both networks.)
Block transactions already had a height, but mempool transactions didn't.
This PR adds a height to mempool transactions, and deletes redundant and
unused fields. It also adds an accessor method for that height.
* Refactor to create `verify_sprout_shielded_data`
Move the join split verification code into a new
`verify_sprout_shielded_data` helper method that returns an
`AsyncChecks` set.
* Test if signed V4 tx. join splits are accepted
Create a fake V4 transaction with a dummy join split, and sign it
appropriately. Check if the transaction verifier accepts the
transaction.
* Test if unsigned V4 tx. joinsplit data is rejected
Create a fake V4 transaction with a dummy join split. Do NOT sign this
transaction's join split data, and check that the verifier rejects the
transaction.
* Join tests to share Tokio runtime
Otherwise one of the tests might fail incorrectly because of a
limitation in the test environment. `Batch` services spawn a task in the
Tokio runtime, but separate tests can have separate runtimes, so sharing
a `Batch` service can lead to the worker task only being available for
one of the tests.
* Describe how a ZIP-213 rule is implemented in the transaction verifier
* Move the only coinbase-specific check outside the ZIP-213 block
This change isn't required to implement the ZIP-213 rule, but it makes
it easier to identify the specific checks for coinbase transactions.
* Add a note about coinbase in the mempool
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Update multiple crates to ensure bitvec 0.22.3 is being used and avoid package conflicts
* Add documentation to zebra-chain::sapling to indicate that ZIP-216 rules are enforced by jubjub
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Add missing documentation
Document methods to describe what they do and why.
* Create an `AsyncChecks` type alias
Make it simpler to write the `FuturesUnordered` type with boxed futures.
This will also end up being used more when refactoring to return the
checks so that the `call` method can wait on them.
* Create `verify_transparent_inputs_and_outputs`
Refactors the verification of the transparent inputs and outputs into a
separate method.
* Refactor transparent checks to use `call_all`
Instead of pushing the verifications into a stream of unordered futures,
use the `ServiceExt::call_all` method to build an equivalent stream
after building a stream of requests.
* Replace `CallAll` with `FuturesUnordered`
Make it more consistent with the rest of the code, and make sure that
the `len()` method is available to use for tracing.
Co-authored-by: teor <teor@riseup.net>
* Refactor to move wait for checks into a new method
Allow the code snipped to be reused by other transaction
version-specific check methods.
* Verify transparent inputs in V5 transactions
Use the script verifier to check the transparent inputs in a V5
transaction.
* Check `has_inputs_and_outputs` for all versions
Check if a transaction has inputs and outputs, independently of the
transaction version.
* Wait for checks in `call` method
Refactor to move the repeated code into the `call` method. Now the
validation methods return the set of asynchronous checks to wait for.
* Add helper function to mock transparent transfers
Creates a fake source UTXO, and then the input and output that represent
spending that UTXO. The initial UTXO can be configured to have a script
that either accepts or rejects any spend attempt.
* Test if transparent V4 transaction is accepted
Create a fake V4 transaction that includes a fake transparent transfer
of funds. The transfer uses a script to allow any UTXO to spend it.
* Test transaction V4 rejection based on script
Create a fake transparent transfer where the source UTXO has a script
that rejects spending. The script verifier should not accept this
transaction.
* Test if transparent V5 transaction is accepted
Create a mock V5 transaction that includes a transparent transfer of
funds. The transaction should be accepted by the verifier.
* Test transaction V5 rejection based on script
Create a fake transparent transfer where the source UTXO has a script
that rejects spending. The script verifier should not accept this
transaction.
* Update `Request::upgrade` getter documentation
Simplify it so that it won't become updated when #1683 is fixed.
Co-authored-by: teor <teor@riseup.net>
* move network_upgrade check into zebra-chain
* fix the errors
* rename function
* typo fix
* rename the check function
* make changes from last code review
* Add a `Transaction::version` getter
Returns the version of the transaction as a `u32`.
* Add `Transaction::is_overwintered` helper method
Returns if the `fOverwintered` flag should be set for the transaction's
version.
* Use new helpers to serialize transaction version
Reduce the repeated code and make it less error-prone with future
changes.
* Add getter methods to `transaction::Request` type
Refactor to move the type deconstruction code into the `Request` type.
The main objective is to make it easier to split the call handler into
methods that receive the request directly.
* Refactor to create `verify_v4_transaction` helper
Split the code specific to V4 transactions into a separate helper
method.
* Create `verify_v5_transaction` helper method
Prepare a separate method to have the validation code.
* Add `UnsupportedByNetworkUpgrade` error variant
An error for when a transaction's version isn't supported by the network
upgrade of the block it's included or for the current network upgrade if
the transaction is for the mempool.
* Verify a V5 transaction's network upgrade
For now, only NU5 supports V5 transactions.
* Test that V5 transaction is rejected on Canopy
Create a fake V5 transaction and try to verify it using a block height
from Canopy's activation. The verifier should reject the transaction
with an error saying that the network upgrade does not support that
transaction version.
* Test if V5 tx. is accepted after NU5 activation
Create a fake V5 transaction and pretend it is placed in a block that
has a height after the NU5 activation. The test should succeed, but
since the NU5 activation height has not been specified yet (neither for
the testnet nor the mainnet), for now this test is marked as
`should_panic`.
* Add `TODO` comment to the code
Add more detail to what's left to do, and link to the appropriate PRs.
* Use `u32` to store transaction version
Use a type consistent with how the version is specified.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* Add a `at_least_one!` macro for testing
Similar to the `vec!` macro, but doesn't allow creating an empty list.
* Test if `has_inputs_and_outputs` considers actions
Create a dummy transaction with no inputs and no outputs, and add a
dummy Orchard action to it. The `check::has_inputs_and_outputs`
should succeed, because the consensus rule considers having Orchard
actions as having inputs and/or outputs.
* Refactor to create helper function
Move the code to create a fake Orchard shielded data instance to a
helper function in `zebra_chain::transaction::arbitrary`, so that other
tests can also use it.
* Test coinbase V5 transaction with enable spends
A V5 coinbase transaction that has Orchard shielded data MUST NOT have
the enable spends flag set.
* Test if coinbase without enable spends is valid
A coinbase transaction with Orchard shielded data and without the enable
spends flag set should be valid.
* Add a security comment about the `at_least_one!` macro
This macro must not be used outside tests, because it allows memory denial
of service.
Co-authored-by: teor <teor@riseup.net>
* Implement `PartialEq<i64>` for `Amount`
Allows to compare an `Amount` instance directly to an integer.
* Add `SerializationError::BadTransactionBalance`
Error variant representing deserialization of a transaction that doesn't
conform to the Sapling consensus rule where the balance MUST be zero if
there aren't any shielded spends and outputs.
* Validate consensus rule when deserializing
Return an error if the deserialized V4 transaction has a non-zero value
balance but doesn't have any Sapling shielded spends nor outputs.
* Add consensus rule link to field documentation
Describe how the consensus rule is validated structurally by
`ShieldedData`.
* Clarify that `value_balance` is zero
Make the description more concise and objective.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Update field documentation
Include information about how the consensus rule is guaranteed during
serialization.
Co-authored-by: teor <teor@riseup.net>
* Remove `check::sapling_balances_match` function
The check is redundant because the respective consensus rule is
validated structurally by `ShieldedData`.
* Test deserialization of invalid V4 transaction
A transaction with no Sapling shielded spends and no outputs but with a
non-zero balance value should fail to deserialize.
* Change least-significant byte of the value balance
State how the byte index is calculated, and change the least
significant-byte to be non-zero.
Co-authored-by: teor <teor@riseup.net>
* Standardise lints across Zebra crates, and add missing docs
The only remaining module with missing docs is `zebra_test::command`
* Todo -> TODO
* Clarify what a transcript ErrorChecker does
Also change `Error` -> `BoxError`
* TransError -> ExpectedTranscriptError
* Output Descriptions -> Output descriptions
* Add a `Transaction::orchard_shielded_data` getter
Allows accessing the Orchard shielded data if it is present in the
transaction, regardless of the transaction version.
* Refactor `orchard_nullifiers` to use new getter
Allows making the method more concise.
* Add `CoinbaseHasEnableSpendsOrchard` error variant
Used when the validation rule is not met.
* Implement `enableSpendsOrchard` in coinbase check
The flag must not be set for the coinbase transaction.
* Refactor `Transaction::orchard_*` getters
Use the fact that `Option<T>` implements `Iterator<T>` to simplify the
code and remove the need for boxing the iterators.
Co-authored-by: teor <teor@riseup.net>
* Make sure the Canopy activation block is a finalized checkpoint block
This enables ZIP-221 chain history from Canopy activation onwards.
* Clarify that the mandatory checkpoint test includes Canopy activation
The test was correct, but the docs and assertion message did not include activation.
* Document that the mandatory checkpoint includes Canopy activation
Co-authored-by: teor <teor@riseup.net>
* Fix documentation comment
Was missing a slash to become documentation.
* Add documentation link to type reference
Just to help navigation a bit.
* Implement `Transaction::orchard_actions()` getter
Returns an iterator to iterator over the actions in the Orchard shielded
data (if there is one, otherwise it returns an empty iterator).
* Add V5 support for `has_inputs_and_outputs`
Checks if the transaction has Orchard actions. If it does, it is
considered to have inputs and outputs.
* Refactor transaction test vectors
Make it easier to reuse the fake V5 transaction converter in other test
vectors.
* Move helper function to `zebra-chain` crate
Place it together with some other helper functions, including the one
that actually creates the fake V5 transaction.
* Test transaction with no inputs
`check::has_inputs_and_outputs` should return an error indicating that
the transaction has no inputs.
* Test transaction with no outputs
`check::has_inputs_and_outputs` should return an error indicating that
the transaction has no outputs.
* Note that transaction is fake in `expect` message
Should make the message easier to find, and also gives emphasis to the
fact that the transaction is a fake conversion to V5.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* validate sapling v5 tx
* Make itertools dependency optional
We only need itertools when the `proptest-impl` feature is enabled.
* Check if V4 and V5 coinbase transactions contain PrevOut transparent inputs
This is a bugfix on V4 transaction validation. The PrevOut consensus
rule was not explicitly stated in the Zcash spec until April 2021.
(But it was implied by Bitcoin, and partially implemented by Zebra.)
Also do the shielded sapling input check for V5 transactions.
* Add spec and orchard TODOs to has_inputs_and_outputs
Also make the variable names match the spec.
* Sort transaction functions to match v5 data order
* Simplify transaction input and output checks
Move counts or iterators into `Transaction` methods, so we can remove
duplicate code, and make the consensus rule logic clearer.
* Update sapling_balances_match for Transaction v5
- Quote from the spec
- Explain why the function is redunant for v5
- Rename the function so it's clear that it is sapling-specific
Co-authored-by: teor <teor@riseup.net>
* start refactoring transaction v4 for transaction v5
- move ShieldedData to sapling
- add AnchorVariant
- rename shielded_data to sapling_shielded data in V4
- move value_balance into ShieldedData
- update prop tests for new structure
* add AnchorVariant to Spend
- make anchor types available from sapling crate
- update serialize
* change shielded_balances_match() arguments
* change variable name anchor to shared_anchor in ShieldedData
* fix empty value balance serialization
* use AnchorV in shielded spends
* Rename anchor to per_spend_anchor
* Use nullifiers function directly in non-finalized state
* Use self.value_balance instead of passing it as an argument
* Add missing fields to ShieldedData PartialEq
* Derive Copy for tag types
* Add doc comments for ShieldedData refactor
* Implement a per-spend anchor compatibility iterator
Co-authored-by: teor <teor@riseup.net>
* Ed25519 async batch verification for JoinSplit signatures
We've been verifying JoinSplitSigs one-by-one pre-ZIP-215. Now as we're post-ZIP-215,
we can take advantage of the batch math to validate this signatures.
I would have pumped all the joinsplits in our MAINNET_BLOCKS test vectors but these
signatures are over the sighash, which needs the NU code to compute, and once we're
doing all that set up, we're basically doing transaction validation, so.
Resolves#1944
* Repoint to latest ed25519-zebra commit with note to point at 3.0 when released
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Zebra's latest alpha checkpoints on Canopy activation, continues our work on NU5, and fixes a security issue.
Some notable changes include:
## Added
- Log address book metrics when PeerSet or CandidateSet don't have many peers (#1906)
- Document test coverage workflow (#1919)
- Add a final job to CI, so we can easily require all the CI jobs to pass (#1927)
## Changed
- Zebra has moved its mandatory checkpoint from Sapling to Canopy (#1898, #1926)
- This is a breaking change for users that depend on the exact height of the mandatory checkpoint.
## Fixed
- tower-batch: wake waiting workers on close to avoid hangs (#1908)
- Assert that pre-Canopy blocks use checkpointing (#1909)
- Fix CI disk space usage by disabling incremental compilation in coverage builds (#1923)
## Security
- Stop relying on unchecked length fields when preallocating vectors (#1925)
* add transaction V5 stub
* add v5_strategy
* deduplicate version group ids
* Update comment for V5 transactions
* Add V5 transactions to non_finalized_state
Currently these are all `unimplemented!(...)`
* Fix struct matches
* Apply trivial panic message changes
* add zcash_deserialize for V5
* make all tx versions explicit in sprout and sapling nullifier functions
* match exhaustively in sprout and sapling nullifier functions
* fix matches in zebra-consensus
* fix NU5 strategy
* We're still deciding if v5 transactions support Sprout
Co-authored-by: teor <teor@riseup.net>
* change `poll_ready()` and `call()` of `ChainVerifier`
* add bound, move max_checkpoint_height
* add buffers to the checkpoint and block verifiers
And rename the chain verifier fields so `block` means `Arc<Block>`, and
`block_verifier` means `Buffer<BlockVerifier, ...>`.
* Fix the error types
* Use `ServiceExt::oneshot` in `ChainVerifier::call`
And:
* make the code look like the `main` branch as much as possible
* document the `poll_ready`/`call` invariant
* Use `ServiceExt::oneshot` in `chain::init`
Co-authored-by: teor <teor@riseup.net>
This PR is the first step in getting a groth16 proving system fully
integrated with the rest of zebra. This PR implements the initial async
API, but none of the actual batching logic necessary for our eventual
verifier design.
Once the batch verification API from bellman has been implemented we
will need to swap out the "Batch" type defined in this crate with the
new `batch::Verifier` defined in bellman.
* Bump versions where appropriate
Tested with cargo install --locked --path etc
* Remove fixed panics from 'Known Issues'
* Change to alpha release series in the README
Co-authored-by: teor <teor@riseup.net>
The clippy unknown lints attribute was deprecated in
nightly in rust-lang/rust#80524. The old lint name now produces a
warning.
Since we're using `allow(unknown_lints)` to suppress warnings, we need to
add the canonical name, so we can continue to build without warnings on
nightly.
But we also need to keep the old name, so we can continue to build
without warnings on stable.
And therefore, we also need to disable the "removed lints" warning,
otherwise we'll get warnings about the old name on nightly.
We'll need to keep this transitional clippy config until rustc 1.51 is
stable.
## Motivation
This PR is motivated by the regression identified in https://github.com/ZcashFoundation/zebra/issues/1349. That PR notes that the metrics stopped working for most of the crates other than `zebrad`.
## Solution
This PR resolves the regression by deduplicating the `metrics` crate dependency. During a recent change we upgraded the metrics version in `zebrad` and a couple other of our crates, but we never updated the dependencies in `zebra-state`, `zebra-consensus`, or `zebra-network`. This caused the metrics macros to attempt to retrieve the current metrics exporter through the wrong function. We would install the metrics exporter in `0.13`, but then attempt to look it up through the `0.12` crate, which contains a different instance of the metrics exporter static variable which is unset. Doing this causes the metrics macros to return `None` for the current exporter after which they just silently give up.
## Related Issues
closes https://github.com/ZcashFoundation/zebra/issues/1349
## Follow Up Work
I noticed we have quite a few duplicate dependencies in our tree. We might be able to save some compilation time by auditing those and deduplicating them as much as possible.
- https://github.com/ZcashFoundation/zebra/issues/1582
Co-authored-by: teor <teor@riseup.net>
* export new precompute api in zebra-script
* remove old API in favor of precompute API
* add multi use test cases and bump version
* update implementation to actually match henry's design
* Add safety comment for zebra-script
Co-authored-by: teor <teor@riseup.net>
* create readme for utils and checkpoints
* add link to checkpoint usage to book
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Previously we set the crate versions to 3.x, so that the major version was
aligned with the NU version. But we want to be able to make API changes
independently of the NU schedule.
Change the Merkle root validation logic to also check that a block does not
contain duplicate transactions. This check is redundant with later
double-spend checks, but is a useful defense-in-depth.
As a side effect of computing Merkle roots, we build a list of
transaction hashes. Instead of discarding these, add them to
PreparedBlock and FinalizedBlock so that they can be reused rather than
recomputed.
This commit adds Merkle root validation to:
1. the block verifier;
2. the checkpoint verifier.
In the first case, Bitcoin Merkle tree malleability has no effect,
because only a single Merkle tree in each malleablity set is valid (the
others have duplicate transactions).
In the second case, we need to check that the Merkle tree does not contain any
duplicate transactions.
Closes#1385Closes#906
UTXO requests during transaction input verification can time out because:
1. The block that creates the UTXO is queued for download or verify, but
it hasn't been committed yet. The creating block might spend UTXOs
that come from other recent blocks, so UTXO verification can depend on
a (non-contiguous) sequence of block verifications.
In this case, Zebra should wait for additional block download and
verify tasks to complete.
2. The block that creates the UTXO isn't queued for download. This can
happen because the block is gossiped block that's much higher than the
current tip, or because a peer sent the syncer a bad list of block
hashes.
In this case, Zebra should discard the timed out block, and restart
the sync.
We need to choose a timeout that balances these two cases, so we time
out after 180 seconds.
Assuming Zebra can download at least 1 MB per second, 180 seconds is
enough time to download a few hundred blocks. So Zebra should be able to
download and verify the next block before the UTXOs that it creates time
out. (Since Zebra has already verified all the blocks before the next
block, its UTXO requests should return immediately.)
Even if some peers time out downloads, a block can only be pending
download for 80 seconds (4 retries * 20 second timeout) before the
download fails. So the UTXO timeout doesn't need to be much larger than
this overall download timeout - because the download timeout will happen
first on slow networks.
Alternately, if the download for the creating block was never queued,
Zebra should timeout as soon as possible - so it can restart the sync
and download the creating block.
As a side-effect, a lower UTXO timeout also makes it slightly easier to
debug UTXO issues, because unsatisfiable queries fail faster.
The state service API says explicitly that AwaitUTXO requests should be coupled
with a timeout layer. I didn't add this when I was testing and fixing the UTXO
lookup code (#1348, #1358) because causing zebrad to hang on a failed
dependency was useful for identifying cases where the code wasn't useful (and
then inspecting execution traces).
As a side effect, this ticket resolves most of the hangs in #1389, because
far-future gossiped blocks will have their UTXO lookups time out, though we
may wish to do other work as part of debugging the combined sync+gossip logic.
This commit changes the state system and database format to track the
provenance of UTXOs, in addition to the outputs themselves.
Specifically, it tracks the following additional metadata:
- the height at which the UTXO was created;
- whether or not the UTXO was created from a coinbase transaction or
not.
This metadata will allow us to:
- check the coinbase maturity consensus rule;
- check the coinbase inputs => no transparent outputs rule;
- implement lookup of transactions by utxo (using the height to find the
block and then scanning the block) for a future RPC mechanism.
Closes#1342
This change introduces two new types:
- `PreparedBlock`, representing a block which has undergone semantic
validation and has been prepared for contextual validation;
- `FinalizedBlock`, representing a block which is ready to be finalized
immediately;
and changes the `Request::CommitBlock`,`Request::CommitFinalizedBlock`
variants to use these types instead of their previous fields.
This change solves the problem of passing data between semantic
validation and contextual validation, and cleans up the state code by
allowing it to pass around a bundle of data. Previously, the state code
just passed around an `Arc<Block>`, which forced it to needlessly
recompute block hashes and other data, and was incompatible with the
already-known but not-yet-implemented data transfer requirements, namely
passing in the Sprout and Sapling anchors computed during contextual
validation.
This commit propagates the `PreparedBlock` and `FinalizedBlock` types
through the state code but only uses their data opportunistically, e.g.,
changing .hash() computations to use the precomputed hash. In the
future, these structures can be extended to pass data through the
verification pipeline for reuse as appropriate. For instance, these
changes allow the sprout and sapling anchors to be propagated through
the state.
The UTXO query system assumes that a transaction will only request
information about UTXOs created in prior blocks. But transactions are
allowed to spend UTXOs created by prior transactions in the same block.
This doesn't fit with the existing query model, so instead of trying to
change it, allow the script verifier to take an additional set of known
UTXOs, and propagate this set from the block.
This consensus rule is supposed to apply to transactions whose
transparent inputs are the *outputs* of previous coinbase
transactions, not to transactions with coinbase inputs. Because that
logic is different enough from this logic, and requires different data
flow, it's cleaner to just remove this check for now.
Making this check's match statement exhaustive revealed a bug similar to
the previous commit. The logic in the spec is written in terms of
numbers, but our data is internally represented in terms of enums
(ADTs). This kind of cross-representation rule translation is a bug
surface, which we can avoid by converting to counts and summing up. (We
should use one style at a time).
This function caused spurious "WrongVersion" errors, because the match
pattern in the first arm was non-exhaustive, but the fallthrough match
arm was present and assumed it would only be reached if the version was
incorrect.
This commit cleans up the implemenation, splits out the error variants,
and renames the check to be more precise.
To avoid this kind of bug in the future, two guidelines are useful:
1. Avoid fallthrough cases that circumvent non-exhaustive match checks;
2. Avoid nested conditionals, preferring a "straight-line" sequence of
match arm => result pairs rather than nested matches or matches with
conditionals inside.
The BlockVerifier constructed a tracing span and manually entered it
inside of an async block. Manually entering spans inside async blocks
can cause problems where the span might not be entered and exited
correctly as the resulting future is polled. Instead, using the
.instrument creates a wrapper future that handles the bookkeeping.
I changed the span name and contents to be consistent with the spans in
the checkpoint verifier.
This change is mostly mechanical, with the exception of the changes to the
`tower-batch` middleware. This middleware was adapted from `tower::buffer`,
and the `tower::buffer` code was changed to implement its own bounded queue,
because Tokio 0.3 removed the `mpsc::Sender::poll_send` method. See
ddc64e8d4d
for more context on the Tower changes. To match Tower as closely as possible
in order to be able to upstream `tower-batch`, those changes are copied from
`tower::Buffer` to `tower-batch`.
This change explicitly documents cancellation contracts for our Tower services,
and tries to correct a bug in the implementation of the CheckpointVerifier,
which duplicates information from the state service but did not ensure that it
would be kept in sync.
* Implement Expanded to Compact Difficulty
* Implement Arbitrary for CompactDifficulty
Remove the derive, and generate values from random block
hashes.
* Implement Arbitrary for ExpandedDifficulty and Work
* Use Arbitrary for CompactDifficulty in Arbitrary for Block
* Test difficulty on all block test vectors
And cleanup some duplicate test code
* Round-trip tests for compact test cases
* Round-trip tests for compact difficulty in block test vectors
* Make Add for Work return PartialCumulativeWork
Remove AddAssign for Work
Rewrite a proptest using Sub for PartialCumulativeWork
Use Arbitrary for Work
* Add roundtrip work sum tests
* Add roundtrip comparison difficulty tests
* Add failing proptest cases due to test bugs
* Use Some(_) rather than _.into()
* Reduce visibility of difficulty type inner values
* Split work and other difficulty proptests
This change makes sure that rejected work values don't disable property
tests on other types.
There's no reason to return a pre-Buffer'd service (there's no need for
internal access to the state service, as in zebra-network), but wrapping
it internally removes control of the buffer size from the caller.
The previous debug output printed a message that the chain verifier had
recieved a block. But this provides no additional information compared
to printing no message in chain::Verifier and a message in whichever
verifier the block was sent to, since the resulting spans indicate where
the block was dispatched.
This commit also removes the "unexpected high block" detection; this was
an artefact of the original sync algorithm failing to handle block
advertisements, but we don't have that problem any more, so we can
simplify the code by eliminating that logic.
teor
Alfredo Garcia
Pili Guerra
Janito Vaqueiro Ferreira Filho
Conrado Gouvea
dependabot[bot]
Deirdre Connolly
Deirdre Connolly
Jack Grigg
Jane Lusby
Jane Lusby
Henry de Valence
Henry de Valence