Change the Merkle root validation logic to also check that a block does not
contain duplicate transactions. This check is redundant with later
double-spend checks, but is a useful defense-in-depth.
As a side effect of computing Merkle roots, we build a list of
transaction hashes. Instead of discarding these, add them to
PreparedBlock and FinalizedBlock so that they can be reused rather than
recomputed.
This commit adds Merkle root validation to:
1. the block verifier;
2. the checkpoint verifier.
In the first case, Bitcoin Merkle tree malleability has no effect,
because only a single Merkle tree in each malleablity set is valid (the
others have duplicate transactions).
In the second case, we need to check that the Merkle tree does not contain any
duplicate transactions.
Closes#1385Closes#906
UTXO requests during transaction input verification can time out because:
1. The block that creates the UTXO is queued for download or verify, but
it hasn't been committed yet. The creating block might spend UTXOs
that come from other recent blocks, so UTXO verification can depend on
a (non-contiguous) sequence of block verifications.
In this case, Zebra should wait for additional block download and
verify tasks to complete.
2. The block that creates the UTXO isn't queued for download. This can
happen because the block is gossiped block that's much higher than the
current tip, or because a peer sent the syncer a bad list of block
hashes.
In this case, Zebra should discard the timed out block, and restart
the sync.
We need to choose a timeout that balances these two cases, so we time
out after 180 seconds.
Assuming Zebra can download at least 1 MB per second, 180 seconds is
enough time to download a few hundred blocks. So Zebra should be able to
download and verify the next block before the UTXOs that it creates time
out. (Since Zebra has already verified all the blocks before the next
block, its UTXO requests should return immediately.)
Even if some peers time out downloads, a block can only be pending
download for 80 seconds (4 retries * 20 second timeout) before the
download fails. So the UTXO timeout doesn't need to be much larger than
this overall download timeout - because the download timeout will happen
first on slow networks.
Alternately, if the download for the creating block was never queued,
Zebra should timeout as soon as possible - so it can restart the sync
and download the creating block.
As a side-effect, a lower UTXO timeout also makes it slightly easier to
debug UTXO issues, because unsatisfiable queries fail faster.
The state service API says explicitly that AwaitUTXO requests should be coupled
with a timeout layer. I didn't add this when I was testing and fixing the UTXO
lookup code (#1348, #1358) because causing zebrad to hang on a failed
dependency was useful for identifying cases where the code wasn't useful (and
then inspecting execution traces).
As a side effect, this ticket resolves most of the hangs in #1389, because
far-future gossiped blocks will have their UTXO lookups time out, though we
may wish to do other work as part of debugging the combined sync+gossip logic.
This commit changes the state system and database format to track the
provenance of UTXOs, in addition to the outputs themselves.
Specifically, it tracks the following additional metadata:
- the height at which the UTXO was created;
- whether or not the UTXO was created from a coinbase transaction or
not.
This metadata will allow us to:
- check the coinbase maturity consensus rule;
- check the coinbase inputs => no transparent outputs rule;
- implement lookup of transactions by utxo (using the height to find the
block and then scanning the block) for a future RPC mechanism.
Closes#1342
This change introduces two new types:
- `PreparedBlock`, representing a block which has undergone semantic
validation and has been prepared for contextual validation;
- `FinalizedBlock`, representing a block which is ready to be finalized
immediately;
and changes the `Request::CommitBlock`,`Request::CommitFinalizedBlock`
variants to use these types instead of their previous fields.
This change solves the problem of passing data between semantic
validation and contextual validation, and cleans up the state code by
allowing it to pass around a bundle of data. Previously, the state code
just passed around an `Arc<Block>`, which forced it to needlessly
recompute block hashes and other data, and was incompatible with the
already-known but not-yet-implemented data transfer requirements, namely
passing in the Sprout and Sapling anchors computed during contextual
validation.
This commit propagates the `PreparedBlock` and `FinalizedBlock` types
through the state code but only uses their data opportunistically, e.g.,
changing .hash() computations to use the precomputed hash. In the
future, these structures can be extended to pass data through the
verification pipeline for reuse as appropriate. For instance, these
changes allow the sprout and sapling anchors to be propagated through
the state.
The UTXO query system assumes that a transaction will only request
information about UTXOs created in prior blocks. But transactions are
allowed to spend UTXOs created by prior transactions in the same block.
This doesn't fit with the existing query model, so instead of trying to
change it, allow the script verifier to take an additional set of known
UTXOs, and propagate this set from the block.
This consensus rule is supposed to apply to transactions whose
transparent inputs are the *outputs* of previous coinbase
transactions, not to transactions with coinbase inputs. Because that
logic is different enough from this logic, and requires different data
flow, it's cleaner to just remove this check for now.
Making this check's match statement exhaustive revealed a bug similar to
the previous commit. The logic in the spec is written in terms of
numbers, but our data is internally represented in terms of enums
(ADTs). This kind of cross-representation rule translation is a bug
surface, which we can avoid by converting to counts and summing up. (We
should use one style at a time).
This function caused spurious "WrongVersion" errors, because the match
pattern in the first arm was non-exhaustive, but the fallthrough match
arm was present and assumed it would only be reached if the version was
incorrect.
This commit cleans up the implemenation, splits out the error variants,
and renames the check to be more precise.
To avoid this kind of bug in the future, two guidelines are useful:
1. Avoid fallthrough cases that circumvent non-exhaustive match checks;
2. Avoid nested conditionals, preferring a "straight-line" sequence of
match arm => result pairs rather than nested matches or matches with
conditionals inside.
The BlockVerifier constructed a tracing span and manually entered it
inside of an async block. Manually entering spans inside async blocks
can cause problems where the span might not be entered and exited
correctly as the resulting future is polled. Instead, using the
.instrument creates a wrapper future that handles the bookkeeping.
I changed the span name and contents to be consistent with the spans in
the checkpoint verifier.
This change is mostly mechanical, with the exception of the changes to the
`tower-batch` middleware. This middleware was adapted from `tower::buffer`,
and the `tower::buffer` code was changed to implement its own bounded queue,
because Tokio 0.3 removed the `mpsc::Sender::poll_send` method. See
ddc64e8d4d
for more context on the Tower changes. To match Tower as closely as possible
in order to be able to upstream `tower-batch`, those changes are copied from
`tower::Buffer` to `tower-batch`.
This change explicitly documents cancellation contracts for our Tower services,
and tries to correct a bug in the implementation of the CheckpointVerifier,
which duplicates information from the state service but did not ensure that it
would be kept in sync.
* Implement Expanded to Compact Difficulty
* Implement Arbitrary for CompactDifficulty
Remove the derive, and generate values from random block
hashes.
* Implement Arbitrary for ExpandedDifficulty and Work
* Use Arbitrary for CompactDifficulty in Arbitrary for Block
* Test difficulty on all block test vectors
And cleanup some duplicate test code
* Round-trip tests for compact test cases
* Round-trip tests for compact difficulty in block test vectors
* Make Add for Work return PartialCumulativeWork
Remove AddAssign for Work
Rewrite a proptest using Sub for PartialCumulativeWork
Use Arbitrary for Work
* Add roundtrip work sum tests
* Add roundtrip comparison difficulty tests
* Add failing proptest cases due to test bugs
* Use Some(_) rather than _.into()
* Reduce visibility of difficulty type inner values
* Split work and other difficulty proptests
This change makes sure that rejected work values don't disable property
tests on other types.
teor
Henry de Valence
Deirdre Connolly
dependabot[bot]
Henry de Valence
Alfredo Garcia