Zebra's latest beta continues implementing zero-knowledge proof and note commitment tree validation. In this release, we have finished implementing transaction header, transaction amount, and Zebra-specific NU5 validation. (NU5 mainnet validation is waiting on an `orchard` crate update, and some consensus parameter updates.)
We also fix a number of security issues that could pose a local denial of service risk, or make it easier for an attacker to make a node follow a false chain.
As of this release, Zebra will automatically download and cache the Sprout and Sapling Groth16 circuit parameters. The cache uses around 1 GB of disk space. These cached parameters are shared across all Zebra and `zcashd` instances run by the same user.
See CHANGELOG.md for the full list of changes in this release.
* validate non coinbase expiration height
* change var name
* move checks to transaction verifier
* Add variants and debug fields to transaction expiry errors
* Fix a failing existing test
Co-authored-by: teor <teor@riseup.net>
* Download and load Sprout parameters using zcash_proofs
Also update some librustzcash dependencies, to avoid duplicate dependencies.
* Update upstream orchard to avoid a compilation error
* Skip librustzcash batch refactor for now, to avoid compilation errors
* Change the cache ID, so we actually cache Sprout
* Move existing file checks into zcash_proofs
* Add a 1 hour timeout to parameter file downloads
* Give other tasks priority, before spawning the download task
* Update to the latest version of our modified librustzcash fork
* Change the cache key for Sprout
* Add 40 minutes to CI timeouts for occasional sprout downloads
* Update to zcash_proofs with split downloads
* Check file sizes to help debug parameter load failures in zcash_proofs
* Start the second download once the first has finished in zcash_proofs
* Document the parameter download task
* Stop hashing existing files twice
* validate consensus rule: negative fee not allowed
* fix a test TODO
* fix imports
* move import back
* fix panic text
* join consensus rule check code
* match assertion better in tests
* fix test
* fix consensus rule validation
* remove panics
* Delete a TODO
Co-authored-by: teor <teor@riseup.net>
* Start network before verifiers
This makes the Groth16 download task start as late as possible.
* Explain why the Groth16 download must happen first
* Speed up Zebra shutdown: skip waiting for the tokio runtime
* Create a `LockTime::unlocked` helper constructor
Returns a `LockTime` that is unlocked at the genesis block.
* Return `Option<LockTime>` from `lock_time` method
Prepare to return `None` for when a transaction has its lock time
disabled.
* Return `None` instead of zero `LockTime`
Because a zero lock time means that the transaction was unlocked at the
genesis block, so it was never actually locked.
* Rephrase zero lock time check comment
Clarify that the check is not redundant, and is necessary for the
genesis transaction.
Co-authored-by: teor <teor@riseup.net>
* Add a `transparent::Input::sequence` getter method
Retrieve a transparent input's sequence number.
* Check if lock time is enabled by a sequence number
Validate the consensus rule that the lock time is only enabled if at
least one transparent input has a value different from `u32::MAX` as its
sequence number.
* Add more Zcash specific details to comment
Explain the Zcash specific lock time behaviors.
Co-authored-by: teor <teor@riseup.net>
* Add `time` field to `Request::Block` variant
The block time to use to check if the transaction was unlocked and
allowed to be included in the block.
* Add `Request::block_time` getter
Returns the block time for the block that owns the transaction being
validated or the current time plus a tolerance for mempool transactions.
* Validate transaction lock times
If they are enabled by a transaction's transparent input sequence
numbers, make sure that they are in the past.
* Add comments with consensus rule parts
Make it easier to map what part of the consensus rule each match arm is
responsible for.
Co-authored-by: teor <teor@riseup.net>
* add testnet test blocks around nu5
* validate coinbase expiration height
* change const name and doc
Co-authored-by: teor <teor@riseup.net>
* change commit location
Co-authored-by: teor <teor@riseup.net>
* use pre Nu5 rules when there is no activation height
* add sapling final root to nu5 test vectors
* fix tests
Co-authored-by: teor <teor@riseup.net>
* Replace Zcash parameters crates with pre-downloaded local parameter files
* Download Zcash parameters using the `zcashd` script in CI and Docker
* Add a zcash_proofs dependency to zebra-consensus
* Download Sapling parameters using zcash_proofs, rather than fetch-params.sh
* Add a new `zebrad download` subcommand
This command isn't required for nomrmal usage.
But it's useful when testing, or launching multiple Zebra instances.
* Use `zebrad download` in CI to pre-download parameters
* Log a helpful hint if downloading fails
* Allow some duplicate dependencies currently hidden by orchard
* Spawn a separate task to download Groth16 parameters
* Run the parameter download with code coverage
This avoids re-compining Zebra with and without coverage.
* Update Cargo.lock after rebase
* Try to pass `download` as an argument to `zebrad` in coverage CI
* Fix copy and paste comment typos
* Add path and download examples, like zcash_proofs
* Download params in CI just like zcash_proofs does
* Delete a redundant build step
* Implement graceful shutdown for zebrad start
* Send coverage summary to /dev/null when getting the params path
* Use the correct parameters path and download commands in CI
* Explain pre-downloads
* Avoid calling params_folder twice
* Rename parameter types and methods for consistency
```sh
fastmod SaplingParams SaplingParameters zebra*
fastmod Groth16Params Groth16Parameters zebra*
fastmod PARAMS GROTH16_PARAMETERS zebra*
fastmod params_folder directory zebra*
```
And a manual variable name tweak.
* rustfmt
* Remove a redundant coverage step
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* First pass at async Halo2 verification service
Stubs out a batch verifier for the future.
The dependencies for orchard, halo2, librustzcash, zcash_primitives, have
not been resolved.
* Halo2 verifier service and test
* Remove redundant conversion
* Test async halo2 verifier service with pre-computed Orchard shielded data test vectors
* Fix typo
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Assert future result is_ok() in Halo2 verifier test
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Shorten tower::Service trait constraints for Halo2 verifier tests
* Remove commented out trait constraints
* .expect() vs .unwrap() to parse orchard::redpallas::VerificationKey
* Use .to_vec() for some test vectors
* Fix self-referential Display impl
* Fix deps
* Distinguish orchard vs zebra_chain::orchard imports
* Add test that halo2 verifier fails with malformed proof inputs
* Use thiserror for Halo2Error
* Use ZcashFoundation/orchard instead of dconnolly/orchard
* Add a link to the issue to remove the zfnd fork of orchard crate
* Update zebra-consensus/Cargo.toml
Co-authored-by: teor <teor@riseup.net>
* Add note
* Move artificial Orchard shielded data test vectors to zebra-test
* Align brackets
* Tidy some trait constraints and debug statements
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* Upgrade aes and fpe
* Upgrade bellman, bls12_381, jubjub to latest
* Upgrade x25519-dalek to 1.2.0 and curve25519-dalek to 3.2.0 in the Cargo.lock
* Skip outdated hdrhistogram rather than its dependencies
Co-authored-by: teor <teor@riseup.net>
* Cleanup a function that calls zcash_script
* Remove zebra_test::prelude macros that conflict with the Rust prelude
* Add sigops count support to zebra-script
* Check MAX_BLOCK_SIGOPS in the block verifier
* Test MAX_BLOCK_SIGOPS on generated and historic blocks
* Add SAFETY comments for all unsafe zebra-script code
* Explain where the consensus rule comes from
* Remove unused pretty_assertions dependency
* Allow large test block generation functions with the proptest-impl feature
* Replace `as` with `try_into` for integer conversions in unsafe code
* Expand SAFETY comments
* validate funding stream addresses
* simplify a bit funder stream address check
* add integer division code comment
* document constant
* replace some unwraps
* fix some doc comments
* check at least one output has calculated address and amount
* create a convinient storage for funding stream addresses
* replace some unwraps
* docs: change `7.7` protocol sections to `7.8`
* change errors text
* change function name
* refactor `FundingStreamReceiver::receivers()`
* refactor FUNDING_STREAM_ADDRESSES
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* remove a `clone()`
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix consensus rule check
* use a constant for testnet first halving height
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* validate funding stream amounts in the coinbase
* clippy
* use `i64::from()` and remove `number()` method from `Amount`
* move tests to their own file
* refactor the funding stream check
* use `Amount`s in funding streams calculation
* remove unused import
* add import to tests
* expand test vectors
* add notes to `funding_stream_values()`
* Update `tower` to version `0.4.9`
Update to latest version to add support for Tokio version 1.
* Replace usage of `ServiceExt::ready_and`
It was deprecated in favor of `ServiceExt::ready`.
* Update Tokio dependency to version `1.13.0`
This will break the build because the code isn't ready for the update,
but future commits will fix the issues.
* Replace import of `tokio::stream::StreamExt`
Use `futures::stream::StreamExt` instead, because newer versions of
Tokio don't have the `stream` feature.
* Use `IntervalStream` in `zebra-network`
In newer versions of Tokio `Interval` doesn't implement `Stream`, so the
wrapper types from `tokio-stream` have to be used instead.
* Use `IntervalStream` in `inventory_registry`
In newer versions of Tokio the `Interval` type doesn't implement
`Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used
instead.
* Use `BroadcastStream` in `inventory_registry`
In newer versions of Tokio `broadcast::Receiver` doesn't implement
`Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This
also requires changing the error type that is used.
* Handle `Semaphore::acquire` error in `tower-batch`
Newer versions of Tokio can return an error if the semaphore is closed.
This shouldn't happen in `tower-batch` because the semaphore is never
closed.
* Handle `Semaphore::acquire` error in `zebrad` test
On newer versions of Tokio `Semaphore::acquire` can return an error if
the semaphore is closed. This shouldn't happen in the test because the
semaphore is never closed.
* Update some `zebra-network` dependencies
Use versions compatible with Tokio version 1.
* Upgrade Hyper to version 0.14
Use a version that supports Tokio version 1.
* Update `metrics` dependency to version 0.17
And also update the `metrics-exporter-prometheus` to version 0.6.1.
These updates are to make sure Tokio 1 is supported.
* Use `f64` as the histogram data type
`u64` isn't supported as the histogram data type in newer versions of
`metrics`.
* Update the initialization of the metrics component
Make it compatible with the new version of `metrics`.
* Simplify build version counter
Remove all constants and use the new `metrics::incement_counter!` macro.
* Change metrics output line to match on
The snapshot string isn't included in the newer version of
`metrics-exporter-prometheus`.
* Update `sentry` to version 0.23.0
Use a version compatible with Tokio version 1.
* Remove usage of `TracingIntegration`
This seems to not be available from `sentry-tracing` anymore, so it
needs to be replaced.
* Add sentry layer to tracing initialization
This seems like the replacement for `TracingIntegration`.
* Remove unnecessary conversion
Suggested by a Clippy lint.
* Update Cargo lock file
Apply all of the updates to dependencies.
* Ban duplicate tokio dependencies
Also ban git sources for tokio dependencies.
* Stop allowing sentry-tracing git repository in `deny.toml`
* Allow remaining duplicates after the tokio upgrade
* Use C: drive for CI build output on Windows
GitHub Actions uses a Windows image with two disk drives, and the
default D: drive is smaller than the C: drive. Zebra currently uses a
lot of space to build, so it has to use the C: drive to avoid CI build
failures because of insufficient space.
Co-authored-by: teor <teor@riseup.net>
* Reorder imports to follow convention
Place the imports from `std` at the top.
* Add transaction errors for double spends
Add a variant for each pool. They represent a double spend inside a
transaction.
* Add `check::spend_conflicts` implementation
Checks if a transaction has spend conflicts, i.e., if a transaction
spends a UTXO more than once or if it reveals a nullifier more than
once.
* Reject transactions with internal spend conflicts
The transaction verifier should reject transactions that spend the same
transparent UTXO or that reveal the same nullifier.
* Add transparent spend consensus rule
Add it to the documentation to help with understanding and auditing it.
Co-authored-by: teor <teor@riseup.net>
* Use different nullifiers by default
Don't use the same nullifier twice when mocking a
`sprout::JoinSplitData` because it will lead to an invalid transaction.
* Test transactions with repeated spend outpoints
Since that represents a spend conflict, they should be rejected.
* Test duplicate nullifiers in joinsplit
Check if a mock transaction with a joinsplit that reveals the same
nullifier twice is rejected.
* Test duplicate nullifiers across joinsplits
Check if a duplicate nullifier in two different joinsplits in the same
transaction is rejected.
* Test V4 transaction with duplicate Sapling spend
Check if a V4 transaction that has a duplicate Sapling spend is
rejected.
* Test V5 transaction with duplicate Sapling spend
Check if a V5 transaction that has a duplicate Sapling spend is
rejected.
* Test V5 transaction with duplicate Orchard actions
Check if a V5 transaction that has duplicate Orchard actions is rejected
by the transaction verifier.
Co-authored-by: teor <teor@riseup.net>
* Increment the crates that have new commits since the last version
* Increment the crates that depend on crates that have changed
* Increment the version of `zebra-script`
* Use the `zebrad` version in the `zebra-network` user agent string
* Use the `v1.0.0-alpha.19` git tag in `README.md`
* Copy the draft changelog into `CHANGELOG.md`
* Delete bumps
* Update CHANGELOG.md
Co-authored-by: teor <teor@riseup.net>
* Add newly merged PRs
Co-authored-by: teor <teor@riseup.net>
* Ignore AlreadyInChain error in the syncer
* Split Cancelled errors; add them to should_restart_sync exceptions
* Also filter 'block is already comitted'; try to detect a wrong downcast
* Create a new VerifiedUnminedTx containing the miner fee
* Use VerifiedUnminedTx in mempool verification responses
And do a bunch of other cleanups.
* Use VerifiedUnminedTx in mempool download and verifier
* Use VerifiedUnminedTx in mempool storage and verified set
* Impl Display for VerifiedUnminedTx, and some convenience methods
* Use VerifiedUnminedTx in existing tests
* Get the transaction fee from utxos
* Return the transaction fee from the verifier
* Avoid calculating the fee for coinbase transactions
Coinbase transactions don't have fees. In case of a coinbase transaction, the
verifier returns a zero fee.
* Update the result obtained by `Downloads`
* Send spent UTXOs from the script verifier to the transaction verifier
* Add temporary assertions for testing spent UTXO sending
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Marek <mail@marek.onl>
* Check for state errors before updating metrics
Previously, the metrics would be updated for some rejected blocks.
* Clarify and expand block verification metrics
Rename checkpoint-specific metrics to clarify their purpose.
Add metrics for:
- finalized blocks on disk
- blocks verified using the full block verifier
(this metric was previously incorrectly called `zcash_chain_verified_block_height`)
* Update dashboard metric names
Also:
- add some extra block height metrics
- fix a dashboard name
* Add exact block heights to Grafana dashboards
* Add a missing comment
* grafana: use 0 decimals for metrics
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* grafana: show the entire height instead of abbreviated
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* grafana: show the entire height instead of abbreviated
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Fix typo in metric name
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Move height gauges to the state, so they are correct
If we update height gauges in futures, they can execute out of order,
so the metrics can be incorrect.
Instead:
- move the height gauges to the state, and update them based on the best tip
- move the verified block counts to the state
- continue to include all verified blocks on all non-finalized chains
(not just the best chain)
* Show exact checkpoint heights in the dashboard
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Remove unused mempool storage errors
Preparation for ticket #2819.
Removing these errors means that we don't have to decide
which type of transaction ID match we want for them.
* Remove unused mempool errors, and deduplicate storage errors
* rustfmt
* Update versions for zebra v1.0.0-alpha.18 release
* WIP: Initial PR list
* Remove uninteresting version bumps from CHANGELOG
* Categorise and group PRs in CHANGELOG, removing uninteresting PRs
* Further refine and categorise changelog entries
* Fix tag url
* Final changes to CHANGELOG
* Add a changelog description
* Spacing
* Clarify and fix changelog PR descriptions
* Add PRs that are about to be merged
* More slight clarifications
* Spacing
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Check return value of zcash_script_new_precomputed_tx
* Set the NU5 testnet activation height to 1_590_000
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Update Nu5 constants to new values
* Update ZIP-244 test vectors for new branch ID
* Squashed commit of the following:
commit bdb120a249
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 11:54:01 2021 -0400
Use pallas::Base::from_str_vartime() in sinsemilla tests
commit e99fa49258
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 11:45:24 2021 -0400
Compiles
commit a520018114
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 10:15:17 2021 -0400
Incomplete upgrade of deps
* Squashed commit of the following:
commit 8d1b76ec5626517817c3a4d9f3950acc90a359df
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 04:02:26 2021 +0000
Update `zcash_script` to support V5 transactions
Use a newer version of `zcash_script` that has been updated to support
V5 transactions.
commit 371233628ae61e0c25d6ba8f31d9dba42823becb
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 03:06:20 2021 +0000
Update Zcash dependencies
Update some Zcash crates:
- `halo2`
- `incrementalmerkletree' (patch version)
- `orchard` (patch version)
- `zcash_history` (patch version)
- `zcash_note_encryption` (patch version)
- `zcash_primitives` (patch version)
And also update the `group` dependency so that the code remains
compatible.
commit de5cf1ec40c3fc08670fc971cdf3e65e13d9f4c7
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 03:04:13 2021 +0000
Update error message assertion
Use the updated message for the expected error variant.
* Update `zcash_script` to support V5 transactions
Use a newer version of `zcash_script` that has been updated to support
V5 transactions.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
* Rename type parameter to be more explicit
Replace the single letter with a proper name.
* Remove imports for `Request` and `Response`
The type names will conflict with the ones for the mempool service.
* Attach `Mempool` service to the `Crawler`
Add a field to the `Crawler` type to store a way to access the `Mempool`
service.
* Forward crawled transactions to downloader
The crawled transactions are now sent to the transaction downloader and
verifier, to be included in the mempool.
* Derive `Eq` and `PartialEq` for `mempool::Request`
Make it simpler to use the `MockService::expect_request` method.
* Test if crawled transactions are downloaded
Create some dummy crawled transactions, and let the crawler discover
them. Then check if they are forwarded to the mempool to be downloaded
and verified.
* Don't send empty transaction ID list to downloader
Ignore response from peers that don't provide any crawled transactions.
* Log errors when forwarding crawled transaction IDs
Calling the Mempool service should not fail, so if an error happens it
should be visible. However, errors when downloading individual
transactions can happen from time to time, so there's no need for them
to be very visible.
* Document existing `mempool::Crawler` test
Provide some depth as to what the test expect from the crawler's
behavior.
* Refactor to create `setup_crawler` helper function
Make it easier to reuse the common test setup code.
* Simplify code to expect requests
Now that `zebra_network::Request` implement `Eq`, the call can be
simplified into `expect_request`.
* Refactor to create `respond_with_transaction_ids`
A helper function that checks for a network crawl request and responds
with the given list of crawled transaction IDs.
* Refactor to create `crawler_iterator` helper
A function to intercept and respond to the fanned-out requests sent
during a single crawl iteration.
* Refactor to create `respond_to_queue_request`
Reduce the repeated code necessary to intercept and reply to a request
for queuing transactions to be downloaded.
* Add `respond_to_queue_request_with_error` helper
Intercepts a mempool request to queue transactions to be downloaded, and
responds with an error, simulating an internal problem in the mempool
service implementation.
* Derive `Arbitrary` for `NetworkUpgrade`
This is required for deriving `Arbitrary` for some error types.
* Derive `Arbitrary` for `TransactionError`
Allow random transaction errors to be generated for property tests.
* Derive `Arbitrary` for `MempoolError`
Allow random Mempool errors to be generated for property tests.
* Test if errors don't stop the mempool crawler
The crawler should be robust enough to continue operating even if the
mempool service fails to download transactions or even fails to handle
requests to enqueue transactions.
* Reduce the log level for download errors
They should happen regularly, so there's no need to have them with a
high visibility level.
Co-authored-by: teor <teor@riseup.net>
* Stop crawler if service stops
If `Mempool::poll_ready` returns an error, it's because the mempool
service has stopped and can't handle any requests, so the crawler should
stop as well.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Allow deliberate instances of the new nightly clippy::derivable_impls lint
We want our config defaults to be explicit.
Not so sure about the application defaults, but they also contain a config.
* Also allow unknown lint names
Stable doesn't know about this lint, but nightly does.
We don't check the authorizing data hash until checkpoint blocks reach the state.
So signatures, proofs, or scripts could be different,
even if the block hash is the same.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Stop precalculating transaction hashes twice during checkpointing
* Refactor a complex type using a new `RequestBlock` type
* Comment formatting
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Store precalculated transactions in an `Arc`
Transaction `Hash`es are 32 bytes,
and the minimun transparent transaction size is 54 bytes.
So a full 2MB block can create 1.1MB of transaction hashes.
We use an `Arc` to avoid repeatedly cloning that much data.
* Remove the unused `Block` from `ChainTipBlock`
This drops the block as soon as it isn't needed any more.
Previously, it would stick around until every `ChainTipReceiver`
dropped their `ChainTipBlock`, even if they didn't use the `Block`
at all.
* Add `Arc<Transaction>` conversions for Transaction IDs
* Use UnminedTxId as the transaction verifier response type
* Use UnminedTx in transaction verifier mempool requests
* Refactor is_mempool into a transaction verifier request method
* Order derives in alphabetical order
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Return a transaction verifier from `zebra_consensus::init`
This verifier is temporarily created separately from the block verifier's
transaction verifier.
* Return the same transaction verifier used by the block verifier
* Clarify that the mempool verifier is the transaction verifier
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add validation of ZIP-221 and ZIP-244 commitments
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Add auth commitment check in the finalized state
* Reset the verifier when comitting to state fails
* Add explanation comment
* Add test with fake activation heights
* Add generate_valid_commitments flag
* Enable fake activation heights using env var instead of feature
* Also update initial_tip_hash; refactor into progress_from_tip()
* Improve comments
* Add fake activation heights test to CI
* Fix bug that caused commitment trees to not match when generating partial arbitrary chains
* Add ChainHistoryBlockTxAuthCommitmentHash::from_commitments to organize and deduplicate code
* Remove stale comment, improve readability
* Allow overriding with PROPTEST_CASES
* partial_chain_strategy(): don't update note commitment trees when not needed; add comment
Co-authored-by: teor <teor@riseup.net>
* add value pools to the database
* remove redundant genesis block check
* use update_with_chain_value_pool_change()
* remove constrains
* remove height from the database
* remove calls to chain_value_pool_change
* clippy
* use the "correct" value balances
* bump the database format
* remove everything that is not finalized state
* clippy
* rustfmt
* use all spent utxos
* add new_outputs utxos to all_utxos_spent_by_block
* remove panic
* add finalized state value pool test
* clippy
* clippy 2
* move import
* fix import
* rustfmt
Co-authored-by: teor <teor@riseup.net>
* Make Amount arithmetic more generic
To modify generated amounts, we need some extra operations on `Amount`.
We also need to extend existing operations to both `NonNegative` and
`NegativeAllowed` amounts.
* Add a constrain method for ValueBalance
* Derive Eq for ValueBalance
* impl Neg for ValueBalance
* Make some Amount arithmetic expectations explicit
* Explain why we use i128 for multiplication
And expand the overflow error details.
* Expand Amount::sum error details
* Make amount::Error field order consistent
* Rename an amount::Error variant to Constraint, so it's clearer
* Add specific pool variants to ValueBalanceError
* Update coinbase remaining value consensus rule comment
This consensus rule was updated recently to include coinbase transactions,
but Zebra doesn't check block subsidy or miner fees yet.
* Add test methods for modifying transparent values and shielded value balances
* Temporarily set values and value balances to zero in proptests
In both generated chains and proptests that construct their own transactions.
Using zero values reduces value calculation and value check test coverage.
A future change will use non-zero values, and fix them so the check passes.
* Add extra fields to remaining transaction value errors
* Swap the transparent value balance sign to match shielded value balances
This makes the signs of all the chain value pools consistent.
* Use a NonNegative constraint for transparent values
This fix:
* makes the type signature match the consensus rules
* avoids having to write code to handle negative values
* Allocate total generated transaction input value to outputs
If there isn't enough input value for an output, set it to zero.
Temporarily reduce all generated values to avoid overflow.
(We'll remove this workaround when we calculate chain value balances.)
* Consistently use ValueBalanceError for ValueBalances
* Make the value balance signs match the spec
And rename and document methods so their signs are clearer.
* Convert amount::Errors to specific pool ValueBalanceErrors
* Move some error changes to the next PR
* Add extra info to remaining transaction value errors (#2585)
* Distinguish between overflow and negative remaining transaction value errors
And make some error types cloneable.
* Add methods for updating chain value pools (#2586)
* Move amount::test to amount::tests:vectors
* Make ValueBalance traits more consistent with Amount
- implement Add and Sub variants with Result and Assign
- derive Hash
* Clarify some comments and expects
* Create ValueBalance update methods for blocks and transactions
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Validate transparent coinbase output maturity and shielding
- Add a CoinbaseSpendRestriction enum and Transaction method
- Validate transparent coinbase spends in non-finalized chains
* Don't use genesis created UTXOs for spends in generated block chains
* Refactor out a new_transaction_ordered_outputs function
* Add Transaction::outputs_mut for tests
* Generate valid transparent spends in arbitrary block chains
* When generating blocks, fixup the block contents, then the block hash
* Test that generated chains contain at least one transparent spend
* Make generated chains long enough for reliable tests
* Add transparent and shielded input and output methods to Transaction
* Split chain generation into 3 functions
* Test that unshielded and immature transparent coinbase spends fail
* Comment punctuation
* Clarify a comment
* Clarify probability calculation
* Test that shielded mature coinbase output spends succeed
* Create a `zebra_state::init_test` helper function
This function will be used as a replacement for `zebra_state::init`
inside tests. It's a simpler alternative because it can ignore any
details that aren't relevant for tests.
* Use `init_test` inside `zebra-state` tests
Update usages of `init` to use `init_test` instead, which simplifies
most cases.
* Use `zebra_state::init_test` in `zebra-consensus`
Replace usages of `zebra_state::init` with the new helper function. This
simplifies the code a bit.
* Add an OrderedUtxo type for validation of spends within a block
This change allows us to check that transparent spends use outputs from
earlier in their block. (But we don't actually do that check yet.)
We need to keep the order of UTXOs when we're contextually verifying
each new block that is added to a chain. But the block order is
irrelevant for UTXOs stored in the state.
* Take ownership in utxos_from_ordered_utxos
* Delete a confusing comment
* Add panic message to `unimplemented!`
So that it is clear why the panic happened upon initial inspection. Also
include a reference to the mempool epic, so that it's easier to find the
issue that tracks the implementation of the missing code.
* Add panic message that references a tracking issue
Make it easy to find the relevant issue if the panic occurs.
* Remove incomplete and currently unnecessary code
The current implementation works, the commented out code was just a
previous improvement idea, which is now tracked by issue #2473.
* Always use librustzcash for sighash and remove old sighash code
Also added ZIP-143 test vectors
* Remove librustzcash_sighash test that is no longer needed
* Make `verify_sapling_shielded_data` more generic
Prepare to support V5 transactions which have a shared anchor.
* Verify Sapling shielded data in V5 transactions
Call the `verify_sapling_shielded_data` method and add the respective
asynchronous checks to the set of V5 checks.
* Fix expect message in V4 transaction test
It was using the same message as the previous test, even though the test
searches with different criteria.
* Test V5 transaction with Sapling spends
Create a fake V5 transaction that has Sapling spends and check that the
verifier accepts the transaction.
* Ignore rejected V5 transaction test for now
Because now it needs the `sighash` implementation for V5 to be ready.
* Reference V5 `sighash` PR in comment
So that it is easier to check if it's possible to remove the
`should_panic` or not.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove `sapling shielded pool` TODO
V5 transactions now have Sapling shielded pool properly validated.
* Link to some extra issues in TODO comment
Some other issues are also necessary for full V5 validation.
* Add a TODO in the main code to fix the tests
Some tests are blocked due to missing features required for full V5
validation. Once those features are implemented, they should be updated
to remove the `#[should_panic]` attribute so that they actually run and
check the code correctly.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Refactor `AsyncChecks` into a proper type
Add some helper methods to it so that checks can be added by
daisy-chaining calls. Also move the code to wait for the checks to
finish into the new type.
* Refactor inclusion of individual Sapling checks
Use `oneshot` instead of `ready_and` so that the method becomes
synchronous.
* Make V4 and V5 verification methods synchronous
There is no longer a need to wait for any internal service to be ready,
since now that's always done as part of an asynchronous check included
in the returned set of checks.
* add disabled sprout pool check
* change method name
* change error name
* fix typo
* make the success test case in other tx than the coinbase
* use new `height` method instead of deriving `PartialOrd` in `NetworkUpgrade`
* move check of network upgrade into function, rename, docs
* increase test coverage
* fix comment
* Update versions for zebra v1.0.0-alpha.12 release
* Update Cargo.lock
* Update release checklist with latest version changes to help keep track for future releases
* Remove reference to the fact that tower-fallback was not updated
* Refactor to create `verify_sapling_shielded_data`
Move the code to verify Sapling shielded data into a new helper method
that returns `AsyncChecks`.
* Test verifying a Sapling transaction with spends
Use the test vectors to find a transaction that has Sapling spends and
test if it the verifier considers it valid.
* Create a helper method to list test transactions
Transforms the block test vectors into a list of transactions and block
heights for each transaction.
* Use new helper function in V4 Sapling spend test
Also use the block height for that transaction as specified in the test
vector.
* Test V4 tx. with Sapling outputs but no spends
Find a transaction V4 vector that has Sapling outputs but no spends, and
check that the verifier accepts it.
* Add a `zebra_test::RUNTIME` shared runtime
Create a lazily instantiated Tokio runtime that can be shared by tests.
* Split tests that require a shared runtime
Split two tests that were previously in one because of the need to share
a single Tokio runtime. With the `zebra_test::RUNTIME`, they can now
share the runtime without having to be a single test.
* Stop trying to verify coinbase inputs using the script verifier
And create tests to catch similar bugs earier.
* Use Testnet in NU5 tests that temporarily should_panic
We've marked these tests as should_panic until there is a NU5 activation
height. But Testnet will have an activation height first, so we should
prefer it in the tests. (Or use both networks.)
Block transactions already had a height, but mempool transactions didn't.
This PR adds a height to mempool transactions, and deletes redundant and
unused fields. It also adds an accessor method for that height.
* Refactor to create `verify_sprout_shielded_data`
Move the join split verification code into a new
`verify_sprout_shielded_data` helper method that returns an
`AsyncChecks` set.
* Test if signed V4 tx. join splits are accepted
Create a fake V4 transaction with a dummy join split, and sign it
appropriately. Check if the transaction verifier accepts the
transaction.
* Test if unsigned V4 tx. joinsplit data is rejected
Create a fake V4 transaction with a dummy join split. Do NOT sign this
transaction's join split data, and check that the verifier rejects the
transaction.
* Join tests to share Tokio runtime
Otherwise one of the tests might fail incorrectly because of a
limitation in the test environment. `Batch` services spawn a task in the
Tokio runtime, but separate tests can have separate runtimes, so sharing
a `Batch` service can lead to the worker task only being available for
one of the tests.
* Describe how a ZIP-213 rule is implemented in the transaction verifier
* Move the only coinbase-specific check outside the ZIP-213 block
This change isn't required to implement the ZIP-213 rule, but it makes
it easier to identify the specific checks for coinbase transactions.
* Add a note about coinbase in the mempool
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Update multiple crates to ensure bitvec 0.22.3 is being used and avoid package conflicts
* Add documentation to zebra-chain::sapling to indicate that ZIP-216 rules are enforced by jubjub
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Add missing documentation
Document methods to describe what they do and why.
* Create an `AsyncChecks` type alias
Make it simpler to write the `FuturesUnordered` type with boxed futures.
This will also end up being used more when refactoring to return the
checks so that the `call` method can wait on them.
* Create `verify_transparent_inputs_and_outputs`
Refactors the verification of the transparent inputs and outputs into a
separate method.
* Refactor transparent checks to use `call_all`
Instead of pushing the verifications into a stream of unordered futures,
use the `ServiceExt::call_all` method to build an equivalent stream
after building a stream of requests.
* Replace `CallAll` with `FuturesUnordered`
Make it more consistent with the rest of the code, and make sure that
the `len()` method is available to use for tracing.
Co-authored-by: teor <teor@riseup.net>
* Refactor to move wait for checks into a new method
Allow the code snipped to be reused by other transaction
version-specific check methods.
* Verify transparent inputs in V5 transactions
Use the script verifier to check the transparent inputs in a V5
transaction.
* Check `has_inputs_and_outputs` for all versions
Check if a transaction has inputs and outputs, independently of the
transaction version.
* Wait for checks in `call` method
Refactor to move the repeated code into the `call` method. Now the
validation methods return the set of asynchronous checks to wait for.
* Add helper function to mock transparent transfers
Creates a fake source UTXO, and then the input and output that represent
spending that UTXO. The initial UTXO can be configured to have a script
that either accepts or rejects any spend attempt.
* Test if transparent V4 transaction is accepted
Create a fake V4 transaction that includes a fake transparent transfer
of funds. The transfer uses a script to allow any UTXO to spend it.
* Test transaction V4 rejection based on script
Create a fake transparent transfer where the source UTXO has a script
that rejects spending. The script verifier should not accept this
transaction.
* Test if transparent V5 transaction is accepted
Create a mock V5 transaction that includes a transparent transfer of
funds. The transaction should be accepted by the verifier.
* Test transaction V5 rejection based on script
Create a fake transparent transfer where the source UTXO has a script
that rejects spending. The script verifier should not accept this
transaction.
* Update `Request::upgrade` getter documentation
Simplify it so that it won't become updated when #1683 is fixed.
Co-authored-by: teor <teor@riseup.net>
* move network_upgrade check into zebra-chain
* fix the errors
* rename function
* typo fix
* rename the check function
* make changes from last code review
* Add a `Transaction::version` getter
Returns the version of the transaction as a `u32`.
* Add `Transaction::is_overwintered` helper method
Returns if the `fOverwintered` flag should be set for the transaction's
version.
* Use new helpers to serialize transaction version
Reduce the repeated code and make it less error-prone with future
changes.
* Add getter methods to `transaction::Request` type
Refactor to move the type deconstruction code into the `Request` type.
The main objective is to make it easier to split the call handler into
methods that receive the request directly.
* Refactor to create `verify_v4_transaction` helper
Split the code specific to V4 transactions into a separate helper
method.
* Create `verify_v5_transaction` helper method
Prepare a separate method to have the validation code.
* Add `UnsupportedByNetworkUpgrade` error variant
An error for when a transaction's version isn't supported by the network
upgrade of the block it's included or for the current network upgrade if
the transaction is for the mempool.
* Verify a V5 transaction's network upgrade
For now, only NU5 supports V5 transactions.
* Test that V5 transaction is rejected on Canopy
Create a fake V5 transaction and try to verify it using a block height
from Canopy's activation. The verifier should reject the transaction
with an error saying that the network upgrade does not support that
transaction version.
* Test if V5 tx. is accepted after NU5 activation
Create a fake V5 transaction and pretend it is placed in a block that
has a height after the NU5 activation. The test should succeed, but
since the NU5 activation height has not been specified yet (neither for
the testnet nor the mainnet), for now this test is marked as
`should_panic`.
* Add `TODO` comment to the code
Add more detail to what's left to do, and link to the appropriate PRs.
* Use `u32` to store transaction version
Use a type consistent with how the version is specified.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* Add a `at_least_one!` macro for testing
Similar to the `vec!` macro, but doesn't allow creating an empty list.
* Test if `has_inputs_and_outputs` considers actions
Create a dummy transaction with no inputs and no outputs, and add a
dummy Orchard action to it. The `check::has_inputs_and_outputs`
should succeed, because the consensus rule considers having Orchard
actions as having inputs and/or outputs.
* Refactor to create helper function
Move the code to create a fake Orchard shielded data instance to a
helper function in `zebra_chain::transaction::arbitrary`, so that other
tests can also use it.
* Test coinbase V5 transaction with enable spends
A V5 coinbase transaction that has Orchard shielded data MUST NOT have
the enable spends flag set.
* Test if coinbase without enable spends is valid
A coinbase transaction with Orchard shielded data and without the enable
spends flag set should be valid.
* Add a security comment about the `at_least_one!` macro
This macro must not be used outside tests, because it allows memory denial
of service.
Co-authored-by: teor <teor@riseup.net>
* Implement `PartialEq<i64>` for `Amount`
Allows to compare an `Amount` instance directly to an integer.
* Add `SerializationError::BadTransactionBalance`
Error variant representing deserialization of a transaction that doesn't
conform to the Sapling consensus rule where the balance MUST be zero if
there aren't any shielded spends and outputs.
* Validate consensus rule when deserializing
Return an error if the deserialized V4 transaction has a non-zero value
balance but doesn't have any Sapling shielded spends nor outputs.
* Add consensus rule link to field documentation
Describe how the consensus rule is validated structurally by
`ShieldedData`.
* Clarify that `value_balance` is zero
Make the description more concise and objective.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Update field documentation
Include information about how the consensus rule is guaranteed during
serialization.
Co-authored-by: teor <teor@riseup.net>
* Remove `check::sapling_balances_match` function
The check is redundant because the respective consensus rule is
validated structurally by `ShieldedData`.
* Test deserialization of invalid V4 transaction
A transaction with no Sapling shielded spends and no outputs but with a
non-zero balance value should fail to deserialize.
* Change least-significant byte of the value balance
State how the byte index is calculated, and change the least
significant-byte to be non-zero.
Co-authored-by: teor <teor@riseup.net>
* Standardise lints across Zebra crates, and add missing docs
The only remaining module with missing docs is `zebra_test::command`
* Todo -> TODO
* Clarify what a transcript ErrorChecker does
Also change `Error` -> `BoxError`
* TransError -> ExpectedTranscriptError
* Output Descriptions -> Output descriptions
* Add a `Transaction::orchard_shielded_data` getter
Allows accessing the Orchard shielded data if it is present in the
transaction, regardless of the transaction version.
* Refactor `orchard_nullifiers` to use new getter
Allows making the method more concise.
* Add `CoinbaseHasEnableSpendsOrchard` error variant
Used when the validation rule is not met.
* Implement `enableSpendsOrchard` in coinbase check
The flag must not be set for the coinbase transaction.
* Refactor `Transaction::orchard_*` getters
Use the fact that `Option<T>` implements `Iterator<T>` to simplify the
code and remove the need for boxing the iterators.
Co-authored-by: teor <teor@riseup.net>
* Make sure the Canopy activation block is a finalized checkpoint block
This enables ZIP-221 chain history from Canopy activation onwards.
* Clarify that the mandatory checkpoint test includes Canopy activation
The test was correct, but the docs and assertion message did not include activation.
* Document that the mandatory checkpoint includes Canopy activation
Co-authored-by: teor <teor@riseup.net>
* Fix documentation comment
Was missing a slash to become documentation.
* Add documentation link to type reference
Just to help navigation a bit.
* Implement `Transaction::orchard_actions()` getter
Returns an iterator to iterator over the actions in the Orchard shielded
data (if there is one, otherwise it returns an empty iterator).
* Add V5 support for `has_inputs_and_outputs`
Checks if the transaction has Orchard actions. If it does, it is
considered to have inputs and outputs.
* Refactor transaction test vectors
Make it easier to reuse the fake V5 transaction converter in other test
vectors.
* Move helper function to `zebra-chain` crate
Place it together with some other helper functions, including the one
that actually creates the fake V5 transaction.
* Test transaction with no inputs
`check::has_inputs_and_outputs` should return an error indicating that
the transaction has no inputs.
* Test transaction with no outputs
`check::has_inputs_and_outputs` should return an error indicating that
the transaction has no outputs.
* Note that transaction is fake in `expect` message
Should make the message easier to find, and also gives emphasis to the
fact that the transaction is a fake conversion to V5.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* validate sapling v5 tx
* Make itertools dependency optional
We only need itertools when the `proptest-impl` feature is enabled.
* Check if V4 and V5 coinbase transactions contain PrevOut transparent inputs
This is a bugfix on V4 transaction validation. The PrevOut consensus
rule was not explicitly stated in the Zcash spec until April 2021.
(But it was implied by Bitcoin, and partially implemented by Zebra.)
Also do the shielded sapling input check for V5 transactions.
* Add spec and orchard TODOs to has_inputs_and_outputs
Also make the variable names match the spec.
* Sort transaction functions to match v5 data order
* Simplify transaction input and output checks
Move counts or iterators into `Transaction` methods, so we can remove
duplicate code, and make the consensus rule logic clearer.
* Update sapling_balances_match for Transaction v5
- Quote from the spec
- Explain why the function is redunant for v5
- Rename the function so it's clear that it is sapling-specific
Co-authored-by: teor <teor@riseup.net>
* start refactoring transaction v4 for transaction v5
- move ShieldedData to sapling
- add AnchorVariant
- rename shielded_data to sapling_shielded data in V4
- move value_balance into ShieldedData
- update prop tests for new structure
* add AnchorVariant to Spend
- make anchor types available from sapling crate
- update serialize
* change shielded_balances_match() arguments
* change variable name anchor to shared_anchor in ShieldedData
* fix empty value balance serialization
* use AnchorV in shielded spends
* Rename anchor to per_spend_anchor
* Use nullifiers function directly in non-finalized state
* Use self.value_balance instead of passing it as an argument
* Add missing fields to ShieldedData PartialEq
* Derive Copy for tag types
* Add doc comments for ShieldedData refactor
* Implement a per-spend anchor compatibility iterator
Co-authored-by: teor <teor@riseup.net>
* Ed25519 async batch verification for JoinSplit signatures
We've been verifying JoinSplitSigs one-by-one pre-ZIP-215. Now as we're post-ZIP-215,
we can take advantage of the batch math to validate this signatures.
I would have pumped all the joinsplits in our MAINNET_BLOCKS test vectors but these
signatures are over the sighash, which needs the NU code to compute, and once we're
doing all that set up, we're basically doing transaction validation, so.
Resolves#1944
* Repoint to latest ed25519-zebra commit with note to point at 3.0 when released
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Zebra's latest alpha checkpoints on Canopy activation, continues our work on NU5, and fixes a security issue.
Some notable changes include:
## Added
- Log address book metrics when PeerSet or CandidateSet don't have many peers (#1906)
- Document test coverage workflow (#1919)
- Add a final job to CI, so we can easily require all the CI jobs to pass (#1927)
## Changed
- Zebra has moved its mandatory checkpoint from Sapling to Canopy (#1898, #1926)
- This is a breaking change for users that depend on the exact height of the mandatory checkpoint.
## Fixed
- tower-batch: wake waiting workers on close to avoid hangs (#1908)
- Assert that pre-Canopy blocks use checkpointing (#1909)
- Fix CI disk space usage by disabling incremental compilation in coverage builds (#1923)
## Security
- Stop relying on unchecked length fields when preallocating vectors (#1925)
* add transaction V5 stub
* add v5_strategy
* deduplicate version group ids
* Update comment for V5 transactions
* Add V5 transactions to non_finalized_state
Currently these are all `unimplemented!(...)`
* Fix struct matches
* Apply trivial panic message changes
* add zcash_deserialize for V5
* make all tx versions explicit in sprout and sapling nullifier functions
* match exhaustively in sprout and sapling nullifier functions
* fix matches in zebra-consensus
* fix NU5 strategy
* We're still deciding if v5 transactions support Sprout
Co-authored-by: teor <teor@riseup.net>
* change `poll_ready()` and `call()` of `ChainVerifier`
* add bound, move max_checkpoint_height
* add buffers to the checkpoint and block verifiers
And rename the chain verifier fields so `block` means `Arc<Block>`, and
`block_verifier` means `Buffer<BlockVerifier, ...>`.
* Fix the error types
* Use `ServiceExt::oneshot` in `ChainVerifier::call`
And:
* make the code look like the `main` branch as much as possible
* document the `poll_ready`/`call` invariant
* Use `ServiceExt::oneshot` in `chain::init`
Co-authored-by: teor <teor@riseup.net>
This PR is the first step in getting a groth16 proving system fully
integrated with the rest of zebra. This PR implements the initial async
API, but none of the actual batching logic necessary for our eventual
verifier design.
Once the batch verification API from bellman has been implemented we
will need to swap out the "Batch" type defined in this crate with the
new `batch::Verifier` defined in bellman.
* Bump versions where appropriate
Tested with cargo install --locked --path etc
* Remove fixed panics from 'Known Issues'
* Change to alpha release series in the README
Co-authored-by: teor <teor@riseup.net>
The clippy unknown lints attribute was deprecated in
nightly in rust-lang/rust#80524. The old lint name now produces a
warning.
Since we're using `allow(unknown_lints)` to suppress warnings, we need to
add the canonical name, so we can continue to build without warnings on
nightly.
But we also need to keep the old name, so we can continue to build
without warnings on stable.
And therefore, we also need to disable the "removed lints" warning,
otherwise we'll get warnings about the old name on nightly.
We'll need to keep this transitional clippy config until rustc 1.51 is
stable.
## Motivation
This PR is motivated by the regression identified in https://github.com/ZcashFoundation/zebra/issues/1349. That PR notes that the metrics stopped working for most of the crates other than `zebrad`.
## Solution
This PR resolves the regression by deduplicating the `metrics` crate dependency. During a recent change we upgraded the metrics version in `zebrad` and a couple other of our crates, but we never updated the dependencies in `zebra-state`, `zebra-consensus`, or `zebra-network`. This caused the metrics macros to attempt to retrieve the current metrics exporter through the wrong function. We would install the metrics exporter in `0.13`, but then attempt to look it up through the `0.12` crate, which contains a different instance of the metrics exporter static variable which is unset. Doing this causes the metrics macros to return `None` for the current exporter after which they just silently give up.
## Related Issues
closes https://github.com/ZcashFoundation/zebra/issues/1349
## Follow Up Work
I noticed we have quite a few duplicate dependencies in our tree. We might be able to save some compilation time by auditing those and deduplicating them as much as possible.
- https://github.com/ZcashFoundation/zebra/issues/1582
Co-authored-by: teor <teor@riseup.net>
* export new precompute api in zebra-script
* remove old API in favor of precompute API
* add multi use test cases and bump version
* update implementation to actually match henry's design
* Add safety comment for zebra-script
Co-authored-by: teor <teor@riseup.net>
* create readme for utils and checkpoints
* add link to checkpoint usage to book
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Previously we set the crate versions to 3.x, so that the major version was
aligned with the NU version. But we want to be able to make API changes
independently of the NU schedule.
Change the Merkle root validation logic to also check that a block does not
contain duplicate transactions. This check is redundant with later
double-spend checks, but is a useful defense-in-depth.
As a side effect of computing Merkle roots, we build a list of
transaction hashes. Instead of discarding these, add them to
PreparedBlock and FinalizedBlock so that they can be reused rather than
recomputed.
This commit adds Merkle root validation to:
1. the block verifier;
2. the checkpoint verifier.
In the first case, Bitcoin Merkle tree malleability has no effect,
because only a single Merkle tree in each malleablity set is valid (the
others have duplicate transactions).
In the second case, we need to check that the Merkle tree does not contain any
duplicate transactions.
Closes#1385Closes#906
UTXO requests during transaction input verification can time out because:
1. The block that creates the UTXO is queued for download or verify, but
it hasn't been committed yet. The creating block might spend UTXOs
that come from other recent blocks, so UTXO verification can depend on
a (non-contiguous) sequence of block verifications.
In this case, Zebra should wait for additional block download and
verify tasks to complete.
2. The block that creates the UTXO isn't queued for download. This can
happen because the block is gossiped block that's much higher than the
current tip, or because a peer sent the syncer a bad list of block
hashes.
In this case, Zebra should discard the timed out block, and restart
the sync.
We need to choose a timeout that balances these two cases, so we time
out after 180 seconds.
Assuming Zebra can download at least 1 MB per second, 180 seconds is
enough time to download a few hundred blocks. So Zebra should be able to
download and verify the next block before the UTXOs that it creates time
out. (Since Zebra has already verified all the blocks before the next
block, its UTXO requests should return immediately.)
Even if some peers time out downloads, a block can only be pending
download for 80 seconds (4 retries * 20 second timeout) before the
download fails. So the UTXO timeout doesn't need to be much larger than
this overall download timeout - because the download timeout will happen
first on slow networks.
Alternately, if the download for the creating block was never queued,
Zebra should timeout as soon as possible - so it can restart the sync
and download the creating block.
As a side-effect, a lower UTXO timeout also makes it slightly easier to
debug UTXO issues, because unsatisfiable queries fail faster.
The state service API says explicitly that AwaitUTXO requests should be coupled
with a timeout layer. I didn't add this when I was testing and fixing the UTXO
lookup code (#1348, #1358) because causing zebrad to hang on a failed
dependency was useful for identifying cases where the code wasn't useful (and
then inspecting execution traces).
As a side effect, this ticket resolves most of the hangs in #1389, because
far-future gossiped blocks will have their UTXO lookups time out, though we
may wish to do other work as part of debugging the combined sync+gossip logic.
This commit changes the state system and database format to track the
provenance of UTXOs, in addition to the outputs themselves.
Specifically, it tracks the following additional metadata:
- the height at which the UTXO was created;
- whether or not the UTXO was created from a coinbase transaction or
not.
This metadata will allow us to:
- check the coinbase maturity consensus rule;
- check the coinbase inputs => no transparent outputs rule;
- implement lookup of transactions by utxo (using the height to find the
block and then scanning the block) for a future RPC mechanism.
Closes#1342
This change introduces two new types:
- `PreparedBlock`, representing a block which has undergone semantic
validation and has been prepared for contextual validation;
- `FinalizedBlock`, representing a block which is ready to be finalized
immediately;
and changes the `Request::CommitBlock`,`Request::CommitFinalizedBlock`
variants to use these types instead of their previous fields.
This change solves the problem of passing data between semantic
validation and contextual validation, and cleans up the state code by
allowing it to pass around a bundle of data. Previously, the state code
just passed around an `Arc<Block>`, which forced it to needlessly
recompute block hashes and other data, and was incompatible with the
already-known but not-yet-implemented data transfer requirements, namely
passing in the Sprout and Sapling anchors computed during contextual
validation.
This commit propagates the `PreparedBlock` and `FinalizedBlock` types
through the state code but only uses their data opportunistically, e.g.,
changing .hash() computations to use the precomputed hash. In the
future, these structures can be extended to pass data through the
verification pipeline for reuse as appropriate. For instance, these
changes allow the sprout and sapling anchors to be propagated through
the state.
The UTXO query system assumes that a transaction will only request
information about UTXOs created in prior blocks. But transactions are
allowed to spend UTXOs created by prior transactions in the same block.
This doesn't fit with the existing query model, so instead of trying to
change it, allow the script verifier to take an additional set of known
UTXOs, and propagate this set from the block.
This consensus rule is supposed to apply to transactions whose
transparent inputs are the *outputs* of previous coinbase
transactions, not to transactions with coinbase inputs. Because that
logic is different enough from this logic, and requires different data
flow, it's cleaner to just remove this check for now.
Making this check's match statement exhaustive revealed a bug similar to
the previous commit. The logic in the spec is written in terms of
numbers, but our data is internally represented in terms of enums
(ADTs). This kind of cross-representation rule translation is a bug
surface, which we can avoid by converting to counts and summing up. (We
should use one style at a time).
This function caused spurious "WrongVersion" errors, because the match
pattern in the first arm was non-exhaustive, but the fallthrough match
arm was present and assumed it would only be reached if the version was
incorrect.
This commit cleans up the implemenation, splits out the error variants,
and renames the check to be more precise.
To avoid this kind of bug in the future, two guidelines are useful:
1. Avoid fallthrough cases that circumvent non-exhaustive match checks;
2. Avoid nested conditionals, preferring a "straight-line" sequence of
match arm => result pairs rather than nested matches or matches with
conditionals inside.
The BlockVerifier constructed a tracing span and manually entered it
inside of an async block. Manually entering spans inside async blocks
can cause problems where the span might not be entered and exited
correctly as the resulting future is polled. Instead, using the
.instrument creates a wrapper future that handles the bookkeeping.
I changed the span name and contents to be consistent with the spans in
the checkpoint verifier.
This change is mostly mechanical, with the exception of the changes to the
`tower-batch` middleware. This middleware was adapted from `tower::buffer`,
and the `tower::buffer` code was changed to implement its own bounded queue,
because Tokio 0.3 removed the `mpsc::Sender::poll_send` method. See
ddc64e8d4d
for more context on the Tower changes. To match Tower as closely as possible
in order to be able to upstream `tower-batch`, those changes are copied from
`tower::Buffer` to `tower-batch`.
This change explicitly documents cancellation contracts for our Tower services,
and tries to correct a bug in the implementation of the CheckpointVerifier,
which duplicates information from the state service but did not ensure that it
would be kept in sync.
Conrado Gouvea
teor
Alfredo Garcia
dependabot[bot]
Janito Vaqueiro Ferreira Filho
Pili Guerra
Deirdre Connolly
Dimitris Apostolou
Marek
Deirdre Connolly
Jack Grigg
Jane Lusby
Jane Lusby
Henry de Valence
Henry de Valence