* validate funding stream amounts in the coinbase
* clippy
* use `i64::from()` and remove `number()` method from `Amount`
* move tests to their own file
* refactor the funding stream check
* use `Amount`s in funding streams calculation
* remove unused import
* add import to tests
* expand test vectors
* add notes to `funding_stream_values()`
* Update `tower` to version `0.4.9`
Update to latest version to add support for Tokio version 1.
* Replace usage of `ServiceExt::ready_and`
It was deprecated in favor of `ServiceExt::ready`.
* Update Tokio dependency to version `1.13.0`
This will break the build because the code isn't ready for the update,
but future commits will fix the issues.
* Replace import of `tokio::stream::StreamExt`
Use `futures::stream::StreamExt` instead, because newer versions of
Tokio don't have the `stream` feature.
* Use `IntervalStream` in `zebra-network`
In newer versions of Tokio `Interval` doesn't implement `Stream`, so the
wrapper types from `tokio-stream` have to be used instead.
* Use `IntervalStream` in `inventory_registry`
In newer versions of Tokio the `Interval` type doesn't implement
`Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used
instead.
* Use `BroadcastStream` in `inventory_registry`
In newer versions of Tokio `broadcast::Receiver` doesn't implement
`Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This
also requires changing the error type that is used.
* Handle `Semaphore::acquire` error in `tower-batch`
Newer versions of Tokio can return an error if the semaphore is closed.
This shouldn't happen in `tower-batch` because the semaphore is never
closed.
* Handle `Semaphore::acquire` error in `zebrad` test
On newer versions of Tokio `Semaphore::acquire` can return an error if
the semaphore is closed. This shouldn't happen in the test because the
semaphore is never closed.
* Update some `zebra-network` dependencies
Use versions compatible with Tokio version 1.
* Upgrade Hyper to version 0.14
Use a version that supports Tokio version 1.
* Update `metrics` dependency to version 0.17
And also update the `metrics-exporter-prometheus` to version 0.6.1.
These updates are to make sure Tokio 1 is supported.
* Use `f64` as the histogram data type
`u64` isn't supported as the histogram data type in newer versions of
`metrics`.
* Update the initialization of the metrics component
Make it compatible with the new version of `metrics`.
* Simplify build version counter
Remove all constants and use the new `metrics::incement_counter!` macro.
* Change metrics output line to match on
The snapshot string isn't included in the newer version of
`metrics-exporter-prometheus`.
* Update `sentry` to version 0.23.0
Use a version compatible with Tokio version 1.
* Remove usage of `TracingIntegration`
This seems to not be available from `sentry-tracing` anymore, so it
needs to be replaced.
* Add sentry layer to tracing initialization
This seems like the replacement for `TracingIntegration`.
* Remove unnecessary conversion
Suggested by a Clippy lint.
* Update Cargo lock file
Apply all of the updates to dependencies.
* Ban duplicate tokio dependencies
Also ban git sources for tokio dependencies.
* Stop allowing sentry-tracing git repository in `deny.toml`
* Allow remaining duplicates after the tokio upgrade
* Use C: drive for CI build output on Windows
GitHub Actions uses a Windows image with two disk drives, and the
default D: drive is smaller than the C: drive. Zebra currently uses a
lot of space to build, so it has to use the C: drive to avoid CI build
failures because of insufficient space.
Co-authored-by: teor <teor@riseup.net>
* Reorder imports to follow convention
Place the imports from `std` at the top.
* Add transaction errors for double spends
Add a variant for each pool. They represent a double spend inside a
transaction.
* Add `check::spend_conflicts` implementation
Checks if a transaction has spend conflicts, i.e., if a transaction
spends a UTXO more than once or if it reveals a nullifier more than
once.
* Reject transactions with internal spend conflicts
The transaction verifier should reject transactions that spend the same
transparent UTXO or that reveal the same nullifier.
* Add transparent spend consensus rule
Add it to the documentation to help with understanding and auditing it.
Co-authored-by: teor <teor@riseup.net>
* Use different nullifiers by default
Don't use the same nullifier twice when mocking a
`sprout::JoinSplitData` because it will lead to an invalid transaction.
* Test transactions with repeated spend outpoints
Since that represents a spend conflict, they should be rejected.
* Test duplicate nullifiers in joinsplit
Check if a mock transaction with a joinsplit that reveals the same
nullifier twice is rejected.
* Test duplicate nullifiers across joinsplits
Check if a duplicate nullifier in two different joinsplits in the same
transaction is rejected.
* Test V4 transaction with duplicate Sapling spend
Check if a V4 transaction that has a duplicate Sapling spend is
rejected.
* Test V5 transaction with duplicate Sapling spend
Check if a V5 transaction that has a duplicate Sapling spend is
rejected.
* Test V5 transaction with duplicate Orchard actions
Check if a V5 transaction that has duplicate Orchard actions is rejected
by the transaction verifier.
Co-authored-by: teor <teor@riseup.net>
* Ignore AlreadyInChain error in the syncer
* Split Cancelled errors; add them to should_restart_sync exceptions
* Also filter 'block is already comitted'; try to detect a wrong downcast
* Create a new VerifiedUnminedTx containing the miner fee
* Use VerifiedUnminedTx in mempool verification responses
And do a bunch of other cleanups.
* Use VerifiedUnminedTx in mempool download and verifier
* Use VerifiedUnminedTx in mempool storage and verified set
* Impl Display for VerifiedUnminedTx, and some convenience methods
* Use VerifiedUnminedTx in existing tests
* Get the transaction fee from utxos
* Return the transaction fee from the verifier
* Avoid calculating the fee for coinbase transactions
Coinbase transactions don't have fees. In case of a coinbase transaction, the
verifier returns a zero fee.
* Update the result obtained by `Downloads`
* Send spent UTXOs from the script verifier to the transaction verifier
* Add temporary assertions for testing spent UTXO sending
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Marek <mail@marek.onl>
* Check for state errors before updating metrics
Previously, the metrics would be updated for some rejected blocks.
* Clarify and expand block verification metrics
Rename checkpoint-specific metrics to clarify their purpose.
Add metrics for:
- finalized blocks on disk
- blocks verified using the full block verifier
(this metric was previously incorrectly called `zcash_chain_verified_block_height`)
* Update dashboard metric names
Also:
- add some extra block height metrics
- fix a dashboard name
* Add exact block heights to Grafana dashboards
* Add a missing comment
* grafana: use 0 decimals for metrics
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* grafana: show the entire height instead of abbreviated
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* grafana: show the entire height instead of abbreviated
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Fix typo in metric name
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Move height gauges to the state, so they are correct
If we update height gauges in futures, they can execute out of order,
so the metrics can be incorrect.
Instead:
- move the height gauges to the state, and update them based on the best tip
- move the verified block counts to the state
- continue to include all verified blocks on all non-finalized chains
(not just the best chain)
* Show exact checkpoint heights in the dashboard
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Remove unused mempool storage errors
Preparation for ticket #2819.
Removing these errors means that we don't have to decide
which type of transaction ID match we want for them.
* Remove unused mempool errors, and deduplicate storage errors
* rustfmt
* Check return value of zcash_script_new_precomputed_tx
* Set the NU5 testnet activation height to 1_590_000
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Update Nu5 constants to new values
* Update ZIP-244 test vectors for new branch ID
* Squashed commit of the following:
commit bdb120a249
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 11:54:01 2021 -0400
Use pallas::Base::from_str_vartime() in sinsemilla tests
commit e99fa49258
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 11:45:24 2021 -0400
Compiles
commit a520018114
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 10:15:17 2021 -0400
Incomplete upgrade of deps
* Squashed commit of the following:
commit 8d1b76ec5626517817c3a4d9f3950acc90a359df
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 04:02:26 2021 +0000
Update `zcash_script` to support V5 transactions
Use a newer version of `zcash_script` that has been updated to support
V5 transactions.
commit 371233628ae61e0c25d6ba8f31d9dba42823becb
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 03:06:20 2021 +0000
Update Zcash dependencies
Update some Zcash crates:
- `halo2`
- `incrementalmerkletree' (patch version)
- `orchard` (patch version)
- `zcash_history` (patch version)
- `zcash_note_encryption` (patch version)
- `zcash_primitives` (patch version)
And also update the `group` dependency so that the code remains
compatible.
commit de5cf1ec40c3fc08670fc971cdf3e65e13d9f4c7
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 03:04:13 2021 +0000
Update error message assertion
Use the updated message for the expected error variant.
* Update `zcash_script` to support V5 transactions
Use a newer version of `zcash_script` that has been updated to support
V5 transactions.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
* Rename type parameter to be more explicit
Replace the single letter with a proper name.
* Remove imports for `Request` and `Response`
The type names will conflict with the ones for the mempool service.
* Attach `Mempool` service to the `Crawler`
Add a field to the `Crawler` type to store a way to access the `Mempool`
service.
* Forward crawled transactions to downloader
The crawled transactions are now sent to the transaction downloader and
verifier, to be included in the mempool.
* Derive `Eq` and `PartialEq` for `mempool::Request`
Make it simpler to use the `MockService::expect_request` method.
* Test if crawled transactions are downloaded
Create some dummy crawled transactions, and let the crawler discover
them. Then check if they are forwarded to the mempool to be downloaded
and verified.
* Don't send empty transaction ID list to downloader
Ignore response from peers that don't provide any crawled transactions.
* Log errors when forwarding crawled transaction IDs
Calling the Mempool service should not fail, so if an error happens it
should be visible. However, errors when downloading individual
transactions can happen from time to time, so there's no need for them
to be very visible.
* Document existing `mempool::Crawler` test
Provide some depth as to what the test expect from the crawler's
behavior.
* Refactor to create `setup_crawler` helper function
Make it easier to reuse the common test setup code.
* Simplify code to expect requests
Now that `zebra_network::Request` implement `Eq`, the call can be
simplified into `expect_request`.
* Refactor to create `respond_with_transaction_ids`
A helper function that checks for a network crawl request and responds
with the given list of crawled transaction IDs.
* Refactor to create `crawler_iterator` helper
A function to intercept and respond to the fanned-out requests sent
during a single crawl iteration.
* Refactor to create `respond_to_queue_request`
Reduce the repeated code necessary to intercept and reply to a request
for queuing transactions to be downloaded.
* Add `respond_to_queue_request_with_error` helper
Intercepts a mempool request to queue transactions to be downloaded, and
responds with an error, simulating an internal problem in the mempool
service implementation.
* Derive `Arbitrary` for `NetworkUpgrade`
This is required for deriving `Arbitrary` for some error types.
* Derive `Arbitrary` for `TransactionError`
Allow random transaction errors to be generated for property tests.
* Derive `Arbitrary` for `MempoolError`
Allow random Mempool errors to be generated for property tests.
* Test if errors don't stop the mempool crawler
The crawler should be robust enough to continue operating even if the
mempool service fails to download transactions or even fails to handle
requests to enqueue transactions.
* Reduce the log level for download errors
They should happen regularly, so there's no need to have them with a
high visibility level.
Co-authored-by: teor <teor@riseup.net>
* Stop crawler if service stops
If `Mempool::poll_ready` returns an error, it's because the mempool
service has stopped and can't handle any requests, so the crawler should
stop as well.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Allow deliberate instances of the new nightly clippy::derivable_impls lint
We want our config defaults to be explicit.
Not so sure about the application defaults, but they also contain a config.
* Also allow unknown lint names
Stable doesn't know about this lint, but nightly does.
We don't check the authorizing data hash until checkpoint blocks reach the state.
So signatures, proofs, or scripts could be different,
even if the block hash is the same.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Stop precalculating transaction hashes twice during checkpointing
* Refactor a complex type using a new `RequestBlock` type
* Comment formatting
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Store precalculated transactions in an `Arc`
Transaction `Hash`es are 32 bytes,
and the minimun transparent transaction size is 54 bytes.
So a full 2MB block can create 1.1MB of transaction hashes.
We use an `Arc` to avoid repeatedly cloning that much data.
* Remove the unused `Block` from `ChainTipBlock`
This drops the block as soon as it isn't needed any more.
Previously, it would stick around until every `ChainTipReceiver`
dropped their `ChainTipBlock`, even if they didn't use the `Block`
at all.
* Add `Arc<Transaction>` conversions for Transaction IDs
* Use UnminedTxId as the transaction verifier response type
* Use UnminedTx in transaction verifier mempool requests
* Refactor is_mempool into a transaction verifier request method
* Order derives in alphabetical order
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Return a transaction verifier from `zebra_consensus::init`
This verifier is temporarily created separately from the block verifier's
transaction verifier.
* Return the same transaction verifier used by the block verifier
* Clarify that the mempool verifier is the transaction verifier
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add validation of ZIP-221 and ZIP-244 commitments
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Add auth commitment check in the finalized state
* Reset the verifier when comitting to state fails
* Add explanation comment
* Add test with fake activation heights
* Add generate_valid_commitments flag
* Enable fake activation heights using env var instead of feature
* Also update initial_tip_hash; refactor into progress_from_tip()
* Improve comments
* Add fake activation heights test to CI
* Fix bug that caused commitment trees to not match when generating partial arbitrary chains
* Add ChainHistoryBlockTxAuthCommitmentHash::from_commitments to organize and deduplicate code
* Remove stale comment, improve readability
* Allow overriding with PROPTEST_CASES
* partial_chain_strategy(): don't update note commitment trees when not needed; add comment
Co-authored-by: teor <teor@riseup.net>
* add value pools to the database
* remove redundant genesis block check
* use update_with_chain_value_pool_change()
* remove constrains
* remove height from the database
* remove calls to chain_value_pool_change
* clippy
* use the "correct" value balances
* bump the database format
* remove everything that is not finalized state
* clippy
* rustfmt
* use all spent utxos
* add new_outputs utxos to all_utxos_spent_by_block
* remove panic
* add finalized state value pool test
* clippy
* clippy 2
* move import
* fix import
* rustfmt
Co-authored-by: teor <teor@riseup.net>
* Make Amount arithmetic more generic
To modify generated amounts, we need some extra operations on `Amount`.
We also need to extend existing operations to both `NonNegative` and
`NegativeAllowed` amounts.
* Add a constrain method for ValueBalance
* Derive Eq for ValueBalance
* impl Neg for ValueBalance
* Make some Amount arithmetic expectations explicit
* Explain why we use i128 for multiplication
And expand the overflow error details.
* Expand Amount::sum error details
* Make amount::Error field order consistent
* Rename an amount::Error variant to Constraint, so it's clearer
* Add specific pool variants to ValueBalanceError
* Update coinbase remaining value consensus rule comment
This consensus rule was updated recently to include coinbase transactions,
but Zebra doesn't check block subsidy or miner fees yet.
* Add test methods for modifying transparent values and shielded value balances
* Temporarily set values and value balances to zero in proptests
In both generated chains and proptests that construct their own transactions.
Using zero values reduces value calculation and value check test coverage.
A future change will use non-zero values, and fix them so the check passes.
* Add extra fields to remaining transaction value errors
* Swap the transparent value balance sign to match shielded value balances
This makes the signs of all the chain value pools consistent.
* Use a NonNegative constraint for transparent values
This fix:
* makes the type signature match the consensus rules
* avoids having to write code to handle negative values
* Allocate total generated transaction input value to outputs
If there isn't enough input value for an output, set it to zero.
Temporarily reduce all generated values to avoid overflow.
(We'll remove this workaround when we calculate chain value balances.)
* Consistently use ValueBalanceError for ValueBalances
* Make the value balance signs match the spec
And rename and document methods so their signs are clearer.
* Convert amount::Errors to specific pool ValueBalanceErrors
* Move some error changes to the next PR
* Add extra info to remaining transaction value errors (#2585)
* Distinguish between overflow and negative remaining transaction value errors
And make some error types cloneable.
* Add methods for updating chain value pools (#2586)
* Move amount::test to amount::tests:vectors
* Make ValueBalance traits more consistent with Amount
- implement Add and Sub variants with Result and Assign
- derive Hash
* Clarify some comments and expects
* Create ValueBalance update methods for blocks and transactions
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Validate transparent coinbase output maturity and shielding
- Add a CoinbaseSpendRestriction enum and Transaction method
- Validate transparent coinbase spends in non-finalized chains
* Don't use genesis created UTXOs for spends in generated block chains
* Refactor out a new_transaction_ordered_outputs function
* Add Transaction::outputs_mut for tests
* Generate valid transparent spends in arbitrary block chains
* When generating blocks, fixup the block contents, then the block hash
* Test that generated chains contain at least one transparent spend
* Make generated chains long enough for reliable tests
* Add transparent and shielded input and output methods to Transaction
* Split chain generation into 3 functions
* Test that unshielded and immature transparent coinbase spends fail
* Comment punctuation
* Clarify a comment
* Clarify probability calculation
* Test that shielded mature coinbase output spends succeed
* Create a `zebra_state::init_test` helper function
This function will be used as a replacement for `zebra_state::init`
inside tests. It's a simpler alternative because it can ignore any
details that aren't relevant for tests.
* Use `init_test` inside `zebra-state` tests
Update usages of `init` to use `init_test` instead, which simplifies
most cases.
* Use `zebra_state::init_test` in `zebra-consensus`
Replace usages of `zebra_state::init` with the new helper function. This
simplifies the code a bit.
* Add an OrderedUtxo type for validation of spends within a block
This change allows us to check that transparent spends use outputs from
earlier in their block. (But we don't actually do that check yet.)
We need to keep the order of UTXOs when we're contextually verifying
each new block that is added to a chain. But the block order is
irrelevant for UTXOs stored in the state.
* Take ownership in utxos_from_ordered_utxos
* Delete a confusing comment
* Add panic message to `unimplemented!`
So that it is clear why the panic happened upon initial inspection. Also
include a reference to the mempool epic, so that it's easier to find the
issue that tracks the implementation of the missing code.
* Add panic message that references a tracking issue
Make it easy to find the relevant issue if the panic occurs.
* Remove incomplete and currently unnecessary code
The current implementation works, the commented out code was just a
previous improvement idea, which is now tracked by issue #2473.
* Always use librustzcash for sighash and remove old sighash code
Also added ZIP-143 test vectors
* Remove librustzcash_sighash test that is no longer needed
* Make `verify_sapling_shielded_data` more generic
Prepare to support V5 transactions which have a shared anchor.
* Verify Sapling shielded data in V5 transactions
Call the `verify_sapling_shielded_data` method and add the respective
asynchronous checks to the set of V5 checks.
* Fix expect message in V4 transaction test
It was using the same message as the previous test, even though the test
searches with different criteria.
* Test V5 transaction with Sapling spends
Create a fake V5 transaction that has Sapling spends and check that the
verifier accepts the transaction.
* Ignore rejected V5 transaction test for now
Because now it needs the `sighash` implementation for V5 to be ready.
* Reference V5 `sighash` PR in comment
So that it is easier to check if it's possible to remove the
`should_panic` or not.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove `sapling shielded pool` TODO
V5 transactions now have Sapling shielded pool properly validated.
* Link to some extra issues in TODO comment
Some other issues are also necessary for full V5 validation.
* Add a TODO in the main code to fix the tests
Some tests are blocked due to missing features required for full V5
validation. Once those features are implemented, they should be updated
to remove the `#[should_panic]` attribute so that they actually run and
check the code correctly.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Refactor `AsyncChecks` into a proper type
Add some helper methods to it so that checks can be added by
daisy-chaining calls. Also move the code to wait for the checks to
finish into the new type.
* Refactor inclusion of individual Sapling checks
Use `oneshot` instead of `ready_and` so that the method becomes
synchronous.
* Make V4 and V5 verification methods synchronous
There is no longer a need to wait for any internal service to be ready,
since now that's always done as part of an asynchronous check included
in the returned set of checks.
* add disabled sprout pool check
* change method name
* change error name
* fix typo
* make the success test case in other tx than the coinbase
* use new `height` method instead of deriving `PartialOrd` in `NetworkUpgrade`
* move check of network upgrade into function, rename, docs
* increase test coverage
* fix comment
* Refactor to create `verify_sapling_shielded_data`
Move the code to verify Sapling shielded data into a new helper method
that returns `AsyncChecks`.
* Test verifying a Sapling transaction with spends
Use the test vectors to find a transaction that has Sapling spends and
test if it the verifier considers it valid.
* Create a helper method to list test transactions
Transforms the block test vectors into a list of transactions and block
heights for each transaction.
* Use new helper function in V4 Sapling spend test
Also use the block height for that transaction as specified in the test
vector.
* Test V4 tx. with Sapling outputs but no spends
Find a transaction V4 vector that has Sapling outputs but no spends, and
check that the verifier accepts it.
* Add a `zebra_test::RUNTIME` shared runtime
Create a lazily instantiated Tokio runtime that can be shared by tests.
* Split tests that require a shared runtime
Split two tests that were previously in one because of the need to share
a single Tokio runtime. With the `zebra_test::RUNTIME`, they can now
share the runtime without having to be a single test.
* Stop trying to verify coinbase inputs using the script verifier
And create tests to catch similar bugs earier.
* Use Testnet in NU5 tests that temporarily should_panic
We've marked these tests as should_panic until there is a NU5 activation
height. But Testnet will have an activation height first, so we should
prefer it in the tests. (Or use both networks.)
Conrado Gouvea
Alfredo Garcia
Janito Vaqueiro Ferreira Filho
teor
Marek
Deirdre Connolly