* doc(utils): remove trailing spaces
* fix(consensus): update Zebra's hard-coded checkpoint lists
This updates the checkpoint lists as at 2022-02-22:
- mainnet: to the latest finalized tip
- testnet: to the latest Canopy checkpoint,
because we expect a NU5 testnet rollback
* doc(consensus): Add example commands to update the checkpoints
* docs: document consensus rules from 4.6 Action Descriptions
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Marek <mail@marek.onl>
* feat(log): log current height when logging sync progress
* fix(test): log the specific error when full sync tests fail
* doc(start): remove an obsolete TODO
We can't decrease this log level, because the tests rely on it.
* fix(test): wait until mempool activates in full sync tests
Changes the log message and log test so that the test only finishes
when the mempool has activated.
There is still a race condition here, between the log timer and
mempool activation. But it should be very rare, because the mempool
is activated immediately when `is_close_to_tip()` becomes true.
* fix(test): warn when Zebra stalls below the maximum checkpoint height
This also improves the full sync tests,
because the warning is checked before logging a successful sync.
* feat(log): warn when sync stalls downloading the genesis block
* fix(test): warn when the state hasn't committed a block for a long time
This also improves the full sync tests,
because the warning is checked before logging a successful sync.
* doc(test): update some sync acceptance test comments
* fix(log): use Display formatting to log chrono::Duration
Debug formatting is complicated and hard to read.
* fix(log): stop saying that we've activated the mempool without checking it
We're not checking if the mempool is active, so we can't say that.
* fix(log): minor tidying and TODOs
* fix(doc): fix a typo in the tests
* fix(log): explain the post-checkpoint blocks in progress warning calculations
* fix(doc): explain what could happen if we don't wait for extra blocks
* fix(log): add a percent symbol to a percent log
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(clippy): for loop with only one item
* fix(clippy): manual Range::contains
Also clarified the surrounding code because it was unclear.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(network): split synthetic NotFoundRegistry from message NotFoundResponse
* docs(network): Improve `notfound` message documentation
* refactor(network): Rename MustUseOneshotSender to MustUseClientResponseSender
```
fastmod MustUseOneshotSender MustUseClientResponseSender zebra*
```
* docs(network): fix a comment typo
* refactor(network): remove generics from MustUseClientResponseSender
* refactor(network): add an inventory collector to Client, but don't use it yet
* feat(network): register missing peer responses as missing inventory
We register this missing inventory based on peer responses,
or connection errors or timeouts.
Inbound message inventory tracking requires peers to send `notfound` messages.
But `zcashd` skips `notfound` for blocks, so we can't rely on peer messages.
This missing inventory tracking works regardless of peer `notfound` messages.
* refactor(network): rename ResponseStatus to InventoryResponse
```sh
fastmod ResponseStatus InventoryResponse zebra*
```
* refactor(network): rename InventoryStatus::inner() to to_inner()
* fix(network): remove a redundant runtime.enter() in a test
* doc(network): the exact time used to filter outbound peers doesn't matter
* fix(network): handle block requests slightly more efficiently
* doc(network): fix a typo
* fmt(network): `cargo fmt` after rename ResponseStatus to InventoryResponse
* doc(test): clarify some test comments
* test(network): test synthetic notfound from connection errors and peer inventory routing
* test(network): improve inbound test diagnostics
* feat(network): add a proptest-impl feature to zebra-network
* feat(network): add a test-only connect_isolated_with_inbound function
* test(network): allow a response on the isolated peer test connection
* test(network): fix failures in test synthetic notfound
* test(network): Simplify SharedPeerError test assertions
* test(network): test synthetic notfound from partially successful requests
* test(network): MissingInventoryCollector ignores local NotFoundRegistry errors
* fix(network): decrease the inventory rotation interval
This stops us waiting 3-4 sync resets (4 minutes) before we retry a missing block.
Now we wait 1-2 sync resets (2 minutes), which is still a reasonable rate limit.
This should speed up syncing near the tip, and on testnet.
* fmt(network): cargo fmt --all
* cleanup(network): remove unnecessary allow(dead_code)
* cleanup(network): stop importing the whole sync module into tests
* doc(network): clarify syncer inventory retry constraint
* doc(network): add a TODO for a fix to ensure API behaviour remains consistent
* doc(network): fix a function doc typo
* doc(network): clarify how we handle peers that don't send `notfound`
* docs(network): clarify a test comment
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Support large block heights
* Document consensus rules referring to expiry heights
* Refactor the docs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix the formatting of an error message
* refactor: Simplify coinbase expiry code so the consensus rule is clear (#3408)
* Fix some outdated TODO comments
* refactor(coinbase expiry): Simplify the code so consensus rule is clear
* Fix the formatting of an error message
* Remove a redundant comment
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Marek <mail@marek.onl>
* Check the max expiry height at parse time
* Test that 2^31 - 1 is the last valid height
* Add tests for nExpiryHeight
* Add tests for expiry heights of V4 transactions
* Add tests for V5 transactions
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feat(network): send notfound messages to the inventory registry
* refactor(network): move the inventory filter into an async function
* feat(network): avoid routing requests to peers that are missing inventory
* test(network): advertised routing is independent of numeric address value
* test(network): peer set routes requests to peers not missing that inventory
* test(network): peer set fails requests if all ready peers are missing that inventory
* fix(clippy): needless-borrow in the peer set
* fix(lint): remove redundant trailing commas in macro calls
There is no clippy lint for this, maybe because some macros
are sensitive to trailing commas.
(But not the ones changed in this commit.)
* test(network): check the exact number of inventory peers
* doc(network): explain why we ignore inventory send failures
* docs(network): explain why a channel error is ignored
* Add all_previous_outputs; load UTXOs in transaction verifier
* Remove UTXO loading and returning from script.rs
* Don't pass state service to script verifier
* Remove output from is_valid()
* Refactor loading UTXOs to separate function
* Pass all_previous_output to sighash
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Create AwaitUtxo only when needed; formatting
* Add comments about output vectors in tests
* Change sighash() to receive reference and avoid cloning
* Expand comments
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* remove founders reward code
* panic if a block before Canopy is validated for subsidy
Co-Authored-By: Daira Hopwood <daira@jacaranda.org>
Co-authored-by: Daira Hopwood <daira@jacaranda.org>
* feat(log): log the state tip height as part of sync progress logs
* fix(log): downgrade some verbose state logs to debug
* feat(log): log successful gossiped block verification at info level
These logs help us diagnose slow progress near the tip.
There won't be very many of these logs,
because they only happen near the tip.
* fix(log): spawn top-level tasks within the global Zebra tracing span
* fix(log): spawn blocking top-level tasks within the global Zebra tracing span
Co-authored-by: teor <teor@riseup.net>
* Updating zebra-test to v1.0.0-beta.4
* Updating tower-fallback to v0.2.16
* Updating tower-batch to v0.2.20
* Updating zebra-chain to v1.0.0-beta.4
* Updating zebra-script to v1.0.0-beta.4
* Updating zebra-network to v1.0.0-beta.4
* Updating zebra-state to v1.0.0-beta.4
* Updating zebra-consensus to v1.0.0-beta.4
* Updating zebra-utils to v1.0.0-beta.4
* Updating zebrad to v1.0.0-beta.4
* Update Cargo.lock with updated crate versions
* Add section for 1.0.0-beta.4 in the CHANGELOG
* Update CHANGELOG.md
Co-authored-by: teor <teor@riseup.net>
* Update README.md
Co-authored-by: teor <teor@riseup.net>
* Update book/src/user/install.md
Co-authored-by: teor <teor@riseup.net>
* Update release description to mention Rust 2021
Update the changelog to mention that all crates now use Rust 2021
Edition.
* Elaborate on Section 3.6 documentation entry
Add the title of the section and mention that it's a section of the
Zcash protocol specification.
* Make changelog entries consistent
Use the same format for entries related to consensus rule documentation.
Co-authored-by: Marek <mail@marek.onl>
* Update README about validated consensus rules
Zebra now validates all documented consensus rules.
* Add changelog entry for newly merged PR
Describe the security fix in the changelog.
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
* Move standard lints into .cargo/config.toml
* Ignore "wrong self convention" in a futures-based trait
This lint might only trigger on beta or nightly at the moment.
* Warn if future incompatibile code is added to Zebra
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* retry download parameters a few times before giving up
* separate sapling and sprout params download to methods
* simpify the new created methods
* Assert the parameters were downloaded to the expected paths
Co-authored-by: teor <teor@riseup.net>
* Ensure that sapling::keys::TransmissionKey jubjub point is always in the prime order group
* Adjust TransmissionKey check; add AuthorizingKey check
* Move ValueCommitment small order check to deserialization
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Use is_torsion_free() instead of is_identity() and is_prime_order()
* Add EphemeralPublicKey small order check on instantiation; remove old checks
* Use VerificationKey instead of VerificationKeyBytes; fix tests
* Use ValidatingKey instead of VerificationKeyBytes for rk
* Reject identity when creating an Orchard EphemeralPublicKey
* Make documentation more consistent, use generator in tests
* s/JubJub/Jubjub/
* Fix zebra-consensus tests (insert_fake_orchard_shielded_data)
* Create NotSmallOrderValueCommitment, since intermediate values can be the identity
* Clarify documentation
* rustdoc
* Tidy rustdoc
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
* Create a strategy for block heights after Sapling
Provides an arbitrary network (mainnet or testnet) and a block height
between the Sapling activation height on that network and the maximum
block height.
* Create a helper function to select block heights
Allows generating block heights inside a range using a scale factor
between 0 and 1.
* Allow specifying the outpoint index for mock UTXOs
Avoid creating multiple transparent transfers in the same transaction
with the same source UTXO, which would lead to a double spend.
* Create helper function to mock multiple transfers
Given relative block height scale factors, create a mock transparent
transfer for each one of them.
Also add a constant that serves as a guideline for the maximum number of
transparent transfers to mock.
* Create helper function to sanitize tx. version
Make sure the arbitrary transaction version is valid for the network
(testnet or mainnet) at the specified block height.
* Create `mock_transparent_transaction` helper func.
Creates a V4 or V5 mock transaction that only includes transparent
inputs and outputs.
* Create helper function for transaction validation
Performs the actual tested action of verifying a transaction. It sets up
the verifier and uses it to obtain the verification result.
* Test if zero lock time means unlocked
Generate arbitrary transactions with zero lock time, and check that they
are accepted by the transaction verifier.
* Allow changing the sequence number of an input
Add a setter method for a `transparent::Input`'s sequence number. This
setter is only available for testing.
* Test if sequence numbers can disable lock time
Create arbitrary transactions and set the sequence numbers of all of its
inputs to `u32::MAX` to see if that disables the lock time and the
transactions are accepted by the verifier.
* Test block height lock times
Make sure that the transaction verifier rejects transactions that are
still locked at a certain block height.
* Test block time lock times
Test that the transaction verifier rejects a transaction that is
validated at a block time that's before the transaction's lock time.
* Test unlocking by block height
Test that transactions unlocked at an earlier block height are accepted
by the transaction verifier.
* Test transactions unlocked by the block time
Test that transactions that were unlocked at a previous block time are
accepted by the transaction verifier.
* Fix an incorrect method comment
Co-authored-by: teor <teor@riseup.net>
* Document the chain verifier
* Drop gossiped blocks that are too far ahead of the tip
* Add extra gossiped block metrics
* Allow extra gossiped blocks, now we have a stricter limit
* Fix a comment
* Check the exact number of blocks in a downloaded block response
* Drop synced blocks that are too far ahead of the tip
* Add extra synced block metrics
* Test dropping gossiped blocks that are too far ahead of the tip
* Allow an extra checkpoint's worth of blocks in the verifier queues
* Actually let's try two extra checkpoints
* Scale extra height limit with lookahead limit
* Also drop blocks that are behind the finalized tip
* Downgrade a noisy log
* Use a debug log for already verified gossiped blocks
* Use debug logs for already verified synced blocks
* Integrate JoinSplit verifier with transaction verifier
* Add test with malformed Groth16 Output proof
* Use TryFrom instead of From in ItemWrapper to correctly propagate malformed proof errors
* Simplify by removing ItemWrapper and directly TryFrom into Item
* Fix existing tests to work with JoinSplit validation
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
Zebra's latest beta continues implementing zero-knowledge proof and note commitment tree validation. In this release, we have finished implementing transaction header, transaction amount, and Zebra-specific NU5 validation. (NU5 mainnet validation is waiting on an `orchard` crate update, and some consensus parameter updates.)
We also fix a number of security issues that could pose a local denial of service risk, or make it easier for an attacker to make a node follow a false chain.
As of this release, Zebra will automatically download and cache the Sprout and Sapling Groth16 circuit parameters. The cache uses around 1 GB of disk space. These cached parameters are shared across all Zebra and `zcashd` instances run by the same user.
See CHANGELOG.md for the full list of changes in this release.
* validate non coinbase expiration height
* change var name
* move checks to transaction verifier
* Add variants and debug fields to transaction expiry errors
* Fix a failing existing test
Co-authored-by: teor <teor@riseup.net>
* Download and load Sprout parameters using zcash_proofs
Also update some librustzcash dependencies, to avoid duplicate dependencies.
* Update upstream orchard to avoid a compilation error
* Skip librustzcash batch refactor for now, to avoid compilation errors
* Change the cache ID, so we actually cache Sprout
* Move existing file checks into zcash_proofs
* Add a 1 hour timeout to parameter file downloads
* Give other tasks priority, before spawning the download task
* Update to the latest version of our modified librustzcash fork
* Change the cache key for Sprout
* Add 40 minutes to CI timeouts for occasional sprout downloads
* Update to zcash_proofs with split downloads
* Check file sizes to help debug parameter load failures in zcash_proofs
* Start the second download once the first has finished in zcash_proofs
* Document the parameter download task
* Stop hashing existing files twice
* validate consensus rule: negative fee not allowed
* fix a test TODO
* fix imports
* move import back
* fix panic text
* join consensus rule check code
* match assertion better in tests
* fix test
* fix consensus rule validation
* remove panics
* Delete a TODO
Co-authored-by: teor <teor@riseup.net>
* Start network before verifiers
This makes the Groth16 download task start as late as possible.
* Explain why the Groth16 download must happen first
* Speed up Zebra shutdown: skip waiting for the tokio runtime
* Create a `LockTime::unlocked` helper constructor
Returns a `LockTime` that is unlocked at the genesis block.
* Return `Option<LockTime>` from `lock_time` method
Prepare to return `None` for when a transaction has its lock time
disabled.
* Return `None` instead of zero `LockTime`
Because a zero lock time means that the transaction was unlocked at the
genesis block, so it was never actually locked.
* Rephrase zero lock time check comment
Clarify that the check is not redundant, and is necessary for the
genesis transaction.
Co-authored-by: teor <teor@riseup.net>
* Add a `transparent::Input::sequence` getter method
Retrieve a transparent input's sequence number.
* Check if lock time is enabled by a sequence number
Validate the consensus rule that the lock time is only enabled if at
least one transparent input has a value different from `u32::MAX` as its
sequence number.
* Add more Zcash specific details to comment
Explain the Zcash specific lock time behaviors.
Co-authored-by: teor <teor@riseup.net>
* Add `time` field to `Request::Block` variant
The block time to use to check if the transaction was unlocked and
allowed to be included in the block.
* Add `Request::block_time` getter
Returns the block time for the block that owns the transaction being
validated or the current time plus a tolerance for mempool transactions.
* Validate transaction lock times
If they are enabled by a transaction's transparent input sequence
numbers, make sure that they are in the past.
* Add comments with consensus rule parts
Make it easier to map what part of the consensus rule each match arm is
responsible for.
Co-authored-by: teor <teor@riseup.net>