* refactor(state): split database access into modules by Zebra types
Also split the genesis block check from the genesis note commitment trees.
* test(db): snapshot column family names
* fix(db): assert that the default column family is empty on open and close
* feat(test): apply `cargo insta` settings to all tests
* doc(db/test): improve test docs
* test(db): snapshot column family data for the empty state
* refactor(db/test): split out the raw database snapshot
* test(db): snapshot raw database for blocks 0-2
* test(db): initial serialized snapshot data for raw database
* test(db): tweak snapshot file names
* test(db): rename snapshots for consistency
* test(db): store empty column families in a single snapshot
* test(db): simplify snapshot files by combining empty snapshots
* doc(db/test): put comment in a better place
* refactor(db): fastmod assert_default_is_empty assert_default_cf_is_empty
* doc(test): explain when insta settings are needed
* fix(state/test): use the network to initialize the state
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
The mempool is only activated once, so we must check for that log first.
After mempool activation, the stop regex is logged at least once.
(It might be logged before as well, but we can't rely on that.)
When checking that the mempool didn't activate,
wait for the `zebrad` command to exit,
then check the entire log.
* Updating zebra-test to v1.0.0-beta.4
* Updating tower-fallback to v0.2.16
* Updating tower-batch to v0.2.20
* Updating zebra-chain to v1.0.0-beta.4
* Updating zebra-script to v1.0.0-beta.4
* Updating zebra-network to v1.0.0-beta.4
* Updating zebra-state to v1.0.0-beta.4
* Updating zebra-consensus to v1.0.0-beta.4
* Updating zebra-utils to v1.0.0-beta.4
* Updating zebrad to v1.0.0-beta.4
* Update Cargo.lock with updated crate versions
* Add section for 1.0.0-beta.4 in the CHANGELOG
* Update CHANGELOG.md
Co-authored-by: teor <teor@riseup.net>
* Update README.md
Co-authored-by: teor <teor@riseup.net>
* Update book/src/user/install.md
Co-authored-by: teor <teor@riseup.net>
* Update release description to mention Rust 2021
Update the changelog to mention that all crates now use Rust 2021
Edition.
* Elaborate on Section 3.6 documentation entry
Add the title of the section and mention that it's a section of the
Zcash protocol specification.
* Make changelog entries consistent
Use the same format for entries related to consensus rule documentation.
Co-authored-by: Marek <mail@marek.onl>
* Update README about validated consensus rules
Zebra now validates all documented consensus rules.
* Add changelog entry for newly merged PR
Describe the security fix in the changelog.
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
* cancel background database tasks in `FinalizedState` destructor
* use `shutdown_timeout()`
* Log info-level messages while waiting for background tasks to shut down
* Cancel background tasks during debug_stop_at_height shutdown
This commit moves the database shutdown code into a common function.
* Create a constant for the tokio timeout
* Add a test script for Zebra shutdown errors
* Increase the shutdown timeout to 20 seconds for slower machines
* add title to building zebra
* use imported duration
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add arti as a zebra-network dependency
* Add a method for isolated anonymised Tor connections to a specific hostname
* Add tests for isolated tor connections
* Use a shared tor client instance for all isolated connections
* Silence a spurious tor warning in tests
* Make tor support optional, activate it via a new "tor" feature
* Extra Cargo.lock changes
* fastmod AsyncReadWrite PeerTransport zebra*
* Remove unnecessary PeerTransport generics
* Refactor common test code into a function
* Don't drop the stream until the end of the test
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Move standard lints into .cargo/config.toml
* Ignore "wrong self convention" in a futures-based trait
This lint might only trigger on beta or nightly at the moment.
* Warn if future incompatibile code is added to Zebra
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Refactor setup of `Connection` test vectors
Add a `new_test_connection` helper function to create a `Connection`
instance that's ready for testing.
* Check that no inbound requests are sent
Return the mock inbound service from `new_test_connection` and assert
that no requests were sent to it in any test.
* Replace `&mut Vec<u8>` with an `mpsc` channel
Make it easier to run the connection task in the background, i.e.,
remove any lifetime constraints from the borrowed buffer so that
`Connection` is `'static`.
It's now also easier to assert on individual messages sent from the
`Connection` instance.
* Make `MockServiceBuilder::finish` public
Allow test functions to be generic when creating a `MockService`, so
that caller functions actually determine if the type of `MockService`
assertions.
* Move `new_test_connection` to parent module
Make it more generic so that it can be used later in property tests as
well.
* Derive `Eq` and `PartialEq` for network `Response`
Allow intercepted `Response` instances to be easily compared in tests.
* Test block request cancel causes an error cascade
This is the scenario that caused the block synchronizer to reset every
few minutes, which made it considerably slower.
* Ignore unexpected block responses
It's likely that it's just a response for a previously cancelled block
request.
* Use `testdir()` instead of `TempDir::new()`
Reduce repeated code and make it easier to change from using `tempdir`
to use `tempfile` instead.
* Replace `tempdir` with `tempfile` in `zebrad`
Use `tempfile`'s `TempDir` instead.
* Use `tempdir()` instead of `TempDir::new()`
Reduce repeated code and make it easier to upgrade to `tempfile`.
* Use `tempfile` instead of `tempdir`
Replace obsoleted `tempdir` dependency with `tempfile`.
* Use `tempfile` instead of `tempdir`
Replace obsoleted `tempdir` dependency with `tempfile`.
* Update `Cargo.lock`
Update it now that `tempdir` has been replaced with `tempfile`.
* Remove `tempdir` from `deny.toml` exceptions
Ban duplicate versions of the `tempdir` dependency.
* Remove `inferno` from `deny.toml` exceptions
It apparently isn't needed anymore.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Refactor so that RetryLimit::Future is std::marker::Sync
* Make the syncer future std::marker::Send by spawning tips futures
* Download synced blocks in chain order, not HashSet order
* Improve MockService failure messages
* Add closure-based responses to the MockService API
* Move MockChainTip to zebra-chain
* Add a MockChainTipSender type alias
* Support MockChainTip in ChainSync and its downloader
* Add syncer tests for obtain tips, extend tips, and wrong block hashes
* Add block too high tests for obtain tips and extend tips
* Add syncer tests for duplicate FindBlocks response hashes
* Allow longer request delays for mocked services in syncer tests
* Move `MockedClientHandle` to `peer` module
It's more closely related to a `Client` than the `PeerSet`, and this
prepares it to be used by other tests.
* Rename `MockedClientHandle` to `ClientTestHarness`
Reduce confusion, and clarify that the client is not mocked.
Co-authored-by: teor <teor@riseup.net>
* Add clarification to `mock_peers` documentation
Explicitly say how the generated data is returned.
* Rename method to `wants_connection_heartbeats`
The `Client` service only represents one direction of a connection, so
`is_connected` is not the exact term.
Co-authored-by: teor <teor@riseup.net>
* Mock `Client` instead of `LoadTrackedClient`
Move where the conversion from mocked `Client` to mocked
`LoadTrackedClient` in order to make the test helper more easily used by
other tests.
* Use `ClientTestHarness` in `initialize` tests
Replace the boilerplate code to create a fake `Client` instance with
usages of the `ClientTestHarness` constructor.
* Allow receiving requests from `Client` instance
Create a helper type to wrap the result, to make it easier to assert on
specific events after trying to receive a request.
* Allow inspecting the current error in the slot
Share the `ErrorSlot` between the `Client` and the handle, so that the
handle can be used to inspect the contents of the `ErrorSlot`.
* Allow placing an error into the `ErrorSlot`
Assuming it is initially empty. If it already has an error, the code
will panic.
* Allow gracefully closing the request receiver
Close the endpoint with the appropriate call to the `close()` method.
* Allow dropping the request receiver endpoint
Forcefully closes the endpoint.
* Rename field to `client_request_receiver`
Also rename the related methods to include
`outbound_client_request_receiver` to make it more precise.
Co-authored-by: teor <teor@riseup.net>
* Allow dropping the heartbeat shutdown receiver
Allows the `Client` to detect that the channel has been closed.
* Rename fn. to `drop_heartbeat_shutdown_receiver`
Make it clear that it affects the heartbeat task.
Co-authored-by: teor <teor@riseup.net>
* Move `NowOrLater` into a new `now-or-later` crate
Make it easily accessible to other crates.
* Add `IsReady` extension trait for `Service`
Simplifies checking if a service is immediately ready to be called.
* Add extension method to check for readiness error
Checks if the `Service` isn't immediately ready because a call to
`ready` immediately returns an error.
* Rename method to `is_failed`
Avoid negated method names.
Co-authored-by: teor <teor@riseup.net>
* Add a `IsReady::is_pending` extension method
Checks if a `Service` is not ready to be called.
* Use `ClientTestHarness` in `Client` test vectors
Reduce repeated code and try to improve readability.
* Create a new `ClientTestHarnessBuilder` type
A builder to create test `Client` instances using mock data which can be
tracked and manipulated through a `ClientTestHarness`.
* Allow configuring the `Client`'s mocked version
Add a `with_version` builder method.
* Use `ClientTestHarnessBuilder` in `PeerVersions`
Use the builder to set the peer version, so that the `version` parameter
can be removed from the constructor later.
* Use a default mock version where possible
Reduce noise when setting up the harness for tests that don't really
care about the remote peer version.
* Remove `Version` parameter from the `build` method
The `with_version` builder method should be used instead.
* Fix some typos and outdated info in the release checklist
* Add extra client tests for zero and multiple readiness checks (#3273)
And document existing tests.
* Replace `NowOrLater` with `futures::poll!` (#3272)
* Replace NowOrLater with the futures::poll! macro in zebrad
* Replace NowOrLater with the futures::poll! macro in zebra-test
* Remove the now-or-later crate
* remove unused imports
* rustfmt
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Do prelim checking of Sprout anchors in non-finalized state
Does not check intra-transaction interstitial states yet
* Populate sprout anchors to allow other state tests to pass
* Preliminary interstitial sprout note commitment tree anchor checks implementation
* Make sure only prior anchors are checked in the same transaction
* Add tests
* Refactor a comment
* Refactor rustdoc
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Use the first `JoinSplit`s from mainnet
* Print debug messages
* Use correct blocks for the tests
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Use a single-thread shared Tokio runtime
This allows it to pause the time and more closely resembles the
environment that's set by default for asynchronous tests.
* Add a `zebra_test::init_async` helper function
Calls `zebra_test::init` but also constructs a single-thread Tokio
runtime and returns it. This makes it simpler to initialize asynchronous
tests that can't use the `#[tokio::test]` attribute.
* Replace usages of `Runtime::new` in tests
Use the new `zebra_test::init_async()` helper function instead.
* Replace `runtime::Builder::new_current_thread()`
Use the new `zebra_test::init_async()` helper function instead.
* Replace `runtime::Builder::new_multi_thread()`
Use the new `zebra_test::init_async()` helper function instead. The test
with the change doesn't necessarily have to use a multi-thread runtime.
Zebra's latest beta continues implementing zero-knowledge proof and note commitment tree validation. In this release, we have finished implementing transaction header, transaction amount, and Zebra-specific NU5 validation. (NU5 mainnet validation is waiting on an `orchard` crate update, and some consensus parameter updates.)
We also fix a number of security issues that could pose a local denial of service risk, or make it easier for an attacker to make a node follow a false chain.
As of this release, Zebra will automatically download and cache the Sprout and Sapling Groth16 circuit parameters. The cache uses around 1 GB of disk space. These cached parameters are shared across all Zebra and `zcashd` instances run by the same user.
See CHANGELOG.md for the full list of changes in this release.
* Add Transaction::sprout_joinsplits()
* Add Anchor variants to ValidateContextError
* Make Chain anchor collections pub(crate)
* tracing::instrument several methods in state
* Add contains_*_anchors methods to FinalizedState
* Add check::anchors module and function
* Verify that anchors_refer_to_earlier_treestates in when updating chains in non-finalized state
* Update zebra-state/src/service/check/anchors.rs
Co-authored-by: teor <teor@riseup.net>
* Add anchors() to sapling::ShieldedData
* Add sapling_anchors() to Transaction
* Use Transaction::sapling_anchors() in the anchors_refer_to_earlier_treestates() check
* Whoops, itertools
* Add a comment for improvement
Co-authored-by: teor <teor@riseup.net>
* Add & use a cfg(test) method on FinalizedState to prep test state with anchors to allow other tests to pass contextual checks
* Allow test nullifier checks to pass by populating anchor sets, allowing test anchor checks to pass
* Add mainnet block 419202 and its sapling note commitment tree root to test vectors
* Test sapling anchor verification using the first few Sapling blocks data
* Correct comment
* assert_eq instead of assert(bool)
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Update zebra-state/src/service/non_finalized_state.rs
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* add testnet test blocks around nu5
* validate coinbase expiration height
* change const name and doc
Co-authored-by: teor <teor@riseup.net>
* change commit location
Co-authored-by: teor <teor@riseup.net>
* use pre Nu5 rules when there is no activation height
* add sapling final root to nu5 test vectors
* fix tests
Co-authored-by: teor <teor@riseup.net>
* Implement incremental note commitment Merkle tree for Sprout
* Add tests for Sprout note commitment tree
* Remove the `Arbitrary` attribute
* Reverse the vector of empty roots
* Add more tests
* Refactor rustdoc
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Refactor rustdoc
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* rustdoc
* Rustdoc
* rustdoc links
* Oops, need the trait in scope to use it
* Avoid accessing the wrapped hash directly
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* rustfmt
* Add typing
* Avoid accessing the wrapped hash directly
* Implement incremental note commitment Merkle tree for Sprout
* Add tests for Sprout note commitment tree
* Remove the `Arbitrary` attribute
* Reverse the vector of empty roots
* Add more tests
* Refactor rustdoc
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Refactor rustdoc
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* rustdoc
* Rustdoc
* rustdoc links
* Oops, need the trait in scope to use it
* Avoid accessing the wrapped hash directly
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* rustfmt
* Add typing
* Avoid accessing the wrapped hash directly
* Add Overwinter final roots (test vectors)
* Test sprout note commitments trees on Overwinter blocks
* Add new test vectors
* Finish the tests for the note commitment trees
* Make the wrapped hash in `Root` private
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* First pass at async Halo2 verification service
Stubs out a batch verifier for the future.
The dependencies for orchard, halo2, librustzcash, zcash_primitives, have
not been resolved.
* Halo2 verifier service and test
* Remove redundant conversion
* Test async halo2 verifier service with pre-computed Orchard shielded data test vectors
* Fix typo
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Assert future result is_ok() in Halo2 verifier test
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Shorten tower::Service trait constraints for Halo2 verifier tests
* Remove commented out trait constraints
* .expect() vs .unwrap() to parse orchard::redpallas::VerificationKey
* Use .to_vec() for some test vectors
* Fix self-referential Display impl
* Fix deps
* Distinguish orchard vs zebra_chain::orchard imports
* Add test that halo2 verifier fails with malformed proof inputs
* Use thiserror for Halo2Error
* Use ZcashFoundation/orchard instead of dconnolly/orchard
* Add a link to the issue to remove the zfnd fork of orchard crate
* Update zebra-consensus/Cargo.toml
Co-authored-by: teor <teor@riseup.net>
* Add note
* Move artificial Orchard shielded data test vectors to zebra-test
* Align brackets
* Tidy some trait constraints and debug statements
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* Cleanup a function that calls zcash_script
* Remove zebra_test::prelude macros that conflict with the Rust prelude
* Add sigops count support to zebra-script
* Check MAX_BLOCK_SIGOPS in the block verifier
* Test MAX_BLOCK_SIGOPS on generated and historic blocks
* Add SAFETY comments for all unsafe zebra-script code
* Explain where the consensus rule comes from
* Remove unused pretty_assertions dependency
* Allow large test block generation functions with the proptest-impl feature
* Replace `as` with `try_into` for integer conversions in unsafe code
* Expand SAFETY comments
* Revert "Remove commented-out code"
This reverts commit 9e69777925f103ee11e5940bba95b896c828839b.
* Implement deserialization for `addrv2` messages
* Limit addr and addrv2 messages to MAX_ADDRS_IN_MESSAGE
* Clarify address version comments
* Minor cleanups and fixes
* Add preallocation tests for AddrV2
* Add serialization tests for AddrV2
* Use prop_assert in AddrV2 proptests
* Use a generic utility method for deserializing IP addresses in `addrv2`
* Document the purpose of a conversion to MetaAddr
* Fix a comment typo, and clarify that comment
* Clarify the unsupported AddrV2 network ID error and enum variant names
```sh
fastmod AddrV2UnimplementedError UnsupportedAddrV2NetworkIdError zebra-network
fastmod Unimplemented Unsupported zebra-network
```
* Fix and clarify unsupported AddrV2 comments
* Replace `panic!` with `unreachable!`
* Clarify a comment about skipping a length check in a test
* Remove a redundant test
* Basic addr (v1) and addrv2 deserialization tests
* Test deserialized IPv4 and IPv6 values in addr messages
* Remove redundant io::Cursor
* Add comments with expected values of address test vectors
* Update `tower` to version `0.4.9`
Update to latest version to add support for Tokio version 1.
* Replace usage of `ServiceExt::ready_and`
It was deprecated in favor of `ServiceExt::ready`.
* Update Tokio dependency to version `1.13.0`
This will break the build because the code isn't ready for the update,
but future commits will fix the issues.
* Replace import of `tokio::stream::StreamExt`
Use `futures::stream::StreamExt` instead, because newer versions of
Tokio don't have the `stream` feature.
* Use `IntervalStream` in `zebra-network`
In newer versions of Tokio `Interval` doesn't implement `Stream`, so the
wrapper types from `tokio-stream` have to be used instead.
* Use `IntervalStream` in `inventory_registry`
In newer versions of Tokio the `Interval` type doesn't implement
`Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used
instead.
* Use `BroadcastStream` in `inventory_registry`
In newer versions of Tokio `broadcast::Receiver` doesn't implement
`Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This
also requires changing the error type that is used.
* Handle `Semaphore::acquire` error in `tower-batch`
Newer versions of Tokio can return an error if the semaphore is closed.
This shouldn't happen in `tower-batch` because the semaphore is never
closed.
* Handle `Semaphore::acquire` error in `zebrad` test
On newer versions of Tokio `Semaphore::acquire` can return an error if
the semaphore is closed. This shouldn't happen in the test because the
semaphore is never closed.
* Update some `zebra-network` dependencies
Use versions compatible with Tokio version 1.
* Upgrade Hyper to version 0.14
Use a version that supports Tokio version 1.
* Update `metrics` dependency to version 0.17
And also update the `metrics-exporter-prometheus` to version 0.6.1.
These updates are to make sure Tokio 1 is supported.
* Use `f64` as the histogram data type
`u64` isn't supported as the histogram data type in newer versions of
`metrics`.
* Update the initialization of the metrics component
Make it compatible with the new version of `metrics`.
* Simplify build version counter
Remove all constants and use the new `metrics::incement_counter!` macro.
* Change metrics output line to match on
The snapshot string isn't included in the newer version of
`metrics-exporter-prometheus`.
* Update `sentry` to version 0.23.0
Use a version compatible with Tokio version 1.
* Remove usage of `TracingIntegration`
This seems to not be available from `sentry-tracing` anymore, so it
needs to be replaced.
* Add sentry layer to tracing initialization
This seems like the replacement for `TracingIntegration`.
* Remove unnecessary conversion
Suggested by a Clippy lint.
* Update Cargo lock file
Apply all of the updates to dependencies.
* Ban duplicate tokio dependencies
Also ban git sources for tokio dependencies.
* Stop allowing sentry-tracing git repository in `deny.toml`
* Allow remaining duplicates after the tokio upgrade
* Use C: drive for CI build output on Windows
GitHub Actions uses a Windows image with two disk drives, and the
default D: drive is smaller than the C: drive. Zebra currently uses a
lot of space to build, so it has to use the C: drive to avoid CI build
failures because of insufficient space.
Co-authored-by: teor <teor@riseup.net>
* Increment the crates that have new commits since the last version
* Increment the crates that depend on crates that have changed
* Increment the version of `zebra-script`
* Use the `zebrad` version in the `zebra-network` user agent string
* Use the `v1.0.0-alpha.19` git tag in `README.md`
* Copy the draft changelog into `CHANGELOG.md`
* Delete bumps
* Update CHANGELOG.md
Co-authored-by: teor <teor@riseup.net>
* Add newly merged PRs
Co-authored-by: teor <teor@riseup.net>
* Get the transaction fee from utxos
* Return the transaction fee from the verifier
* Avoid calculating the fee for coinbase transactions
Coinbase transactions don't have fees. In case of a coinbase transaction, the
verifier returns a zero fee.
* Update the result obtained by `Downloads`
* Update versions for zebra v1.0.0-alpha.18 release
* WIP: Initial PR list
* Remove uninteresting version bumps from CHANGELOG
* Categorise and group PRs in CHANGELOG, removing uninteresting PRs
* Further refine and categorise changelog entries
* Fix tag url
* Final changes to CHANGELOG
* Add a changelog description
* Spacing
* Clarify and fix changelog PR descriptions
* Add PRs that are about to be merged
* More slight clarifications
* Spacing
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
teor
Marek
dependabot[bot]
Janito Vaqueiro Ferreira Filho
Alfredo Garcia
Deirdre Connolly
Conrado Gouvea
Pili Guerra
Dimitris Apostolou