* Refactor so that RetryLimit::Future is std::marker::Sync
* Make the syncer future std::marker::Send by spawning tips futures
* Download synced blocks in chain order, not HashSet order
* Improve MockService failure messages
* Add closure-based responses to the MockService API
* Move MockChainTip to zebra-chain
* Add a MockChainTipSender type alias
* Support MockChainTip in ChainSync and its downloader
* Add syncer tests for obtain tips, extend tips, and wrong block hashes
* Add block too high tests for obtain tips and extend tips
* Add syncer tests for duplicate FindBlocks response hashes
* Allow longer request delays for mocked services in syncer tests
* Add mergify merging queues
* Fix Mergify invalid configuration
* Improve adaptability to the actual workflow
Do not merge if the pull-request test is not green. Do not move draft PRs to the queue. And update keep all open PRs updated.
* Fix a typo on check-success condition
* Stop holding completed messages until the next inbound message
* Add more info to network message block download debug logs
* Simplify address metrics logs
* Try handling inbound messages as responses, then try as a new request
* Improve address book logging
* Fix a race between the first heartbeat and getaddr requests
* Temporarily reduce the getaddr fanout to 1
* Update metrics when exiting the Connection run loop
* Downgrade some debug logs to trace
* retry download parameters a few times before giving up
* separate sapling and sprout params download to methods
* simpify the new created methods
* Assert the parameters were downloaded to the expected paths
Co-authored-by: teor <teor@riseup.net>
* create a test for message broadcast to peers
* remove dbg comment
* use `prop_assert_eq`
* use a random number of peers with constant version
* fix some docs
* check total peers is equal to active peers
* increase coverage
Co-authored-by: teor <teor@riseup.net>
* Ensure that sapling::keys::TransmissionKey jubjub point is always in the prime order group
* Adjust TransmissionKey check; add AuthorizingKey check
* Move ValueCommitment small order check to deserialization
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Use is_torsion_free() instead of is_identity() and is_prime_order()
* Add EphemeralPublicKey small order check on instantiation; remove old checks
* Use VerificationKey instead of VerificationKeyBytes; fix tests
* Use ValidatingKey instead of VerificationKeyBytes for rk
* Reject identity when creating an Orchard EphemeralPublicKey
* Make documentation more consistent, use generator in tests
* s/JubJub/Jubjub/
* Fix zebra-consensus tests (insert_fake_orchard_shielded_data)
* Create NotSmallOrderValueCommitment, since intermediate values can be the identity
* Clarify documentation
* rustdoc
* Tidy rustdoc
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
* Draft design for lightwalletd database support
* Explain how to create blocks from headers and transactions
* Make keys smaller to improve database performance
* Support transparent address balances and UTXOs
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Add a copy-state command, which copies blocks between two state services
* Check blocks were written correctly
* Add extra logging to debug shutdown
* Add a block height limit argument
* Let the target state start from any height
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Refactor to create heartbeat sender function
Move the code that's part of the heartbeat task into a separate helper
function.
* Move `Client` initialization down
Keep it closer to where it's actually used, and make it easier to add
new fields to `Client` for the connection and heartbeat tasks.
* Add background task handles to `Client` type
Prepare it to be able to check for panics or errors from the background
tasks.
* Add dummy background tasks to `ClientTestHarness`
Spawn simple timeout tasks as mock connection and heartbeat tasks.
* Fix `PeerSet` tests that use `ClientTestHarness`
Building a `ClientTestHarness` requires a Tokio runtime to be set up, so
the calls were moved into the `async` block.
* Refactor to create `set_task_exited_error`
Make the code reusable for both background tasks.
* Check heartbeat task for errors
Periodically poll it to check if the task has unexpectedly stopped.
* Check if connection background task has stopped
The client service should stop if the connection background task has
exited, because then it's not able to receive any replies.
* Allow aborting mocked `Client` background tasks
Wrap the background tasks in `Abortable`, so that they can be aborted
through the `ClientTestHarness`.
* Test if stopped connection task is detected
Check that stopping the background connection task is something that the
`Client` instance detects and handles correctly.
* Test if stopped heartbeat task is detected
Check that stopping the background heartbeat task is something that the
`Client` instance detects and handles correctly.
* Allow setting custom background tasks
Will be used later to create background tasks that panic.
* Test if `Client` handles panics in connection task
Use a mock background connection task that panics immediately, and check
that the `Client` handles it gracefully.
* Test if `Client` handles panics in heartbeat task
Use a mock background heartbeat task that panics immediately, and check
that the `Client` handles it gracefully.
* Change ticket referenced by `TODO`
The previously linked issue was a broad plan to improve Zebra's shutdown
behavior, while the new issue is more specific, and can be scheduled
sooner.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* Re-enable code coverage comments on PRs
* Post a coverage comment, even if coverage hasn't changed
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Move `MockedClientHandle` to `peer` module
It's more closely related to a `Client` than the `PeerSet`, and this
prepares it to be used by other tests.
* Rename `MockedClientHandle` to `ClientTestHarness`
Reduce confusion, and clarify that the client is not mocked.
Co-authored-by: teor <teor@riseup.net>
* Add clarification to `mock_peers` documentation
Explicitly say how the generated data is returned.
* Rename method to `wants_connection_heartbeats`
The `Client` service only represents one direction of a connection, so
`is_connected` is not the exact term.
Co-authored-by: teor <teor@riseup.net>
* Mock `Client` instead of `LoadTrackedClient`
Move where the conversion from mocked `Client` to mocked
`LoadTrackedClient` in order to make the test helper more easily used by
other tests.
* Use `ClientTestHarness` in `initialize` tests
Replace the boilerplate code to create a fake `Client` instance with
usages of the `ClientTestHarness` constructor.
* Allow receiving requests from `Client` instance
Create a helper type to wrap the result, to make it easier to assert on
specific events after trying to receive a request.
* Allow inspecting the current error in the slot
Share the `ErrorSlot` between the `Client` and the handle, so that the
handle can be used to inspect the contents of the `ErrorSlot`.
* Allow placing an error into the `ErrorSlot`
Assuming it is initially empty. If it already has an error, the code
will panic.
* Allow gracefully closing the request receiver
Close the endpoint with the appropriate call to the `close()` method.
* Allow dropping the request receiver endpoint
Forcefully closes the endpoint.
* Rename field to `client_request_receiver`
Also rename the related methods to include
`outbound_client_request_receiver` to make it more precise.
Co-authored-by: teor <teor@riseup.net>
* Allow dropping the heartbeat shutdown receiver
Allows the `Client` to detect that the channel has been closed.
* Rename fn. to `drop_heartbeat_shutdown_receiver`
Make it clear that it affects the heartbeat task.
Co-authored-by: teor <teor@riseup.net>
* Move `NowOrLater` into a new `now-or-later` crate
Make it easily accessible to other crates.
* Add `IsReady` extension trait for `Service`
Simplifies checking if a service is immediately ready to be called.
* Add extension method to check for readiness error
Checks if the `Service` isn't immediately ready because a call to
`ready` immediately returns an error.
* Rename method to `is_failed`
Avoid negated method names.
Co-authored-by: teor <teor@riseup.net>
* Add a `IsReady::is_pending` extension method
Checks if a `Service` is not ready to be called.
* Use `ClientTestHarness` in `Client` test vectors
Reduce repeated code and try to improve readability.
* Create a new `ClientTestHarnessBuilder` type
A builder to create test `Client` instances using mock data which can be
tracked and manipulated through a `ClientTestHarness`.
* Allow configuring the `Client`'s mocked version
Add a `with_version` builder method.
* Use `ClientTestHarnessBuilder` in `PeerVersions`
Use the builder to set the peer version, so that the `version` parameter
can be removed from the constructor later.
* Use a default mock version where possible
Reduce noise when setting up the harness for tests that don't really
care about the remote peer version.
* Remove `Version` parameter from the `build` method
The `with_version` builder method should be used instead.
* Fix some typos and outdated info in the release checklist
* Add extra client tests for zero and multiple readiness checks (#3273)
And document existing tests.
* Replace `NowOrLater` with `futures::poll!` (#3272)
* Replace NowOrLater with the futures::poll! macro in zebrad
* Replace NowOrLater with the futures::poll! macro in zebra-test
* Remove the now-or-later crate
* remove unused imports
* rustfmt
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Revert "Stop ignoring some completed Responses"
This reverts commit 0383562e10 from PR #3120,
but keeps the metrics and logging changes since that commit.
* Document why the request handling needs to happen in this order
* Justify that the ErrorSlot Mutex is deadlock-safe
* Document cancellation safety in the async RFC
* Document task starvation in the async RFC
Co-authored-by: Marek <mail@marek.onl>
* Tweak crawler timings so peers are more likely to be available
* Tweak min peer connection interval so we try all peers
* Let other tasks run between fanouts, so we're more likely to choose different peers
* Let other tasks run between retries, so we're more likely to choose different peers
* Let other tasks run after peer crawler DemandDrop
This makes it more likely that peers will become ready.
* Spawn the address book updater on a blocking thread
* Spawn CandidateSet address book operations on blocking threads
* Replace the PeerSet address book with a metrics watch channel
* Fix comment
* Await spawned address book tasks
* Run the address book update tasks concurrently (except for the mutex)
* Explain an internal-only method better
* Fix a typo
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Create a strategy for block heights after Sapling
Provides an arbitrary network (mainnet or testnet) and a block height
between the Sapling activation height on that network and the maximum
block height.
* Create a helper function to select block heights
Allows generating block heights inside a range using a scale factor
between 0 and 1.
* Allow specifying the outpoint index for mock UTXOs
Avoid creating multiple transparent transfers in the same transaction
with the same source UTXO, which would lead to a double spend.
* Create helper function to mock multiple transfers
Given relative block height scale factors, create a mock transparent
transfer for each one of them.
Also add a constant that serves as a guideline for the maximum number of
transparent transfers to mock.
* Create helper function to sanitize tx. version
Make sure the arbitrary transaction version is valid for the network
(testnet or mainnet) at the specified block height.
* Create `mock_transparent_transaction` helper func.
Creates a V4 or V5 mock transaction that only includes transparent
inputs and outputs.
* Create helper function for transaction validation
Performs the actual tested action of verifying a transaction. It sets up
the verifier and uses it to obtain the verification result.
* Test if zero lock time means unlocked
Generate arbitrary transactions with zero lock time, and check that they
are accepted by the transaction verifier.
* Allow changing the sequence number of an input
Add a setter method for a `transparent::Input`'s sequence number. This
setter is only available for testing.
* Test if sequence numbers can disable lock time
Create arbitrary transactions and set the sequence numbers of all of its
inputs to `u32::MAX` to see if that disables the lock time and the
transactions are accepted by the verifier.
* Test block height lock times
Make sure that the transaction verifier rejects transactions that are
still locked at a certain block height.
* Test block time lock times
Test that the transaction verifier rejects a transaction that is
validated at a block time that's before the transaction's lock time.
* Test unlocking by block height
Test that transactions unlocked at an earlier block height are accepted
by the transaction verifier.
* Test transactions unlocked by the block time
Test that transactions that were unlocked at a previous block time are
accepted by the transaction verifier.
* Fix an incorrect method comment
Co-authored-by: teor <teor@riseup.net>
* Tweak crawler timings so peers are more likely to be available
* Tweak min peer connection interval so we try all peers
* Let other tasks run between fanouts, so we're more likely to choose different peers
* Let other tasks run between retries, so we're more likely to choose different peers
* Let other tasks run after peer crawler DemandDrop
This makes it more likely that peers will become ready.
* Document the chain verifier
* Drop gossiped blocks that are too far ahead of the tip
* Add extra gossiped block metrics
* Allow extra gossiped blocks, now we have a stricter limit
* Fix a comment
* Check the exact number of blocks in a downloaded block response
* Drop synced blocks that are too far ahead of the tip
* Add extra synced block metrics
* Test dropping gossiped blocks that are too far ahead of the tip
* Allow an extra checkpoint's worth of blocks in the verifier queues
* Actually let's try two extra checkpoints
* Scale extra height limit with lookahead limit
* Also drop blocks that are behind the finalized tip
* Downgrade a noisy log
* Use a debug log for already verified gossiped blocks
* Use debug logs for already verified synced blocks
* Drop peer services if their cancel handles are dropped
* Exit the client task if the heartbeat task exits
* Allow multiple errors on a connection without panicking
* Explain why we don't need to send an error when the request is cancelled
* Document connection fields
* Make sure connections don't hang due to spurious timer or channel usage
* Actually shut down the client when the heartbeat task exits
* Add tests for unready services
* Close all senders to peer when `Client` is dropped
* Return a Client error if the error slot has an error
* Add tests for peer Client service errors
* Make Client drop and error cleanups consistent
* Use a ClientDropped error when the Client struct is dropped
* Test channel and error state in peer Client tests
* Move all Connection cleanup into a single method
* Add tests for Connection
* fix typo in comment
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Use a named CancelHeartbeatTask unit struct for the channel type
* Prefer cancel handles in selects, if both are ready
* Fix message metrics to just show the command name
* Add metrics for internal requests and responses
* Add internal requests and responses to the messages dashboard
* Add a canceled metric, and peer addresses to request and response metrics
* Add a canceled messages graph
* Add connection state metrics for currently open connections
* Fix the connection state graph with new metrics
* Always send an error before dropping pending responses
* Move error detail logging into `fail_with`
* Delete an unused timer future
* Make error strings in metrics less verbose
* Downgrade some error logs to info
* Remove a redundant expect
* Avoid unnecessary allocations for connection state metrics
* Fix missed updates to mempool and block gossip metrics
* Integrate JoinSplit verifier with transaction verifier
* Add test with malformed Groth16 Output proof
* Use TryFrom instead of From in ItemWrapper to correctly propagate malformed proof errors
* Simplify by removing ItemWrapper and directly TryFrom into Item
* Fix existing tests to work with JoinSplit validation
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
ZIP-201 describes how a Zcash node should behave when it reaches a
network upgrade activation height. Zebra doesn't implement all the
details specified there, so we need to document what it does implement
and what it doesn't and why.