* add a test that call all the lightwalletd grpc methods
* fix some docs
* use ZF address in tests for balance and utxos
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make the lightwalletd integration test take a test type
* Configure lightwalletd tests based on the test type
* Remove obsolete kill_on_error() in the lightwalletd test
* Refactor to simplify the test function
* Move LightwalletdTestType to the lightwalletd module
* Create a test function that runs the full lightwalletd test suite
* Actually use the cached Zebra state
* Add checks for the new integration test modes
* Populate the lightwalletd state dir in the FullSyncFromGenesis test
* Fix up state handling, fail earlier if state is invalid
* Adjust timeouts and regex escapes
* Make state requirements for each test stricter
* Move configs to the top of the test function
* Allow unexpected lightwalletd cached state in some tests
* Speed up tests slightly by removing an intermittent log check
* Move timeout selection into test type methods
* Move failure messages into test type methods
* Turn a function argument into an enum field
* Check lightwalletd state directly, rather than Zebra RPC results
* Update gRPC tests for function argument changes
* Remove duplicate env var constant and redundant code
* Export the `zebra_state::Config::db_path` method
Make it easier for tests to discover the sub-directory used to store
the chain state data.
* Generate code for interfacing with lightwalletd
Use the `tonic-build` crate to generate Rust code for communicating with
lightwalletd using gRPC.
The `*.proto` files were obtained from the Zcash lightwalletd
repository.
* Use `block::Height` instead of `Height`
Import the `block` instead to make it slightly clearer.
* Add helper function to remove a file if it exists
Try to remove it and ignore an error if it says that the file doesn't
exist. This will be used later to remove the lock file from a copied
chain state directory.
* Add helper function to copy chain state dirs
Copy an existing chain state directory into a new temporary directory.
* Add a `BoxStateService` type alias
Make it easier to write and read a boxed version of a state service.
* Add a helper function to start the state service
Make it easier to specify the state service to use an existing state
cache directory.
* Import `eyre!` macro at the module level
Allow it to be used in different places without having to repeat the
imports.
* Add `load_tip_height_from_state_directory` helper
A function to discover the current chain tip height stored in a state
cache.
* Add helper function to prepare partial sync. state
Loads a partially synchronized cached state directory into a temporary
directory that can be used by a zebrad instance, and also returns the
chain tip block height of that state.
* Add `perform_full_sync_starting_from` helper
Runs a zebrad with an existing partially synchronized state, and
finishes synchronizing it to the network chain tip.
* Add function to load transactions from a block
Use a provided state service to load all transactions from a block at a
specified height.
The state service is a generic type parameter, because
`zebra_state::service::ReadStateService` is not exported publicly. Using
a generic type parameter also allows the service to be wrapped in layers
if needed in the future.
* Add `load_transactions_from_block_after` helper
A function to load transactions from a block stored in a cached state
directory. The cached state must be synchronized to a chain tip higher
than the requested height.
* Add helper function to load some test transactions
Given a partially synchronized chain state, it will extend that chain by
performing a full synchronization, and obtain some transactions from one
of the newly added blocks.
* Update `spawn_zebrad_for_rpc_without_initial_peers`
Wait until the mempool is activated.
* Add method to start lightwalletd with RPC server
Returns the lightwalletd instance and the port that it's listening for
RPC connections.
The instance can reuse an existing cached lightwalletd state if the
`LIGHTWALLETD_DATA_DIR` environment variable is set.
* Add a `LightwalletdRpcClient` type alias
To make it easier to identify the type generated from the Protobuf
files.
* Add helper function to connect to lightwalletd
Prepare an RPC client to send requests to a lightwalletd instance.
* Add a `prepare_send_transaction_request` helper
Creates a request message for lightwalletd to send a transaction.
* Add test to send transactions using lightwalletd
Obtain some valid transactions from future blocks and try to send them
to a lightwalletd instance connected to a zebrad instance that hasn't
seen those transactions yet. The transactions should be successfully
queued in Zebra's mempool.
* Make `zebra_directory` parameter generic
Allow using a `TempDir` or a `PathBuf`.
* Move lightwalletd protobuf files
Place them closer to the module directory, so that it's clearer that
they specify the RPC protocol for lightwalletd, and not Zebra itself.
* Don't use coinbase transactions in the test
Coinbase transactions are rejected by the mempool.
* Don't remove state lock file
It is removed automatically by Zebra when it shuts down, so if it exists
it should be reported as a bug.
* Force mempool to be enabled in Zebrad instance
Speed up the initialization of the Zebrad instance used for lightwalletd
to connect to.
* Refactor to create `LIGHTWALLETD_DATA_DIR_VAR`
Document how the environment variable can be used to speed up the test.
* Check for process errors in spawned Zebra instance
Enable checking for known process failure messages.
* Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant
Document why it exists and how the choice of the value affects the test.
* Add `LIGHTWALLETD_TEST_TIMEOUT` constant
And use it for the Zebrad and the Lightwalletd instances used in the
send transaction integration test.
* Check `lightwalletd` process for errors
Enable checking the lightwalletd process for known failure messages.
* Update `tonic` and `prost` dependencies
Use the latest version and fix CI failures because `rustfmt` isn't
installed in the build environment.
* Create `send_transaction_test` module
Move the send transaction using lightwalletd test and its helper
functions into a new module.
* Move `LIGHTWALLETD_TEST_TIMEOUT` constant
Place it in the parent `lightwalletd` module.
* Move gRPC helper functions and types to `rpc` mod.
Make them more accessible so that they can be used by other tests.
* Create a `cached_state` module
Move the test utility functions related to using a cached Zebra state
into the module.
* Move `perform_full_sync_starting_from` to `sync`
Keep to closer to the synchronization utility functions.
* Move Zebra cached state path variable constant
Place it in the `cached_state` module.
* Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set
Make it part of the set of tests ignored as a whole if no lightwalletd
tests should be executed.
* Move `spawn_zebrad_for_rpc_without_initial_peers`
Place it in the `launch` sub-module.
* Rename `rpc` module into `wallet_grpc`
Avoid any potential misunderstandings when the name is seen out of
context.
* Allow duplicate `heck` dependency
At least until `structopt` is updated or `zebra-utils` is updated to use
`clap` 3.
* Fix a deny.toml typo
* fix(build): CMake is required by `prost` crate
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Increase the block download retry limit to 3
* Remove an obsolete timing check from the tests
We cancel all block downloads for each sync restart,
so we don't need to wait for the maximum hedged download time.
* Join similar imports
Avoid the confusion that might cause one to think that they come from
different modules or crates.
* Create an `Arguments` helper type
A type to keep track of a list of arguments for a sub-process. It makes
it easier for overriding parameters with new values.
* Create an `args!` helper macro
Make it simpler to create `Arguments` instances with known values.
* Require `Arguments` for `spawn_child` method
Change the method to have an `Arguments` parameter, and merge it with
some default values before passing them forward.
* Use `Arguments` in `spawn_lightwalletd_child`
Change the method to use an `Arguments` instance, and merge it with some
default options.
* Use `Arguments` in `spawn_child_with_command`
Require an `Arguments` instance in the `spawn_child_with_command`
extension method. Makes it simpler to call from `spawn_child` and
`spawn_lightwalletd_child` extension methods.
* Test if argument order is preserved
Check that when building an `Arguments` instance, the order that the
arguments are set is preserved in the generated list of strings.
* Refactor test to improve readability
Also separates some common code to be reused by later tests.
* Test overriding arguments
Check to see if overriding arguments behaves as expected, by keeping the
argument order when overriding and not introducing duplicates.
* Refactor test to improve readability
Move out a chunk of code so that the test itself is easier to read and
to make that code reusable by a later test.
* Test that `Arguments` instances can be merged
Merge two `Arguments` instances built from two lists of arguments, and
check that the expanded strings preserve order and override rules.
* Add Eq derives on Arguments
Co-authored-by: teor <teor@riseup.net>
* Update changelog for 1.0.0-beta.8
* Bump Zebra crate versions
The tower crates and their dependencies haven't changed,
so their versions don't need to be bumped.
Some Zebra crates haven't changed,
but I bumped all the versions to keep them consistent.
* Explicitly say that we support NU5 testnet 2
* Expand PR #3799 into changelog entries
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add a rpc queue
* Implement the rpc queue
* Add rpc queue tests
* Remove mutex, use broadcast channel
* Have order and limit in the queue
* fix multiple transactions channel
* Use a network argument
* Use chain tip to calculate block spacing
* Add extra time
* Finalize the state check test
* Add a retry test
* Fix description
* fix some docs
* add additional empty check to `Runner::run`
* remove non used method
* ignore some errors
* fix some docs
* add a panic checker to the queue
* add missing file changes for panic checker
* skip checks and retries if height has not changed
* change constants
* reduce the number of queue test cases
* remove suggestion
* change best tip check
* fix(rpc): Check for panics in the transaction queue (#4046)
* Check for panics in the RPC transaction queue
* Add missing pin! and abort in the start task
* Check for transaction queue panics in tests
* Fixup a new RPC test from the main branch
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Update changelog for v1.0.0-beta.7
* Increment all crate versions
* Remove redundant release test that is now covered by CI
* Remove completed NU5 README check task from the release template
* Add Merge Freeze tool to the release checklist
* Simplify release checklist by removing unused steps
* Return error code -8 for missing blocks, like zcashd does
* Test that getblock returns error code -8 for missing blocks
* Make RegexSet failures easier to read in logs
* Update lightwalletd acceptance tests for log message changes
* Add extra failure strings for lightwalletd logs
* Improve link formatting
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Make proptest-impl features consistently depend on each other
* Use a non-test method for consensus checks
* Make proptest dependencies the same between crates
* Repoint zebra image links to our new zfnd.org site for now
* Remove images/
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
- use info level, there is nothing the user needs to do,
particularly for a single error
- explain that the errors are temporary
- hide backtraces, because they look like crashes
* Implement `getblockchaininfo` RPC method
* add a test for `get_blockchain_info`
* fix tohex/fromhex
* move comment
* Update lightwalletd acceptance test for getblockchaininfo RPC (#3914)
* change(rpc): Return getblockchaininfo network upgrades in height order (#3915)
* Update lightwalletd acceptance test for getblockchaininfo RPC
* Update some doc comments for network upgrades
* List network upgrades in order in the getblockchaininfo RPC
Also:
- Use a constant for the "missing consensus branch ID" RPC value
- Simplify fetching consensus branch IDs
- Make RPC type derives consistent
- Update RPC type documentation
* Make RPC type derives consistent
* Fix a confusing test comment
* get hashand height at the same time
* fix estimated_height
* fix lint
* add extra check
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix typo
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* split test
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix(rpc): ignore an expected error in the RPC acceptance tests (#3961)
* Add ignored regexes to test command failure regex methods
* Ignore empty chain error in getblockchaininfo
We expect this error when zebrad starts up with an empty state.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make command test matching code accept generic regexes
And add generic conversions to regexes.
* Document test command structs
* Support matching multiple regexes internally in the test command
* Make it easier to call the generic regex methods
* Add a missing API usage comment
* Fix a potential hang in test child error reports
* Revert Option<Child> process handling
* Revert "Revert Option<Child> process handling"
This reverts commit 2af30086858d104dcb0ec87383996c36bcaa7371.
* Add a set of failure regexes to test command output
* Allow debug-printing TestChild again
* When the child is dropped, check any remaining output
* Document a wait_with_output edge case
* Improve failure regex panic output
* Improve builder ergonomics
* Add internal tests for failure regex panics
It would be easy to disable these panics, and never realise.
* Add some module structure TODOs
* Stop panicking if the child process has already been taken
* Add test APIs for consuming child output lines
* Fix a hang on child process drop
* Handle output being already taken in wait_with_output
And document some edge cases we don't handle yet
* Use bash's read command in the TestChild stderr test
And check the actual command we're using to see if it errors.
* Pretty print full failure regex list
* Add the test child command line to the failure regex logs
* Simplify the sync_until test code
* Check logs for failure messages in lightwalletd integration tests
* Add more TODOs and improve comments
* Allow missing branch ID 00000000 provided by zcashd
* Revert "Revert Option<Child> process handling"
This reverts commit 2af30086858d104dcb0ec87383996c36bcaa7371.
* Add a set of failure regexes to test command output
* Allow debug-printing TestChild again
* When the child is dropped, check any remaining output
* Document a wait_with_output edge case
* Improve failure regex panic output
* Improve builder ergonomics
* Add internal tests for failure regex panics
It would be easy to disable these panics, and never realise.
* Add some module structure TODOs
* Stop panicking if the child process has already been taken
* Add test APIs for consuming child output lines
* Fix a hang on child process drop
* Handle output being already taken in wait_with_output
And document some edge cases we don't handle yet
* Use bash's read command in the TestChild stderr test
And check the actual command we're using to see if it errors.
* Pretty print full failure regex list
* Add the test child command line to the failure regex logs
* Refactor to split `ChainSync::sync` method in two
Replace the use of loop labels and `continue` for control flow, and use
early return from a separate method instead. This also allows removing
the `started_once` flag.
* Refactor to create `handle_block_response` helper
Reduce duplicate code and make the main synchronization methods a little
more concise to improve readability.
* Only cancel downloads in case of error
Leave active downloads running if the tips have been exhausted, because
it could have reached the chain tip.
* Add the ChainTip and Network to RpcImpl
* Add the new RpcImpl fields to the tests
* Simplify RPC version field using generics
* Temporarily allow unused struct fields
* Use the read-only state service for RPCs
* Refactor non-finalized block lookup into Chain
* Implement the read-only state block request
* Drop the Chain watch channel lock before accessing the finalized state
* feat(rpc): implement `getrawmempool` rpc method
* tests(rpc): start implementing a test
* Return a `Vec<String>` from `get_raw_mempool`
Ensure the returned type is serialized as an array of hexadecimal
strings.
* Make `getrawmempool` test a property test
Test it with random sets of transaction IDs.
* Fix code formatting inside `proptest!`
Run `rustfmt` inside the `proptest!` block.
* tests(rpc): fix lightwalletd test
* fix(rpc): remove non used return type
* tests(rpc): remove assert
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Allow extra time for lightwalletd integration tests
* Wait for a successful block ingestion in lightwalletd integration
* Check for any ingestor block, not just block 1
* Add state service module docs and cleanup
* Move and add finalized state methods
* Add chain and non-finalized state methods
* Cleanup methods and imports
* Create a ReadStateService type
* Add a stub service implementation
* Add a TODO
* Update ReadStateService request stubs with RPC names and tickets
* Documentation updates
* Make RPC State generic bounds accept a buffered state and a read-only state
* Doc updates
* Add missing proptest-impl feature in RPC dev dependencies
* feature(rpc): start adding a `getblock` method
* fix(rpc): replace oneshot
* fix(rpc): replace a panic with error
* fix(rpc): fix test
* feature(rpc): add hex to response
* refactor(rpc): use generic instead of alias
* docs(rpc): improve docs for getblock method
* test(rpc): add a test for getblock method
* deps(rpc): remove non needed tower features
Co-authored-by: teor <teor@riseup.net>
* docs(rpc): add a note to getblock doc
* refactor(rpc): replace alias
* fix(rpc): use `zcash_serialize_to_vec()` instead of logging format
* tests(rpc): add network argument to `populated_state()`
* refactor(rpc): use an error for state service readiness
* fix(rpc): add parameter
* fix(rpc): clippy
* nit(rpc): remove new line from imports
* fix(rpc): remove commented code
* fix(rpc): simplify error
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Use a `SerializedBlock` type to help serializing blocks (#3725)
* Create a `SerializedBlock` helper type
Create a type that can be used as a byte slice, but is guaranteed to
represent a valid block.
* Use `into_iter` instead of `iter`
There's no need to borrow the elements, they can be moved out directly.
This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`,
so a `SerializedBlock` can't be built directly from an `&Arc<Block>`.
* Use `SerializedBlock` in `GetBlock`
Make the type stricter to avoid storing possibly invalid values. The
bytes are still serialized as a hexadecimal string, through the usage of
`hex`.
The `serde::Deserialize` can't be derived because `hex` requires the
type to also implement `FromHex`.
* feature(rpc): add suggestions from code review
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* tests(rpc): make sure mempool has no requests in get_block test
* fix(rpc): change height argument type in getblock method
* fix(rpc): rustfmt
* fix(rpc): replace panic
* fix(rpc): change getblock response
* fix(rpc): fix lightwalletd test
* tests(rpc): add a getblock error test
* fix(rpc): try another regex
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Stub `sendrawtransaction` RPC method
Register the RPC method, and stub an implementation that currently just
panics. The method has a single `String` parameter with the hexadecimal
string of the raw transaction's bytes and returns a
`SentTransactionHash` wrapper type that's just a hexadecimal `String` of
the sent transaction's hash.
* Add mempool service instance to `RpcImpl`
Use a type parameter to represent the mempool service using the
interface defined by `zebra-node-services`.
* Update test vector to use a mock mempool service
Update the test to be compatible with the changes to `RpcImpl`. The mock
mempool service is expected to not be used during the test.
* Use a `tower::Buffer` for the mempool service
Make it simpler to send requests to the service in a concurrent manner.
* Return a `Future` from `send_raw_transaction`
Make the call asynchronous.
* Implement `sendrawtransaction` RPC
Deserialize the transaction and send it to be queued for verification
and subsequent inclusion in the mempool.
* Test if mempool receives sent raw transaction
Use a mock service as the mempool service and check that it receives a
sent raw transaction.
* Test using non-hexadecimal string parameter
The method should return an error.
* Test with bytes that fail deserialization
Check that the method returns an invalid parameters error if the input
can't be deserialized as a `Transaction`.
* Test if mempool errors are forwarded to caller
Mempool service errors should be sent back to the remote caller as
server errors.
* Test transactions rejected by the mempool service
Transactions that are rejected by the mempool service should result in
a server error being sent to the caller.
* Improve error message
Add the word "structurally" to make it clear that the issue is in the
transaction's deserialization.
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Add note regarding missing `allowhighfees` param.
The parameter isn't supported yet because `lightwalletd` doesn't use it.
* Update the documentation to be consistent
Follow the convention adopted by the `get_info` RPC method.
* Implement `ToHex` and `FromHex` for `Hash`
Make it easier to generate hexadecimal strings from `transaction::Hash`
instances.
* Use `ToHex` in `Debug` and `Display`
Reduce repeated code.
* Refactor to add `bytes_in_display_order` method
Use it to remove repeated code and improve clarity a bit.
* Use `hex::serialize` to serialize transaction hash
Make the type stricter in its contents, while still serializing the
transaction has as a hexadecimal string.
* Simplify serialization attribute
Deserialization should also use `hex::deserialize`, so using the shorter
attribute makes things easier to read and more future proof.
* Update zebra-chain/src/transaction/hash.rs
* Remove unnecessary lifetime
The anonymous lifetime is automatically inferred by the compiler.
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Sometimes lightwalletd goes straight to "Another refresh in progress",
without logging "Mempool refresh error" first.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Upgrade some dependencies
* Upgrade some dependencies
* Upgrade dependencies for zebrad
* Upgrade tracing dependencies
* Revert `tor` & `arti`
* Upgrade `criterion` & `pin-project` in `deny.toml`
* Remove some dependencies from `skip-tree` in `deny.toml`
* Revert some the versions of dependencies because of duplicates
* Revert proptest regressions
* Upgrade dependencies, then ignore some more duplicates (#3716)
* feat(actions)!: add full sync test (#3582)
* add(tests): full sync test
* fix(test): add build
* fix(deploy): escape double dashes '--' correctly
* fix(test): remove unexpected --no-capture arg
error: Found argument '--nocapture' which wasn't expected, or isn't valid in this context
* refactor(docker): use default executable as entrypoint
* refactor(startup): add a custom entrypoint
* fix(test): add missing TEST_FULL_SYNC variable
* test(timeout): use the biggest machine
* fix
* fix(deploy): use latest successful image
* typo
* refactor(docker): generate config file at startup
* revert(build): changes were made to docker
* fix(docker): send variables correctly to the entrypoint
* test different conf file approach
* fix(env): add RUN_TEST env variable
* ref: use previous approach
* fix(color): use environment variable
* fix(resources): use our normal machine size
* fix(ci): double CPU and RAM for full sync test
* fix(test): check for zebrad test output in the correct order
The mempool is only activated once, so we must check for that log first.
After mempool activation, the stop regex is logged at least once.
(It might be logged before as well, but we can't rely on that.)
When checking that the mempool didn't activate,
wait for the `zebrad` command to exit,
then check the entire log.
* fix(ci): run full sync test with full compiler optimisations
* fix(tests): reintroduce tests and run full sync on approval
* fix(tests): reduce the changelog
Co-authored-by: teor <teor@riseup.net>
* fix(ci): update CI job path triggers (#3692)
* ci(test): re-run tests when snapshot data changes
* fix(ci): rebuild state when disk format changes
* fix(ci): rebuild rust docs when code or dependencies change
* doc(ci): explain why we run jobs when files change
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(build): use the right multistage target (#3700)
* fix(review): only assign one reviewer to general Rust reviews (#3708)
If we assign two teams, GitHub assigns two reviewers.
* fix(ci): change the color-eyre ignore to a tracing-subscriber ignore
* fix(ci): ignore duplicate darling dependencies
* doc(ci): remove an alternative resolution doc
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
dependabot[bot]
Alfredo Garcia
teor
Janito Vaqueiro Ferreira Filho
Conrado Gouvea
Dimitris Apostolou
Deirdre Connolly
Marek