Commit Graph

161 Commits

Author SHA1 Message Date
Marek 3b839b7976
Warn the user when Zebra doesn't recognize the format of `zebrad.toml` (#4689)
* Warn the user when Zebra cannot parse `zebrad.toml`

* Test that Zebra warns the user when it cannot parse `zebrad.toml`

* Fix up a mistaken merge change

* Suggest how to fix `zebrad.toml` when Zebra cannot parse it

Co-authored-by: teor <teor@riseup.net>
2022-06-28 00:36:36 +00:00
teor d37d8aa8c6
fix(ci): Ignore lightwalletd hangs for now (#4663)
* Ignore lightwalletd hangs for now

* cargo fmt --all

* cargo +stable fmt --all

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-27 19:50:47 +00:00
Alfredo Garcia 83aa42e649
tests(config): Add tests for old configs (#4676)
* change `initial_mainnet_peers` and `initial_testnet_peers` type to `IndexSet`

* add tests for zebra config files

* add serde feature to indexmap

* remove async

* update config

* fix `stored_config_path()`

* skip tests if config is not found

* improve error

* use CARGO_MANIFEST_DIR

* remove `stored_config_is_newest` test

* move `stored_config_works` test to the end of `valid_generated_config_test`

* space
2022-06-27 00:07:37 +00:00
Alfredo Garcia 769d069d0a
feat(state): delete old database directories (#4586)
* delete old database directories

* check if state directory exists

* skip deleting when ephemeral

* split `check_and_delete_old_databases`

* move `check_and_delete_old_databases` to state

* spawn `check_and_delete_old_databases`

* simplity a bit

* fix(state): only delete old database directories inside the cache directory (#4631)

* Add function comments, tweak log

* Simplify version parsing

* Use spawn_blocking to launch the task on a separate thread, do the cleanup last

* Abort the cleanup task when Zebra exits

* Split directory deletion into its own function, handle ownership

* Rename cache_dir to state_dir

* If an outdated state directory is outside the cache directory, don't delete it

* Minimise diffs

* add test

* fix typos

Co-authored-by: teor <teor@riseup.net>

* add `canonicalize` to test regex

* add another match to test

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-21 00:59:51 +00:00
teor 29e73b3f3e
breaking(diagnostics): make zebrad diagnostics into optional compile-time features (#4539)
* Disable the flamegraph feature by default at compile time

* Disable the journald feature by default at compile time

* Also disable inferno dependency, and rearrange features

* Disable the prometheus feature by default at compile time

* Disable the tracing filter reload feature by default at compile time

* Disable tests when corresponding features are disabled

* Add compile-time tracing features to user docs

* Add compile-time features to the metrics user docs

* Document diagnostics as part of the start command tasks and services

* breaking(diagnostics): rename "enable-sentry" feature to "sentry" (#4623)

* Also skip conflict tests when those ports are disabled

* breaking(diagnostics): rename "enable-sentry" feature to "sentry"

This is mostly:
```sh
fastmod enable-sentry sentry
```

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-16 19:56:40 +00:00
Alfredo Garcia 9975d5077d
doc(tests): full sync and lightwalletd tests (#4523)
* document full sync and lightwalletd tests

* clippy

* upgrade error

* move env variables

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-15 08:05:02 +00:00
teor a1074954e0
Disable zcash_rpc_conflict test on macOS (#4614) 2022-06-15 00:31:18 +00:00
Conrado Gouvea e016bbe8e3
increase lightwalletd timeout, remove testnet tests (#4584)
* increase lightwalletd timeout

* switch back to aditya's fork

* manually point to new aditya's lightwalletd image

* disable sync_one_checkpoint_testnet test

* disable restart_stop_at_height in testnet

* rever to 'latest' lightwalletd image
2022-06-13 20:13:30 +00:00
teor 61afd02a98
fix(ci): test RPCs with zcash/lightwalletd, to fix post-NU5 failures in adityapk00/lightwalletd (#4553)
* Remove a duplicate lightwalletd error message

* Reactivate some error messages that have been fixed

* Fix confusing lightwalletd cached state path logs

* Add the gRPC tests to the lightwalletd test suite function

* Make test regexes compatible with zcash/lightwalletd

* Add logging to gRPC tests

* Switch to zcash/lightwalletd for testing
2022-06-01 07:36:59 -04:00
teor ca0520b2e8
change(deps): Upgrade tracing-subscriber and related dependencies (#4517)
* Upgrade tracing and related dependencies

```sh
cargo upgrade --workspace
tracing-error
tracing-subscrber

color-eyre

tracing-flame
tracing-journald

sentry
sentry-tracing

metrics
metrics-exporter-prometheus
reqwest
```

* Update duplicate dependency checks

* Enable the tracing/env-filter feature

* Fix type inference for metrics

Manual changes, plus:
```sh
fastmod "as _" "as f64"
```

* Tidy up some unrelated test code

* Update metrics-exporter-prometheus API

And make unused dependencies optional.

* Adjust test regexes to new tracing format

Also fix some regex bugs, and refactor to simplify.

* Disable color-eyre span traces and track caller in release builds

* Add a feature that enables extra debugging in release builds

* Clean up some redundant features

* Increase a test timeout
2022-06-01 13:53:51 +10:00
Marek 6f896ef5a5
fix(doc): Fix the syntax of links in comments (#4494)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fa.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-05-30 20:12:11 +00:00
teor 4add7fc53c
Permanently fix unreliable sync finished log regex (#4504)
Unlike the other recent changes, this is a permanent fix.
2022-05-26 13:40:28 +10:00
teor be91ab29ee
fix(clippy): resolve various clippy warnings (#4473)
* clippy: unused import on non-linux platforms

* Fix some instances of clippy::derive_partial_eq_without_eq

* Move a deref to fix clippy::significant_drop_in_scrutinee

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-05-25 20:56:32 +00:00
teor 89e79474a6
Revert "Stop failing the send transaction test (#4416)" (#4487)
This reverts commit 35aaee5308.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-05-25 19:04:15 +00:00
teor 97b9911c7d
Stop full sync at 97%, but send transactions at 100% (#4483) 2022-05-25 13:14:38 +10:00
teor 49406f3132
fix(ci): Temporarily finish full sync at 99% (#4457)
* Finish full sync at 99.5%

* Finish full sync test harness at 99.5%

* Try 99.2% instead

* Try 99%
2022-05-24 17:59:09 +10:00
Marek 083674de4e
feat(rpc): Add `z_gettreestate` gRPC tests (#4455)
* Add gRPC tests for `z_gettreestate`

* Add a Sapling treestate test vector
2022-05-24 03:09:53 +00:00
Alfredo Garcia d50cf8b77a
tests(rpc): Add grpc test for `GetTaddressBalanceStream` and `GetAddressUtxosStream` (#4407)
* add test for `GetTaddressBalanceStream`

* add test for `GetAddressUtxosStream`
2022-05-19 02:30:50 +00:00
Marek ec9b569ebf
Fix some failing gRPC tests. (#4424) 2022-05-19 02:30:34 +00:00
teor 35aaee5308
Stop failing the send transaction test (#4416)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-05-18 19:14:39 +00:00
teor 94ba4b46a2
change(test): Require cached lightwalletd state for the send transaction tests (#4303)
* Revert "Temporarily stop requiring cached lightwalletd state for the send transaction tests"

This reverts commit f6b29b151e.

* fix(ci): add a lightwalletd cached state to the test

Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-05-13 23:01:31 +00:00
Gustavo Valverde e3a65d86e0
feat(ci): add `lightwalletd_full_sync` test to CI (#4268)
* Temporarily use an earlier lightwalletd version

This checks if commit
e146dbf5c2
contains a mempool refresh deadlock bug.

* Actually rebuild the lightwalletd image

* Delete an unfinished comment

* Remove duplicate test in entrypoint.sh

* Keep a recent change to make tests consistent

* fix(ci): remove not used variable `lwd_state_dir`

* fix(ci): state wast not being added to the image name

* fix(ci): mount a docker volume with lightwalletd dir

If the volume doesn't mount this lwd cached state dir, the content won't be saved to the mounted disk in the VM

* fix(ci): lwd state condition

* docs(ci): explain disk mounting logic

* docs(ci): explain disk mounting decision better

* docs(ci): add a description for confusing input names

Co-authored-by: teor <teor@riseup.net>
2022-05-13 15:20:17 +00:00
teor f7a3a0f6bc
fix(build): put gRPC tests behind an optional feature flag to fix production build issues (#4369)
* fix(ci): sentry is not longer being activated in test builds

This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.

* fix(build): workaround the failed to fetch oauth token error

* Drop sentry dependencies when enable-sentry feature is disabled

* Make lightwalletd gRPC tests depend on a new lightwalletd-grpc-tests feature

* fix(ci): remove enable-sentry feature from tests

* Add lightwalletd-grpc-tests feature for functionality or efficiency

And document where it is just used to stop re-compilations.

* Remove redundant `cmake` and `protobuf-compiler` dependencies

* Document Zebra's optional production and test feature flags

* Minimise dependencies in zcash-params/Dockerfile

* Minimise dependencies in docker/Dockerfile

* Add a workflow TODO

Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-05-11 14:06:58 +00:00
Gustavo Valverde 9226adea97
feat(ci): add `sending_transactions_using_lightwalletd` test to CI (#4267)
* feat(ci): add lightwalletd_*_sync tests to CI

* feat(ci): add lightwalletd RPC call test

* feat(ci): add send transactions test with lwd to CI

* fix(ci): create a variable to run transactions test

* refactor(ci): use docker in docker

This is a workaround for an issue related to disk partitioning, caused by a GCP service called Konlet, while mounting the cached disks to the VM and then to the container

* fix(build): persist docker login credentials

* fix(ci): get sync height from docker logs instead of gcp

* try: use gha cache for faster building

* fix(ci): mount disk in container to make it available in vm

* fix(build): do not invalidate cache between images

* try(docker): invalidate cache as less as possible

* fix(ci): GHA terminal is not a TTY

* fix(build): do not ignore entrypoint.sh

* fix

* fix(ci): mount using root priveleges

* fix(ci): use existing disk as cached state

* fix(ci): wait for disks to get mounted

* force rebuild

* fix failed force

* fix failed commit

* WIP

* fix(ci): some tests does not use a cached state

* wip

* refactor(ci): disk names and job segregation

* fix(ci): do not name boot and attached disk the same

* fix(ci): attach a disk to full sync, to snapshot the state

* fix(ci): use correct disk implementations

* fix(ci): use different disk name to allow test concurrency

* feat(ci): add lightwalledt send transaction test

* cleanup(ci): remove extra tests

* fix(ci): allow disk concurrency with tests

* fix(ci): add considerations for different tests

* fix(reusable): last fixes

* feat(ci): use reusable workflow for tests

* fix(rw): remove nested worflow

* fix(rw): minor fixes

* force rebuild

* fix(rw): do not use an input as job name

* fix(rw): remove variable id

* fix(ci): remove explicit conditions and id

* fix(ci): docker does not need the variable sign ($) to work

* fix(ci): mount typo

* fix(ci): if a sync fails, always delete the instance

This also reduces the amount of jobs needed.

* refactor(ci): make all test depend on the same build

* fix(ci): some tests require multiple variables

* fix(docker): variable substitution

* fix(ci): allow to run multiple commits from a PR at once

* fix(docker): lower the NETWORK env var for test names

* reduce uneeded diff

* imp(keys): use better naming for builds_disks

* imp(ci): use input defaults

* imp(ci): remove test_name in favor of test_id

* fix(ci): better key naming

* fix(ci): long disk names breaks GCP naming convention

* feat(ci): validate local state version with cached state

* fix(ci): add condition to run tests

* fix: typo

* fix: app_name should not be required

* fix: zebra_state_path shouldn't be required

* fix: reduce diff

* fix(ci): checkout to grep local state version

* Update .github/workflows/test.yml

Co-authored-by: teor <teor@riseup.net>

* revert: merge all tests into a single workflow

* Remove unused STATE_VERSION env var

* fix: minor fixes

* fix(ci): make test.patch the same as test

* fix(ci): negate the input value

* imp(ci): better cached state conditional handling

* imp(ci): exit code is captured by `docker run`

* fix(deploy): mount disks with better write performance

* fix(ci): change sync id to a broader id name

* fix(ci): use correct input validation

* fix(ci): do not make test with cached state dependant on other

* imp(ci): organiza keys better

* fix(ci): use appropiate naming

* fix(ci): create docker volume before mounting

* fix(lint): do not fail on all new changes

* imp(ci): do not report in pr review

* fix(ci): partition clean disks

* fix: typo

* fix: test called the wrong way

* fix(build): stop using gha cache

* ref(ci): validate run condition before calling reusable workflow

* fix(ci): use a better filesystem dir and fix other values

* fix: linting errors

* fix(ci): typo

* Revert "fix(build): stop using gha cache"

This reverts commit a8fbc5f416.

Cache expiration is a lesser evil than not using caching at all and then failing with a 401

* imp(ci): do not set a default for needs_zebra_state

* Update .github/workflows/test.yml

Co-authored-by: teor <teor@riseup.net>

* fix(deps): remove dependencies

* force build

* Update .github/workflows/test.yml

Co-authored-by: teor <teor@riseup.net>

* fix(docker): add RUST_LOG as an ARG and ENV

* fix(test): add `#[ignore]` to send transactions test

This test needs state then it should be marked as #[ignore]

* fix(ci): differentiate between root cache path and its dir

* Remove extra `state` directory

That was a workaround for an issue that has been fixed.

* imp(docs): use better test descriptions

Co-authored-by: teor <teor@riseup.net>

* fix: reduce unwanted diff with main

* fix(ci): make lwd conditions consistent

* Remove another extra `state` directory

Was also part of a workaround for an issue that has been fixed.

* fix(ci): use better conditionals to run test jobs

Co-authored-by: teor <teor@riseup.net>

* Tweak to support different lightwalletd versions

Some versions print `Waiting for block`, and some versions print
`Ingestor waiting for block`.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2022-05-06 05:30:38 +00:00
teor d720ab011a
fix(test): make the full sync tests cache state at `/zebrad-cache` (#4308)
* Rewrite the full sync tests to cache at /zebrad-cache

* Improve error reporting from cached state tests
2022-05-05 05:25:46 -04:00
teor 7cc8f7e745
Fix confusing Rust "environmental variable name" variable names (#4301)
```sh
fastmod _VAR "" zebra*
```
2022-05-05 05:59:31 +00:00
teor 5cd1133584
change(test): Stop requiring cached lightwalletd state for the send transaction test (#4302)
* Expect a Zebra cached state with at least a million blocks

* Set the zebrad timeout and failure messages based on the test type

* Temporarily stop requiring cached lightwalletd state for the send transaction tests
2022-05-05 04:44:14 +00:00
Alfredo Garcia d0e81001bc
fix(rpc): Use a structure for parameters of getaddresstxids (#4264)
* add info to lightwalletd_state_path()

* fix getaddresstxids rpc

* normalize more the env vars in lightwalletd tests

* extra logging

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2022-05-05 01:08:27 +00:00
Alfredo Garcia 25712f1935
tests(rpc): Add wallet grpc tests (#4253)
* add a test that call all the lightwalletd grpc methods

* fix some docs

* use ZF address in tests for balance and utxos

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-05-04 04:00:46 +00:00
teor 59bdab17bf
test(rpc): Add Rust tests for lightwalletd sync from Zebra (#4177)
* Make the lightwalletd integration test take a test type

* Configure lightwalletd tests based on the test type

* Remove obsolete kill_on_error() in the lightwalletd test

* Refactor to simplify the test function

* Move LightwalletdTestType to the lightwalletd module

* Create a test function that runs the full lightwalletd test suite

* Actually use the cached Zebra state

* Add checks for the new integration test modes

* Populate the lightwalletd state dir in the FullSyncFromGenesis test

* Fix up state handling, fail earlier if state is invalid

* Adjust timeouts and regex escapes

* Make state requirements for each test stricter

* Move configs to the top of the test function

* Allow unexpected lightwalletd cached state in some tests

* Speed up tests slightly by removing an intermittent log check

* Move timeout selection into test type methods

* Move failure messages into test type methods

* Turn a function argument into an enum field

* Check lightwalletd state directly, rather than Zebra RPC results

* Update gRPC tests for function argument changes

* Remove duplicate env var constant and redundant code
2022-04-29 19:56:11 +00:00
Janito Vaqueiro Ferreira Filho 5a94a09292
add(test): Integration test to send transactions using lightwalletd (#4068)
* Export the `zebra_state::Config::db_path` method

Make it easier for tests to discover the sub-directory used to store
the chain state data.

* Generate code for interfacing with lightwalletd

Use the `tonic-build` crate to generate Rust code for communicating with
lightwalletd using gRPC.

The `*.proto` files were obtained from the Zcash lightwalletd
repository.

* Use `block::Height` instead of `Height`

Import the `block` instead to make it slightly clearer.

* Add helper function to remove a file if it exists

Try to remove it and ignore an error if it says that the file doesn't
exist. This will be used later to remove the lock file from a copied
chain state directory.

* Add helper function to copy chain state dirs

Copy an existing chain state directory into a new temporary directory.

* Add a `BoxStateService` type alias

Make it easier to write and read a boxed version of a state service.

* Add a helper function to start the state service

Make it easier to specify the state service to use an existing state
cache directory.

* Import `eyre!` macro at the module level

Allow it to be used in different places without having to repeat the
imports.

* Add `load_tip_height_from_state_directory` helper

A function to discover the current chain tip height stored in a state
cache.

* Add helper function to prepare partial sync. state

Loads a partially synchronized cached state directory into a temporary
directory that can be used by a zebrad instance, and also returns the
chain tip block height of that state.

* Add `perform_full_sync_starting_from` helper

Runs a zebrad with an existing partially synchronized state, and
finishes synchronizing it to the network chain tip.

* Add function to load transactions from a block

Use a provided state service to load all transactions from a block at a
specified height.

The state service is a generic type parameter, because
`zebra_state::service::ReadStateService` is not exported publicly. Using
a generic type parameter also allows the service to be wrapped in layers
if needed in the future.

* Add `load_transactions_from_block_after` helper

A function to load transactions from a block stored in a cached state
directory. The cached state must be synchronized to a chain tip higher
than the requested height.

* Add helper function to load some test transactions

Given a partially synchronized chain state, it will extend that chain by
performing a full synchronization, and obtain some transactions from one
of the newly added blocks.

* Update `spawn_zebrad_for_rpc_without_initial_peers`

Wait until the mempool is activated.

* Add method to start lightwalletd with RPC server

Returns the lightwalletd instance and the port that it's listening for
RPC connections.

The instance can reuse an existing cached lightwalletd state if the
`LIGHTWALLETD_DATA_DIR` environment variable is set.

* Add a `LightwalletdRpcClient` type alias

To make it easier to identify the type generated from the Protobuf
files.

* Add helper function to connect to lightwalletd

Prepare an RPC client to send requests to a lightwalletd instance.

* Add a `prepare_send_transaction_request` helper

Creates a request message for lightwalletd to send a transaction.

* Add test to send transactions using lightwalletd

Obtain some valid transactions from future blocks and try to send them
to a lightwalletd instance connected to a zebrad instance that hasn't
seen those transactions yet. The transactions should be successfully
queued in Zebra's mempool.

* Make `zebra_directory` parameter generic

Allow using a `TempDir` or a `PathBuf`.

* Move lightwalletd protobuf files

Place them closer to the module directory, so that it's clearer that
they specify the RPC protocol for lightwalletd, and not Zebra itself.

* Don't use coinbase transactions in the test

Coinbase transactions are rejected by the mempool.

* Don't remove state lock file

It is removed automatically by Zebra when it shuts down, so if it exists
it should be reported as a bug.

* Force mempool to be enabled in Zebrad instance

Speed up the initialization of the Zebrad instance used for lightwalletd
to connect to.

* Refactor to create `LIGHTWALLETD_DATA_DIR_VAR`

Document how the environment variable can be used to speed up the test.

* Check for process errors in spawned Zebra instance

Enable checking for known process failure messages.

* Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant

Document why it exists and how the choice of the value affects the test.

* Add `LIGHTWALLETD_TEST_TIMEOUT` constant

And use it for the Zebrad and the Lightwalletd instances used in the
send transaction integration test.

* Check `lightwalletd` process for errors

Enable checking the lightwalletd process for known failure messages.

* Update `tonic` and `prost` dependencies

Use the latest version and fix CI failures because `rustfmt` isn't
installed in the build environment.

* Create `send_transaction_test` module

Move the send transaction using lightwalletd test and its helper
functions into a new module.

* Move `LIGHTWALLETD_TEST_TIMEOUT` constant

Place it in the parent `lightwalletd` module.

* Move gRPC helper functions and types to `rpc` mod.

Make them more accessible so that they can be used by other tests.

* Create a `cached_state` module

Move the test utility functions related to using a cached Zebra state
into the module.

* Move `perform_full_sync_starting_from` to `sync`

Keep to closer to the synchronization utility functions.

* Move Zebra cached state path variable constant

Place it in the `cached_state` module.

* Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set

Make it part of the set of tests ignored as a whole if no lightwalletd
tests should be executed.

* Move `spawn_zebrad_for_rpc_without_initial_peers`

Place it in the `launch` sub-module.

* Rename `rpc` module into `wallet_grpc`

Avoid any potential misunderstandings when the name is seen out of
context.

* Allow duplicate `heck` dependency

At least until `structopt` is updated or `zebra-utils` is updated to use
`clap` 3.

* Fix a deny.toml typo

* fix(build): CMake is required by `prost` crate

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-04-27 23:06:11 +00:00
Conrado Gouvea 910f4cbf40
RPC test with fully synced Zebra (#4157)
* partial implementation; starts zebra from cached state

* add fully_synced_rpc_test()

* use block from MAINNET_BLOCKS
2022-04-26 05:32:27 +00:00
Conrado Gouvea 315eebf09c
fix(test): don't create another 'state' directory when implementing ZebradTestDirExt (#4158)
* fix(test): don't create another 'state' directory when implementing ZebradTestDirExt

* fix zebra_state_conflict() test
2022-04-22 01:10:28 +00:00
Janito Vaqueiro Ferreira Filho c47dac8d5f
change(test): Refactor how extra arguments are handled when spawing lightwalled (#4067)
* Join similar imports

Avoid the confusion that might cause one to think that they come from
different modules or crates.

* Create an `Arguments` helper type

A type to keep track of a list of arguments for a sub-process. It makes
it easier for overriding parameters with new values.

* Create an `args!` helper macro

Make it simpler to create `Arguments` instances with known values.

* Require `Arguments` for `spawn_child` method

Change the method to have an `Arguments` parameter, and merge it with
some default values before passing them forward.

* Use `Arguments` in `spawn_lightwalletd_child`

Change the method to use an `Arguments` instance, and merge it with some
default options.

* Use `Arguments` in `spawn_child_with_command`

Require an `Arguments` instance in the `spawn_child_with_command`
extension method. Makes it simpler to call from `spawn_child` and
`spawn_lightwalletd_child` extension methods.

* Test if argument order is preserved

Check that when building an `Arguments` instance, the order that the
arguments are set is preserved in the generated list of strings.

* Refactor test to improve readability

Also separates some common code to be reused by later tests.

* Test overriding arguments

Check to see if overriding arguments behaves as expected, by keeping the
argument order when overriding and not introducing duplicates.

* Refactor test to improve readability

Move out a chunk of code so that the test itself is easier to read and
to make that code reusable by a later test.

* Test that `Arguments` instances can be merged

Merge two `Arguments` instances built from two lists of arguments, and
check that the expanded strings preserve order and override rules.

* Add Eq derives on Arguments

Co-authored-by: teor <teor@riseup.net>
2022-04-19 10:28:52 +00:00
teor ce51ad060f
fix(rpc): use the correct RPC error code for missing blocks (#3977)
* Return error code -8 for missing blocks, like zcashd does

* Test that getblock returns error code -8 for missing blocks

* Make RegexSet failures easier to read in logs

* Update lightwalletd acceptance tests for log message changes

* Add extra failure strings for lightwalletd logs

* Improve link formatting

Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>

Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
2022-03-30 23:34:52 +00:00
Alfredo Garcia f687ab947f
feat(rpc): Implement `getblockchaininfo` RPC method (#3891)
* Implement `getblockchaininfo` RPC method

* add a test for `get_blockchain_info`

* fix tohex/fromhex

* move comment

* Update lightwalletd acceptance test for getblockchaininfo RPC (#3914)

* change(rpc): Return getblockchaininfo network upgrades in height order (#3915)

* Update lightwalletd acceptance test for getblockchaininfo RPC

* Update some doc comments for network upgrades

* List network upgrades in order in the getblockchaininfo RPC

Also:
- Use a constant for the "missing consensus branch ID" RPC value
- Simplify fetching consensus branch IDs
- Make RPC type derives consistent
- Update RPC type documentation

* Make RPC type derives consistent

* Fix a confusing test comment

* get hashand height at the same time

* fix estimated_height

* fix lint

* add extra check

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* fix typo

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* split test

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* fix(rpc): ignore an expected error in the RPC acceptance tests (#3961)

* Add ignored regexes to test command failure regex methods

* Ignore empty chain error in getblockchaininfo

We expect this error when zebrad starts up with an empty state.

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-25 12:25:31 +00:00
teor 65c1ef8177
T3. add(test): check for failure messages in lightwalletd and Zebra logs (#3903)
* Make command test matching code accept generic regexes

And add generic conversions to regexes.

* Document test command structs

* Support matching multiple regexes internally in the test command

* Make it easier to call the generic regex methods

* Add a missing API usage comment

* Fix a potential hang in test child error reports

* Revert Option<Child> process handling

* Revert "Revert Option<Child> process handling"

This reverts commit 2af30086858d104dcb0ec87383996c36bcaa7371.

* Add a set of failure regexes to test command output

* Allow debug-printing TestChild again

* When the child is dropped, check any remaining output

* Document a wait_with_output edge case

* Improve failure regex panic output

* Improve builder ergonomics

* Add internal tests for failure regex panics

It would be easy to disable these panics, and never realise.

* Add some module structure TODOs

* Stop panicking if the child process has already been taken

* Add test APIs for consuming child output lines

* Fix a hang on child process drop

* Handle output being already taken in wait_with_output

And document some edge cases we don't handle yet

* Use bash's read command in the TestChild stderr test

And check the actual command we're using to see if it errors.

* Pretty print full failure regex list

* Add the test child command line to the failure regex logs

* Simplify the sync_until test code

* Check logs for failure messages in lightwalletd integration tests

* Add more TODOs and improve comments

* Allow missing branch ID 00000000 provided by zcashd
2022-03-23 01:34:37 +00:00
teor a5d7b9c1e0
T2. add(test): add test API that checks process logs for failures (#3899)
* Revert "Revert Option<Child> process handling"

This reverts commit 2af30086858d104dcb0ec87383996c36bcaa7371.

* Add a set of failure regexes to test command output

* Allow debug-printing TestChild again

* When the child is dropped, check any remaining output

* Document a wait_with_output edge case

* Improve failure regex panic output

* Improve builder ergonomics

* Add internal tests for failure regex panics

It would be easy to disable these panics, and never realise.

* Add some module structure TODOs

* Stop panicking if the child process has already been taken

* Add test APIs for consuming child output lines

* Fix a hang on child process drop

* Handle output being already taken in wait_with_output

And document some edge cases we don't handle yet

* Use bash's read command in the TestChild stderr test

And check the actual command we're using to see if it errors.

* Pretty print full failure regex list

* Add the test child command line to the failure regex logs
2022-03-22 23:53:24 +00:00
teor 9a8ab9468d
T0. refactor(test): split zebrad acceptance tests into sub-modules (#3901)
* Improve launch delay docs

* Initial split of zebrad acceptance tests into modules

* Split shared lightwalletd test code into a module
2022-03-18 16:02:22 +00:00
Alfredo Garcia 53e2d2e298
feat(rpc): implement `getrawmempool` RPC method (#3851)
* feat(rpc): implement `getrawmempool` rpc method

* tests(rpc): start implementing a test

* Return a `Vec<String>` from `get_raw_mempool`

Ensure the returned type is serialized as an array of hexadecimal
strings.

* Make `getrawmempool` test a property test

Test it with random sets of transaction IDs.

* Fix code formatting inside `proptest!`

Run `rustfmt` inside the `proptest!` block.

* tests(rpc): fix lightwalletd test

* fix(rpc): remove non used return type

* tests(rpc): remove assert

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2022-03-12 16:51:49 +00:00
teor ebecfd078c
test(lightwalletd): wait for successful block ingestion in integration tests (#3824)
* Allow extra time for lightwalletd integration tests

* Wait for a successful block ingestion in lightwalletd integration

* Check for any ingestor block, not just block 1
2022-03-12 00:20:17 +00:00
Alfredo Garcia 833560411f
feature(rpc): implement getblock api call (#3707)
* feature(rpc): start adding a `getblock` method

* fix(rpc): replace oneshot

* fix(rpc): replace a panic with error

* fix(rpc): fix test

* feature(rpc): add hex to response

* refactor(rpc): use generic instead of alias

* docs(rpc): improve docs for getblock method

* test(rpc): add a test for getblock method

* deps(rpc): remove non needed tower features

Co-authored-by: teor <teor@riseup.net>

* docs(rpc): add a note to getblock doc

* refactor(rpc): replace alias

* fix(rpc): use `zcash_serialize_to_vec()` instead of logging format

* tests(rpc): add network argument to `populated_state()`

* refactor(rpc): use an error for state service readiness

* fix(rpc): add parameter

* fix(rpc): clippy

* nit(rpc): remove new line from imports

* fix(rpc): remove commented code

* fix(rpc): simplify error

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Use a `SerializedBlock` type to help serializing blocks (#3725)

* Create a `SerializedBlock` helper type

Create a type that can be used as a byte slice, but is guaranteed to
represent a valid block.

* Use `into_iter` instead of `iter`

There's no need to borrow the elements, they can be moved out directly.
This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`,
so a `SerializedBlock` can't be built directly from an `&Arc<Block>`.

* Use `SerializedBlock` in `GetBlock`

Make the type stricter to avoid storing possibly invalid values. The
bytes are still serialized as a hexadecimal string, through the usage of
`hex`.

The `serde::Deserialize` can't be derived because `hex` requires the
type to also implement `FromHex`.

* feature(rpc): add suggestions from code review

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* tests(rpc): make sure mempool has no requests in get_block test

* fix(rpc): change height argument type in getblock method

* fix(rpc): rustfmt

* fix(rpc): replace panic

* fix(rpc): change getblock response

* fix(rpc): fix lightwalletd test

* tests(rpc): add a getblock error test

* fix(rpc): try another regex

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2022-03-10 01:12:41 +00:00
teor cef146edbd
lint(clippy): warn on manual printing to stdout or stderr (#3767)
Most logging should use `tracing::trace!()` or `tracing::debug!()` instead.
2022-03-08 09:14:15 +00:00
teor 886a4f585b
fix(test/lightwalletd): add an alternative matching log line (#3758)
Sometimes lightwalletd goes straight to "Another refresh in progress",
without logging "Mempool refresh error" first.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-08 02:46:03 +00:00
teor ac662df179
fix(zebrad/ci): skip RPC conflict test when network tests are disabled (#3726) 2022-03-04 03:01:41 -04:00
teor 9ce3a9a62f
fix(ci): check for adityapk00/lightwalletd behaviour in test harness (#3705)
* fix(ci): remove an unused trigger path

* doc(ci): explain lightwalletd trigger paths

* fix(test): check for adityapk00/lightwalletd behaviour in test harness

* fix(ci): work around buildx command error

* fix(ci): revert the workaround
2022-03-03 13:03:06 +00:00
Alfredo Garcia 675fa3621d
tests(rpc): Add some RPC acceptance tests (#3641)
* tests(rpc): add an rpc endpoint test

* tests(rpc): add an rpc port conflict test

* tests(rpc): refactor some imports

* tests(rpc): fix failures, make test more complete

* tests(rpc): parse json response for better coverage

* tests(rpc): change request

* tests(rpc): wait until port is open in rpc_endpoint test

* tests(rpc): add a delay between launching 2 nodes

* tests(rpc): try 5 seconds

* refactor(rpc): open rpc server faster

* tests(rpc): extend `LAUNCH_DELAY` to 15 seconds

* fix(rpc): disable rpc_conflict test for windows

* fix(ci): skip the RPC tests if the network is disabled

* rustfmt

* fix(zebrad/test): test function return type

* tests(rpc): print server output in assert

* fix(rpc): fix acceptance test looking for string in `build` field

* fix(rpc): reduce the number of acceptable characters in version output

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-03 00:39:47 +00:00
teor d08b13da73
fix(zebrad/test): use the correct stop condition for the cached state tests (#3688) 2022-03-02 08:53:00 +00:00
teor 41d61a62f9
refactor(test): split lightwalletd test launch into separate methods (#3628)
* fix(test): only run lightwalletd test when the ZEBRA_TEST_LIGHTWALLETD env var is set

* fix(test): actually skip the test

* doc(zebrad): add some test TODOs

* doc(test): document zebrad-specific process launch methods

* refactor(zebrad): split lightwalletd launch into its own testing methods

* fix(zebrad/test): simplify file writing

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* refactor(zebrad/test): rename argument variables

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* lint(zebrad/tests): remove unused import

* fix(zebrad/test): restore SocketAddr import that was removed on main

* rustfmt

* fix(zebrad/test): update integration test to match adityapk00/lightwalletd behaviour

* rustfmt

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-02 02:01:57 +00:00
teor 729535cf25
fix(test): check for zebrad test output in the correct order (#3643)
The mempool is only activated once, so we must check for that log first.
After mempool activation, the stop regex is logged at least once.
(It might be logged before as well, but we can't rely on that.)

When checking that the mempool didn't activate,
wait for the `zebrad` command to exit,
then check the entire log.
2022-02-25 23:36:20 +00:00