Commit Graph

141 Commits

Author SHA1 Message Date
teor 6eaf83b4bf
fix(security): Randomly drop connections when inbound service is overloaded (#6790)
* fix(security): Randomly drop connections when inbound service is overloaded

* Uses progressively higher drop probabilities

* Replaces Error::Overloaded with Fatal when internal services shutdown

* Applies suggestions from code review.

* Quickens initial drop probability decay and updates comment

* Applies suggestions from code review.

* Fixes drop connection probablity calc

* Update connection state metrics for different overload/error outcomes

* Split overload handler into separate methods

* Add unit test for drop probability function properties

* Add respond_error methods to zebra-test to help with type resolution

* Initial test that Overloaded errors cause some continues and some closes

* Tune the number of test runs and test timing

* Fix doctests and replace some confusing example requests

---------

Co-authored-by: arya2 <aryasolhi@gmail.com>
2023-05-31 19:04:15 +00:00
teor 1461c912f9
change(ci): Generate mainnet checkpoints in CI (#6550)
* Add extra test type modes to support zebra-checkpoints

* Add Mainnet and Testnet zebra-checkpoints test harnesses

* Add zebra-checkpoints to test docker images

* Add zebra-checkpoints test entrypoints

* Add Mainnet CI workflow for zebra-checkpoints

* Enable zebra-checkpoints feature in the test image

* Use the same features for (almost) all the docker tests

* Make workflow features match Docker features

* Add a feature note

* Add a zebra-checkpoints test feature to zebrad

* Remove the "no cached state" testnet code

* Log a startup message to standard error when launching zebra-checkpoints

* Rename tests to avoid partial name conflicts

* Fix log formatting

* Add sentry feature to experimental docker image build

* Explain what ENTRYPOINT_FEATURES is used for

* Use the correct zebra-checkpoints path

* Silence zebrad logs while generating checkpoints

* Fix zebra-checkpoints log handling

* Re-enable waiting for zebrad to fully sync

* Add documentation for how to run these tests individually

* Start generating checkpoints from the last compiled-in checkpoint

* Fix clippy lints

* Revert changes to TestType

* Wait for all the checkpoints before finishing

* Add more stderr debugging to zebra-checkpoints

* Fix an outdated module comment

* Add a workaround for zebra-checkpoints launch/run issues

* Use temp dir and log what it is

* Log extra metadata about the zebra-checkpoints binary

* Add note about unstable feature -Z bindeps

* Temporarily make the test run faster and with debug info

* Log the original test command name when showing stdout and stderr

* Try zebra-checkpoints in the system path first, then the cargo path

* Fix slow thread close bug in dual process test harness

* If the logs are shown, don't say they are hidden

* Run `zebra-checkpoints --help` to work out what's going on in CI

* Build `zebra-utils` binaries for `zebrad` integration tests

* Revert temporary debugging changes

* Revert changes that were moved to another PR
2023-04-27 04:39:43 +00:00
teor 019ae25847
Remove redundant track_callers which now cause errors on nightly (#5839) 2022-12-13 00:59:38 +00:00
teor b722a34102
fix(rpc): Wait for 3 minutes to check Zebra is synced to the tip, rather than 2 (#5840)
* Wait for 3 minutes to check Zebra is synced to the tip, rather than 2

* Increase the MockService request timeout
2022-12-12 23:20:03 +00:00
teor cb1045ae5f
change(mempool): Re-verify mempool transactions after a chain fork, rather than re-downloading them all (#5841)
* Move Drop from mempool::ActiveState to mempool::Downloads, to avoid bugs

* Re-verify mempool transactions after a fork

And add a marker struct for mempool download cancellation.

* Update README based on recent mitigations for some issues, tidy format

* Make mempool proptests easier to debug

* Make UnminedTx Display text much smaller

* Update tests for mempool transaction re-verification after forks

* Retry all stored and pending transactions

* Fix a test to check for mempool reset retries
2022-12-12 23:19:45 +00:00
teor 47073ab30a
Fix a test child process panic bug (#5842) 2022-12-12 23:19:30 +00:00
teor 2041fda7bb
fix(test): Reduce verbose test logs (#5825)
* Remove verbose continuous_blockchain test logs

* Downgrade verbose zebra-network logs to debug

* Downgrade some state logs to debug during tests

* Mark were we would add always-on log filters, if we needed to

* Reduce the number of mempool property tests, to reduce logging
2022-12-08 23:56:01 +00:00
teor 09836d2800
fix(clippy): Put Rust format variables inline (#5783)
* cargo clippy --fix --all-features --all-targets

With rustc 1.67.0-nightly (234151769 2022-12-03)

* cargo fmt --all
2022-12-08 01:05:57 +00:00
Arya beb45fc514
change(tests): use OS-assigned unallocated ports for getting random known ports (#5607)
* updates mod docs for tests that use future blocks

* updates submitblock test to use TestType methods

* prunes redundant code

* adds check_sync_logs_until

* adds assertion for needs cached state & rpc server

* updates get_raw_future_blocks fn with rpc calls

* updates to get_raw_future_blocks fn and submit_block test

* Rename LightwalletdTestType to TestType

* moves TestType and random_known_rpc_port_config to test_type.rs and config.rs

* moves get_raw_future_blocks to cached_state.rs

* updates ci workflows to include submit block test

* adds get_future_blocks fn and uses it in load_transactions_from_future_blocks

* updates CI docker

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Applies suggestions from code review

* Updates misnamed closure param

* updates mod docs for test_type.rs

* updates random_known_port to ask the OS for a port

* uses old random_known_port for windows

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-11-10 23:23:12 +00:00
teor 34313b8857
fix(build): Disable new Rust 1.65 lints and fix some test errors (#5549)
* allow(clippy::result_large_err)

* Increase the async executor delay expected by tests

* Split getblocktemplate-rpcs OS tests into a separate matrix job

* Add new patch jobs

* allow(unknown_lints)
2022-11-04 02:00:56 +00:00
teor c812f880cf
cleanup(clippy): Use inline format strings (#5489)
* Inline format strings using an automated clippy fix

```sh
cargo clippy --fix --all-features --all-targets -- -A clippy::all -W clippy::uninlined_format_args
cargo fmt --all
```

* Remove unused & and &mut using an automated clippy fix

```sh
cargo clippy --fix --all-features --all-targets -- -A clippy::all -W clippy::uninlined_format_args
```
2022-10-27 13:25:18 +00:00
teor cea622307a
Allow more time between thread CPU slices in db_init_outside_future_executor (#5310) 2022-10-04 02:51:53 +00:00
teor 3d8f4e6064
fix(ci): improve test output and test reliability (#5014)
* Rename a function to prepare_block_header_and_transaction_data_batch()

* Fix formatting of test command timeouts and child process output

* Put some #[cfg()]s in the standard Rust location

* Update some test timings

* Allow code timers to be ignored
2022-09-02 08:54:40 +00:00
teor f46d0115e5
fix(test): Show full Zebra test panic details in CI logs (#4942)
* Handle test failure regexes using Result::Err, rather than panicking

* Add output logs to test context, and add tests for that

* Let empty test child logs be read again (and produce empty output)

* Ignore missing test children when killing with ignore_exited

* Fix a clippy lint

* Rename `line` to `line_result` for clarity

* Revert a redundant context_from() on kill()

* Only ignore "no such process" kill() errors in sync_until() tests

* Log the command timeout when an acceptance test fails

* fix clippy

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-08-28 23:52:19 +00:00
Conrado Gouvea 6fd750e168
build(deps): bump insta from 1.15.0 to 1.17.1 (#4884)
* build(deps): bump insta from 1.15.0 to 1.17.1

Bumps [insta](https://github.com/mitsuhiko/insta) from 1.15.0 to 1.17.1.
- [Release notes](https://github.com/mitsuhiko/insta/releases)
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mitsuhiko/insta/compare/1.15.0...1.17.1)

---
updated-dependencies:
- dependency-name: insta
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* make zebra_test::init() return the insta drop guard

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-08-04 15:44:44 +00:00
teor 89a0410e23
fix(ci): fix hangs in lightwalletd tests by checking concurrent process output in different threads (#4828)
* Make code execution time logs shorter

* Do ZK parameter preloads in the lightwalletd tests that need them

* Try to re-launch `lightwalletd` when it hangs during sync tests

* Increase full sync timeout

* Clear the `zebrad` logs during `lightwalletd` tests, to avoid logging deadlocks

* Actually clear more than one line of logs

* Check zebrad and lightwalletd output in parallel threads, while waiting for zebrad

* Check zebrad and lightwalletd output in parallel threads, while waiting for lightwalletd

* Improve test logging

* Fix a log typo

* Only wait for lightwalletd once, because its logs stop after the initial sync

* Look for cached state disks for this commit and branch first

* Only copy the state once in the send transactions test

* Wait longer for lightwalletd gRPC server startup

* Add some function docs

* cargo fmt --all
2022-07-29 07:06:18 +10:00
teor 9b9cd55097
fix(batch): Improve batch verifier async, correctness, and performance (#4750)
* Use a new channel for each batch

* Prefer the batch timer if there are also new batch requests

* Allow other tasks to run after each batch

* Label each batch worker with the verifier's type

* Rename Handle to ErrorHandle, and fix up some docs

* Check batch worker tasks for panics and task termination

* Use tokio's PollSemaphore instead of an outdated Semaphore impl

* Run all verifier cryptography on a blocking thread

Also use a new verifier channel for each batch.

* Make flush and drop behaviour consistent for all verifiers

* Partly fix an incorrect NU5 test

* Switch batch tests to the multi-threaded runtime

* Export all verifier primitive modules from zebra-consensus

* Remove outdated test code in tower-batch

* Use a watch channel to send batch verifier results

* Use spawn_blocking for batch fallback verifiers

* Spawn cryptography batches onto blocking tokio threads

* Use smaller batches for halo2

* Minor tower-batch cleanups

* Fix doc link in zebra-test

* Drop previous permit before acquiring another to avoid a deadlock edge case
2022-07-18 08:41:18 +10:00
Alfredo Garcia 97fb85dca9
lint(clippy): add `unwrap_in_result` lint (#4667)
* `unwrap_in_result` in zebra-chain crate

* `unwrap_in_result` in zebra-script crate

* `unwrap_in_result` in zebra-state crate

* `unwrap_in_result` in zebra-consensus crate

* `unwrap_in_result` in zebra-test crate

* `unwrap_in_result` in zebra-network crate

* `unwrap_in_result` in zebra-rpc crate

* `unwrap_in_result` in zebrad crate

* rustfmt

* revert `?` and add exceptions

* explain some panics better

* move some lint positions

* replace a panic with error

* Fix rustfmt?

Co-authored-by: teor <teor@riseup.net>
2022-06-28 06:22:07 +00:00
Marek b91aaf7863
Use `config.toml` instead of inner attributes for private links (#4627) 2022-06-15 23:21:27 +00:00
Marek cc75c3f5f9
fix(doc): Fix various doc warnings, part 3 (#4611)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fa.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

* Fix some rustdoc warnings

* Fix some rustdoc warnings

* Resolve various TODOs

Co-authored-by: teor <teor@riseup.net>

* Fix some unresolved links

* Allow links to private items

* Fix some unresolved links

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-15 03:57:19 +00:00
Marek 2e50ccc8f3
fix(doc): Fix various doc warnings, part 2 (#4561)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fa.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

* Fix some rustdoc warnings

* Fix some rustdoc warnings

* Resolve various TODOs

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-14 01:22:16 +00:00
Marek b8b35f8da9
fix(doc): Fix various doc warnings, part 1 (#4514)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fa.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-06-02 15:07:35 +00:00
Marek 6f896ef5a5
fix(doc): Fix the syntax of links in comments (#4494)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fa.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-05-30 20:12:11 +00:00
Marek 083674de4e
feat(rpc): Add `z_gettreestate` gRPC tests (#4455)
* Add gRPC tests for `z_gettreestate`

* Add a Sapling treestate test vector
2022-05-24 03:09:53 +00:00
Dimitris Apostolou 04b0bb9358
Fix typos (#4164) 2022-04-22 01:10:33 +00:00
Janito Vaqueiro Ferreira Filho c47dac8d5f
change(test): Refactor how extra arguments are handled when spawing lightwalled (#4067)
* Join similar imports

Avoid the confusion that might cause one to think that they come from
different modules or crates.

* Create an `Arguments` helper type

A type to keep track of a list of arguments for a sub-process. It makes
it easier for overriding parameters with new values.

* Create an `args!` helper macro

Make it simpler to create `Arguments` instances with known values.

* Require `Arguments` for `spawn_child` method

Change the method to have an `Arguments` parameter, and merge it with
some default values before passing them forward.

* Use `Arguments` in `spawn_lightwalletd_child`

Change the method to use an `Arguments` instance, and merge it with some
default options.

* Use `Arguments` in `spawn_child_with_command`

Require an `Arguments` instance in the `spawn_child_with_command`
extension method. Makes it simpler to call from `spawn_child` and
`spawn_lightwalletd_child` extension methods.

* Test if argument order is preserved

Check that when building an `Arguments` instance, the order that the
arguments are set is preserved in the generated list of strings.

* Refactor test to improve readability

Also separates some common code to be reused by later tests.

* Test overriding arguments

Check to see if overriding arguments behaves as expected, by keeping the
argument order when overriding and not introducing duplicates.

* Refactor test to improve readability

Move out a chunk of code so that the test itself is easier to read and
to make that code reusable by a later test.

* Test that `Arguments` instances can be merged

Merge two `Arguments` instances built from two lists of arguments, and
check that the expanded strings preserve order and override rules.

* Add Eq derives on Arguments

Co-authored-by: teor <teor@riseup.net>
2022-04-19 10:28:52 +00:00
Conrado Gouvea d79e71e969
change(nu5): use new V5 transaction script verification API (#3799)
* update librustzcash; adapt to new API

* add ticket reference for removing zcash_proofs duplicated dependencies

* update to new zcash_script V5 API

* use zp_tx shorthand

* update to Zcash 4.7.0 dependencies

* update protocol versions

* feat(rpc): Implement `getblockchaininfo` RPC method (#3891)

* Implement `getblockchaininfo` RPC method

* add a test for `get_blockchain_info`

* fix tohex/fromhex

* move comment

* Update lightwalletd acceptance test for getblockchaininfo RPC (#3914)

* change(rpc): Return getblockchaininfo network upgrades in height order (#3915)

* Update lightwalletd acceptance test for getblockchaininfo RPC

* Update some doc comments for network upgrades

* List network upgrades in order in the getblockchaininfo RPC

Also:
- Use a constant for the "missing consensus branch ID" RPC value
- Simplify fetching consensus branch IDs
- Make RPC type derives consistent
- Update RPC type documentation

* Make RPC type derives consistent

* Fix a confusing test comment

* get hashand height at the same time

* fix estimated_height

* fix lint

* add extra check

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* fix typo

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* split test

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* fix(rpc): ignore an expected error in the RPC acceptance tests (#3961)

* Add ignored regexes to test command failure regex methods

* Ignore empty chain error in getblockchaininfo

We expect this error when zebrad starts up with an empty state.

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Make sync error logs more user-friendly (#3944)

- use info level, there is nothing the user needs to do,
  particularly for a single error
- explain that the errors are temporary
- hide backtraces, because they look like crashes

* Update test.patch.yml with lightwalletd job (#3970)

* Update test.patch.yml with lightwalletd job

* Remove a workflow condition that will always be false

In general, patch workflows need the
opposite conditions to the original workflow.

But in this case, we know the result of the
condition will always be true, so we can just delete it.

Co-authored-by: teor <teor@riseup.net>

* fix(doc): Fix bugs in the lightwalletd database design (#3964)

* Re-order column families in design in dependency order

* Minor RFC design tweaks and fixes

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Repoint zebra image links to our new zfnd.org site for now (#3949)

* Repoint zebra image links to our new zfnd.org site for now

* Remove images/

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Fix typos (#3956)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* bump database version to trigger testnet rollback

* reduce minimum protocol version for now (will be changed later)

* update dependencies

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* update versions to match zcash 4.7.0

* deny.toml: update 'darling'

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Dimitris Apostolou <dimitris.apostolou@icloud.com>
2022-04-19 10:14:16 +10:00
teor ce51ad060f
fix(rpc): use the correct RPC error code for missing blocks (#3977)
* Return error code -8 for missing blocks, like zcashd does

* Test that getblock returns error code -8 for missing blocks

* Make RegexSet failures easier to read in logs

* Update lightwalletd acceptance tests for log message changes

* Add extra failure strings for lightwalletd logs

* Improve link formatting

Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>

Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
2022-03-30 23:34:52 +00:00
Deirdre Connolly 0821e30623
Repoint zebra image links to our new zfnd.org site for now (#3949)
* Repoint zebra image links to our new zfnd.org site for now

* Remove images/

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-27 23:42:47 +00:00
Alfredo Garcia f687ab947f
feat(rpc): Implement `getblockchaininfo` RPC method (#3891)
* Implement `getblockchaininfo` RPC method

* add a test for `get_blockchain_info`

* fix tohex/fromhex

* move comment

* Update lightwalletd acceptance test for getblockchaininfo RPC (#3914)

* change(rpc): Return getblockchaininfo network upgrades in height order (#3915)

* Update lightwalletd acceptance test for getblockchaininfo RPC

* Update some doc comments for network upgrades

* List network upgrades in order in the getblockchaininfo RPC

Also:
- Use a constant for the "missing consensus branch ID" RPC value
- Simplify fetching consensus branch IDs
- Make RPC type derives consistent
- Update RPC type documentation

* Make RPC type derives consistent

* Fix a confusing test comment

* get hashand height at the same time

* fix estimated_height

* fix lint

* add extra check

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* fix typo

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* split test

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* fix(rpc): ignore an expected error in the RPC acceptance tests (#3961)

* Add ignored regexes to test command failure regex methods

* Ignore empty chain error in getblockchaininfo

We expect this error when zebrad starts up with an empty state.

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-25 12:25:31 +00:00
teor a5d7b9c1e0
T2. add(test): add test API that checks process logs for failures (#3899)
* Revert "Revert Option<Child> process handling"

This reverts commit 2af30086858d104dcb0ec87383996c36bcaa7371.

* Add a set of failure regexes to test command output

* Allow debug-printing TestChild again

* When the child is dropped, check any remaining output

* Document a wait_with_output edge case

* Improve failure regex panic output

* Improve builder ergonomics

* Add internal tests for failure regex panics

It would be easy to disable these panics, and never realise.

* Add some module structure TODOs

* Stop panicking if the child process has already been taken

* Add test APIs for consuming child output lines

* Fix a hang on child process drop

* Handle output being already taken in wait_with_output

And document some edge cases we don't handle yet

* Use bash's read command in the TestChild stderr test

And check the actual command we're using to see if it errors.

* Pretty print full failure regex list

* Add the test child command line to the failure regex logs
2022-03-22 23:53:24 +00:00
teor 16872f3ba6
T1. add(test): add test API that checks logs for multiple regexes (#3892)
* Make command test matching code accept generic regexes

And add generic conversions to regexes.

* Document test command structs

* Support matching multiple regexes internally in the test command

* Make it easier to call the generic regex methods

* Add a missing API usage comment

* Fix a potential hang in test child error reports

* Revert Option<Child> process handling

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-22 15:58:27 +00:00
teor 9a8ab9468d
T0. refactor(test): split zebrad acceptance tests into sub-modules (#3901)
* Improve launch delay docs

* Initial split of zebrad acceptance tests into modules

* Split shared lightwalletd test code into a module
2022-03-18 16:02:22 +00:00
teor 39dfca8e64
5. change(state): split ReadStateService requests into a ReadRequest enum (#3866)
* Split out ReadRequest and ReadResponse state service enums

* Simplify RPC test vectors

* Split state requests into Request and ReadRequest

* Make zebra-rpc use the new state ReadRequest
2022-03-17 22:59:46 +00:00
teor cef146edbd
lint(clippy): warn on manual printing to stdout or stderr (#3767)
Most logging should use `tracing::trace!()` or `tracing::debug!()` instead.
2022-03-08 09:14:15 +00:00
teor 744aca9d45
7. test(database): snapshot raw RocksDB column family data (#3630)
* refactor(state): split database access into modules by Zebra types

Also split the genesis block check from the genesis note commitment trees.

* test(db): snapshot column family names

* fix(db): assert that the default column family is empty on open and close

* feat(test): apply `cargo insta` settings to all tests

* doc(db/test): improve test docs

* test(db): snapshot column family data for the empty state

* refactor(db/test): split out the raw database snapshot

* test(db): snapshot raw database for blocks 0-2

* test(db): initial serialized snapshot data for raw database

* test(db): tweak snapshot file names

* test(db): rename snapshots for consistency

* test(db): store empty column families in a single snapshot

* test(db): simplify snapshot files by combining empty snapshots

* doc(db/test): put comment in a better place

* refactor(db): fastmod assert_default_is_empty assert_default_cf_is_empty

* doc(test): explain when insta settings are needed

* fix(state/test): use the network to initialize the state

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2022-03-02 02:44:39 +00:00
teor 729535cf25
fix(test): check for zebrad test output in the correct order (#3643)
The mempool is only activated once, so we must check for that log first.
After mempool activation, the stop regex is logged at least once.
(It might be logged before as well, but we can't rely on that.)

When checking that the mempool didn't activate,
wait for the `zebrad` command to exit,
then check the entire log.
2022-02-25 23:36:20 +00:00
teor 499ae89c80
T2. Add isolated Tor connection API, but don't enable it by default (#3303)
* Add arti as a zebra-network dependency

* Add a method for isolated anonymised Tor connections to a specific hostname

* Add tests for isolated tor connections

* Use a shared tor client instance for all isolated connections

* Silence a spurious tor warning in tests

* Make tor support optional, activate it via a new "tor" feature

* Extra Cargo.lock changes

* fastmod AsyncReadWrite PeerTransport zebra*

* Remove unnecessary PeerTransport generics

* Refactor common test code into a function

* Don't drop the stream until the end of the test

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-01-25 01:46:31 +00:00
teor 00aa5d96a3
Consolidate standard lints into a cargo config file (#3386)
* Move standard lints into .cargo/config.toml

* Ignore "wrong self convention" in a futures-based trait

This lint might only trigger on beta or nightly at the moment.

* Warn if future incompatibile code is added to Zebra

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-01-24 16:25:06 +00:00
Janito Vaqueiro Ferreira Filho ec207cfa95
Ignore unexpected block responses to fix error cascade when synchronizing blocks (#3374)
* Refactor setup of `Connection` test vectors

Add a `new_test_connection` helper function to create a `Connection`
instance that's ready for testing.

* Check that no inbound requests are sent

Return the mock inbound service from `new_test_connection` and assert
that no requests were sent to it in any test.

* Replace `&mut Vec<u8>` with an `mpsc` channel

Make it easier to run the connection task in the background, i.e.,
remove any lifetime constraints from the borrowed buffer so that
`Connection` is `'static`.

It's now also easier to assert on individual messages sent from the
`Connection` instance.

* Make `MockServiceBuilder::finish` public

Allow test functions to be generic when creating a `MockService`, so
that caller functions actually determine if the type of `MockService`
assertions.

* Move `new_test_connection` to parent module

Make it more generic so that it can be used later in property tests as
well.

* Derive `Eq` and `PartialEq` for network `Response`

Allow intercepted `Response` instances to be easily compared in tests.

* Test block request cancel causes an error cascade

This is the scenario that caused the block synchronizer to reset every
few minutes, which made it considerably slower.

* Ignore unexpected block responses

It's likely that it's just a response for a previously cancelled block
request.
2022-01-20 08:14:16 +00:00
Deirdre Connolly 89b0403582
Enforce Rust edition 2021 (#3332)
* Rust edition 2021: zebra-network, cargo fix --edition and clippy --fix

* Rust edition 2021: zebra-chain, cargo fix --edition

* Rust edition 2021: tower-batch, cargo fix --edition

* Rust edition 2021: tower-fallback, cargo fix --edition

* Rust edition 2021: zebra-client, cargo fix --edition

* Rust edition 2021: zebra-consensus, cargo fix --edition

* Rust edition 2021: zebra-rpc, cargo fix --edition

* Rust edition 2021: zebra-state, cargo fix --edition

* Rust edition 2021: zebra-state, cargo fix --edition

* Rust edition 2021: zebra-test, cargo fix --edition

* Rust edition 2021: zebra-utils, cargo fix --edition

* Rust edition 2021: zebrad, cargo fix --edition

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-01-14 12:10:18 +00:00
teor d076b999f3
Fix syncer download order and add sync tests (#3168)
* Refactor so that RetryLimit::Future is std::marker::Sync

* Make the syncer future std::marker::Send by spawning tips futures

* Download synced blocks in chain order, not HashSet order

* Improve MockService failure messages

* Add closure-based responses to the MockService API

* Move MockChainTip to zebra-chain

* Add a MockChainTipSender type alias

* Support MockChainTip in ChainSync and its downloader

* Add syncer tests for obtain tips, extend tips, and wrong block hashes

* Add block too high tests for obtain tips and extend tips

* Add syncer tests for duplicate FindBlocks response hashes

* Allow longer request delays for mocked services in syncer tests
2022-01-11 14:11:35 -03:00
Janito Vaqueiro Ferreira Filho b71833292d
Use `MockedClientHandle` in other tests (#3241)
* Move `MockedClientHandle` to `peer` module

It's more closely related to a `Client` than the `PeerSet`, and this
prepares it to be used by other tests.

* Rename `MockedClientHandle` to `ClientTestHarness`

Reduce confusion, and clarify that the client is not mocked.

Co-authored-by: teor <teor@riseup.net>

* Add clarification to `mock_peers` documentation

Explicitly say how the generated data is returned.

* Rename method to `wants_connection_heartbeats`

The `Client` service only represents one direction of a connection, so
`is_connected` is not the exact term.

Co-authored-by: teor <teor@riseup.net>

* Mock `Client` instead of `LoadTrackedClient`

Move where the conversion from mocked `Client` to mocked
`LoadTrackedClient` in order to make the test helper more easily used by
other tests.

* Use `ClientTestHarness` in `initialize` tests

Replace the boilerplate code to create a fake `Client` instance with
usages of the `ClientTestHarness` constructor.

* Allow receiving requests from `Client` instance

Create a helper type to wrap the result, to make it easier to assert on
specific events after trying to receive a request.

* Allow inspecting the current error in the slot

Share the `ErrorSlot` between the `Client` and the handle, so that the
handle can be used to inspect the contents of the `ErrorSlot`.

* Allow placing an error into the `ErrorSlot`

Assuming it is initially empty. If it already has an error, the code
will panic.

* Allow gracefully closing the request receiver

Close the endpoint with the appropriate call to the `close()` method.

* Allow dropping the request receiver endpoint

Forcefully closes the endpoint.

* Rename field to `client_request_receiver`

Also rename the related methods to include
`outbound_client_request_receiver` to make it more precise.

Co-authored-by: teor <teor@riseup.net>

* Allow dropping the heartbeat shutdown receiver

Allows the `Client` to detect that the channel has been closed.

* Rename fn. to `drop_heartbeat_shutdown_receiver`

Make it clear that it affects the heartbeat task.

Co-authored-by: teor <teor@riseup.net>

* Move `NowOrLater` into a new `now-or-later` crate

Make it easily accessible to other crates.

* Add `IsReady` extension trait for `Service`

Simplifies checking if a service is immediately ready to be called.

* Add extension method to check for readiness error

Checks if the `Service` isn't immediately ready because a call to
`ready` immediately returns an error.

* Rename method to `is_failed`

Avoid negated method names.

Co-authored-by: teor <teor@riseup.net>

* Add a `IsReady::is_pending` extension method

Checks if a `Service` is not ready to be called.

* Use `ClientTestHarness` in `Client` test vectors

Reduce repeated code and try to improve readability.

* Create a new `ClientTestHarnessBuilder` type

A builder to create test `Client` instances using mock data which can be
tracked and manipulated through a `ClientTestHarness`.

* Allow configuring the `Client`'s mocked version

Add a `with_version` builder method.

* Use `ClientTestHarnessBuilder` in `PeerVersions`

Use the builder to set the peer version, so that the `version` parameter
can be removed from the constructor later.

* Use a default mock version where possible

Reduce noise when setting up the harness for tests that don't really
care about the remote peer version.

* Remove `Version` parameter from the `build` method

The `with_version` builder method should be used instead.

* Fix some typos and outdated info in the release checklist

* Add extra client tests for zero and multiple readiness checks (#3273)

And document existing tests.

* Replace `NowOrLater` with `futures::poll!` (#3272)

* Replace NowOrLater with the futures::poll! macro in zebrad

* Replace NowOrLater with the futures::poll! macro in zebra-test

* Remove the now-or-later crate

* remove unused imports

* rustfmt

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-12-22 06:13:26 +10:00
teor ea01e305ed
Silence expected anchor errors in tests (#3237) 2021-12-15 22:20:53 +00:00
Deirdre Connolly b973b7a622
Checking of Sprout anchors in non-finalized state (#3123)
* Do prelim checking of Sprout anchors in non-finalized state

Does not check intra-transaction interstitial states yet

* Populate sprout anchors to allow other state tests to pass

* Preliminary interstitial sprout note commitment tree anchor checks implementation

* Make sure only prior anchors are checked in the same transaction

* Add tests

* Refactor a comment

* Refactor rustdoc

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Use the first `JoinSplit`s from mainnet

* Print debug messages

* Use correct blocks for the tests

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-12-09 16:50:26 +00:00
Janito Vaqueiro Ferreira Filho 1f756fcc81
Add `zebra_test::init_async` helper function (#3169)
* Use a single-thread shared Tokio runtime

This allows it to pause the time and more closely resembles the
environment that's set by default for asynchronous tests.

* Add a `zebra_test::init_async` helper function

Calls `zebra_test::init` but also constructs a single-thread Tokio
runtime and returns it. This makes it simpler to initialize asynchronous
tests that can't use the `#[tokio::test]` attribute.

* Replace usages of `Runtime::new` in tests

Use the new `zebra_test::init_async()` helper function instead.

* Replace `runtime::Builder::new_current_thread()`

Use the new `zebra_test::init_async()` helper function instead.

* Replace `runtime::Builder::new_multi_thread()`

Use the new `zebra_test::init_async()` helper function instead. The test
with the change doesn't necessarily have to use a multi-thread runtime.
2021-12-09 00:18:17 +00:00
Deirdre Connolly e6ffe374d4
Validate sapling, orchard anchors (#3084)
* Add Transaction::sprout_joinsplits()

* Add Anchor variants to ValidateContextError

* Make Chain anchor collections pub(crate)

* tracing::instrument several methods in state

* Add contains_*_anchors methods to FinalizedState

* Add check::anchors module and function

* Verify that anchors_refer_to_earlier_treestates in when updating chains in non-finalized state

* Update zebra-state/src/service/check/anchors.rs

Co-authored-by: teor <teor@riseup.net>

* Add anchors() to sapling::ShieldedData

* Add sapling_anchors() to Transaction

* Use Transaction::sapling_anchors() in the anchors_refer_to_earlier_treestates() check

* Whoops, itertools

* Add a comment for improvement

Co-authored-by: teor <teor@riseup.net>

* Add & use a cfg(test) method on FinalizedState to prep test state with anchors to allow other tests to pass contextual checks

* Allow test nullifier checks to pass by populating anchor sets, allowing test anchor checks to pass

* Add mainnet block 419202 and its sapling note commitment tree root to test vectors

* Test sapling anchor verification using the first few Sapling blocks data

* Correct comment

* assert_eq instead of assert(bool)

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Update zebra-state/src/service/non_finalized_state.rs

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-11-30 16:05:35 +00:00
Alfredo Garcia dbd49a3f00
Validate coinbase expiration height (#3082)
* add testnet test blocks around nu5

* validate coinbase expiration height

* change const name and doc

Co-authored-by: teor <teor@riseup.net>

* change commit location

Co-authored-by: teor <teor@riseup.net>

* use pre Nu5 rules when there is no activation height

* add sapling final root to nu5 test vectors

* fix tests

Co-authored-by: teor <teor@riseup.net>
2021-11-23 05:17:05 +00:00
Marek 8963007397
Sprout note commitment trees (#3051)
* Implement incremental note commitment Merkle tree for Sprout

* Add tests for Sprout note commitment tree

* Remove the `Arbitrary` attribute

* Reverse the vector of empty roots

* Add more tests

* Refactor rustdoc

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Refactor rustdoc

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* rustdoc

* Rustdoc

* rustdoc links

* Oops, need the trait in scope to use it

* Avoid accessing the wrapped hash directly

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* rustfmt

* Add typing

* Avoid accessing the wrapped hash directly

* Implement incremental note commitment Merkle tree for Sprout

* Add tests for Sprout note commitment tree

* Remove the `Arbitrary` attribute

* Reverse the vector of empty roots

* Add more tests

* Refactor rustdoc

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Refactor rustdoc

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* rustdoc

* Rustdoc

* rustdoc links

* Oops, need the trait in scope to use it

* Avoid accessing the wrapped hash directly

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* rustfmt

* Add typing

* Avoid accessing the wrapped hash directly

* Add Overwinter final roots (test vectors)

* Test sprout note commitments trees on Overwinter blocks

* Add new test vectors

* Finish the tests for the note commitment trees

* Make the wrapped hash in `Root` private

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
2021-11-18 23:05:52 +00:00
Deirdre Connolly eda83ebe0e
Async Halo2 verifier service (#2645)
* First pass at async Halo2 verification service

Stubs out a batch verifier for the future.
The dependencies for orchard, halo2, librustzcash, zcash_primitives, have
not been resolved.

* Halo2 verifier service and test

* Remove redundant conversion

* Test async halo2 verifier service with pre-computed Orchard shielded data test vectors

* Fix typo

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Assert future result is_ok() in Halo2 verifier test

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Shorten tower::Service trait constraints for Halo2 verifier tests

* Remove commented out trait constraints

* .expect() vs .unwrap() to parse orchard::redpallas::VerificationKey

* Use .to_vec() for some test vectors

* Fix self-referential Display impl

* Fix deps

* Distinguish orchard vs zebra_chain::orchard imports

* Add test that halo2 verifier fails with malformed proof inputs

* Use thiserror for Halo2Error

* Use ZcashFoundation/orchard instead of dconnolly/orchard

* Add a link to the issue to remove the zfnd fork of orchard crate

* Update zebra-consensus/Cargo.toml

Co-authored-by: teor <teor@riseup.net>

* Add note

* Move artificial Orchard shielded data test vectors to zebra-test

* Align brackets

* Tidy some trait constraints and debug statements

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
2021-11-17 13:26:15 +10:00