Commit Graph

38 Commits

Author SHA1 Message Date
Alfredo Garcia 887d4a2457
fix(beta-toolchain): Warnings (#8404)
* fix beta imports and other warnings

* clippy lints beta

* move attributes to after docs

Co-authored-by: Marek <mail@marek.onl>

* move some code to test module

* implement display instead of direct tostring for LongPollId

Co-authored-by: Arya <aryasolhi@gmail.com>

* fix typo

---------

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>
2024-04-18 01:45:22 +00:00
idky137 4579722833
change(chain): Remove `Copy` trait impl from `Network` (#8354)
* removed derive copy from network, address and ledgerstate

* changed is_max_block_time_enforced to accept ref

* changed NetworkUpgrade::Current to accept ref

* changed NetworkUpgrade::Next to accept ref

* changed NetworkUpgrade::IsActivationHeight to accept ref

* changed NetworkUpgrade::TargetSpacingForHeight to accept ref

* changed NetworkUpgrade::TargetSpacings to accept ref

* changed NetworkUpgrade::MinimumDifficultySpacing_forHeight to accept ref

* changed NetworkUpgrade::IsTestnetMinDifficultyBlock to accept ref

* changed NetworkUpgrade::AveragingWindowTimespanForHeight to accept ref

* changed NetworkUpgrade::ActivationHeight to accept ref

* changed sapling_activation_height to accept ref

* fixed lifetime for target_spacings

* fixed sapling_activation_height

* changed transaction_to_fake_v5 and fake_v5_transactions_for_network to accept ref to network

* changed Input::vec_strategy to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_history.rs to accept ref to network

* changed functions in zebra-chain/src/history_tree.rs to accept ref to network

* changed functions in zebra-chain/src/history_tree.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/address.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/viewing_key* to accept ref to network

* changed functions in zebra-chain/src/transparent/address.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_primitives.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_note_encryption.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/history_tree* to accept ref to network

* changed functions in zebra-chain/src/block* to accept ref to network

* fixed errors in zebra-chain::parameters::network

* fixed errors in zebra-chain::parameters::network

* fixed errors in zebra-chain

* changed NonEmptyHistoryTree and InnerHistoryTree to hold value instead of ref

* changed NonEmptyHistoryTree and InnerHistoryTree to hold value instead of ref

* fixed errors in zebra-chain/src/block/arbitrary.rs

* finished fixing errors in zebra-chain - all crate tests pass

* changed functions in zebra-state::service::finalized_state to accept &Network

* changed functions in zebra-state::service::non_finalized_state to accept &Network

* zebra-state tests run but fail with overflow error

* zebra-state tests all pass

* converted zebra-network -- all crate tests pass

* applied all requested changes from review

* converted zebra-consensus -- all crate tests pass

* converted zebra-scan -- all crate tests pass

* converted zebra-rpc -- all crate tests pass

* converted zebra-grpc -- all crate tests pass

* converted zebrad -- all crate tests pass

* applied all requested changes from review

* fixed all clippy errors

* fixed build error in zebrad/src/components/mempool/crawler.rs
2024-03-19 20:45:27 +00:00
Arya 0a3790b73e
change(consensus): Remove Sprout and Sapling parameter download task and debug_skip_preload config (#7844)
* removes groth16 download task

* updates docs.

* Adds new config to test configs

* fixes compatibility with past configs

* uses inner config to deserialize removed field for compatibility

* update docs

* updates latest config

* Applies suggestions from code review

* Avoid duplicating hard-coded default values

---------

Co-authored-by: teor <teor@riseup.net>
2023-10-27 06:12:57 +00:00
teor 08ce2ad500
feature(mining): Enable mining RPCs by default in production builds (#7740)
* Enable getblocktemplate-rpcs in production builds

* Update mining instructions

* Consistently use ZF funding stream addresses in examples

* Simplify skip_getblocktemplate()

* Set the test miner address based on the network

* Correctly skip checks in lightwalletd send transactions test

* Wait longer for transaction query

* Wait longer before re-checking the mempool

* Skip mempool checks if it wasn't queried by lightwalletd

* rustfmt

* clippy and rustfmt

* Fix attribute issue

* Fix typo

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-10-18 04:15:17 +00:00
teor be5cfad07f
change(state): Prepare for in-place database format upgrades, but don't make any format changes yet (#7031)
* Move format upgrades to their own module and enum

* Launch a format change thread if needed, and shut it down during shutdown

* Add some TODOs and remove a redundant timer

* Regularly check for panics in the state upgrade task

* Only run example upgrade once, change version field names

* Increment database format to 25.0.2: add format change task

* Log the running and initial disk database format versions on startup

* Add initial disk and running state versions to cached state images in CI

* Fix missing imports

* Fix typo in logs workflow command

* Add a force_save_to_disk argument to the CI workflow

* Move use_internet_connection into zebrad_config()

* fastmod can_spawn_zebrad_for_rpc can_spawn_zebrad_for_test_type zebra*

* Add a spawn_zebrad_without_rpc() function

* Remove unused copy_state() test code

* Assert that upgrades and downgrades happen with the correct versions

* Add a kill_and_return_output() method for tests

* Add a test for new_state_format() versions (no upgrades or downgrades)

* Add use_internet_connection to can_spawn_zebrad_for_test_type()

* Fix workflow parameter passing

* Check that reopening a new database doesn't upgrade (or downgrade) the format

* Allow ephemeral to be set to false even if we don't have a cached state

* Add a test type that will accept any kind of state

* When re-using a directory, configure the state test config with that path

* Actually mark newly created databases with their format versions

* Wait for the state to be opened before testing the format

* Run state format tests on mainnet and testnet configs (no network access)

* run multiple reopens in tests

* Test upgrades run correctly

* Test that version downgrades work as expected (best effort)

* Add a TODO for testing partial updates

* Fix missing test arguments

* clippy if chain

* Fix typo

* another typo

* Pass a database instance to the format upgrade task

* Fix a timing issue in the tests

* Fix version matching in CI

* Use correct env var reference

* Use correct github env file

* Wait for the database to be written before killing Zebra

* Use correct workflow syntax

* Version changes aren't always upgrades

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-07-13 21:36:15 +00:00
teor 38206373b4
fix(ci): Add jobs to allow the full sync to finish before the first stable release (#6846)
* Delete an unused CI job that was previously partially deleted

* Add 2 more jobs to the full sync test

* Increase Rust test time: current expected time is 60 hours
2023-06-07 15:35:32 +00:00
Alfredo Garcia eb07bb31d6
rename(state): Rename state verifiers and related code (#6762)
* rename verifiers

* rename `PreparedBlock` to `SemanticallyVerifiedBlock`

* rename `CommitBlock` to `SemanticallyVerifiedBlock`

* rename `FinalizedBlock` to `CheckpointVerifiedBlock`

* rename `CommitFinalizedBlock` to `CommitCheckpointVerifiedBlock`

* rename `FinalizedWithTrees` to `ContextuallyVerifiedBlockWithTrees`

* rename `ContextuallyValidBlock` to `ContextuallyVerifiedBlock`

* change some `finalized` variables or function arguments to `checkpoint_verified`

* fix docs

* document the difference between `CheckpointVerifiedBlock` and `ContextuallyVerifiedBlock`

* fix doc links

* apply suggestions to request

Co-authored-by: Marek <mail@marek.onl>

* apply suggestions to service

Co-authored-by: Marek <mail@marek.onl>

* apply suggestions to finalized_state.rs and write.rs

Co-authored-by: Marek <mail@marek.onl>

* fmt

* change some more variable names

* change a few missing generics

* fix checkpoint log issue

* rename more `prepared` vars `semantically_verified`

* fix test regex

* fix test regex 2

---------

Co-authored-by: Marek <mail@marek.onl>
2023-06-01 12:29:03 +00:00
teor 1461c912f9
change(ci): Generate mainnet checkpoints in CI (#6550)
* Add extra test type modes to support zebra-checkpoints

* Add Mainnet and Testnet zebra-checkpoints test harnesses

* Add zebra-checkpoints to test docker images

* Add zebra-checkpoints test entrypoints

* Add Mainnet CI workflow for zebra-checkpoints

* Enable zebra-checkpoints feature in the test image

* Use the same features for (almost) all the docker tests

* Make workflow features match Docker features

* Add a feature note

* Add a zebra-checkpoints test feature to zebrad

* Remove the "no cached state" testnet code

* Log a startup message to standard error when launching zebra-checkpoints

* Rename tests to avoid partial name conflicts

* Fix log formatting

* Add sentry feature to experimental docker image build

* Explain what ENTRYPOINT_FEATURES is used for

* Use the correct zebra-checkpoints path

* Silence zebrad logs while generating checkpoints

* Fix zebra-checkpoints log handling

* Re-enable waiting for zebrad to fully sync

* Add documentation for how to run these tests individually

* Start generating checkpoints from the last compiled-in checkpoint

* Fix clippy lints

* Revert changes to TestType

* Wait for all the checkpoints before finishing

* Add more stderr debugging to zebra-checkpoints

* Fix an outdated module comment

* Add a workaround for zebra-checkpoints launch/run issues

* Use temp dir and log what it is

* Log extra metadata about the zebra-checkpoints binary

* Add note about unstable feature -Z bindeps

* Temporarily make the test run faster and with debug info

* Log the original test command name when showing stdout and stderr

* Try zebra-checkpoints in the system path first, then the cargo path

* Fix slow thread close bug in dual process test harness

* If the logs are shown, don't say they are hidden

* Run `zebra-checkpoints --help` to work out what's going on in CI

* Build `zebra-utils` binaries for `zebrad` integration tests

* Revert temporary debugging changes

* Revert changes that were moved to another PR
2023-04-27 04:39:43 +00:00
teor 9c15b14f86
fix(ci): Avoid inbound service overloads in tests (#6537)
* Silence an extremely verbose error in zebra-consensus tests

This disables around 10,000 logs like:
2023-04-18T02:46:28.441662Z  WARN init{config=Config { checkpoint_sync: true, debug_skip_parameter_preload: false } network=Mainnet debug_skip_parameter_preload=true}: unexpected error: Closed in state request while verifying previous state checkpoints

* Increase the outbound connection interval to 100ms

* Start the inbound service as soon as possible, and the syncer last

* Increase acceptance test time limits to get more debug info

* Add more debug info to inbound service overload tracing messages
2023-04-23 13:41:38 +00:00
teor 4daedbc789
fix(consensus): Check that Zebra's state contains the social consensus chain on startup (#6163)
* Allow missing docs directly on derived error types

* Make Request::BestChainBlockHash redirect to the ReadStateService

* Re-write the checkpoint_sync documentation based on the latest consensus rules

* Expose the underlying iterator for CheckpointList

* Validate existing state block hashes at startup, but ignore the result

* Monitor state block hash checkpoint task in the start command

* Fix indentation

* Make logging consistent

* Explain the config needed for full security

* Tidy required checkpoints docs, expand other docs

* Add security and deprecation changelog entries

* Replace task handle vector with a struct

* Add a test that this consensus-critical code actually runs and finishes

* Make some state methods and types available in tests

* Add missing docs

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-02-21 05:30:29 +00:00
teor 311daaa19e
Add 2 full sync CI jobs and 10 hours to the Rust test time limit (#6103) 2023-02-06 21:16:28 +00:00
teor 2f23de80f1
fix(clippy): Fix nightly clippy redundant code lints (#5916)
* Remove needless borrows

* Remove redundant into_iter()

* Remove unnecessary clone()
2023-01-05 15:46:04 +00:00
teor d8834c010e
fix(ci): Increase full sync jobs and timeout (#5781)
* Remove a redundant sprout full sync job

* Add two new full sync jobs

* Allow the full sync test to run for 48 hours (estimated current time 40-45 hours)
2022-12-06 11:36:05 +10:00
teor 353eee9910
Allow full sync tests to take 42 hours (#5708) 2022-11-23 23:07:37 +00:00
Arya c447b03223
fix(tests): add submitblock test to CI, and avoid copying the cached state directory in other tests (#5589)
* updates mod docs for tests that use future blocks

* updates submitblock test to use TestType methods

* prunes redundant code

* adds check_sync_logs_until

* adds assertion for needs cached state & rpc server

* updates get_raw_future_blocks fn with rpc calls

* updates to get_raw_future_blocks fn and submit_block test

* Rename LightwalletdTestType to TestType

* moves TestType and random_known_rpc_port_config to test_type.rs and config.rs

* moves get_raw_future_blocks to cached_state.rs

* updates ci workflows to include submit block test

* adds get_future_blocks fn and uses it in load_transactions_from_future_blocks

* updates CI docker

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Applies suggestions from code review

* Updates misnamed closure param

* updates mod docs for test_type.rs

Co-authored-by: teor <teor@riseup.net>
2022-11-10 03:40:21 +00:00
teor c4fad29824
fix(sync): Pause new downloads when Zebra reaches the lookahead limit (#5561)
* Use correct release for getblocktemplate config

* Include at least 2 full checkpoints in the lookahead limit

* Increase full sync timeout to 36 hours

* Only log "synced block height too far ahead of the tip" once

* Replace AboveLookaheadHeightLimit error with pausing the syncer

* Use AboveLookaheadHeightLimit for blocks a very long way from the tip

* Also add the getblocktemplate config, and fix the test message

* Remove an outdated TODO comment

* Allow syncing again when a small number of blocks are in the queue

* Allow some dead code
2022-11-09 04:42:04 +00:00
teor c812f880cf
cleanup(clippy): Use inline format strings (#5489)
* Inline format strings using an automated clippy fix

```sh
cargo clippy --fix --all-features --all-targets -- -A clippy::all -W clippy::uninlined_format_args
cargo fmt --all
```

* Remove unused & and &mut using an automated clippy fix

```sh
cargo clippy --fix --all-features --all-targets -- -A clippy::all -W clippy::uninlined_format_args
```
2022-10-27 13:25:18 +00:00
Alfredo Garcia 1937b6cdc0
fix(test): Wait for zebrad and lightwalletd to reach the tip in tests, to improve test coverage (#5164)
* Add RPC timing to zcash-rpc-diff

* Use transaction hash index for verbose block requests, rather than block data

* check if we are at tip for lightwallet wallet tests

* move function

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Combine the lightwalletd sync and gRPC test APIs

* Rewrite the gRPC and full sync tests for the new APIs

* Make zebra_rpc_address optional because only some tests need it

* Check for the zebrad RPC port to open in the right place

* Do the quick lightwalletd integration tests first in the sequential test function

* Ignore the lightwalletd cached state env var in tests that don't want it

* Don't replace the state path in RPC tests

* Enable IO (and timers) on the tip check tokio runtime

* Stop waiting for sync if either waiter thread errors or panics

* Try to speed up slow lightwalletd full syncs

* Don't wait for the tip in send transaction tests, and try to speed up full lightwalletd syncs

* Remove redundant is_lightwalletd_finished store

Co-authored-by: Arya <aryasolhi@gmail.com>

* Fix unused variable error

* Actually create the lightwalletd cached state

* Fix lwd cache check logic

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-10-06 04:12:27 +00:00
teor d6781de5e6
Increase full sync time to 32 hours (#5172) 2022-09-15 14:55:24 +00:00
teor dcbbf444b9
fix(ci): Increase the full sync timeout (#5129)
* Increase the full sync timeout to 28 hours

And put the timeout constant with the rest of them.

* Update developer documentation for the new constant location

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-09-13 03:52:23 +00:00
teor ea34baa8b8
add(test): test disabled `lightwalletd` mempool gRPCs via zebrad logs (#5016)
* add grpc mempool test research

* add a config flag for mempool injection of transactions in test

* Only copy the inner state directory in the send transactions test

* Preload Zcash parameters in some transaction verification tests

* Add a block and transaction Hash method to convert from display order bytes

* Update test coverage docs

* Add debugging output for mempool transaction verification

* Test fetching sent mempool transactions using gRPC

* Add extra log checks to the send transaction test

* Wait for zebrad mempool activation before running gRPC tests

* Update send transaction test for lightwalletd not returning mempool transactions

* Check zebrad logs instead of disabled lightwalletd gRPCs

* Add a debug option that makes RPCs pretend the sync is finished

* Remove an unused debug option

* Remove unused test code and downgrade some logs

* Fix test log checks

* Fix some rustdoc warnings

* Fix a compilation error due to new function arguments

* Make zebrad sync timeouts consistent and remove outdated code

* Document how to increase temporary directory space for tests

* Stop checking for a log that doesn't always happen

* Remove some commented-out code

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Update a comment about run time

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Add new config to new tests from the `main` branch

* Add transactions to the list, rather than replacing the list with each new block

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-09-06 13:32:33 +00:00
teor f46d0115e5
fix(test): Show full Zebra test panic details in CI logs (#4942)
* Handle test failure regexes using Result::Err, rather than panicking

* Add output logs to test context, and add tests for that

* Let empty test child logs be read again (and produce empty output)

* Ignore missing test children when killing with ignore_exited

* Fix a clippy lint

* Rename `line` to `line_result` for clarity

* Revert a redundant context_from() on kill()

* Only ignore "no such process" kill() errors in sync_until() tests

* Log the command timeout when an acceptance test fails

* fix clippy

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-08-28 23:52:19 +00:00
teor 4cda4eef66
fix(ci): Improve Zebra acceptance test diagnostics (#4958)
* Show the arguments of acceptance test functions in the logs

* Show all the logs in the "Run tests" jobs

* Document expected "broken pipe" error from `tee`

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-08-28 17:08:43 +00:00
teor 1d861b0d20
fix(ci): Increase full sync timeouts for longer syncs (#4961)
* Increase full sync timeout to 24 hours

Expected sync time is ~21 hours as of August 2022.

* Split final checkpoint job into two smaller jobs to avoid timeouts

Also make regexes easier to read.

* Fix a job name typo
2022-08-28 05:42:20 +10:00
Alfredo Garcia 9fb87425b7
fix(tests): Update timeout for Zebra sync tests (#4918)
* update timeout

* update the doc comment

* Increase test timeouts for Zebra update syncs

* Stop failing the 1740k job if the cached state is after block 1740k

Co-authored-by: teor <teor@riseup.net>
2022-08-24 10:06:18 +10:00
Conrado Gouvea 6fd750e168
build(deps): bump insta from 1.15.0 to 1.17.1 (#4884)
* build(deps): bump insta from 1.15.0 to 1.17.1

Bumps [insta](https://github.com/mitsuhiko/insta) from 1.15.0 to 1.17.1.
- [Release notes](https://github.com/mitsuhiko/insta/releases)
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mitsuhiko/insta/compare/1.15.0...1.17.1)

---
updated-dependencies:
- dependency-name: insta
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* make zebra_test::init() return the insta drop guard

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-08-04 15:44:44 +00:00
teor 65b0a8b6fa
fix(ci): split NU5 sync into two GitHub actions jobs (#4840)
* Split the NU5 job at block 1,740,000

* Fix the split regex

* Fix the lightwalletd sync to tip regex
2022-07-29 00:43:47 +00:00
teor 87f4308caf
fix(sync): Temporarily set full verification concurrency to 30 blocks (#4726)
* Return the maximum checkpoint height from the chain verifier

* Return the verified block height from the sync downloader

* Track the verified height in the syncer

* Use a lower concurrency limit during full verification

* Get the tip from the state before the first verified block

* Limit the number of submitted download and verify blocks in a batch

* Adjust lookahead limits when transitioning to full verification

* Keep unused extra hashes and submit them to the downloader later

* Remove redundant verified_height and state_tip()

* Split the checkpoint and full verify concurrency configs

* Decrease full verification concurrency to 5 blocks

10 concurrent blocks causes 3 minute stalls on some blocks on my machine.
(And it has about 4x as many cores as a standard machine.)

* cargo +stable fmt --all

* Remove a log that's verbose with smaller lookahead limits

* Apply the full verify concurrency limit to the inbound service

* Add a summary of the config changes to the CHANGELOG

* Increase the default full verify concurrency limit to 30
2022-07-06 10:13:57 -04:00
teor 11dcc13b84
fix(ci): make full sync go all the way to the tip (#4709)
* Checkout zebra in each job to avoid warnings

But put TODOs where we might be able to skip checkouts

* Split log following into sprout checkpoints, sapling/orchard checkpoints, and full validation

* Make job IDs shorter

* Use /dev/stderr because docker doesn't have a tty

* remove pipefail

* Revert "remove pipefail"

This reverts commit a7ee37bebdc107a4215e7dd307b189d925969234.

* Make tee ignore errors writing to a grep pipe

* Avoid launching multiple docker instances for duplicate jobs

* Ignore broken pipe error messages and statuses

* fix(ci): docker wait not finding container

We had this issue before, I can't recall if this was a parsing error between GitHub Actions and gcloud `--command` parsing, but we had to change this into two pieces.

This implementation keeps it how we did it before 9b9578c999/.github/workflows/test.yml (L235-L243)

* docs: remove pending TODO

We can't remove  `actions/checkout` nor set `create_credentials_file` to `false` as next steps won't be able to authenticate to GCP.

We can surely remove `actions/checkout` and leave `create_credentials_file` as `true`, but this will raise a warning on each step, and there's no benefit of doing so.

* Show `docker wait` and `gcloud ssh` output

* If `docker wait` fails, get the exit code using `docker inspect`

* Make full sync tests go all the way to the tip

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-06-30 15:14:30 +00:00
teor ca0520b2e8
change(deps): Upgrade tracing-subscriber and related dependencies (#4517)
* Upgrade tracing and related dependencies

```sh
cargo upgrade --workspace
tracing-error
tracing-subscrber

color-eyre

tracing-flame
tracing-journald

sentry
sentry-tracing

metrics
metrics-exporter-prometheus
reqwest
```

* Update duplicate dependency checks

* Enable the tracing/env-filter feature

* Fix type inference for metrics

Manual changes, plus:
```sh
fastmod "as _" "as f64"
```

* Tidy up some unrelated test code

* Update metrics-exporter-prometheus API

And make unused dependencies optional.

* Adjust test regexes to new tracing format

Also fix some regex bugs, and refactor to simplify.

* Disable color-eyre span traces and track caller in release builds

* Add a feature that enables extra debugging in release builds

* Clean up some redundant features

* Increase a test timeout
2022-06-01 13:53:51 +10:00
teor 4add7fc53c
Permanently fix unreliable sync finished log regex (#4504)
Unlike the other recent changes, this is a permanent fix.
2022-05-26 13:40:28 +10:00
teor 97b9911c7d
Stop full sync at 97%, but send transactions at 100% (#4483) 2022-05-25 13:14:38 +10:00
teor 49406f3132
fix(ci): Temporarily finish full sync at 99% (#4457)
* Finish full sync at 99.5%

* Finish full sync test harness at 99.5%

* Try 99.2% instead

* Try 99%
2022-05-24 17:59:09 +10:00
teor f7a3a0f6bc
fix(build): put gRPC tests behind an optional feature flag to fix production build issues (#4369)
* fix(ci): sentry is not longer being activated in test builds

This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.

* fix(build): workaround the failed to fetch oauth token error

* Drop sentry dependencies when enable-sentry feature is disabled

* Make lightwalletd gRPC tests depend on a new lightwalletd-grpc-tests feature

* fix(ci): remove enable-sentry feature from tests

* Add lightwalletd-grpc-tests feature for functionality or efficiency

And document where it is just used to stop re-compilations.

* Remove redundant `cmake` and `protobuf-compiler` dependencies

* Document Zebra's optional production and test feature flags

* Minimise dependencies in zcash-params/Dockerfile

* Minimise dependencies in docker/Dockerfile

* Add a workflow TODO

Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-05-11 14:06:58 +00:00
Janito Vaqueiro Ferreira Filho 5a94a09292
add(test): Integration test to send transactions using lightwalletd (#4068)
* Export the `zebra_state::Config::db_path` method

Make it easier for tests to discover the sub-directory used to store
the chain state data.

* Generate code for interfacing with lightwalletd

Use the `tonic-build` crate to generate Rust code for communicating with
lightwalletd using gRPC.

The `*.proto` files were obtained from the Zcash lightwalletd
repository.

* Use `block::Height` instead of `Height`

Import the `block` instead to make it slightly clearer.

* Add helper function to remove a file if it exists

Try to remove it and ignore an error if it says that the file doesn't
exist. This will be used later to remove the lock file from a copied
chain state directory.

* Add helper function to copy chain state dirs

Copy an existing chain state directory into a new temporary directory.

* Add a `BoxStateService` type alias

Make it easier to write and read a boxed version of a state service.

* Add a helper function to start the state service

Make it easier to specify the state service to use an existing state
cache directory.

* Import `eyre!` macro at the module level

Allow it to be used in different places without having to repeat the
imports.

* Add `load_tip_height_from_state_directory` helper

A function to discover the current chain tip height stored in a state
cache.

* Add helper function to prepare partial sync. state

Loads a partially synchronized cached state directory into a temporary
directory that can be used by a zebrad instance, and also returns the
chain tip block height of that state.

* Add `perform_full_sync_starting_from` helper

Runs a zebrad with an existing partially synchronized state, and
finishes synchronizing it to the network chain tip.

* Add function to load transactions from a block

Use a provided state service to load all transactions from a block at a
specified height.

The state service is a generic type parameter, because
`zebra_state::service::ReadStateService` is not exported publicly. Using
a generic type parameter also allows the service to be wrapped in layers
if needed in the future.

* Add `load_transactions_from_block_after` helper

A function to load transactions from a block stored in a cached state
directory. The cached state must be synchronized to a chain tip higher
than the requested height.

* Add helper function to load some test transactions

Given a partially synchronized chain state, it will extend that chain by
performing a full synchronization, and obtain some transactions from one
of the newly added blocks.

* Update `spawn_zebrad_for_rpc_without_initial_peers`

Wait until the mempool is activated.

* Add method to start lightwalletd with RPC server

Returns the lightwalletd instance and the port that it's listening for
RPC connections.

The instance can reuse an existing cached lightwalletd state if the
`LIGHTWALLETD_DATA_DIR` environment variable is set.

* Add a `LightwalletdRpcClient` type alias

To make it easier to identify the type generated from the Protobuf
files.

* Add helper function to connect to lightwalletd

Prepare an RPC client to send requests to a lightwalletd instance.

* Add a `prepare_send_transaction_request` helper

Creates a request message for lightwalletd to send a transaction.

* Add test to send transactions using lightwalletd

Obtain some valid transactions from future blocks and try to send them
to a lightwalletd instance connected to a zebrad instance that hasn't
seen those transactions yet. The transactions should be successfully
queued in Zebra's mempool.

* Make `zebra_directory` parameter generic

Allow using a `TempDir` or a `PathBuf`.

* Move lightwalletd protobuf files

Place them closer to the module directory, so that it's clearer that
they specify the RPC protocol for lightwalletd, and not Zebra itself.

* Don't use coinbase transactions in the test

Coinbase transactions are rejected by the mempool.

* Don't remove state lock file

It is removed automatically by Zebra when it shuts down, so if it exists
it should be reported as a bug.

* Force mempool to be enabled in Zebrad instance

Speed up the initialization of the Zebrad instance used for lightwalletd
to connect to.

* Refactor to create `LIGHTWALLETD_DATA_DIR_VAR`

Document how the environment variable can be used to speed up the test.

* Check for process errors in spawned Zebra instance

Enable checking for known process failure messages.

* Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant

Document why it exists and how the choice of the value affects the test.

* Add `LIGHTWALLETD_TEST_TIMEOUT` constant

And use it for the Zebrad and the Lightwalletd instances used in the
send transaction integration test.

* Check `lightwalletd` process for errors

Enable checking the lightwalletd process for known failure messages.

* Update `tonic` and `prost` dependencies

Use the latest version and fix CI failures because `rustfmt` isn't
installed in the build environment.

* Create `send_transaction_test` module

Move the send transaction using lightwalletd test and its helper
functions into a new module.

* Move `LIGHTWALLETD_TEST_TIMEOUT` constant

Place it in the parent `lightwalletd` module.

* Move gRPC helper functions and types to `rpc` mod.

Make them more accessible so that they can be used by other tests.

* Create a `cached_state` module

Move the test utility functions related to using a cached Zebra state
into the module.

* Move `perform_full_sync_starting_from` to `sync`

Keep to closer to the synchronization utility functions.

* Move Zebra cached state path variable constant

Place it in the `cached_state` module.

* Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set

Make it part of the set of tests ignored as a whole if no lightwalletd
tests should be executed.

* Move `spawn_zebrad_for_rpc_without_initial_peers`

Place it in the `launch` sub-module.

* Rename `rpc` module into `wallet_grpc`

Avoid any potential misunderstandings when the name is seen out of
context.

* Allow duplicate `heck` dependency

At least until `structopt` is updated or `zebra-utils` is updated to use
`clap` 3.

* Fix a deny.toml typo

* fix(build): CMake is required by `prost` crate

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-04-27 23:06:11 +00:00
Janito Vaqueiro Ferreira Filho c47dac8d5f
change(test): Refactor how extra arguments are handled when spawing lightwalled (#4067)
* Join similar imports

Avoid the confusion that might cause one to think that they come from
different modules or crates.

* Create an `Arguments` helper type

A type to keep track of a list of arguments for a sub-process. It makes
it easier for overriding parameters with new values.

* Create an `args!` helper macro

Make it simpler to create `Arguments` instances with known values.

* Require `Arguments` for `spawn_child` method

Change the method to have an `Arguments` parameter, and merge it with
some default values before passing them forward.

* Use `Arguments` in `spawn_lightwalletd_child`

Change the method to use an `Arguments` instance, and merge it with some
default options.

* Use `Arguments` in `spawn_child_with_command`

Require an `Arguments` instance in the `spawn_child_with_command`
extension method. Makes it simpler to call from `spawn_child` and
`spawn_lightwalletd_child` extension methods.

* Test if argument order is preserved

Check that when building an `Arguments` instance, the order that the
arguments are set is preserved in the generated list of strings.

* Refactor test to improve readability

Also separates some common code to be reused by later tests.

* Test overriding arguments

Check to see if overriding arguments behaves as expected, by keeping the
argument order when overriding and not introducing duplicates.

* Refactor test to improve readability

Move out a chunk of code so that the test itself is easier to read and
to make that code reusable by a later test.

* Test that `Arguments` instances can be merged

Merge two `Arguments` instances built from two lists of arguments, and
check that the expanded strings preserve order and override rules.

* Add Eq derives on Arguments

Co-authored-by: teor <teor@riseup.net>
2022-04-19 10:28:52 +00:00
teor a5d7b9c1e0
T2. add(test): add test API that checks process logs for failures (#3899)
* Revert "Revert Option<Child> process handling"

This reverts commit 2af30086858d104dcb0ec87383996c36bcaa7371.

* Add a set of failure regexes to test command output

* Allow debug-printing TestChild again

* When the child is dropped, check any remaining output

* Document a wait_with_output edge case

* Improve failure regex panic output

* Improve builder ergonomics

* Add internal tests for failure regex panics

It would be easy to disable these panics, and never realise.

* Add some module structure TODOs

* Stop panicking if the child process has already been taken

* Add test APIs for consuming child output lines

* Fix a hang on child process drop

* Handle output being already taken in wait_with_output

And document some edge cases we don't handle yet

* Use bash's read command in the TestChild stderr test

And check the actual command we're using to see if it errors.

* Pretty print full failure regex list

* Add the test child command line to the failure regex logs
2022-03-22 23:53:24 +00:00
teor 9a8ab9468d
T0. refactor(test): split zebrad acceptance tests into sub-modules (#3901)
* Improve launch delay docs

* Initial split of zebrad acceptance tests into modules

* Split shared lightwalletd test code into a module
2022-03-18 16:02:22 +00:00