* readd the scanner tests in the new binary context
* remove commented out tests from zebrad
* add Cargo.lock
* add local copy of `ZECPAGES_SAPLING_VIEWING_KEY` to binary scanning tests
* moves scanner-results-reader to zebra-scan, adds zebrad bin to zebra-scan, adds a short sleep before trying to connect to gRPC server in test
* always include zebrad dependency in zebra-scan
* renames zebrad bin in zebra-scan to zebrad-for-scanner
* skip test when a cached state is missing
* remove scanning config from zebrad
* move `scan_task_commands` out of zebrad acceptance tests
* remove scanner config from zebrad
* do not panic if env var is not there
* re enable start_scan_where left, remove shielded-scan feature foir scanning tests in CI
* change app names
* uncomment test
* fix the scanner tests in CI by changing the test command
* add log needed by CI?
* move test
* rename trait, docs fix
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* Always activate Canopy at Height(1) on Regtest
* Ignores the zip 212 grace period on configured Testnets and Regtest
* - Returns early when there is no Heartwood activation height when creating or updating HistoryTree
- Skips call to `check::legacy_chain()` when no NU5 activation height is set (it would return immediately anyway)
- Replaces `.map()` with `.filter_map()` in `NetworkUpgrade::target_spacings()`
- Removes outdated TODO
* - When proof of work is disabled, skips checking if Zebra is synced to the network tip in the getblocktemplate method
* Sets full_verifier_utxo_lookahead to Height::MIN instead of panicking
* When network is regtest, skips starting sync task and commits the genesis block if it's missing in the state
* updates/fixes test config
* Adds test for committing Canopy blocks on Regtest
* - Updates median time past and difficulty checks to use fewer than 11/17/28 blocks
* uses SLOW_START_INTERVAL of 0 if PoW is disabled, adds TODOs
* Update getblocktemplate method to return reserved chain history activation root hash, uses Nu5 at height 1
Test passes.
* Updates test to expect NU5 as the default nu activation at Height(1)
* Removes invalid difficulty snapshot
* fixes tests
* removes regtest NU5 activation height config field
* Apply suggestions from code review
* Update zebra-state/src/service/check/difficulty.rs
Co-authored-by: Marek <mail@marek.onl>
* Update zebra-state/src/service/check/difficulty.rs
* Updates docs/comments, renames an argument
---------
Co-authored-by: Marek <mail@marek.onl>
* moves ScanTask to its own module in service module
* moves `process_messages()` method to scan_task.rs
* uses get_mut() and returns new keys
* updates types and adds scan_until() function
* adds and uses wait_for_height() function
* spawns scan_until_task_handler to poll scan_until_tasks FuturesUnordered
* corrects comment
* adds TODO
* updates expected test log, corrects panic msg, fixes fmt
* moves scan functions to scan_task module
* moves ScanTaskCommand and related method impls to its own module
* moves `ScanTask::mock()` test constructor to its own module
* creates `add_keys` mod and moves `scan_until()` function there
* moves scan task executor to its own module and adds ScanRangeTaskBuilder type
* renames add_keys to scan_range, moves executor to scan_task mod
* adds test for process_messages
* updates scan_height_and_store_results() to skip last_scanned_height check if there is no key height for a key in `parsed_keys`
* updates `default_test_config()`
* adds acceptance test for registering keys in ScanTask
* uses the right feature for the new acceptance test
* Applies suggestions from code review
* Add extra test type modes to support zebra-checkpoints
* Add Mainnet and Testnet zebra-checkpoints test harnesses
* Add zebra-checkpoints to test docker images
* Add zebra-checkpoints test entrypoints
* Add Mainnet CI workflow for zebra-checkpoints
* Enable zebra-checkpoints feature in the test image
* Use the same features for (almost) all the docker tests
* Make workflow features match Docker features
* Add a feature note
* Add a zebra-checkpoints test feature to zebrad
* Remove the "no cached state" testnet code
* Log a startup message to standard error when launching zebra-checkpoints
* Rename tests to avoid partial name conflicts
* Fix log formatting
* Add sentry feature to experimental docker image build
* Explain what ENTRYPOINT_FEATURES is used for
* Use the correct zebra-checkpoints path
* Silence zebrad logs while generating checkpoints
* Fix zebra-checkpoints log handling
* Re-enable waiting for zebrad to fully sync
* Add documentation for how to run these tests individually
* Start generating checkpoints from the last compiled-in checkpoint
* Fix clippy lints
* Revert changes to TestType
* Wait for all the checkpoints before finishing
* Add more stderr debugging to zebra-checkpoints
* Fix an outdated module comment
* Add a workaround for zebra-checkpoints launch/run issues
* Use temp dir and log what it is
* Log extra metadata about the zebra-checkpoints binary
* Add note about unstable feature -Z bindeps
* Temporarily make the test run faster and with debug info
* Log the original test command name when showing stdout and stderr
* Try zebra-checkpoints in the system path first, then the cargo path
* Fix slow thread close bug in dual process test harness
* If the logs are shown, don't say they are hidden
* Run `zebra-checkpoints --help` to work out what's going on in CI
* Build `zebra-utils` binaries for `zebrad` integration tests
* Revert temporary debugging changes
* Revert changes that were moved to another PR
* Rename variables so it's clearer what they do
* Fully document zebra-checkpoints arguments
* Remove some outdated references to zcashd
* Add a json-conversion feature for converting JSON to valid Heights
* Simplify zebra-checkpoints code using conversion methods
* Track the last checkpoint height rather than the height gap
* Move all the CLI-specific code into a single function
* Remove cfg(feature) from the test RPC client API
* Move the RpcRequestClient into zebra-node-services
* Fix the case of RpcRequestClient
* Add transport and addr arguments to zebra-checkpoints
* Make zebra-checkpoints support both CLI and direct JSON-RPC connections
* Fix RpcRequestClient compilation
* Add a suggestion for zcashd authentication failures
* Document required features
* Handle differences in CLI & direct parameter and response formats
* Replace a custom function with an existing dependency function
* Add a checkpoint list test for minimum height gaps
* updates mod docs for tests that use future blocks
* updates submitblock test to use TestType methods
* prunes redundant code
* adds check_sync_logs_until
* adds assertion for needs cached state & rpc server
* updates get_raw_future_blocks fn with rpc calls
* updates to get_raw_future_blocks fn and submit_block test
* Rename LightwalletdTestType to TestType
* moves TestType and random_known_rpc_port_config to test_type.rs and config.rs
* moves get_raw_future_blocks to cached_state.rs
* updates ci workflows to include submit block test
* adds get_future_blocks fn and uses it in load_transactions_from_future_blocks
* updates CI docker
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Applies suggestions from code review
* Updates misnamed closure param
* updates mod docs for test_type.rs
Co-authored-by: teor <teor@riseup.net>
* adds submitblock rpc method
* re-orders imports
* replaces thread::yield_now with async yield_now
* Fix doc warnings and unused variable warnings, add missing docs
* Mark work_id as optional
* Use the same ChainVerifier for downloaded and submitted blocks
* Revert unused changes & minor cleanups
* Document currently-unreachable code
* updates tests and submit_block response for AlreadyVerified error
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* changes names from BlockVerifier to ChainVerifier and block_verifier to chain_verifier to keep it consistent with naming in zebra-consensus
* move how to run the submit_block test example to acceptance.rs
* updates snapshot tests
* moved acceptance test to a separate file
* removes extra tower::ServiceBuilder::new(), updates docs
* updates vectors and snapshot tests, changes hex decoding error in submit_block method from server error to parse error
* hides errors module in zebra-rpc behind a feature flag and adds docs.
* Updates snapshot test, adds mod docs, moves HexData to its own mod, and removes the unrelated make_server_error refactor for now
* update submit block acceptance test mod doc
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Fix the syntax of links in comments
* Fix a mistake in the docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove unnecessary angle brackets from a link
* Revert the changes for links that serve as references
* Revert "Revert the changes for links that serve as references"
This reverts commit 8b091aa9fa.
* Remove `<` `>` from links that serve as references
This reverts commit 046ef25620.
* Don't use `<` `>` in normal comments
* Don't use `<` `>` for normal comments
* Revert changes for comments starting with `//`
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Make the lightwalletd integration test take a test type
* Configure lightwalletd tests based on the test type
* Remove obsolete kill_on_error() in the lightwalletd test
* Refactor to simplify the test function
* Move LightwalletdTestType to the lightwalletd module
* Create a test function that runs the full lightwalletd test suite
* Actually use the cached Zebra state
* Add checks for the new integration test modes
* Populate the lightwalletd state dir in the FullSyncFromGenesis test
* Fix up state handling, fail earlier if state is invalid
* Adjust timeouts and regex escapes
* Make state requirements for each test stricter
* Move configs to the top of the test function
* Allow unexpected lightwalletd cached state in some tests
* Speed up tests slightly by removing an intermittent log check
* Move timeout selection into test type methods
* Move failure messages into test type methods
* Turn a function argument into an enum field
* Check lightwalletd state directly, rather than Zebra RPC results
* Update gRPC tests for function argument changes
* Remove duplicate env var constant and redundant code
* Export the `zebra_state::Config::db_path` method
Make it easier for tests to discover the sub-directory used to store
the chain state data.
* Generate code for interfacing with lightwalletd
Use the `tonic-build` crate to generate Rust code for communicating with
lightwalletd using gRPC.
The `*.proto` files were obtained from the Zcash lightwalletd
repository.
* Use `block::Height` instead of `Height`
Import the `block` instead to make it slightly clearer.
* Add helper function to remove a file if it exists
Try to remove it and ignore an error if it says that the file doesn't
exist. This will be used later to remove the lock file from a copied
chain state directory.
* Add helper function to copy chain state dirs
Copy an existing chain state directory into a new temporary directory.
* Add a `BoxStateService` type alias
Make it easier to write and read a boxed version of a state service.
* Add a helper function to start the state service
Make it easier to specify the state service to use an existing state
cache directory.
* Import `eyre!` macro at the module level
Allow it to be used in different places without having to repeat the
imports.
* Add `load_tip_height_from_state_directory` helper
A function to discover the current chain tip height stored in a state
cache.
* Add helper function to prepare partial sync. state
Loads a partially synchronized cached state directory into a temporary
directory that can be used by a zebrad instance, and also returns the
chain tip block height of that state.
* Add `perform_full_sync_starting_from` helper
Runs a zebrad with an existing partially synchronized state, and
finishes synchronizing it to the network chain tip.
* Add function to load transactions from a block
Use a provided state service to load all transactions from a block at a
specified height.
The state service is a generic type parameter, because
`zebra_state::service::ReadStateService` is not exported publicly. Using
a generic type parameter also allows the service to be wrapped in layers
if needed in the future.
* Add `load_transactions_from_block_after` helper
A function to load transactions from a block stored in a cached state
directory. The cached state must be synchronized to a chain tip higher
than the requested height.
* Add helper function to load some test transactions
Given a partially synchronized chain state, it will extend that chain by
performing a full synchronization, and obtain some transactions from one
of the newly added blocks.
* Update `spawn_zebrad_for_rpc_without_initial_peers`
Wait until the mempool is activated.
* Add method to start lightwalletd with RPC server
Returns the lightwalletd instance and the port that it's listening for
RPC connections.
The instance can reuse an existing cached lightwalletd state if the
`LIGHTWALLETD_DATA_DIR` environment variable is set.
* Add a `LightwalletdRpcClient` type alias
To make it easier to identify the type generated from the Protobuf
files.
* Add helper function to connect to lightwalletd
Prepare an RPC client to send requests to a lightwalletd instance.
* Add a `prepare_send_transaction_request` helper
Creates a request message for lightwalletd to send a transaction.
* Add test to send transactions using lightwalletd
Obtain some valid transactions from future blocks and try to send them
to a lightwalletd instance connected to a zebrad instance that hasn't
seen those transactions yet. The transactions should be successfully
queued in Zebra's mempool.
* Make `zebra_directory` parameter generic
Allow using a `TempDir` or a `PathBuf`.
* Move lightwalletd protobuf files
Place them closer to the module directory, so that it's clearer that
they specify the RPC protocol for lightwalletd, and not Zebra itself.
* Don't use coinbase transactions in the test
Coinbase transactions are rejected by the mempool.
* Don't remove state lock file
It is removed automatically by Zebra when it shuts down, so if it exists
it should be reported as a bug.
* Force mempool to be enabled in Zebrad instance
Speed up the initialization of the Zebrad instance used for lightwalletd
to connect to.
* Refactor to create `LIGHTWALLETD_DATA_DIR_VAR`
Document how the environment variable can be used to speed up the test.
* Check for process errors in spawned Zebra instance
Enable checking for known process failure messages.
* Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant
Document why it exists and how the choice of the value affects the test.
* Add `LIGHTWALLETD_TEST_TIMEOUT` constant
And use it for the Zebrad and the Lightwalletd instances used in the
send transaction integration test.
* Check `lightwalletd` process for errors
Enable checking the lightwalletd process for known failure messages.
* Update `tonic` and `prost` dependencies
Use the latest version and fix CI failures because `rustfmt` isn't
installed in the build environment.
* Create `send_transaction_test` module
Move the send transaction using lightwalletd test and its helper
functions into a new module.
* Move `LIGHTWALLETD_TEST_TIMEOUT` constant
Place it in the parent `lightwalletd` module.
* Move gRPC helper functions and types to `rpc` mod.
Make them more accessible so that they can be used by other tests.
* Create a `cached_state` module
Move the test utility functions related to using a cached Zebra state
into the module.
* Move `perform_full_sync_starting_from` to `sync`
Keep to closer to the synchronization utility functions.
* Move Zebra cached state path variable constant
Place it in the `cached_state` module.
* Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set
Make it part of the set of tests ignored as a whole if no lightwalletd
tests should be executed.
* Move `spawn_zebrad_for_rpc_without_initial_peers`
Place it in the `launch` sub-module.
* Rename `rpc` module into `wallet_grpc`
Avoid any potential misunderstandings when the name is seen out of
context.
* Allow duplicate `heck` dependency
At least until `structopt` is updated or `zebra-utils` is updated to use
`clap` 3.
* Fix a deny.toml typo
* fix(build): CMake is required by `prost` crate
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
Marek
Alfredo Garcia
Arya
teor
Janito Vaqueiro Ferreira Filho