* Expect a Zebra cached state with at least a million blocks
* Set the zebrad timeout and failure messages based on the test type
* Temporarily stop requiring cached lightwalletd state for the send transaction tests
* add info to lightwalletd_state_path()
* fix getaddresstxids rpc
* normalize more the env vars in lightwalletd tests
* extra logging
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* add a test that call all the lightwalletd grpc methods
* fix some docs
* use ZF address in tests for balance and utxos
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make the lightwalletd integration test take a test type
* Configure lightwalletd tests based on the test type
* Remove obsolete kill_on_error() in the lightwalletd test
* Refactor to simplify the test function
* Move LightwalletdTestType to the lightwalletd module
* Create a test function that runs the full lightwalletd test suite
* Actually use the cached Zebra state
* Add checks for the new integration test modes
* Populate the lightwalletd state dir in the FullSyncFromGenesis test
* Fix up state handling, fail earlier if state is invalid
* Adjust timeouts and regex escapes
* Make state requirements for each test stricter
* Move configs to the top of the test function
* Allow unexpected lightwalletd cached state in some tests
* Speed up tests slightly by removing an intermittent log check
* Move timeout selection into test type methods
* Move failure messages into test type methods
* Turn a function argument into an enum field
* Check lightwalletd state directly, rather than Zebra RPC results
* Update gRPC tests for function argument changes
* Remove duplicate env var constant and redundant code
* Export the `zebra_state::Config::db_path` method
Make it easier for tests to discover the sub-directory used to store
the chain state data.
* Generate code for interfacing with lightwalletd
Use the `tonic-build` crate to generate Rust code for communicating with
lightwalletd using gRPC.
The `*.proto` files were obtained from the Zcash lightwalletd
repository.
* Use `block::Height` instead of `Height`
Import the `block` instead to make it slightly clearer.
* Add helper function to remove a file if it exists
Try to remove it and ignore an error if it says that the file doesn't
exist. This will be used later to remove the lock file from a copied
chain state directory.
* Add helper function to copy chain state dirs
Copy an existing chain state directory into a new temporary directory.
* Add a `BoxStateService` type alias
Make it easier to write and read a boxed version of a state service.
* Add a helper function to start the state service
Make it easier to specify the state service to use an existing state
cache directory.
* Import `eyre!` macro at the module level
Allow it to be used in different places without having to repeat the
imports.
* Add `load_tip_height_from_state_directory` helper
A function to discover the current chain tip height stored in a state
cache.
* Add helper function to prepare partial sync. state
Loads a partially synchronized cached state directory into a temporary
directory that can be used by a zebrad instance, and also returns the
chain tip block height of that state.
* Add `perform_full_sync_starting_from` helper
Runs a zebrad with an existing partially synchronized state, and
finishes synchronizing it to the network chain tip.
* Add function to load transactions from a block
Use a provided state service to load all transactions from a block at a
specified height.
The state service is a generic type parameter, because
`zebra_state::service::ReadStateService` is not exported publicly. Using
a generic type parameter also allows the service to be wrapped in layers
if needed in the future.
* Add `load_transactions_from_block_after` helper
A function to load transactions from a block stored in a cached state
directory. The cached state must be synchronized to a chain tip higher
than the requested height.
* Add helper function to load some test transactions
Given a partially synchronized chain state, it will extend that chain by
performing a full synchronization, and obtain some transactions from one
of the newly added blocks.
* Update `spawn_zebrad_for_rpc_without_initial_peers`
Wait until the mempool is activated.
* Add method to start lightwalletd with RPC server
Returns the lightwalletd instance and the port that it's listening for
RPC connections.
The instance can reuse an existing cached lightwalletd state if the
`LIGHTWALLETD_DATA_DIR` environment variable is set.
* Add a `LightwalletdRpcClient` type alias
To make it easier to identify the type generated from the Protobuf
files.
* Add helper function to connect to lightwalletd
Prepare an RPC client to send requests to a lightwalletd instance.
* Add a `prepare_send_transaction_request` helper
Creates a request message for lightwalletd to send a transaction.
* Add test to send transactions using lightwalletd
Obtain some valid transactions from future blocks and try to send them
to a lightwalletd instance connected to a zebrad instance that hasn't
seen those transactions yet. The transactions should be successfully
queued in Zebra's mempool.
* Make `zebra_directory` parameter generic
Allow using a `TempDir` or a `PathBuf`.
* Move lightwalletd protobuf files
Place them closer to the module directory, so that it's clearer that
they specify the RPC protocol for lightwalletd, and not Zebra itself.
* Don't use coinbase transactions in the test
Coinbase transactions are rejected by the mempool.
* Don't remove state lock file
It is removed automatically by Zebra when it shuts down, so if it exists
it should be reported as a bug.
* Force mempool to be enabled in Zebrad instance
Speed up the initialization of the Zebrad instance used for lightwalletd
to connect to.
* Refactor to create `LIGHTWALLETD_DATA_DIR_VAR`
Document how the environment variable can be used to speed up the test.
* Check for process errors in spawned Zebra instance
Enable checking for known process failure messages.
* Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant
Document why it exists and how the choice of the value affects the test.
* Add `LIGHTWALLETD_TEST_TIMEOUT` constant
And use it for the Zebrad and the Lightwalletd instances used in the
send transaction integration test.
* Check `lightwalletd` process for errors
Enable checking the lightwalletd process for known failure messages.
* Update `tonic` and `prost` dependencies
Use the latest version and fix CI failures because `rustfmt` isn't
installed in the build environment.
* Create `send_transaction_test` module
Move the send transaction using lightwalletd test and its helper
functions into a new module.
* Move `LIGHTWALLETD_TEST_TIMEOUT` constant
Place it in the parent `lightwalletd` module.
* Move gRPC helper functions and types to `rpc` mod.
Make them more accessible so that they can be used by other tests.
* Create a `cached_state` module
Move the test utility functions related to using a cached Zebra state
into the module.
* Move `perform_full_sync_starting_from` to `sync`
Keep to closer to the synchronization utility functions.
* Move Zebra cached state path variable constant
Place it in the `cached_state` module.
* Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set
Make it part of the set of tests ignored as a whole if no lightwalletd
tests should be executed.
* Move `spawn_zebrad_for_rpc_without_initial_peers`
Place it in the `launch` sub-module.
* Rename `rpc` module into `wallet_grpc`
Avoid any potential misunderstandings when the name is seen out of
context.
* Allow duplicate `heck` dependency
At least until `structopt` is updated or `zebra-utils` is updated to use
`clap` 3.
* Fix a deny.toml typo
* fix(build): CMake is required by `prost` crate
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Join similar imports
Avoid the confusion that might cause one to think that they come from
different modules or crates.
* Create an `Arguments` helper type
A type to keep track of a list of arguments for a sub-process. It makes
it easier for overriding parameters with new values.
* Create an `args!` helper macro
Make it simpler to create `Arguments` instances with known values.
* Require `Arguments` for `spawn_child` method
Change the method to have an `Arguments` parameter, and merge it with
some default values before passing them forward.
* Use `Arguments` in `spawn_lightwalletd_child`
Change the method to use an `Arguments` instance, and merge it with some
default options.
* Use `Arguments` in `spawn_child_with_command`
Require an `Arguments` instance in the `spawn_child_with_command`
extension method. Makes it simpler to call from `spawn_child` and
`spawn_lightwalletd_child` extension methods.
* Test if argument order is preserved
Check that when building an `Arguments` instance, the order that the
arguments are set is preserved in the generated list of strings.
* Refactor test to improve readability
Also separates some common code to be reused by later tests.
* Test overriding arguments
Check to see if overriding arguments behaves as expected, by keeping the
argument order when overriding and not introducing duplicates.
* Refactor test to improve readability
Move out a chunk of code so that the test itself is easier to read and
to make that code reusable by a later test.
* Test that `Arguments` instances can be merged
Merge two `Arguments` instances built from two lists of arguments, and
check that the expanded strings preserve order and override rules.
* Add Eq derives on Arguments
Co-authored-by: teor <teor@riseup.net>
* Return error code -8 for missing blocks, like zcashd does
* Test that getblock returns error code -8 for missing blocks
* Make RegexSet failures easier to read in logs
* Update lightwalletd acceptance tests for log message changes
* Add extra failure strings for lightwalletd logs
* Improve link formatting
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Implement `getblockchaininfo` RPC method
* add a test for `get_blockchain_info`
* fix tohex/fromhex
* move comment
* Update lightwalletd acceptance test for getblockchaininfo RPC (#3914)
* change(rpc): Return getblockchaininfo network upgrades in height order (#3915)
* Update lightwalletd acceptance test for getblockchaininfo RPC
* Update some doc comments for network upgrades
* List network upgrades in order in the getblockchaininfo RPC
Also:
- Use a constant for the "missing consensus branch ID" RPC value
- Simplify fetching consensus branch IDs
- Make RPC type derives consistent
- Update RPC type documentation
* Make RPC type derives consistent
* Fix a confusing test comment
* get hashand height at the same time
* fix estimated_height
* fix lint
* add extra check
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix typo
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* split test
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix(rpc): ignore an expected error in the RPC acceptance tests (#3961)
* Add ignored regexes to test command failure regex methods
* Ignore empty chain error in getblockchaininfo
We expect this error when zebrad starts up with an empty state.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make command test matching code accept generic regexes
And add generic conversions to regexes.
* Document test command structs
* Support matching multiple regexes internally in the test command
* Make it easier to call the generic regex methods
* Add a missing API usage comment
* Fix a potential hang in test child error reports
* Revert Option<Child> process handling
* Revert "Revert Option<Child> process handling"
This reverts commit 2af30086858d104dcb0ec87383996c36bcaa7371.
* Add a set of failure regexes to test command output
* Allow debug-printing TestChild again
* When the child is dropped, check any remaining output
* Document a wait_with_output edge case
* Improve failure regex panic output
* Improve builder ergonomics
* Add internal tests for failure regex panics
It would be easy to disable these panics, and never realise.
* Add some module structure TODOs
* Stop panicking if the child process has already been taken
* Add test APIs for consuming child output lines
* Fix a hang on child process drop
* Handle output being already taken in wait_with_output
And document some edge cases we don't handle yet
* Use bash's read command in the TestChild stderr test
And check the actual command we're using to see if it errors.
* Pretty print full failure regex list
* Add the test child command line to the failure regex logs
* Simplify the sync_until test code
* Check logs for failure messages in lightwalletd integration tests
* Add more TODOs and improve comments
* Allow missing branch ID 00000000 provided by zcashd
* Revert "Revert Option<Child> process handling"
This reverts commit 2af30086858d104dcb0ec87383996c36bcaa7371.
* Add a set of failure regexes to test command output
* Allow debug-printing TestChild again
* When the child is dropped, check any remaining output
* Document a wait_with_output edge case
* Improve failure regex panic output
* Improve builder ergonomics
* Add internal tests for failure regex panics
It would be easy to disable these panics, and never realise.
* Add some module structure TODOs
* Stop panicking if the child process has already been taken
* Add test APIs for consuming child output lines
* Fix a hang on child process drop
* Handle output being already taken in wait_with_output
And document some edge cases we don't handle yet
* Use bash's read command in the TestChild stderr test
And check the actual command we're using to see if it errors.
* Pretty print full failure regex list
* Add the test child command line to the failure regex logs
* feat(rpc): implement `getrawmempool` rpc method
* tests(rpc): start implementing a test
* Return a `Vec<String>` from `get_raw_mempool`
Ensure the returned type is serialized as an array of hexadecimal
strings.
* Make `getrawmempool` test a property test
Test it with random sets of transaction IDs.
* Fix code formatting inside `proptest!`
Run `rustfmt` inside the `proptest!` block.
* tests(rpc): fix lightwalletd test
* fix(rpc): remove non used return type
* tests(rpc): remove assert
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Allow extra time for lightwalletd integration tests
* Wait for a successful block ingestion in lightwalletd integration
* Check for any ingestor block, not just block 1
* feature(rpc): start adding a `getblock` method
* fix(rpc): replace oneshot
* fix(rpc): replace a panic with error
* fix(rpc): fix test
* feature(rpc): add hex to response
* refactor(rpc): use generic instead of alias
* docs(rpc): improve docs for getblock method
* test(rpc): add a test for getblock method
* deps(rpc): remove non needed tower features
Co-authored-by: teor <teor@riseup.net>
* docs(rpc): add a note to getblock doc
* refactor(rpc): replace alias
* fix(rpc): use `zcash_serialize_to_vec()` instead of logging format
* tests(rpc): add network argument to `populated_state()`
* refactor(rpc): use an error for state service readiness
* fix(rpc): add parameter
* fix(rpc): clippy
* nit(rpc): remove new line from imports
* fix(rpc): remove commented code
* fix(rpc): simplify error
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Use a `SerializedBlock` type to help serializing blocks (#3725)
* Create a `SerializedBlock` helper type
Create a type that can be used as a byte slice, but is guaranteed to
represent a valid block.
* Use `into_iter` instead of `iter`
There's no need to borrow the elements, they can be moved out directly.
This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`,
so a `SerializedBlock` can't be built directly from an `&Arc<Block>`.
* Use `SerializedBlock` in `GetBlock`
Make the type stricter to avoid storing possibly invalid values. The
bytes are still serialized as a hexadecimal string, through the usage of
`hex`.
The `serde::Deserialize` can't be derived because `hex` requires the
type to also implement `FromHex`.
* feature(rpc): add suggestions from code review
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* tests(rpc): make sure mempool has no requests in get_block test
* fix(rpc): change height argument type in getblock method
* fix(rpc): rustfmt
* fix(rpc): replace panic
* fix(rpc): change getblock response
* fix(rpc): fix lightwalletd test
* tests(rpc): add a getblock error test
* fix(rpc): try another regex
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Sometimes lightwalletd goes straight to "Another refresh in progress",
without logging "Mempool refresh error" first.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(ci): remove an unused trigger path
* doc(ci): explain lightwalletd trigger paths
* fix(test): check for adityapk00/lightwalletd behaviour in test harness
* fix(ci): work around buildx command error
* fix(ci): revert the workaround
* tests(rpc): add an rpc endpoint test
* tests(rpc): add an rpc port conflict test
* tests(rpc): refactor some imports
* tests(rpc): fix failures, make test more complete
* tests(rpc): parse json response for better coverage
* tests(rpc): change request
* tests(rpc): wait until port is open in rpc_endpoint test
* tests(rpc): add a delay between launching 2 nodes
* tests(rpc): try 5 seconds
* refactor(rpc): open rpc server faster
* tests(rpc): extend `LAUNCH_DELAY` to 15 seconds
* fix(rpc): disable rpc_conflict test for windows
* fix(ci): skip the RPC tests if the network is disabled
* rustfmt
* fix(zebrad/test): test function return type
* tests(rpc): print server output in assert
* fix(rpc): fix acceptance test looking for string in `build` field
* fix(rpc): reduce the number of acceptable characters in version output
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(test): only run lightwalletd test when the ZEBRA_TEST_LIGHTWALLETD env var is set
* fix(test): actually skip the test
* doc(zebrad): add some test TODOs
* doc(test): document zebrad-specific process launch methods
* refactor(zebrad): split lightwalletd launch into its own testing methods
* fix(zebrad/test): simplify file writing
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* refactor(zebrad/test): rename argument variables
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* lint(zebrad/tests): remove unused import
* fix(zebrad/test): restore SocketAddr import that was removed on main
* rustfmt
* fix(zebrad/test): update integration test to match adityapk00/lightwalletd behaviour
* rustfmt
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
The mempool is only activated once, so we must check for that log first.
After mempool activation, the stop regex is logged at least once.
(It might be logged before as well, but we can't rely on that.)
When checking that the mempool didn't activate,
wait for the `zebrad` command to exit,
then check the entire log.
* fix(test): make full sync tests more efficient
Increasing the lookahead limit should increase sync speed,
at the cost of increasing memory usage.
* doc(test): remove a redundant TODO
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feat(log): log current height when logging sync progress
* fix(test): log the specific error when full sync tests fail
* doc(start): remove an obsolete TODO
We can't decrease this log level, because the tests rely on it.
* fix(test): wait until mempool activates in full sync tests
Changes the log message and log test so that the test only finishes
when the mempool has activated.
There is still a race condition here, between the log timer and
mempool activation. But it should be very rare, because the mempool
is activated immediately when `is_close_to_tip()` becomes true.
* fix(test): warn when Zebra stalls below the maximum checkpoint height
This also improves the full sync tests,
because the warning is checked before logging a successful sync.
* feat(log): warn when sync stalls downloading the genesis block
* fix(test): warn when the state hasn't committed a block for a long time
This also improves the full sync tests,
because the warning is checked before logging a successful sync.
* doc(test): update some sync acceptance test comments
* fix(log): use Display formatting to log chrono::Duration
Debug formatting is complicated and hard to read.
* fix(log): stop saying that we've activated the mempool without checking it
We're not checking if the mempool is active, so we can't say that.
* fix(log): minor tidying and TODOs
* fix(doc): fix a typo in the tests
* fix(log): explain the post-checkpoint blocks in progress warning calculations
* fix(doc): explain what could happen if we don't wait for extra blocks
* fix(log): add a percent symbol to a percent log
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Allow forcing colored output in `zebrad`
Add a configuration item that allows forcing Zebra to output in color
mode even if the output device is not a terminal.
* Allow enabling colored output from Zebra in tests
Force Zebrad instances to use colored output if the
`ZEBRA_FORCE_USE_COLOR` environment variable is set.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Create a `MempoolBehavior` helper type
Prepare to replace the `Option<Height>` with a custom type that is more
flexible and provides more meaning when used.
* Use `MempoolBehavior` instead of `Option<Height>`
Clarify what the argument is, and prepare the code so that a new variant
can be added to the `MempoolBehavior` later.
* Allow the mempool to be automatically activated
Add a new variant to `MempoolBehavior` that indicates that the mempool
should become active during the test without needing a forced activation
height.
* Test full synchronization
Run `zebrad` and wait until full synchronization completes. Check if the
mempool is automatically activated when it reaches the tip.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Previously, the test would wait 10 seconds for the process to launch.
Now it waits until the process has used the conflicting resource.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Move standard lints into .cargo/config.toml
* Ignore "wrong self convention" in a futures-based trait
This lint might only trigger on beta or nightly at the moment.
* Warn if future incompatibile code is added to Zebra
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Use `testdir()` instead of `TempDir::new()`
Reduce repeated code and make it easier to change from using `tempdir`
to use `tempfile` instead.
* Replace `tempdir` with `tempfile` in `zebrad`
Use `tempfile`'s `TempDir` instead.
* Use `tempdir()` instead of `TempDir::new()`
Reduce repeated code and make it easier to upgrade to `tempfile`.
* Use `tempfile` instead of `tempdir`
Replace obsoleted `tempdir` dependency with `tempfile`.
* Use `tempfile` instead of `tempdir`
Replace obsoleted `tempdir` dependency with `tempfile`.
* Update `Cargo.lock`
Update it now that `tempdir` has been replaced with `tempfile`.
* Remove `tempdir` from `deny.toml` exceptions
Ban duplicate versions of the `tempdir` dependency.
* Remove `inferno` from `deny.toml` exceptions
It apparently isn't needed anymore.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Implement graceful shutdown for the peer set
* Use the minimum lookahead limit in acceptance tests
* Enable a doctest that compiles with newly public modules
* Update `tower` to version `0.4.9`
Update to latest version to add support for Tokio version 1.
* Replace usage of `ServiceExt::ready_and`
It was deprecated in favor of `ServiceExt::ready`.
* Update Tokio dependency to version `1.13.0`
This will break the build because the code isn't ready for the update,
but future commits will fix the issues.
* Replace import of `tokio::stream::StreamExt`
Use `futures::stream::StreamExt` instead, because newer versions of
Tokio don't have the `stream` feature.
* Use `IntervalStream` in `zebra-network`
In newer versions of Tokio `Interval` doesn't implement `Stream`, so the
wrapper types from `tokio-stream` have to be used instead.
* Use `IntervalStream` in `inventory_registry`
In newer versions of Tokio the `Interval` type doesn't implement
`Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used
instead.
* Use `BroadcastStream` in `inventory_registry`
In newer versions of Tokio `broadcast::Receiver` doesn't implement
`Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This
also requires changing the error type that is used.
* Handle `Semaphore::acquire` error in `tower-batch`
Newer versions of Tokio can return an error if the semaphore is closed.
This shouldn't happen in `tower-batch` because the semaphore is never
closed.
* Handle `Semaphore::acquire` error in `zebrad` test
On newer versions of Tokio `Semaphore::acquire` can return an error if
the semaphore is closed. This shouldn't happen in the test because the
semaphore is never closed.
* Update some `zebra-network` dependencies
Use versions compatible with Tokio version 1.
* Upgrade Hyper to version 0.14
Use a version that supports Tokio version 1.
* Update `metrics` dependency to version 0.17
And also update the `metrics-exporter-prometheus` to version 0.6.1.
These updates are to make sure Tokio 1 is supported.
* Use `f64` as the histogram data type
`u64` isn't supported as the histogram data type in newer versions of
`metrics`.
* Update the initialization of the metrics component
Make it compatible with the new version of `metrics`.
* Simplify build version counter
Remove all constants and use the new `metrics::incement_counter!` macro.
* Change metrics output line to match on
The snapshot string isn't included in the newer version of
`metrics-exporter-prometheus`.
* Update `sentry` to version 0.23.0
Use a version compatible with Tokio version 1.
* Remove usage of `TracingIntegration`
This seems to not be available from `sentry-tracing` anymore, so it
needs to be replaced.
* Add sentry layer to tracing initialization
This seems like the replacement for `TracingIntegration`.
* Remove unnecessary conversion
Suggested by a Clippy lint.
* Update Cargo lock file
Apply all of the updates to dependencies.
* Ban duplicate tokio dependencies
Also ban git sources for tokio dependencies.
* Stop allowing sentry-tracing git repository in `deny.toml`
* Allow remaining duplicates after the tokio upgrade
* Use C: drive for CI build output on Windows
GitHub Actions uses a Windows image with two disk drives, and the
default D: drive is smaller than the C: drive. Zebra currently uses a
lot of space to build, so it has to use the C: drive to avoid CI build
failures because of insufficient space.
Co-authored-by: teor <teor@riseup.net>
This matches the settings for `sync_large_checkpoints_mainnet`.
Also reduce the number of blocks synced to reduce network load.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Increase the restart test timeout to 10 seconds
It shouldn't take this long.
But maybe the CI VMs are under a lot of load?
* Add extensive logging to diagnose CI state reload failures
* add some additional checks to the acceptance mempool test
* add an additional mempool test
* do proposed fixes to `sync_until`
* Ignore "can't kill an exited process" errors
Co-authored-by: teor <teor@riseup.net>
* Update some comments
* Add a mempool debug_enable_at_height config
* Rename a field in the mempool crawler
* Propagate syncer channel errors through the crawler
We don't want to ignore these errors, because they might indicate a shutdown.
(Or a bug that we should fix.)
* Use debug_enable_at_height in the mempool crawler
* Log when the mempool is activated or deactivated
* Deny unknown fields and apply defaults for all configs
* Move Duration last, as required for TOML tables
* Add a basic mempool acceptance test
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Use the block verifier and non-finalized state in the cached state tests
This substantially increases test coverage.
Previously, the cached state tests were configured with
`checkpoint_sync = true`, which only uses the checkpoint
verifier and the finalized state.
* Log the source of blocks in commit_finalized_direct
This lets us check that we're actually testing the non-finalized state
and block verifier in the cached state tests.
It also improves diagnostics for state errors.
* Fail cached state tests if they're using incorrect heights or configs
This makes sure that the cached state tests actually test the transition
from checkpoint to block verification, and the non-finalized state.
* add legacy chain check and tests
* improve has_network_upgrade check
* add docs to legacy_chain_check()
* change arbitrary module structure
* change the panic message
* move legacy chain acceptance into existing tests
* use a reduced_branch_id_strategy()
* add docs to strategy function
* add argument to check for legacy chain into sync_until()
* Disable IPv6 tests when $ZEBRA_SKIP_IPV6_TESTS is set
This allows users to disable IPv6 tests in environments where IPv6 is not
configured.
* Add network test env var constants
* Replace env strings with constants
fastmod '"ZEBRA_SKIP_NETWORK_TESTS"' zebra_test::net::ZEBRA_SKIP_NETWORK_TESTS
fastmod '"ZEBRA_SKIP_IPV6_TESTS"' zebra_test::net::ZEBRA_SKIP_IPV6_TESTS
* Add functions to skip network tests
* Replace test network env var checks with test function
fastmod --fixed-strings 'env::var_os(zebra_test::net::ZEBRA_SKIP_NETWORK_TESTS).is_some()' 'zebra_test::net::zebra_skip_network_tests()'
fastmod --fixed-strings 'env::var_os(zebra_test::net::ZEBRA_SKIP_IPV6_TESTS).is_some()' 'zebra_test::net::zebra_skip_ipv6_tests()'
* Remove redundant logging and use statements
* Gossip dynamically allocated listener ports to peers
Previously, Zebra would either gossip port `0`, which is invalid, or skip
gossiping its own dynamically allocated listener port.
* Improve "no configured peers" warning
And downgrade from error to warning, because inbound-only nodes are a
valid use case.
* Move random_known_port to zebra-test
* Add tests for dynamic local listener ports and the AddressBook
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
- Add a custom semver match for `zebrad` versions
- Prefer "line contains string" matches, so tests ignore minor changes
- Escape regex meta-characters when a literal string match is intended
- Rename test functions so they are more precise
- Rewrite match internals to remove duplicate code and enable custom matches
- Document match functions
* Standardise lints across Zebra crates, and add missing docs
The only remaining module with missing docs is `zebra_test::command`
* Todo -> TODO
* Clarify what a transcript ErrorChecker does
Also change `Error` -> `BoxError`
* TransError -> ExpectedTranscriptError
* Output Descriptions -> Output descriptions
`node2.is_running()` can return `true` on Windows, even though `node2`
has logged a panic. This cleanup code only runs if `node2` fails to panic
and exit as expected. So it's ok for us to skip it.
See #1781 for details.
On Windows, if a process is killed after it is dead, it returns `true`
for `was_killed`. Instead, check if the process is running before killing
it.
Also make the section where processes are running as short as possible,
and include context for both processes in every error.
* Increase the conflict acceptance test launch delay
Also rename the tests - the listener is for the Zcash protocol,
but the state, metrics, and tracing are Zebra-specific.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* add hint for port error
* add issue filter for port panic
* add lock file hint
* add metrics endpoint port conflict hint
* add hint for tracing endpoint port conflict
* add acceptance test for resource conflics
* Split out common conflict test code into a function
* Add state, metrics, and tracing conflict tests
* Add a full set of stderr acceptance test functions
This change makes the stdout and stderr acceptance test interfaces
identical.
* move Zcash listener opening
* add todo about hint for disk full
* add constant for lock file
* match path in state cache
* don't match windows cache path
* Use Display for state path logs
Avoids weird escaping on Windows when using Debug
* Add Windows conflict error messages
* Turn PORT_IN_USE_ERROR into a regex
And add another alternative Windows-specific port error
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Jane Lusby <jane@zfnd.org>
The clippy unknown lints attribute was deprecated in
nightly in rust-lang/rust#80524. The old lint name now produces a
warning.
Since we're using `allow(unknown_lints)` to suppress warnings, we need to
add the canonical name, so we can continue to build without warnings on
nightly.
But we also need to keep the old name, so we can continue to build
without warnings on stable.
And therefore, we also need to disable the "removed lints" warning,
otherwise we'll get warnings about the old name on nightly.
We'll need to keep this transitional clippy config until rustc 1.51 is
stable.
* Stop failing acceptance tests if their directories already exist
* Add an immutable config writing helper
and use it in the cached sapling acceptance tests.
Also:
* consistently create missing config and state directories
* refactor the common config writing code into a separate function
* only ignore NotFound errors in replace_config
* enforce config immutability using the type system
Check misconfigured ephemeral doesn't create a state dir
Add extra misconfigured `zebrad` ephemeral mode checks:
* doesn't create a state directory
* doesn't create unexpected files or directories in the working directory
Check ephemeral doesn't delete an existing state directory
Refactor all the ephemeral configs and checks into a single test
function.
Also:
* cleanup acceptance tests using utility functions
* make some checks consistent between tests
* make error messages consistent
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
This change avoids errors when tests are cancelled and re-run within a
short period of time, for example, using `cargo watch`.
It introduces a slight risk of port conflicts between the endpoint tests,
and with (ephemeral) ports used by other services. The risk of conflicts
across 2 tests is very low, and tests should be run in an isolated
environment on busy servers.
## Motivation
Prior to this PR we've been using `sled` as our database for storing persistent chain data on the disk between boots. We picked sled over rocksdb to minimize our c++ dependencies despite it being a less mature codebase. The theory was if it worked well enough we'd prefer to have a pure rust codebase, but if we ever ran into problems we knew we could easily swap it out with rocksdb.
Well, we ran into problems. Sled's memory usage was particularly high, and it seemed to be leaking memory. On top of all that, the performance for writes was pretty poor, causing us to become bottle-necked on sled instead of the network.
## Solution
This PR replaces `sled` with `rocksdb`. We've seen a 10x improvement in memory usage out of the box, no more leaking, and much better write performance. With this change writing chain data to disk is no longer a limiting factor in how quickly we can sync the chain.
The code in this pull request has:
- [x] Documentation Comments
- [x] Unit Tests and Property Tests
## Review
@hdevalence
* Use the default memory limit in the acceptance tests
PR #1233 changed the default `memory_cache_bytes`, but left the
acceptance tests with their old value.
* Run large checkpoint sync tests in CI
* Improve test child output match error context
* Add a debug_stop_at_height config
* Use stop at height in acceptance tests
And add some restart acceptance tests, to make sure the stop at
height feature works correctly.
* Reverse displayed endianness of transaction and block hashes
* fix zebra-checkpoints utility for new hash order
* Stop using "zebrad revhex" in zebrad-hash-lookup
* Rebuild checkpoint lists in new hash order
This change also adds additional checkpoints to the end of each list.
* Replace TransactionHash with transaction::Hash
This change should have been made in #905, but we missed Debug impls
and some docs.
Co-authored-by: Ramana Venkata <vramana@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
Remove the seed command entirely, and make the behavior it provided
(responding to `Request::Peers`) part of the ordinary functioning of the
start command.
The new `Inbound` service should be expanded to handle all request
types.
* add test for first checkpoint sync
Prior this this change we've not had any tests that verify our sync /
network logic is well behaved. This PR cleans up the test helper code to
make error reports more consistent and uses this cleaned up API to
implement a checkpoint sync test which runs zebrad until it reads the
first checkpoint event from stdout.
Co-authored-by: teor <teor@riseup.net>
* move include out of unix cfg
Co-authored-by: teor <teor@riseup.net>
* add metrics and tracking endpoint tests
* test endpoints more
* add change filter test for tracing
* add await to post
* separate metrics and tracing tests
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* Remove in-memory state service
* make the config compatible with toml again
* checkpoint commit to see how much I still have to revert
* back to the starting point...
* remove unused dependency
* reorganize error handling a bit
* need to make a new color-eyre release now
* reorder again because I have problems
* remove unnecessary helpers
* revert changes to config loading
* add back missing space
* Switch to released color-eyre version
* add back missing newline again...
* improve error message on unix when terminated by signal
* add context to last few asserts in acceptance tests
* instrument some of the helpers
* remove accidental extra space
* try to make this compile on windows
* reorg platform specific code
* hide on_disk module and fix broken link
* add valid generated config test
* change to pathbuf
* use -c to make sure we are using the generated file
* add and use a ZebraTestDir type
* change approach to generate tempdir in top of each test
* pass tempdir to test_cmd and set current dir to it
* add and use a `generated_config_path` variable in tests
* make sure no info is printed in non server tests
* check exact full output for validity instead of log msgs
* add end of output character to version regex
* use coercions, use equality operator
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
* Split tracing component code into modules.
* Repatriate Tracing and simplify config handling.
We upstreamed our Tracing component, expecting not to have to exert fine
control over the tracing settings. But this turned out not to be the case, and
now that we want to do other things (flamegraphs, journalctl, opentelemetry,
etc), we end up with really awkward code (as in the current flamegraph
handling).
This also makes use of the changes to `init()` to load the config early to pass
configuration data into the components, which avoids the need for the
refactoring in #775.
Finally, we restore support for the `-v` flag when the filter is unset. Closes#831.
* Disable tracing and metrics endpoints by default.
Closes#660.
* Switch back to upstream Abscissa.
* Integrate flamegraph support into the new Tracing component.
* Pass -v in acceptance tests to get info-level output.
* Clean up acceptance test code.
* add zebrad acceptance tests
* add custom command test helpers that work with kill
* add and use info event for start and seed commands
* combine conflicting tests into one test case
Co-authored-by: Jane Lusby <jane@zfnd.org>
* Add a TracingConfig and some components
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Restructure, use dependency injection, initialize tracing
* Start a placeholder loop in start command
* Add hyper alpha.1, bump tokio to alpha.4
* Hello world endpoint using async/await from hyper 0.13 alpha
Also cleaned up some linter messages.
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
* Update to tracing_subscriber 0.1
* fmt
* add rust-toolchain
* Remove hyper::Version import
* wip: start filter_handler impl
* Add .rustfmt.toml
* rustfmt
* Tidy up .rustfmt.toml
* Add filter reloading handling.
* bump toolchain
* Remove generated hello world acceptance tests.
These test the behaviour of the autogenerated binary and work as examples of
how to test the behaviour of abscissa binaries. Since we don't print "Hello
World" any more, they fail, but we don't yet have replacement behaviour to add
tests for, so they're removed for now.
* Clean up config file handling with Option::and_then.
teor
Alfredo Garcia
Janito Vaqueiro Ferreira Filho
Conrado Gouvea
Dimitris Apostolou
Conrado P. L. Gouvea
Deirdre Connolly
Jane Lusby
Jane Lusby
Henry de Valence
Ramana Venkata
Tony Arcieri