* Revert "Temporarily stop requiring cached lightwalletd state for the send transaction tests"
This reverts commit f6b29b151e.
* fix(ci): add a lightwalletd cached state to the test
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Temporarily use an earlier lightwalletd version
This checks if commit
e146dbf5c2
contains a mempool refresh deadlock bug.
* Actually rebuild the lightwalletd image
* Delete an unfinished comment
* Remove duplicate test in entrypoint.sh
* Keep a recent change to make tests consistent
* fix(ci): remove not used variable `lwd_state_dir`
* fix(ci): state wast not being added to the image name
* fix(ci): mount a docker volume with lightwalletd dir
If the volume doesn't mount this lwd cached state dir, the content won't be saved to the mounted disk in the VM
* fix(ci): lwd state condition
* docs(ci): explain disk mounting logic
* docs(ci): explain disk mounting decision better
* docs(ci): add a description for confusing input names
Co-authored-by: teor <teor@riseup.net>
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* Drop sentry dependencies when enable-sentry feature is disabled
* Make lightwalletd gRPC tests depend on a new lightwalletd-grpc-tests feature
* fix(ci): remove enable-sentry feature from tests
* Add lightwalletd-grpc-tests feature for functionality or efficiency
And document where it is just used to stop re-compilations.
* Remove redundant `cmake` and `protobuf-compiler` dependencies
* Document Zebra's optional production and test feature flags
* Minimise dependencies in zcash-params/Dockerfile
* Minimise dependencies in docker/Dockerfile
* Add a workflow TODO
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* feat(ci): add lightwalletd_*_sync tests to CI
* feat(ci): add lightwalletd RPC call test
* feat(ci): add send transactions test with lwd to CI
* fix(ci): create a variable to run transactions test
* refactor(ci): use docker in docker
This is a workaround for an issue related to disk partitioning, caused by a GCP service called Konlet, while mounting the cached disks to the VM and then to the container
* fix(build): persist docker login credentials
* fix(ci): get sync height from docker logs instead of gcp
* try: use gha cache for faster building
* fix(ci): mount disk in container to make it available in vm
* fix(build): do not invalidate cache between images
* try(docker): invalidate cache as less as possible
* fix(ci): GHA terminal is not a TTY
* fix(build): do not ignore entrypoint.sh
* fix
* fix(ci): mount using root priveleges
* fix(ci): use existing disk as cached state
* fix(ci): wait for disks to get mounted
* force rebuild
* fix failed force
* fix failed commit
* WIP
* fix(ci): some tests does not use a cached state
* wip
* refactor(ci): disk names and job segregation
* fix(ci): do not name boot and attached disk the same
* fix(ci): attach a disk to full sync, to snapshot the state
* fix(ci): use correct disk implementations
* fix(ci): use different disk name to allow test concurrency
* feat(ci): add lightwalledt send transaction test
* cleanup(ci): remove extra tests
* fix(ci): allow disk concurrency with tests
* fix(ci): add considerations for different tests
* fix(reusable): last fixes
* feat(ci): use reusable workflow for tests
* fix(rw): remove nested worflow
* fix(rw): minor fixes
* force rebuild
* fix(rw): do not use an input as job name
* fix(rw): remove variable id
* fix(ci): remove explicit conditions and id
* fix(ci): docker does not need the variable sign ($) to work
* fix(ci): mount typo
* fix(ci): if a sync fails, always delete the instance
This also reduces the amount of jobs needed.
* refactor(ci): make all test depend on the same build
* fix(ci): some tests require multiple variables
* fix(docker): variable substitution
* fix(ci): allow to run multiple commits from a PR at once
* fix(docker): lower the NETWORK env var for test names
* reduce uneeded diff
* imp(keys): use better naming for builds_disks
* imp(ci): use input defaults
* imp(ci): remove test_name in favor of test_id
* fix(ci): better key naming
* fix(ci): long disk names breaks GCP naming convention
* feat(ci): validate local state version with cached state
* fix(ci): add condition to run tests
* fix: typo
* fix: app_name should not be required
* fix: zebra_state_path shouldn't be required
* fix: reduce diff
* fix(ci): checkout to grep local state version
* Update .github/workflows/test.yml
Co-authored-by: teor <teor@riseup.net>
* revert: merge all tests into a single workflow
* Remove unused STATE_VERSION env var
* fix: minor fixes
* fix(ci): make test.patch the same as test
* fix(ci): negate the input value
* imp(ci): better cached state conditional handling
* imp(ci): exit code is captured by `docker run`
* fix(deploy): mount disks with better write performance
* fix(ci): change sync id to a broader id name
* fix(ci): use correct input validation
* fix(ci): do not make test with cached state dependant on other
* imp(ci): organiza keys better
* fix(ci): use appropiate naming
* fix(ci): create docker volume before mounting
* fix(lint): do not fail on all new changes
* imp(ci): do not report in pr review
* fix(ci): partition clean disks
* fix: typo
* fix: test called the wrong way
* fix(build): stop using gha cache
* ref(ci): validate run condition before calling reusable workflow
* fix(ci): use a better filesystem dir and fix other values
* fix: linting errors
* fix(ci): typo
* Revert "fix(build): stop using gha cache"
This reverts commit a8fbc5f416.
Cache expiration is a lesser evil than not using caching at all and then failing with a 401
* imp(ci): do not set a default for needs_zebra_state
* Update .github/workflows/test.yml
Co-authored-by: teor <teor@riseup.net>
* fix(deps): remove dependencies
* force build
* Update .github/workflows/test.yml
Co-authored-by: teor <teor@riseup.net>
* fix(docker): add RUST_LOG as an ARG and ENV
* fix(test): add `#[ignore]` to send transactions test
This test needs state then it should be marked as #[ignore]
* fix(ci): differentiate between root cache path and its dir
* Remove extra `state` directory
That was a workaround for an issue that has been fixed.
* imp(docs): use better test descriptions
Co-authored-by: teor <teor@riseup.net>
* fix: reduce unwanted diff with main
* fix(ci): make lwd conditions consistent
* Remove another extra `state` directory
Was also part of a workaround for an issue that has been fixed.
* fix(ci): use better conditionals to run test jobs
Co-authored-by: teor <teor@riseup.net>
* Tweak to support different lightwalletd versions
Some versions print `Waiting for block`, and some versions print
`Ingestor waiting for block`.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Expect a Zebra cached state with at least a million blocks
* Set the zebrad timeout and failure messages based on the test type
* Temporarily stop requiring cached lightwalletd state for the send transaction tests
* add info to lightwalletd_state_path()
* fix getaddresstxids rpc
* normalize more the env vars in lightwalletd tests
* extra logging
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* add a test that call all the lightwalletd grpc methods
* fix some docs
* use ZF address in tests for balance and utxos
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make the lightwalletd integration test take a test type
* Configure lightwalletd tests based on the test type
* Remove obsolete kill_on_error() in the lightwalletd test
* Refactor to simplify the test function
* Move LightwalletdTestType to the lightwalletd module
* Create a test function that runs the full lightwalletd test suite
* Actually use the cached Zebra state
* Add checks for the new integration test modes
* Populate the lightwalletd state dir in the FullSyncFromGenesis test
* Fix up state handling, fail earlier if state is invalid
* Adjust timeouts and regex escapes
* Make state requirements for each test stricter
* Move configs to the top of the test function
* Allow unexpected lightwalletd cached state in some tests
* Speed up tests slightly by removing an intermittent log check
* Move timeout selection into test type methods
* Move failure messages into test type methods
* Turn a function argument into an enum field
* Check lightwalletd state directly, rather than Zebra RPC results
* Update gRPC tests for function argument changes
* Remove duplicate env var constant and redundant code
* Export the `zebra_state::Config::db_path` method
Make it easier for tests to discover the sub-directory used to store
the chain state data.
* Generate code for interfacing with lightwalletd
Use the `tonic-build` crate to generate Rust code for communicating with
lightwalletd using gRPC.
The `*.proto` files were obtained from the Zcash lightwalletd
repository.
* Use `block::Height` instead of `Height`
Import the `block` instead to make it slightly clearer.
* Add helper function to remove a file if it exists
Try to remove it and ignore an error if it says that the file doesn't
exist. This will be used later to remove the lock file from a copied
chain state directory.
* Add helper function to copy chain state dirs
Copy an existing chain state directory into a new temporary directory.
* Add a `BoxStateService` type alias
Make it easier to write and read a boxed version of a state service.
* Add a helper function to start the state service
Make it easier to specify the state service to use an existing state
cache directory.
* Import `eyre!` macro at the module level
Allow it to be used in different places without having to repeat the
imports.
* Add `load_tip_height_from_state_directory` helper
A function to discover the current chain tip height stored in a state
cache.
* Add helper function to prepare partial sync. state
Loads a partially synchronized cached state directory into a temporary
directory that can be used by a zebrad instance, and also returns the
chain tip block height of that state.
* Add `perform_full_sync_starting_from` helper
Runs a zebrad with an existing partially synchronized state, and
finishes synchronizing it to the network chain tip.
* Add function to load transactions from a block
Use a provided state service to load all transactions from a block at a
specified height.
The state service is a generic type parameter, because
`zebra_state::service::ReadStateService` is not exported publicly. Using
a generic type parameter also allows the service to be wrapped in layers
if needed in the future.
* Add `load_transactions_from_block_after` helper
A function to load transactions from a block stored in a cached state
directory. The cached state must be synchronized to a chain tip higher
than the requested height.
* Add helper function to load some test transactions
Given a partially synchronized chain state, it will extend that chain by
performing a full synchronization, and obtain some transactions from one
of the newly added blocks.
* Update `spawn_zebrad_for_rpc_without_initial_peers`
Wait until the mempool is activated.
* Add method to start lightwalletd with RPC server
Returns the lightwalletd instance and the port that it's listening for
RPC connections.
The instance can reuse an existing cached lightwalletd state if the
`LIGHTWALLETD_DATA_DIR` environment variable is set.
* Add a `LightwalletdRpcClient` type alias
To make it easier to identify the type generated from the Protobuf
files.
* Add helper function to connect to lightwalletd
Prepare an RPC client to send requests to a lightwalletd instance.
* Add a `prepare_send_transaction_request` helper
Creates a request message for lightwalletd to send a transaction.
* Add test to send transactions using lightwalletd
Obtain some valid transactions from future blocks and try to send them
to a lightwalletd instance connected to a zebrad instance that hasn't
seen those transactions yet. The transactions should be successfully
queued in Zebra's mempool.
* Make `zebra_directory` parameter generic
Allow using a `TempDir` or a `PathBuf`.
* Move lightwalletd protobuf files
Place them closer to the module directory, so that it's clearer that
they specify the RPC protocol for lightwalletd, and not Zebra itself.
* Don't use coinbase transactions in the test
Coinbase transactions are rejected by the mempool.
* Don't remove state lock file
It is removed automatically by Zebra when it shuts down, so if it exists
it should be reported as a bug.
* Force mempool to be enabled in Zebrad instance
Speed up the initialization of the Zebrad instance used for lightwalletd
to connect to.
* Refactor to create `LIGHTWALLETD_DATA_DIR_VAR`
Document how the environment variable can be used to speed up the test.
* Check for process errors in spawned Zebra instance
Enable checking for known process failure messages.
* Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant
Document why it exists and how the choice of the value affects the test.
* Add `LIGHTWALLETD_TEST_TIMEOUT` constant
And use it for the Zebrad and the Lightwalletd instances used in the
send transaction integration test.
* Check `lightwalletd` process for errors
Enable checking the lightwalletd process for known failure messages.
* Update `tonic` and `prost` dependencies
Use the latest version and fix CI failures because `rustfmt` isn't
installed in the build environment.
* Create `send_transaction_test` module
Move the send transaction using lightwalletd test and its helper
functions into a new module.
* Move `LIGHTWALLETD_TEST_TIMEOUT` constant
Place it in the parent `lightwalletd` module.
* Move gRPC helper functions and types to `rpc` mod.
Make them more accessible so that they can be used by other tests.
* Create a `cached_state` module
Move the test utility functions related to using a cached Zebra state
into the module.
* Move `perform_full_sync_starting_from` to `sync`
Keep to closer to the synchronization utility functions.
* Move Zebra cached state path variable constant
Place it in the `cached_state` module.
* Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set
Make it part of the set of tests ignored as a whole if no lightwalletd
tests should be executed.
* Move `spawn_zebrad_for_rpc_without_initial_peers`
Place it in the `launch` sub-module.
* Rename `rpc` module into `wallet_grpc`
Avoid any potential misunderstandings when the name is seen out of
context.
* Allow duplicate `heck` dependency
At least until `structopt` is updated or `zebra-utils` is updated to use
`clap` 3.
* Fix a deny.toml typo
* fix(build): CMake is required by `prost` crate
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Join similar imports
Avoid the confusion that might cause one to think that they come from
different modules or crates.
* Create an `Arguments` helper type
A type to keep track of a list of arguments for a sub-process. It makes
it easier for overriding parameters with new values.
* Create an `args!` helper macro
Make it simpler to create `Arguments` instances with known values.
* Require `Arguments` for `spawn_child` method
Change the method to have an `Arguments` parameter, and merge it with
some default values before passing them forward.
* Use `Arguments` in `spawn_lightwalletd_child`
Change the method to use an `Arguments` instance, and merge it with some
default options.
* Use `Arguments` in `spawn_child_with_command`
Require an `Arguments` instance in the `spawn_child_with_command`
extension method. Makes it simpler to call from `spawn_child` and
`spawn_lightwalletd_child` extension methods.
* Test if argument order is preserved
Check that when building an `Arguments` instance, the order that the
arguments are set is preserved in the generated list of strings.
* Refactor test to improve readability
Also separates some common code to be reused by later tests.
* Test overriding arguments
Check to see if overriding arguments behaves as expected, by keeping the
argument order when overriding and not introducing duplicates.
* Refactor test to improve readability
Move out a chunk of code so that the test itself is easier to read and
to make that code reusable by a later test.
* Test that `Arguments` instances can be merged
Merge two `Arguments` instances built from two lists of arguments, and
check that the expanded strings preserve order and override rules.
* Add Eq derives on Arguments
Co-authored-by: teor <teor@riseup.net>
* Return error code -8 for missing blocks, like zcashd does
* Test that getblock returns error code -8 for missing blocks
* Make RegexSet failures easier to read in logs
* Update lightwalletd acceptance tests for log message changes
* Add extra failure strings for lightwalletd logs
* Improve link formatting
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Implement `getblockchaininfo` RPC method
* add a test for `get_blockchain_info`
* fix tohex/fromhex
* move comment
* Update lightwalletd acceptance test for getblockchaininfo RPC (#3914)
* change(rpc): Return getblockchaininfo network upgrades in height order (#3915)
* Update lightwalletd acceptance test for getblockchaininfo RPC
* Update some doc comments for network upgrades
* List network upgrades in order in the getblockchaininfo RPC
Also:
- Use a constant for the "missing consensus branch ID" RPC value
- Simplify fetching consensus branch IDs
- Make RPC type derives consistent
- Update RPC type documentation
* Make RPC type derives consistent
* Fix a confusing test comment
* get hashand height at the same time
* fix estimated_height
* fix lint
* add extra check
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix typo
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* split test
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix(rpc): ignore an expected error in the RPC acceptance tests (#3961)
* Add ignored regexes to test command failure regex methods
* Ignore empty chain error in getblockchaininfo
We expect this error when zebrad starts up with an empty state.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make command test matching code accept generic regexes
And add generic conversions to regexes.
* Document test command structs
* Support matching multiple regexes internally in the test command
* Make it easier to call the generic regex methods
* Add a missing API usage comment
* Fix a potential hang in test child error reports
* Revert Option<Child> process handling
* Revert "Revert Option<Child> process handling"
This reverts commit 2af30086858d104dcb0ec87383996c36bcaa7371.
* Add a set of failure regexes to test command output
* Allow debug-printing TestChild again
* When the child is dropped, check any remaining output
* Document a wait_with_output edge case
* Improve failure regex panic output
* Improve builder ergonomics
* Add internal tests for failure regex panics
It would be easy to disable these panics, and never realise.
* Add some module structure TODOs
* Stop panicking if the child process has already been taken
* Add test APIs for consuming child output lines
* Fix a hang on child process drop
* Handle output being already taken in wait_with_output
And document some edge cases we don't handle yet
* Use bash's read command in the TestChild stderr test
And check the actual command we're using to see if it errors.
* Pretty print full failure regex list
* Add the test child command line to the failure regex logs
* Simplify the sync_until test code
* Check logs for failure messages in lightwalletd integration tests
* Add more TODOs and improve comments
* Allow missing branch ID 00000000 provided by zcashd
* Revert "Revert Option<Child> process handling"
This reverts commit 2af30086858d104dcb0ec87383996c36bcaa7371.
* Add a set of failure regexes to test command output
* Allow debug-printing TestChild again
* When the child is dropped, check any remaining output
* Document a wait_with_output edge case
* Improve failure regex panic output
* Improve builder ergonomics
* Add internal tests for failure regex panics
It would be easy to disable these panics, and never realise.
* Add some module structure TODOs
* Stop panicking if the child process has already been taken
* Add test APIs for consuming child output lines
* Fix a hang on child process drop
* Handle output being already taken in wait_with_output
And document some edge cases we don't handle yet
* Use bash's read command in the TestChild stderr test
And check the actual command we're using to see if it errors.
* Pretty print full failure regex list
* Add the test child command line to the failure regex logs
* feat(rpc): implement `getrawmempool` rpc method
* tests(rpc): start implementing a test
* Return a `Vec<String>` from `get_raw_mempool`
Ensure the returned type is serialized as an array of hexadecimal
strings.
* Make `getrawmempool` test a property test
Test it with random sets of transaction IDs.
* Fix code formatting inside `proptest!`
Run `rustfmt` inside the `proptest!` block.
* tests(rpc): fix lightwalletd test
* fix(rpc): remove non used return type
* tests(rpc): remove assert
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Allow extra time for lightwalletd integration tests
* Wait for a successful block ingestion in lightwalletd integration
* Check for any ingestor block, not just block 1
* feature(rpc): start adding a `getblock` method
* fix(rpc): replace oneshot
* fix(rpc): replace a panic with error
* fix(rpc): fix test
* feature(rpc): add hex to response
* refactor(rpc): use generic instead of alias
* docs(rpc): improve docs for getblock method
* test(rpc): add a test for getblock method
* deps(rpc): remove non needed tower features
Co-authored-by: teor <teor@riseup.net>
* docs(rpc): add a note to getblock doc
* refactor(rpc): replace alias
* fix(rpc): use `zcash_serialize_to_vec()` instead of logging format
* tests(rpc): add network argument to `populated_state()`
* refactor(rpc): use an error for state service readiness
* fix(rpc): add parameter
* fix(rpc): clippy
* nit(rpc): remove new line from imports
* fix(rpc): remove commented code
* fix(rpc): simplify error
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Use a `SerializedBlock` type to help serializing blocks (#3725)
* Create a `SerializedBlock` helper type
Create a type that can be used as a byte slice, but is guaranteed to
represent a valid block.
* Use `into_iter` instead of `iter`
There's no need to borrow the elements, they can be moved out directly.
This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`,
so a `SerializedBlock` can't be built directly from an `&Arc<Block>`.
* Use `SerializedBlock` in `GetBlock`
Make the type stricter to avoid storing possibly invalid values. The
bytes are still serialized as a hexadecimal string, through the usage of
`hex`.
The `serde::Deserialize` can't be derived because `hex` requires the
type to also implement `FromHex`.
* feature(rpc): add suggestions from code review
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* tests(rpc): make sure mempool has no requests in get_block test
* fix(rpc): change height argument type in getblock method
* fix(rpc): rustfmt
* fix(rpc): replace panic
* fix(rpc): change getblock response
* fix(rpc): fix lightwalletd test
* tests(rpc): add a getblock error test
* fix(rpc): try another regex
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Sometimes lightwalletd goes straight to "Another refresh in progress",
without logging "Mempool refresh error" first.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(ci): remove an unused trigger path
* doc(ci): explain lightwalletd trigger paths
* fix(test): check for adityapk00/lightwalletd behaviour in test harness
* fix(ci): work around buildx command error
* fix(ci): revert the workaround
* tests(rpc): add an rpc endpoint test
* tests(rpc): add an rpc port conflict test
* tests(rpc): refactor some imports
* tests(rpc): fix failures, make test more complete
* tests(rpc): parse json response for better coverage
* tests(rpc): change request
* tests(rpc): wait until port is open in rpc_endpoint test
* tests(rpc): add a delay between launching 2 nodes
* tests(rpc): try 5 seconds
* refactor(rpc): open rpc server faster
* tests(rpc): extend `LAUNCH_DELAY` to 15 seconds
* fix(rpc): disable rpc_conflict test for windows
* fix(ci): skip the RPC tests if the network is disabled
* rustfmt
* fix(zebrad/test): test function return type
* tests(rpc): print server output in assert
* fix(rpc): fix acceptance test looking for string in `build` field
* fix(rpc): reduce the number of acceptable characters in version output
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(test): only run lightwalletd test when the ZEBRA_TEST_LIGHTWALLETD env var is set
* fix(test): actually skip the test
* doc(zebrad): add some test TODOs
* doc(test): document zebrad-specific process launch methods
* refactor(zebrad): split lightwalletd launch into its own testing methods
* fix(zebrad/test): simplify file writing
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* refactor(zebrad/test): rename argument variables
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* lint(zebrad/tests): remove unused import
* fix(zebrad/test): restore SocketAddr import that was removed on main
* rustfmt
* fix(zebrad/test): update integration test to match adityapk00/lightwalletd behaviour
* rustfmt
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
The mempool is only activated once, so we must check for that log first.
After mempool activation, the stop regex is logged at least once.
(It might be logged before as well, but we can't rely on that.)
When checking that the mempool didn't activate,
wait for the `zebrad` command to exit,
then check the entire log.
* fix(test): make full sync tests more efficient
Increasing the lookahead limit should increase sync speed,
at the cost of increasing memory usage.
* doc(test): remove a redundant TODO
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feat(log): log current height when logging sync progress
* fix(test): log the specific error when full sync tests fail
* doc(start): remove an obsolete TODO
We can't decrease this log level, because the tests rely on it.
* fix(test): wait until mempool activates in full sync tests
Changes the log message and log test so that the test only finishes
when the mempool has activated.
There is still a race condition here, between the log timer and
mempool activation. But it should be very rare, because the mempool
is activated immediately when `is_close_to_tip()` becomes true.
* fix(test): warn when Zebra stalls below the maximum checkpoint height
This also improves the full sync tests,
because the warning is checked before logging a successful sync.
* feat(log): warn when sync stalls downloading the genesis block
* fix(test): warn when the state hasn't committed a block for a long time
This also improves the full sync tests,
because the warning is checked before logging a successful sync.
* doc(test): update some sync acceptance test comments
* fix(log): use Display formatting to log chrono::Duration
Debug formatting is complicated and hard to read.
* fix(log): stop saying that we've activated the mempool without checking it
We're not checking if the mempool is active, so we can't say that.
* fix(log): minor tidying and TODOs
* fix(doc): fix a typo in the tests
* fix(log): explain the post-checkpoint blocks in progress warning calculations
* fix(doc): explain what could happen if we don't wait for extra blocks
* fix(log): add a percent symbol to a percent log
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Allow forcing colored output in `zebrad`
Add a configuration item that allows forcing Zebra to output in color
mode even if the output device is not a terminal.
* Allow enabling colored output from Zebra in tests
Force Zebrad instances to use colored output if the
`ZEBRA_FORCE_USE_COLOR` environment variable is set.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Create a `MempoolBehavior` helper type
Prepare to replace the `Option<Height>` with a custom type that is more
flexible and provides more meaning when used.
* Use `MempoolBehavior` instead of `Option<Height>`
Clarify what the argument is, and prepare the code so that a new variant
can be added to the `MempoolBehavior` later.
* Allow the mempool to be automatically activated
Add a new variant to `MempoolBehavior` that indicates that the mempool
should become active during the test without needing a forced activation
height.
* Test full synchronization
Run `zebrad` and wait until full synchronization completes. Check if the
mempool is automatically activated when it reaches the tip.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Previously, the test would wait 10 seconds for the process to launch.
Now it waits until the process has used the conflicting resource.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Move standard lints into .cargo/config.toml
* Ignore "wrong self convention" in a futures-based trait
This lint might only trigger on beta or nightly at the moment.
* Warn if future incompatibile code is added to Zebra
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Use `testdir()` instead of `TempDir::new()`
Reduce repeated code and make it easier to change from using `tempdir`
to use `tempfile` instead.
* Replace `tempdir` with `tempfile` in `zebrad`
Use `tempfile`'s `TempDir` instead.
* Use `tempdir()` instead of `TempDir::new()`
Reduce repeated code and make it easier to upgrade to `tempfile`.
* Use `tempfile` instead of `tempdir`
Replace obsoleted `tempdir` dependency with `tempfile`.
* Use `tempfile` instead of `tempdir`
Replace obsoleted `tempdir` dependency with `tempfile`.
* Update `Cargo.lock`
Update it now that `tempdir` has been replaced with `tempfile`.
* Remove `tempdir` from `deny.toml` exceptions
Ban duplicate versions of the `tempdir` dependency.
* Remove `inferno` from `deny.toml` exceptions
It apparently isn't needed anymore.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Implement graceful shutdown for the peer set
* Use the minimum lookahead limit in acceptance tests
* Enable a doctest that compiles with newly public modules
* Update `tower` to version `0.4.9`
Update to latest version to add support for Tokio version 1.
* Replace usage of `ServiceExt::ready_and`
It was deprecated in favor of `ServiceExt::ready`.
* Update Tokio dependency to version `1.13.0`
This will break the build because the code isn't ready for the update,
but future commits will fix the issues.
* Replace import of `tokio::stream::StreamExt`
Use `futures::stream::StreamExt` instead, because newer versions of
Tokio don't have the `stream` feature.
* Use `IntervalStream` in `zebra-network`
In newer versions of Tokio `Interval` doesn't implement `Stream`, so the
wrapper types from `tokio-stream` have to be used instead.
* Use `IntervalStream` in `inventory_registry`
In newer versions of Tokio the `Interval` type doesn't implement
`Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used
instead.
* Use `BroadcastStream` in `inventory_registry`
In newer versions of Tokio `broadcast::Receiver` doesn't implement
`Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This
also requires changing the error type that is used.
* Handle `Semaphore::acquire` error in `tower-batch`
Newer versions of Tokio can return an error if the semaphore is closed.
This shouldn't happen in `tower-batch` because the semaphore is never
closed.
* Handle `Semaphore::acquire` error in `zebrad` test
On newer versions of Tokio `Semaphore::acquire` can return an error if
the semaphore is closed. This shouldn't happen in the test because the
semaphore is never closed.
* Update some `zebra-network` dependencies
Use versions compatible with Tokio version 1.
* Upgrade Hyper to version 0.14
Use a version that supports Tokio version 1.
* Update `metrics` dependency to version 0.17
And also update the `metrics-exporter-prometheus` to version 0.6.1.
These updates are to make sure Tokio 1 is supported.
* Use `f64` as the histogram data type
`u64` isn't supported as the histogram data type in newer versions of
`metrics`.
* Update the initialization of the metrics component
Make it compatible with the new version of `metrics`.
* Simplify build version counter
Remove all constants and use the new `metrics::incement_counter!` macro.
* Change metrics output line to match on
The snapshot string isn't included in the newer version of
`metrics-exporter-prometheus`.
* Update `sentry` to version 0.23.0
Use a version compatible with Tokio version 1.
* Remove usage of `TracingIntegration`
This seems to not be available from `sentry-tracing` anymore, so it
needs to be replaced.
* Add sentry layer to tracing initialization
This seems like the replacement for `TracingIntegration`.
* Remove unnecessary conversion
Suggested by a Clippy lint.
* Update Cargo lock file
Apply all of the updates to dependencies.
* Ban duplicate tokio dependencies
Also ban git sources for tokio dependencies.
* Stop allowing sentry-tracing git repository in `deny.toml`
* Allow remaining duplicates after the tokio upgrade
* Use C: drive for CI build output on Windows
GitHub Actions uses a Windows image with two disk drives, and the
default D: drive is smaller than the C: drive. Zebra currently uses a
lot of space to build, so it has to use the C: drive to avoid CI build
failures because of insufficient space.
Co-authored-by: teor <teor@riseup.net>
This matches the settings for `sync_large_checkpoints_mainnet`.
Also reduce the number of blocks synced to reduce network load.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Increase the restart test timeout to 10 seconds
It shouldn't take this long.
But maybe the CI VMs are under a lot of load?
* Add extensive logging to diagnose CI state reload failures
* add some additional checks to the acceptance mempool test
* add an additional mempool test
* do proposed fixes to `sync_until`
* Ignore "can't kill an exited process" errors
Co-authored-by: teor <teor@riseup.net>
Alfredo Garcia
Marek
teor
Gustavo Valverde
Janito Vaqueiro Ferreira Filho
Conrado Gouvea
Dimitris Apostolou