* Handle test failure regexes using Result::Err, rather than panicking
* Add output logs to test context, and add tests for that
* Let empty test child logs be read again (and produce empty output)
* Ignore missing test children when killing with ignore_exited
* Fix a clippy lint
* Rename `line` to `line_result` for clarity
* Revert a redundant context_from() on kill()
* Only ignore "no such process" kill() errors in sync_until() tests
* Log the command timeout when an acceptance test fails
* fix clippy
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Show the arguments of acceptance test functions in the logs
* Show all the logs in the "Run tests" jobs
* Document expected "broken pipe" error from `tee`
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Increase full sync timeout to 24 hours
Expected sync time is ~21 hours as of August 2022.
* Split final checkpoint job into two smaller jobs to avoid timeouts
Also make regexes easier to read.
* Fix a job name typo
* update timeout
* update the doc comment
* Increase test timeouts for Zebra update syncs
* Stop failing the 1740k job if the cached state is after block 1740k
Co-authored-by: teor <teor@riseup.net>
* truncate the number of transactions in send transaction test
* Limit send transaction test to RPC queue length
Co-authored-by: teor <teor@riseup.net>
* update zebra to 1.0.0-beta.13
* add changelog
* update the release date
* Update changelog for cryptographic batch PRs
* Add `bitvec` to the changelog
* Update CHANGELOG.md
* Add zebra-network fix and tokio version bump
* Recommend that users update their rust compiler
* Update disk and network usage for recent large blocks
* Fix spacing
* Add disk and network usage update to README
* apply size suggestions
Co-authored-by: teor <teor@riseup.net>
* add known performance issues
* update changelog
* change release date
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make code execution time logs shorter
* Do ZK parameter preloads in the lightwalletd tests that need them
* Try to re-launch `lightwalletd` when it hangs during sync tests
* Increase full sync timeout
* Clear the `zebrad` logs during `lightwalletd` tests, to avoid logging deadlocks
* Actually clear more than one line of logs
* Check zebrad and lightwalletd output in parallel threads, while waiting for zebrad
* Check zebrad and lightwalletd output in parallel threads, while waiting for lightwalletd
* Improve test logging
* Fix a log typo
* Only wait for lightwalletd once, because its logs stop after the initial sync
* Look for cached state disks for this commit and branch first
* Only copy the state once in the send transactions test
* Wait longer for lightwalletd gRPC server startup
* Add some function docs
* cargo fmt --all
* Fix clippy::let_and_return
* Increase lightwalletd test timeouts for zebrad slowness
* Add a `zebrad_update_sync()` test, that update syncs Zebra without lightwalletd
* Run the zebrad-update-sync test in CI
* Add extra zebrad time to workaround lightwalletd bugs
* Split disk reads from CPU-heavy Sprout interstitial tree cryptography
* Improve anchor validation debugging and error messages
* Work around a test data bug, and save some CPU
* Remove redundant checks for empty shielded data
* Skip generating unused interstitial treestates
* Do disk fetches and quick checks, then CPU-heavy cryptography
* Wrap HistoryTree in an Arc in the state
* Run CPU-intensive chain validation and updates in parallel rayon threads
* Refactor to prepare for parallel tree root calculations
* Run finalized state note commitment tree root updates in parallel rayon threads
* Update finalized state note commitment trees using parallel rayon threads
* Fix a comment typo and add a TODO
* Split sprout treestate fetch into its own function
* Move parallel note commitment trees to zebra-chain
* Re-calculate the tree roots in the same parallel batches
* Do non-finalized note commitment tree updates in parallel threads
* Update comments about note commitment tree rebuilds
* Do post-fork tree updates in parallel threads
* Add a TODO for parallel tree updates in tests
* Fix broken intra-doc links
* Clarify documentation for sprout treestates
* Spawn large database reads into blocking tokio threads
* Concurrently read all blocks, headers, and transactions from disk
* Run zebra-state transaction deserialization on a rayon thread
* Update comments about note commitment tree rebuilds
* Add info-level logs to time note commitment tree rebuilds
* Log the number of rebuilt blocks after a fork, and the time per block
* Move humantime formats to zebra-chain
* Use human-friendly time formatting
* Only fetch block headers from the database to answer headers requests
* Move writing to the database to a blocking thread
* Add blocking threads to tests that need them
* Remove mempool downloader requirement for a populated state
And stop populating states that don't need it,
so we can use tokio::time::pause() in those tests.
* Improve debugging for an intermittent test failure
* Try to avoid a race condition populating the mempool in tests
* try to do deserialization of transaction in a rayon thread
* Try tokio::task::block_in_place instead
* fix tests
* add deserialize block into rayon pool
* fill some docs
Co-authored-by: teor <teor@riseup.net>
* Initialize the rayon threadpool with a new config for CPU-bound threads
* Verify proofs and signatures on the rayon thread pool
* Only spawn one concurrent batch per verifier, for now
* Allow tower-batch to queue multiple batches
* Fix up a potentially incorrect comment
* Rename some variables for concurrent batches
* Spawn multiple batches concurrently, without any limits
* Simplify batch worker loop using OptionFuture
* Clear pending batches once they finish
* Stop accepting new items when we're at the concurrent batch limit
* Fail queued requests on drop
* Move pending_items and the batch timer into the worker struct
* Add worker fields to batch trace logs
* Run docker tests on PR series
* During full verification, process 20 blocks concurrently
* Remove an outdated comment about yielding to other tasks
* Return the maximum checkpoint height from the chain verifier
* Return the verified block height from the sync downloader
* Track the verified height in the syncer
* Use a lower concurrency limit during full verification
* Get the tip from the state before the first verified block
* Limit the number of submitted download and verify blocks in a batch
* Adjust lookahead limits when transitioning to full verification
* Keep unused extra hashes and submit them to the downloader later
* Remove redundant verified_height and state_tip()
* Split the checkpoint and full verify concurrency configs
* Decrease full verification concurrency to 5 blocks
10 concurrent blocks causes 3 minute stalls on some blocks on my machine.
(And it has about 4x as many cores as a standard machine.)
* cargo +stable fmt --all
* Remove a log that's verbose with smaller lookahead limits
* Apply the full verify concurrency limit to the inbound service
* Add a summary of the config changes to the CHANGELOG
* Increase the default full verify concurrency limit to 30
* Checkout zebra in each job to avoid warnings
But put TODOs where we might be able to skip checkouts
* Split log following into sprout checkpoints, sapling/orchard checkpoints, and full validation
* Make job IDs shorter
* Use /dev/stderr because docker doesn't have a tty
* remove pipefail
* Revert "remove pipefail"
This reverts commit a7ee37bebdc107a4215e7dd307b189d925969234.
* Make tee ignore errors writing to a grep pipe
* Avoid launching multiple docker instances for duplicate jobs
* Ignore broken pipe error messages and statuses
* fix(ci): docker wait not finding container
We had this issue before, I can't recall if this was a parsing error between GitHub Actions and gcloud `--command` parsing, but we had to change this into two pieces.
This implementation keeps it how we did it before 9b9578c999/.github/workflows/test.yml (L235-L243)
* docs: remove pending TODO
We can't remove `actions/checkout` nor set `create_credentials_file` to `false` as next steps won't be able to authenticate to GCP.
We can surely remove `actions/checkout` and leave `create_credentials_file` as `true`, but this will raise a warning on each step, and there's no benefit of doing so.
* Show `docker wait` and `gcloud ssh` output
* If `docker wait` fails, get the exit code using `docker inspect`
* Make full sync tests go all the way to the tip
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Update Zebra to 1.0.0-beta.12
* Simplify tower-* versioning by updating both to 0.2.27
* Simplify version search and replace by removing an old version from the docs
* Update Cargo.lock
* Add draft changelog as of PR #4693
* Update CHANGELOG to PR #4680
* Document configs that need compile-time features
* Document features in zebrad/src/lib.rs
* Link to the feature list from the README
* Remove some outdated README info
* Move some detailed README info to the `zebrad` crate docs
* Improve time logging using humantime
* Only log full seconds, ignore the fractional part
* Move humantime_seconds to tracing::fmt
* Move the progress task to its own module
* Add missing humantime dependency
* Log the network upgrade in progress logs
* Log when Zebra verifies the final checkpoint
* Warn the user when Zebra cannot parse `zebrad.toml`
* Test that Zebra warns the user when it cannot parse `zebrad.toml`
* Fix up a mistaken merge change
* Suggest how to fix `zebrad.toml` when Zebra cannot parse it
Co-authored-by: teor <teor@riseup.net>
dependabot[bot]
teor
Alfredo Garcia
Conrado Gouvea
Marek