* ci(concurrency)!: run a single CI workflow as required
Previous behavior:
Multiple Mainnet full syncs were able to run on the main branch at the
same time, and pushing multiple commits to the same branch would run
multiple CI workflows, when only the run from last commit was relevant
Expected behavior:
Ensure that only a single CI workflow runs at the same time in PRs.
The latest commit should cancel any previous running workflows from the
same PR.
Solution:
Use GitHub actions concurrency feature https://docs.github.com/en/actions/using-jobs/using-concurrency
Fixes https://github.com/ZcashFoundation/zebra/issues/4977
Fixes https://github.com/ZcashFoundation/zebra/issues/4857
* docs: typo
* ci(concurrency): do not cancel running full syncs
Co-authored-by: teor <teor@riseup.net>
* fix(concurrency): explain the behavior better & add new ones
Co-authored-by: teor <teor@riseup.net>
Previous behavior:
If warnings or error are added in `.cargo/config.toml` or `clippy.toml`,
and those could generate CI failures, we wouldn't catch those new as the
pipelines are not run when this files are changed
Expected behavior:
If warnings or error are added in `.cargo/config.toml` or `clippy.toml`,
run all the builds and test jobs which also track a `Cargo.toml`.
Solution:
Add `.cargo/config.toml` and `clippy.toml` as paths to all the required
jobs which needs to be triggered when these files changes.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Save cached state on full syncs and updates
* Add an -update suffix to CI images created by updating cached state
* Make disk image names unique by adding a time suffix
* Use the latest image from any branch, but prefer the current commit if available
* Document Zebra's continuous integration tests
* Fix typos in environmental variable names
* Expand documentation
* Fix variable name typo
* Fix shell syntax
Previous behavior:
Sometimes Google Cloud authentication fails, this might happen before
IAM permissions are fully propagated
Expected behavior:
If the authentication fails, retry at least 3 times before exiting with
a non zero exit code
Applied solution:
Google GitHub Actions for auth recently added this a `retries` feature
which is now implemented to workaround this issue.
Note: 95a6bc2a27
Fixes https://github.com/ZcashFoundation/zebra/issues/4846
* Fix clippy::let_and_return
* Increase lightwalletd test timeouts for zebrad slowness
* Add a `zebrad_update_sync()` test, that update syncs Zebra without lightwalletd
* Run the zebrad-update-sync test in CI
* Add extra zebrad time to workaround lightwalletd bugs
* Initialize the rayon threadpool with a new config for CPU-bound threads
* Verify proofs and signatures on the rayon thread pool
* Only spawn one concurrent batch per verifier, for now
* Allow tower-batch to queue multiple batches
* Fix up a potentially incorrect comment
* Rename some variables for concurrent batches
* Spawn multiple batches concurrently, without any limits
* Simplify batch worker loop using OptionFuture
* Clear pending batches once they finish
* Stop accepting new items when we're at the concurrent batch limit
* Fail queued requests on drop
* Move pending_items and the batch timer into the worker struct
* Add worker fields to batch trace logs
* Run docker tests on PR series
* During full verification, process 20 blocks concurrently
* Remove an outdated comment about yielding to other tasks
* Upgrade tracing and related dependencies
```sh
cargo upgrade --workspace
tracing-error
tracing-subscrber
color-eyre
tracing-flame
tracing-journald
sentry
sentry-tracing
metrics
metrics-exporter-prometheus
reqwest
```
* Update duplicate dependency checks
* Enable the tracing/env-filter feature
* Fix type inference for metrics
Manual changes, plus:
```sh
fastmod "as _" "as f64"
```
* Tidy up some unrelated test code
* Update metrics-exporter-prometheus API
And make unused dependencies optional.
* Adjust test regexes to new tracing format
Also fix some regex bugs, and refactor to simplify.
* Disable color-eyre span traces and track caller in release builds
* Add a feature that enables extra debugging in release builds
* Clean up some redundant features
* Increase a test timeout
* fix(ci): allow for the lightwalletd-full-sync to mount the lwd-cache dir
* fix(ci): compare with a string
* imp(ci): run a lightwalletd tip if there's no lwd tip disk available
* docs(ci): add TODO explaining this is a temporal condition
* Require a cached state rebuild if the state version changes
* Find cached state disks with the same state version
And prefer `main` to other branches.
* Tweak filters to make them more specific
* Try adding inner quotes
* Try brackets instead
* Try two filters, rather than three
* Use Mainnet as the default network, remove duplicate env var
* Match the exact disk name format in one regular expression
* Log the exact expected disk name, including the network
* Consistently use CACHED_DISK_NAME as the env var name
* Temporary allow missing $NETWORK in disk names
* Print the exact search string
* Debug log the search string
* Use a generic alphabetical pattern rather than a regex group
Google Cloud doesn't seem to support regex groups.
* Add network name to disk match docs
* Fix the logged network name
* Make jobs that use cached state wait for state rebuilds
* Run jobs that need cached state even if the rebuild was skipped
* Fix missing dependencies
And update a TODO
* Revert "Use a generic alphabetical pattern rather than a regex group"
This reverts commit 970afe7b17.
* Revert "Temporary allow missing $NETWORK in disk names"
This reverts commit f1f66500c3.
* Make jobs that use cached state wait for state rebuilds
* Run jobs that need cached state even if the rebuild was skipped
* Fix missing dependencies
And update a TODO
* refactor(ci): look for available disks instead of files changed
This ensure that if the constants.rs file was changed, we search for disks available in the whole repository with the same state.
If there's no disk available a rebuild is triggered depending the missing disk. And if there's a disk available, tests are run with this one.
* fix(ci): lwd syncs needs to wait for zebra disk rebuild
* docs(ci): use better comments on integration tests
* fix(ci): we must authenticate to GCP to find disks
* fix(ci): add needed permissions for google auth
* fix(ci): the output needs to be echoed
* imp(ci): reduce diff with main
* fix(ci): remove redundant dependency
Co-authored-by: teor <teor@riseup.net>
* fix(ci): also add `false` to the JSON object output
* fix(ci): hasty copy/paste
* fix(ci): standardize comments
* fix(ci): run disk rebuilds if no disk was found
* fix(ci): build on any event if a cached disk is not found
* fix(ci): reduce diff with main
* docs(ci): reduce main diff
* fix(ci): sync .patch file with changes on the workflow
* fix(ci): consider network changes in new get-available-disks
* force GHA trigger
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Make jobs that use cached state wait for state rebuilds
* Run jobs that need cached state even if the rebuild was skipped
* Fix missing dependencies
And update a TODO
* refactor(ci): look for available disks instead of files changed
This ensure that if the constants.rs file was changed, we search for disks available in the whole repository with the same state.
If there's no disk available a rebuild is triggered depending the missing disk. And if there's a disk available, tests are run with this one.
* fix(ci): lwd syncs needs to wait for zebra disk rebuild
* docs(ci): use better comments on integration tests
* fix(ci): we must authenticate to GCP to find disks
* fix(ci): add needed permissions for google auth
* fix(ci): the output needs to be echoed
* imp(ci): reduce diff with main
* fix(ci): remove redundant dependency
Co-authored-by: teor <teor@riseup.net>
* fix(ci): also add `false` to the JSON object output
* fix(ci): hasty copy/paste
* fix(ci): standardize comments
* fix(ci): run disk rebuilds if no disk was found
* fix(ci): build on any event if a cached disk is not found
* fix(ci): sync .patch file with changes on the workflow
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Revert "Temporarily stop requiring cached lightwalletd state for the send transaction tests"
This reverts commit f6b29b151e.
* fix(ci): add a lightwalletd cached state to the test
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): lwd state condition
* fix(ci): differentiate tests that need a lwd cached state
* fix(ci): use the right state and save name for each test
* docs(ci): minor comment fixes
* docs(ci): better input description
* fix(ci): end `if` condition correctly
* fix(images): pass the state version to following steps
* fix(ci): $needs_lwd_state condition was inverted
* fix(ci): reduce disk selection code
* docs(ci): better disk search conditional explanation
* fix(ci): end if condition correctly
* fix(ci): evaluate $needs_zebra_state correctly
* fix(ci): use nested condition for readability
* fix(ci): disk search was using the wrong variable
* Temporarily use an earlier lightwalletd version
This checks if commit
e146dbf5c2
contains a mempool refresh deadlock bug.
* Actually rebuild the lightwalletd image
* Delete an unfinished comment
* Remove duplicate test in entrypoint.sh
* Keep a recent change to make tests consistent
* fix(ci): remove not used variable `lwd_state_dir`
* fix(ci): state wast not being added to the image name
* fix(ci): mount a docker volume with lightwalletd dir
If the volume doesn't mount this lwd cached state dir, the content won't be saved to the mounted disk in the VM
* fix(ci): lwd state condition
* docs(ci): explain disk mounting logic
* docs(ci): explain disk mounting decision better
* docs(ci): add a description for confusing input names
Co-authored-by: teor <teor@riseup.net>
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* Drop sentry dependencies when enable-sentry feature is disabled
* Make lightwalletd gRPC tests depend on a new lightwalletd-grpc-tests feature
* fix(ci): remove enable-sentry feature from tests
* Add lightwalletd-grpc-tests feature for functionality or efficiency
And document where it is just used to stop re-compilations.
* Remove redundant `cmake` and `protobuf-compiler` dependencies
* Document Zebra's optional production and test feature flags
* Minimise dependencies in zcash-params/Dockerfile
* Minimise dependencies in docker/Dockerfile
* Add a workflow TODO
* Catch more errors in entrypoint.sh
Also makes entrypoint.sh compatible with more distributions
* Remove unnecessary quoting in entrypoint.sh
* Use exactly the same arguments to call CI tests
* Remove a redundant CI build
* Rename Cargo.lock check job
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* Drop sentry dependencies when enable-sentry feature is disabled
* Make lightwalletd gRPC tests depend on a new lightwalletd-grpc-tests feature
* fix(ci): remove enable-sentry feature from tests
* Add lightwalletd-grpc-tests feature for functionality or efficiency
And document where it is just used to stop re-compilations.
* Remove redundant `cmake` and `protobuf-compiler` dependencies
* Document Zebra's optional production and test feature flags
* Minimise dependencies in zcash-params/Dockerfile
* Minimise dependencies in docker/Dockerfile
* Add a workflow TODO
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* refactor(build): use better stage naming and document it
* refactort(build): use multiple cache sources
* docs(build): add a comment for cache
* fix(build): remove gcr.io as it does not supports OCI images
Gustavo Valverde
teor
Alfredo Garcia
dependabot[bot]