* Use correct release for getblocktemplate config
* Include at least 2 full checkpoints in the lookahead limit
* Increase full sync timeout to 36 hours
* Only log "synced block height too far ahead of the tip" once
* Replace AboveLookaheadHeightLimit error with pausing the syncer
* Use AboveLookaheadHeightLimit for blocks a very long way from the tip
* Also add the getblocktemplate config, and fix the test message
* Remove an outdated TODO comment
* Allow syncing again when a small number of blocks are in the queue
* Allow some dead code
* Only run multiple test jobs if they are needed for a long test
* Remove unused job steps
* Remove trailing whitespace
* Follow logs in the Run step
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Run CI jobs on dependent PRs
* Change job names to be unique
* Fix outdated workflow name
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make "test all" log output shorter
* Use different docker instance names
* Spell out command-line arguments
* Fix option names
* Use nocapture on basic tests but not ignored tests
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* only run the send transaction test on the main branch
* adds patch job
* Add concurrency rule to the send transactions test
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Allow send tx test to be triggered manually
Co-authored-by: teor <teor@riseup.net>
* test all features
* increase task timeout
* add a new task for feature tests
* add `getblocktemplate-rpcs` tests to docker integration
* run the getblocktemplate-rpcs feature as a separated step in docker
* move getblocktemplate-rpcs in docker as a separated task
* ci(sync): only run the `lightwalletd` full sync on the `main` branch
Previous behavior:
In PR #5164, we made lightwalletd sync all the way to the tip in its full
sync test.
This increases that test's time from 1 hour to 4 hours, which makes the CI
we run on each PR change increase from 3 hours to 6 hours.
Expected behavior:
Run the lightwalletd full sync just on `main` or if a state disk for the
actual version is not found.
Solution:
Add the `github.event_name == 'push' && github.ref_name == 'main'` condition
to the `lightwalletd-full-sync` test.
Fixes#5316
* Allow lwd full syncs to be triggered manually (#5400)
* Limit checkpoint and lwd full sync concurrency
* Add a patch job for lightwalletd-full-sync
Co-authored-by: teor <teor@riseup.net>
* Revert "ci(ssh): connect using `ssh-compute` action by Google (#5330)"
This reverts commit b366d6e7bb.
* ci(ssh): use sudo for docker commands if user is not root
* ci(ssh): specify the service account to connect with
* ci(ssh): increase the Google Cloud instance sshd connection limit
* chore: add a new line at the end of the script
* chore: update our VM image to bullseye
* chore: fix `tj-actions/changed-files` file comparison
* refactor(ssh): connect using `ssh-compute` action by Google
Previous behavior:
From time to time SSH connections to deployed VMs fails with the following
error: `kex_exchange_identification: Connection closed by remote host`
This was still happening after implementing https://github.com/ZcashFoundation/zebra/pull/5292
Excpected behavior:
Ensure we're not creating SSH key pairs on the fly to improve our connections
guarantees
Solution:
- Enable the Cloud Identity-Aware Proxy API in GCP
- Create a firewall rule to enable connections from IAP
- Grant the required IAM permissions to enable IAP TCP forwarding
- Generate an SSH keys pair and set a private key as an input param
- Set the GitHub Action SA to have authorized ssh connection to the VMs
- Implement the `google-github-actions/ssh-compute` action to connect
* fix(ssh): id `compute-ssh` cannot be used more than once within the same scope
* fix(ci): try to enclose commands to override parsing issues
* tmp: remove ssh_args
* fix(action): secrets must be inherited to be used
* tmp: validate command enclosing fixes executin
* fix(ssh): ssh_args are not implemented correctly
* fix(ssh): login with the root user
* fix(privelege): uso sudo with docker commands
* tmp: add sudo
* fix(ssh): use sudo for all docker commands
* fix(ssh): add missing `sudo` commands
* fix(ssh): get sync height from ssh stdout
* fix(height): get the height correctly
* Increase search range for sync height
* Update sync height regexes for zebrad and lwd cached states
* Add labels to cached state images
* Update deploy-gcp-tests.yml
* Don't create new cached states for lwd updates
* Add a missing line continuation
* Fix a comment
* Revert a mistaken comment change
* Clarify a TODO comment
* Partially revert to old docker height log handling
* Use an output for the cached disk name
* Increase search range for sync height
* Update sync height regexes for zebrad and lwd cached states
* Add labels to cached state images
* Add a missing line continuation
* ci(concurrency)!: run a single CI workflow as required
Previous behavior:
Multiple Mainnet full syncs were able to run on the main branch at the
same time, and pushing multiple commits to the same branch would run
multiple CI workflows, when only the run from last commit was relevant
Expected behavior:
Ensure that only a single CI workflow runs at the same time in PRs.
The latest commit should cancel any previous running workflows from the
same PR.
Solution:
Use GitHub actions concurrency feature https://docs.github.com/en/actions/using-jobs/using-concurrency
Fixes https://github.com/ZcashFoundation/zebra/issues/4977
Fixes https://github.com/ZcashFoundation/zebra/issues/4857
* docs: typo
* ci(concurrency): do not cancel running full syncs
Co-authored-by: teor <teor@riseup.net>
* fix(concurrency): explain the behavior better & add new ones
Co-authored-by: teor <teor@riseup.net>
Previous behavior:
If warnings or error are added in `.cargo/config.toml` or `clippy.toml`,
and those could generate CI failures, we wouldn't catch those new as the
pipelines are not run when this files are changed
Expected behavior:
If warnings or error are added in `.cargo/config.toml` or `clippy.toml`,
run all the builds and test jobs which also track a `Cargo.toml`.
Solution:
Add `.cargo/config.toml` and `clippy.toml` as paths to all the required
jobs which needs to be triggered when these files changes.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Save cached state on full syncs and updates
* Add an -update suffix to CI images created by updating cached state
* Make disk image names unique by adding a time suffix
* Use the latest image from any branch, but prefer the current commit if available
* Document Zebra's continuous integration tests
* Fix typos in environmental variable names
* Expand documentation
* Fix variable name typo
* Fix shell syntax
Previous behavior:
Sometimes Google Cloud authentication fails, this might happen before
IAM permissions are fully propagated
Expected behavior:
If the authentication fails, retry at least 3 times before exiting with
a non zero exit code
Applied solution:
Google GitHub Actions for auth recently added this a `retries` feature
which is now implemented to workaround this issue.
Note: 95a6bc2a27
Fixes https://github.com/ZcashFoundation/zebra/issues/4846
* Fix clippy::let_and_return
* Increase lightwalletd test timeouts for zebrad slowness
* Add a `zebrad_update_sync()` test, that update syncs Zebra without lightwalletd
* Run the zebrad-update-sync test in CI
* Add extra zebrad time to workaround lightwalletd bugs
* Initialize the rayon threadpool with a new config for CPU-bound threads
* Verify proofs and signatures on the rayon thread pool
* Only spawn one concurrent batch per verifier, for now
* Allow tower-batch to queue multiple batches
* Fix up a potentially incorrect comment
* Rename some variables for concurrent batches
* Spawn multiple batches concurrently, without any limits
* Simplify batch worker loop using OptionFuture
* Clear pending batches once they finish
* Stop accepting new items when we're at the concurrent batch limit
* Fail queued requests on drop
* Move pending_items and the batch timer into the worker struct
* Add worker fields to batch trace logs
* Run docker tests on PR series
* During full verification, process 20 blocks concurrently
* Remove an outdated comment about yielding to other tasks
* Upgrade tracing and related dependencies
```sh
cargo upgrade --workspace
tracing-error
tracing-subscrber
color-eyre
tracing-flame
tracing-journald
sentry
sentry-tracing
metrics
metrics-exporter-prometheus
reqwest
```
* Update duplicate dependency checks
* Enable the tracing/env-filter feature
* Fix type inference for metrics
Manual changes, plus:
```sh
fastmod "as _" "as f64"
```
* Tidy up some unrelated test code
* Update metrics-exporter-prometheus API
And make unused dependencies optional.
* Adjust test regexes to new tracing format
Also fix some regex bugs, and refactor to simplify.
* Disable color-eyre span traces and track caller in release builds
* Add a feature that enables extra debugging in release builds
* Clean up some redundant features
* Increase a test timeout
* fix(ci): allow for the lightwalletd-full-sync to mount the lwd-cache dir
* fix(ci): compare with a string
* imp(ci): run a lightwalletd tip if there's no lwd tip disk available
* docs(ci): add TODO explaining this is a temporal condition
* Require a cached state rebuild if the state version changes
* Find cached state disks with the same state version
And prefer `main` to other branches.
* Tweak filters to make them more specific
* Try adding inner quotes
* Try brackets instead
* Try two filters, rather than three
* Use Mainnet as the default network, remove duplicate env var
* Match the exact disk name format in one regular expression
* Log the exact expected disk name, including the network
* Consistently use CACHED_DISK_NAME as the env var name
* Temporary allow missing $NETWORK in disk names
* Print the exact search string
* Debug log the search string
* Use a generic alphabetical pattern rather than a regex group
Google Cloud doesn't seem to support regex groups.
* Add network name to disk match docs
* Fix the logged network name
* Make jobs that use cached state wait for state rebuilds
* Run jobs that need cached state even if the rebuild was skipped
* Fix missing dependencies
And update a TODO
* Revert "Use a generic alphabetical pattern rather than a regex group"
This reverts commit 970afe7b17.
* Revert "Temporary allow missing $NETWORK in disk names"
This reverts commit f1f66500c3.
* Make jobs that use cached state wait for state rebuilds
* Run jobs that need cached state even if the rebuild was skipped
* Fix missing dependencies
And update a TODO
* refactor(ci): look for available disks instead of files changed
This ensure that if the constants.rs file was changed, we search for disks available in the whole repository with the same state.
If there's no disk available a rebuild is triggered depending the missing disk. And if there's a disk available, tests are run with this one.
* fix(ci): lwd syncs needs to wait for zebra disk rebuild
* docs(ci): use better comments on integration tests
* fix(ci): we must authenticate to GCP to find disks
* fix(ci): add needed permissions for google auth
* fix(ci): the output needs to be echoed
* imp(ci): reduce diff with main
* fix(ci): remove redundant dependency
Co-authored-by: teor <teor@riseup.net>
* fix(ci): also add `false` to the JSON object output
* fix(ci): hasty copy/paste
* fix(ci): standardize comments
* fix(ci): run disk rebuilds if no disk was found
* fix(ci): build on any event if a cached disk is not found
* fix(ci): reduce diff with main
* docs(ci): reduce main diff
* fix(ci): sync .patch file with changes on the workflow
* fix(ci): consider network changes in new get-available-disks
* force GHA trigger
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Make jobs that use cached state wait for state rebuilds
* Run jobs that need cached state even if the rebuild was skipped
* Fix missing dependencies
And update a TODO
* refactor(ci): look for available disks instead of files changed
This ensure that if the constants.rs file was changed, we search for disks available in the whole repository with the same state.
If there's no disk available a rebuild is triggered depending the missing disk. And if there's a disk available, tests are run with this one.
* fix(ci): lwd syncs needs to wait for zebra disk rebuild
* docs(ci): use better comments on integration tests
* fix(ci): we must authenticate to GCP to find disks
* fix(ci): add needed permissions for google auth
* fix(ci): the output needs to be echoed
* imp(ci): reduce diff with main
* fix(ci): remove redundant dependency
Co-authored-by: teor <teor@riseup.net>
* fix(ci): also add `false` to the JSON object output
* fix(ci): hasty copy/paste
* fix(ci): standardize comments
* fix(ci): run disk rebuilds if no disk was found
* fix(ci): build on any event if a cached disk is not found
* fix(ci): sync .patch file with changes on the workflow
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Revert "Temporarily stop requiring cached lightwalletd state for the send transaction tests"
This reverts commit f6b29b151e.
* fix(ci): add a lightwalletd cached state to the test
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): lwd state condition
* fix(ci): differentiate tests that need a lwd cached state
* fix(ci): use the right state and save name for each test
* docs(ci): minor comment fixes
* docs(ci): better input description
* fix(ci): end `if` condition correctly
* fix(images): pass the state version to following steps
* fix(ci): $needs_lwd_state condition was inverted
* fix(ci): reduce disk selection code
* docs(ci): better disk search conditional explanation
* fix(ci): end if condition correctly
* fix(ci): evaluate $needs_zebra_state correctly
* fix(ci): use nested condition for readability
* fix(ci): disk search was using the wrong variable
* Temporarily use an earlier lightwalletd version
This checks if commit
e146dbf5c2
contains a mempool refresh deadlock bug.
* Actually rebuild the lightwalletd image
* Delete an unfinished comment
* Remove duplicate test in entrypoint.sh
* Keep a recent change to make tests consistent
* fix(ci): remove not used variable `lwd_state_dir`
* fix(ci): state wast not being added to the image name
* fix(ci): mount a docker volume with lightwalletd dir
If the volume doesn't mount this lwd cached state dir, the content won't be saved to the mounted disk in the VM
* fix(ci): lwd state condition
* docs(ci): explain disk mounting logic
* docs(ci): explain disk mounting decision better
* docs(ci): add a description for confusing input names
Co-authored-by: teor <teor@riseup.net>
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* Drop sentry dependencies when enable-sentry feature is disabled
* Make lightwalletd gRPC tests depend on a new lightwalletd-grpc-tests feature
* fix(ci): remove enable-sentry feature from tests
* Add lightwalletd-grpc-tests feature for functionality or efficiency
And document where it is just used to stop re-compilations.
* Remove redundant `cmake` and `protobuf-compiler` dependencies
* Document Zebra's optional production and test feature flags
* Minimise dependencies in zcash-params/Dockerfile
* Minimise dependencies in docker/Dockerfile
* Add a workflow TODO
* Catch more errors in entrypoint.sh
Also makes entrypoint.sh compatible with more distributions
* Remove unnecessary quoting in entrypoint.sh
* Use exactly the same arguments to call CI tests
* Remove a redundant CI build
* Rename Cargo.lock check job
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* Drop sentry dependencies when enable-sentry feature is disabled
* Make lightwalletd gRPC tests depend on a new lightwalletd-grpc-tests feature
* fix(ci): remove enable-sentry feature from tests
* Add lightwalletd-grpc-tests feature for functionality or efficiency
And document where it is just used to stop re-compilations.
* Remove redundant `cmake` and `protobuf-compiler` dependencies
* Document Zebra's optional production and test feature flags
* Minimise dependencies in zcash-params/Dockerfile
* Minimise dependencies in docker/Dockerfile
* Add a workflow TODO
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* refactor(build): use better stage naming and document it
* refactort(build): use multiple cache sources
* docs(build): add a comment for cache
* fix(build): remove gcr.io as it does not supports OCI images
teor
Arya
Alfredo Garcia
Gustavo Valverde
dependabot[bot]