* updates mod docs for tests that use future blocks
* updates submitblock test to use TestType methods
* prunes redundant code
* adds check_sync_logs_until
* adds assertion for needs cached state & rpc server
* updates get_raw_future_blocks fn with rpc calls
* updates to get_raw_future_blocks fn and submit_block test
* Rename LightwalletdTestType to TestType
* moves TestType and random_known_rpc_port_config to test_type.rs and config.rs
* moves get_raw_future_blocks to cached_state.rs
* updates ci workflows to include submit block test
* adds get_future_blocks fn and uses it in load_transactions_from_future_blocks
* updates CI docker
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Applies suggestions from code review
* Updates misnamed closure param
* updates mod docs for test_type.rs
Co-authored-by: teor <teor@riseup.net>
* Use correct release for getblocktemplate config
* Include at least 2 full checkpoints in the lookahead limit
* Increase full sync timeout to 36 hours
* Only log "synced block height too far ahead of the tip" once
* Replace AboveLookaheadHeightLimit error with pausing the syncer
* Use AboveLookaheadHeightLimit for blocks a very long way from the tip
* Also add the getblocktemplate config, and fix the test message
* Remove an outdated TODO comment
* Allow syncing again when a small number of blocks are in the queue
* Allow some dead code
* Only run multiple test jobs if they are needed for a long test
* Remove unused job steps
* Remove trailing whitespace
* Follow logs in the Run step
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Run CI jobs on dependent PRs
* Change job names to be unique
* Fix outdated workflow name
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make "test all" log output shorter
* Use different docker instance names
* Spell out command-line arguments
* Fix option names
* Use nocapture on basic tests but not ignored tests
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* allow(clippy::result_large_err)
* Increase the async executor delay expected by tests
* Split getblocktemplate-rpcs OS tests into a separate matrix job
* Add new patch jobs
* allow(unknown_lints)
* only run the send transaction test on the main branch
* adds patch job
* Add concurrency rule to the send transactions test
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Allow send tx test to be triggered manually
Co-authored-by: teor <teor@riseup.net>
* test all features
* increase task timeout
* add a new task for feature tests
* add `getblocktemplate-rpcs` tests to docker integration
* run the getblocktemplate-rpcs feature as a separated step in docker
* move getblocktemplate-rpcs in docker as a separated task
* ci(sync): only run the `lightwalletd` full sync on the `main` branch
Previous behavior:
In PR #5164, we made lightwalletd sync all the way to the tip in its full
sync test.
This increases that test's time from 1 hour to 4 hours, which makes the CI
we run on each PR change increase from 3 hours to 6 hours.
Expected behavior:
Run the lightwalletd full sync just on `main` or if a state disk for the
actual version is not found.
Solution:
Add the `github.event_name == 'push' && github.ref_name == 'main'` condition
to the `lightwalletd-full-sync` test.
Fixes#5316
* Allow lwd full syncs to be triggered manually (#5400)
* Limit checkpoint and lwd full sync concurrency
* Add a patch job for lightwalletd-full-sync
Co-authored-by: teor <teor@riseup.net>
* Revert "ci(ssh): connect using `ssh-compute` action by Google (#5330)"
This reverts commit b366d6e7bb.
* ci(ssh): use sudo for docker commands if user is not root
* ci(ssh): specify the service account to connect with
* ci(ssh): increase the Google Cloud instance sshd connection limit
* chore: add a new line at the end of the script
* chore: update our VM image to bullseye
* chore: fix `tj-actions/changed-files` file comparison
* ci(disk): use an official image on CI VMs for disk auto-resizing
Previous behavior:
We've presented issues in the past with resizing as the device is busy,
for example:
```
e2fsck: Cannot continue, aborting.
/dev/sdb is in use.
```
Expected behavior:
We've been manually resizing the disk as this task was not being done
automatically, but having an official Public Image from GCP would make
this easier (automatic) and it also integrates better with other GCP
services
Configuration differences: https://cloud.google.com/compute/docs/images/os-details#notable-difference-debian
Solution:
- Use `debian-11` from the official public images https://cloud.google.com/compute/docs/images/os-details#debian
- Remove the manual disk resizing from the pipeline
* ci: increase VM disk size to fit future cached states sizes
Some GCP disk images are 160 GB, which means they could get to the current
200 GB size soon.
* Revert "ci(ssh): connect using `ssh-compute` action by Google (#5330)"
This reverts commit b366d6e7bb.
* ci(ssh): use sudo for docker commands if user is not root
* ci(ssh): specify the service account to connect with
* ci(ssh): increase the Google Cloud instance sshd connection limit
* chore: add a new line at the end of the script
* chore: update our VM image to bullseye
* chore: fix `tj-actions/changed-files` file comparison
* Add latest and edge tags to Docker images
* Document how latest tag actually works
* Try a different syntax for is_default_branch
* Try again
* One last try
* Revert changes that don't work
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* refactor(ssh): connect using `ssh-compute` action by Google
Previous behavior:
From time to time SSH connections to deployed VMs fails with the following
error: `kex_exchange_identification: Connection closed by remote host`
This was still happening after implementing https://github.com/ZcashFoundation/zebra/pull/5292
Excpected behavior:
Ensure we're not creating SSH key pairs on the fly to improve our connections
guarantees
Solution:
- Enable the Cloud Identity-Aware Proxy API in GCP
- Create a firewall rule to enable connections from IAP
- Grant the required IAM permissions to enable IAP TCP forwarding
- Generate an SSH keys pair and set a private key as an input param
- Set the GitHub Action SA to have authorized ssh connection to the VMs
- Implement the `google-github-actions/ssh-compute` action to connect
* fix(ssh): id `compute-ssh` cannot be used more than once within the same scope
* fix(ci): try to enclose commands to override parsing issues
* tmp: remove ssh_args
* fix(action): secrets must be inherited to be used
* tmp: validate command enclosing fixes executin
* fix(ssh): ssh_args are not implemented correctly
* fix(ssh): login with the root user
* fix(privelege): uso sudo with docker commands
* tmp: add sudo
* fix(ssh): use sudo for all docker commands
* fix(ssh): add missing `sudo` commands
* fix(ssh): get sync height from ssh stdout
* fix(height): get the height correctly
Previous behavior:
The following error was causing an exit 1 in GitHub Actions when a pushing
to the `main` branch
```
Error: Similar commit hashes detected: previous sha is equivalent to the
current sha
```
Expeceted behavior:
Allow the linter to run succesfully even if the previous SHA has no files
changed
Solution:
Add `fetch-depth: 2` to retrieve the preceding commit
Previous behavior
From time to time SSH connections to deployed VMs fails with the following
error: `kex_exchange_identification: Connection closed by remote host`
Expected behavior
If the connection fails, attempt to reconnect once again (or multiple times)
Solution
Add the `ConnectionAttempts` and `ConnectTimeout` with 20 and 5 values
respectively, which attempst to reconnect 19 more times every 5 seconds
* Update release-drafter.yml
* Explain where we got the workflow from
* Automatically add "trivial" label to dependabot updates
* Add categories and auto-labels to release drafter
* Update release PR template for automatic release drafter versions
* Also strip PR series numbers and leading spaces from changelog entries
* Update release note version check
* Update label names
* Add missing ! in conventional commits regex
Co-authored-by: Marek <mail@marek.onl>
* Make versioning steps more specific
Co-authored-by: Marek <mail@marek.onl>
* Remove conflicting detailed versioning explanations
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Use branch then main build caches
* Revert cache order to try branch cache first
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Try running coverage with Rust 1.63
* Run GitHub Actions tests with Rust 1.63
* Change from stable to 1.63 in the patch file
* Use Rust 1.63 to download Zcash parameters
* Use Rust 1.63 to build Docker zebrad images
* Make Rust 1.63 a supported platform, and make stable temporarily unsupported
* Delete test instances after 3 days
* Use correct delete command, improve shell quoting
* Use sed to provide the correct zone or region
* Fix quoting
* Fix IFS
* Fix IFS for multiple disks
* Document why we can't quote some shell variables
* Document that instances can get deleted
* Fix exact names in deletion docs
* feat(release): create Docker hub binaries when tagging
* fix(release): add a release workflow for binaries
* fix(release): trigger on tag creation, not pushing to it
* fix(release): use the same conditions for logging into DockerHub
* fix(release): add missing parameter to access GH secrets
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* ci(release): just publish to DockerHub when a release is published
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* ci(release): filter prerelease event correctly
* ci(release): fix tags
* ci(release): use `zebra` and not `zebrad` as the repository
* ci(release): do not try to login to Docker if not a release
* Update .github/workflows/build-docker-image.yml
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* Increase search range for sync height
* Update sync height regexes for zebrad and lwd cached states
* Add labels to cached state images
* Update deploy-gcp-tests.yml
* Don't create new cached states for lwd updates
* Add a missing line continuation
* Fix a comment
* Revert a mistaken comment change
* Clarify a TODO comment
* Partially revert to old docker height log handling
* Use an output for the cached disk name
* Disable fmt cache and create shared clippy cache
* Make Cargo.lock check use the shared clippy cache
* Add a TODO for Windows Rust cache path
* Fix quoting for Windows path
* Use correct sharedKey spelling
* Increase search range for sync height
* Update sync height regexes for zebrad and lwd cached states
* Add labels to cached state images
* Add a missing line continuation
Previous behavior:
The `tj-actions/changed-files` crashed when making pushes to main, as no
fetch depth was defined on the previous checkout action. Which is now r
required after b216561b5b
Expected behavior:
Do not fail with this new requirement
Solution:
Change the chekout action `fetch-depth` to 2, allowing to compare with
the previous commit
* bump prost, tonic and tonic-build
* add protoc as a dependency step in the CI
* bump console-subscriber
* add protoc to `build-crates-individually`
* add protoc to docs build
* install protoc in lint.yml
* change protoc installation location in lint.yml
* add protoc to `Check Cargo.lock is up to date`
* ci(build): keep protoc pinned to the same major version
* ci(build): avoid rate limiting with `arduino/setup-protoc@v1`
* cargo upgrade --workspace console-subscriber
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
Co-authored-by: teor <teor@riseup.net>
* Fix delete GCP resources commands
* Don't create a GCP credentials file
* Keep the latest 2 images
* Explain time
* Show the names of disks that are being deleted
* Actually run the image delete steps
* Only delete commit-based instance templates
* Document automated deletion
Previous behavior:
Disk images are being accumulated in GCP for a few years, but this
generates unneeded costs as we're not using images older than 1-2 weeks.
Expected behavior:
Delete previously unused images based on a timefrime.
Solution:
Delete images created on a pull request older than 30 days, from the
`main` branch if older than 60 days, and any other image older than 90
days.
A TODO is on place as we'd like to keep at least the 2 latest images of
each type (zebra checkpoint, zebra tip, lwd tip). Once we've excluded
those images, we can delete any older images after 1 week.
Previous behavior:
As we disabled beta Rust tests in PR #4930, because the parameter
downloads were unstable with beta Rust, we're no longer testing it.
Expected behavior:
Re-enable beta rust tests in CI OSes
Solution:
Remove the parameter exluding beta Rust
* ci(concurrency)!: run a single CI workflow as required
Previous behavior:
Multiple Mainnet full syncs were able to run on the main branch at the
same time, and pushing multiple commits to the same branch would run
multiple CI workflows, when only the run from last commit was relevant
Expected behavior:
Ensure that only a single CI workflow runs at the same time in PRs.
The latest commit should cancel any previous running workflows from the
same PR.
Solution:
Use GitHub actions concurrency feature https://docs.github.com/en/actions/using-jobs/using-concurrency
Fixes https://github.com/ZcashFoundation/zebra/issues/4977
Fixes https://github.com/ZcashFoundation/zebra/issues/4857
* docs: typo
* ci(concurrency): do not cancel running full syncs
Co-authored-by: teor <teor@riseup.net>
* fix(concurrency): explain the behavior better & add new ones
Co-authored-by: teor <teor@riseup.net>
Previous behavior:
When a push was detected in the `main` branch, the workflow would run the
`versioning` job and crash trying to detect the version being deployed as
there was none.
Expected behavior:
Do not fail the `versioning` job when pushing to `main`
Solution:
Limit the `versioning` job to only run when a release event is triggered
and allow the `deploy-nodes` job to run even if `versioning` is skipped
* Show the arguments of acceptance test functions in the logs
* Show all the logs in the "Run tests" jobs
* Document expected "broken pipe" error from `tee`
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feat(build): deploy long running instances on release
Previous behavior:
Each time we merged to main new nodes would be deployed, this is an
expected behavior as we need to ensure nodes get deployed and run
without issues, but this could also replace nodes very hastily.
Expected behavior:
We want instances which would run for a longer time, to allow us to
troubleshoot issues or inspect the behavior of this instances for longer
periods of time (2+ weeks)
Applied solution:
Deploy a versioned manage instance group (MiG) using the major version
of the release semver. We just use the first part of the version to
replace old instances, and change it when a major version is released
to keep a segregation between new and old versions.
* ci(build): allow v0 as a major version tag
* fix(build): use rust conventions for versioning
* fix(deploy): improve documentation and trigger on release
* Update .github/workflows/continous-delivery.yml
Co-authored-by: teor <teor@riseup.net>
* fix(versioning): typo
* fix(deploy): use `zebrad-v1` as the instance name, with no SHA
* fix(deploy): create and update MiG must use the same name
* docs(deployments): add Continuous Delivery process
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Expand cached state disks before running tests
* Install partition management tool
* There isn't actually a partition on the cached state image
* Make e2fsck non-interactive
* Limit the length of image names to 63 characters
* Ignore possibly long branch names when matching images, just match the commit
* Increase full sync timeout to 24 hours
Expected sync time is ~21 hours as of August 2022.
* Split final checkpoint job into two smaller jobs to avoid timeouts
Also make regexes easier to read.
* Fix a job name typo
Previous behavior:
If warnings or error are added in `.cargo/config.toml` or `clippy.toml`,
and those could generate CI failures, we wouldn't catch those new as the
pipelines are not run when this files are changed
Expected behavior:
If warnings or error are added in `.cargo/config.toml` or `clippy.toml`,
run all the builds and test jobs which also track a `Cargo.toml`.
Solution:
Add `.cargo/config.toml` and `clippy.toml` as paths to all the required
jobs which needs to be triggered when these files changes.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Save cached state on full syncs and updates
* Add an -update suffix to CI images created by updating cached state
* Make disk image names unique by adding a time suffix
* Use the latest image from any branch, but prefer the current commit if available
* Document Zebra's continuous integration tests
* Fix typos in environmental variable names
* Expand documentation
* Fix variable name typo
* Fix shell syntax
Previous behavior:
Sometimes Google Cloud authentication fails, this might happen before
IAM permissions are fully propagated
Expected behavior:
If the authentication fails, retry at least 3 times before exiting with
a non zero exit code
Applied solution:
Google GitHub Actions for auth recently added this a `retries` feature
which is now implemented to workaround this issue.
Note: 95a6bc2a27
Fixes https://github.com/ZcashFoundation/zebra/issues/4846
* update timeout
* update the doc comment
* Increase test timeouts for Zebra update syncs
* Stop failing the 1740k job if the cached state is after block 1740k
Co-authored-by: teor <teor@riseup.net>
* Apply the same Rust logging settings to all GitHub workflows
* Enable full optimisations in dev builds for downloading large parameter files
* Disable beta Rust tests in CI
* Make code execution time logs shorter
* Do ZK parameter preloads in the lightwalletd tests that need them
* Try to re-launch `lightwalletd` when it hangs during sync tests
* Increase full sync timeout
* Clear the `zebrad` logs during `lightwalletd` tests, to avoid logging deadlocks
* Actually clear more than one line of logs
* Check zebrad and lightwalletd output in parallel threads, while waiting for zebrad
* Check zebrad and lightwalletd output in parallel threads, while waiting for lightwalletd
* Improve test logging
* Fix a log typo
* Only wait for lightwalletd once, because its logs stop after the initial sync
* Look for cached state disks for this commit and branch first
* Only copy the state once in the send transactions test
* Wait longer for lightwalletd gRPC server startup
* Add some function docs
* cargo fmt --all
* Fix clippy::let_and_return
* Increase lightwalletd test timeouts for zebrad slowness
* Add a `zebrad_update_sync()` test, that update syncs Zebra without lightwalletd
* Run the zebrad-update-sync test in CI
* Add extra zebrad time to workaround lightwalletd bugs
* Initialize the rayon threadpool with a new config for CPU-bound threads
* Verify proofs and signatures on the rayon thread pool
* Only spawn one concurrent batch per verifier, for now
* Allow tower-batch to queue multiple batches
* Fix up a potentially incorrect comment
* Rename some variables for concurrent batches
* Spawn multiple batches concurrently, without any limits
* Simplify batch worker loop using OptionFuture
* Clear pending batches once they finish
* Stop accepting new items when we're at the concurrent batch limit
* Fail queued requests on drop
* Move pending_items and the batch timer into the worker struct
* Add worker fields to batch trace logs
* Run docker tests on PR series
* During full verification, process 20 blocks concurrently
* Remove an outdated comment about yielding to other tasks
* Checkout zebra in each job to avoid warnings
But put TODOs where we might be able to skip checkouts
* Split log following into sprout checkpoints, sapling/orchard checkpoints, and full validation
* Make job IDs shorter
* Use /dev/stderr because docker doesn't have a tty
* remove pipefail
* Revert "remove pipefail"
This reverts commit a7ee37bebdc107a4215e7dd307b189d925969234.
* Make tee ignore errors writing to a grep pipe
* Avoid launching multiple docker instances for duplicate jobs
* Ignore broken pipe error messages and statuses
* fix(ci): docker wait not finding container
We had this issue before, I can't recall if this was a parsing error between GitHub Actions and gcloud `--command` parsing, but we had to change this into two pieces.
This implementation keeps it how we did it before 9b9578c999/.github/workflows/test.yml (L235-L243)
* docs: remove pending TODO
We can't remove `actions/checkout` nor set `create_credentials_file` to `false` as next steps won't be able to authenticate to GCP.
We can surely remove `actions/checkout` and leave `create_credentials_file` as `true`, but this will raise a warning on each step, and there's no benefit of doing so.
* Show `docker wait` and `gcloud ssh` output
* If `docker wait` fails, get the exit code using `docker inspect`
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Put arguments to "docker run" on different lines
And update some comments.
* Split docker run into launch, logs, and wait
* Remove mistaken "needs state" condition on log and results job
* Exit the ssh and the job with the container test's exit status
* Split full sync into checkpoint and full validation
* Sort workflow variables into categories and add descriptions
* Split Create instance/volume and Run test into separate jobs
* Copy initial conditions to all jobs in the series
* Actually create a cached state image
* fix(state): use same disk naming convention for all test instances
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* feat(ci): build each crate individually
* fix(ci): use valid names for each job
* feat(ci): builds and checks with and without all features
* refactor(ci): build job matrix dinamically
* fix: use a JSON_CRATES variable with resulting values
* test: check-matrix
* fix(ci): use "crate" in singular for reference
* imp(ci): use a matrix for feature build arguments
* fix(ci): use correct naming and includes
* fix(ci): implement most recommendations given in review
* fix(ci): use simpler shell script
* fix: typo
* fix: add string to file, not cmd
* fix: some shellchecks
* fix(ci): remove warnings and errors from shellcheck
* imp(ci): add patch file for `Build crates individually` workflow
* Remove unused configs in patch job
Co-authored-by: teor <teor@riseup.net>
* feat(actions): delete old GCP resources
* fix(ci): delete old instances templates
* fix(actions): use correct date arguments and conversion
* fix(actions): missing command in gcloud
* fix(gcp): if an instance can't be deleted, continue
* refacor(action): cleanup and execute monthly
* increase lightwalletd timeout
* switch back to aditya's fork
* manually point to new aditya's lightwalletd image
* disable sync_one_checkpoint_testnet test
* disable restart_stop_at_height in testnet
* rever to 'latest' lightwalletd image
* Remove a duplicate lightwalletd error message
* Reactivate some error messages that have been fixed
* Fix confusing lightwalletd cached state path logs
* Add the gRPC tests to the lightwalletd test suite function
* Make test regexes compatible with zcash/lightwalletd
* Add logging to gRPC tests
* Switch to zcash/lightwalletd for testing
* Upgrade tracing and related dependencies
```sh
cargo upgrade --workspace
tracing-error
tracing-subscrber
color-eyre
tracing-flame
tracing-journald
sentry
sentry-tracing
metrics
metrics-exporter-prometheus
reqwest
```
* Update duplicate dependency checks
* Enable the tracing/env-filter feature
* Fix type inference for metrics
Manual changes, plus:
```sh
fastmod "as _" "as f64"
```
* Tidy up some unrelated test code
* Update metrics-exporter-prometheus API
And make unused dependencies optional.
* Adjust test regexes to new tracing format
Also fix some regex bugs, and refactor to simplify.
* Disable color-eyre span traces and track caller in release builds
* Add a feature that enables extra debugging in release builds
* Clean up some redundant features
* Increase a test timeout
* refactor(ci): keep tests jobs under the 6 hour timeout
When running a full sync or any other test which takes almost 5 hours, having those jobs running with other actions that might take several minutes, also reduces the overall time from the job_id.
We use a separate job for image creation and deletion to handle this cases.
* fix(ci): instance deletion can't run on non finished tests
* fix(ci): tests without a cached state might save to disk
* fix(ci): ignore failures when deleting an instance
* fix(ci): remove delete step `needs` redundancy
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feat(ci): add a codespell linting action
* fix(ci): run this job if the lint workflow is changed
* ci(codespell): add configuration file
* ci(codespell): exclude mermaid.min.js
* fix: wrong mermaid.min.js location
* ci(codespell): Sur from "Big Sur" is being considered as misspelled
* ci(codespell): make warning the max level
This won't restrict PRs from merging
* ci(codespell): lint on every push
* test: create a misspelling
* Revert "test: create a misspelling"
This reverts commit a2c91cda1e.
* fix(ci): allow for the lightwalletd-full-sync to mount the lwd-cache dir
* fix(ci): compare with a string
* imp(ci): run a lightwalletd tip if there's no lwd tip disk available
* docs(ci): add TODO explaining this is a temporal condition
* Require a cached state rebuild if the state version changes
* Find cached state disks with the same state version
And prefer `main` to other branches.
* Tweak filters to make them more specific
* Try adding inner quotes
* Try brackets instead
* Try two filters, rather than three
* Use Mainnet as the default network, remove duplicate env var
* Match the exact disk name format in one regular expression
* Log the exact expected disk name, including the network
* Consistently use CACHED_DISK_NAME as the env var name
* Temporary allow missing $NETWORK in disk names
* Print the exact search string
* Debug log the search string
* Use a generic alphabetical pattern rather than a regex group
Google Cloud doesn't seem to support regex groups.
* Add network name to disk match docs
* Fix the logged network name
* Make jobs that use cached state wait for state rebuilds
* Run jobs that need cached state even if the rebuild was skipped
* Fix missing dependencies
And update a TODO
* Revert "Use a generic alphabetical pattern rather than a regex group"
This reverts commit 970afe7b17.
* Revert "Temporary allow missing $NETWORK in disk names"
This reverts commit f1f66500c3.
* Make jobs that use cached state wait for state rebuilds
* Run jobs that need cached state even if the rebuild was skipped
* Fix missing dependencies
And update a TODO
* refactor(ci): look for available disks instead of files changed
This ensure that if the constants.rs file was changed, we search for disks available in the whole repository with the same state.
If there's no disk available a rebuild is triggered depending the missing disk. And if there's a disk available, tests are run with this one.
* fix(ci): lwd syncs needs to wait for zebra disk rebuild
* docs(ci): use better comments on integration tests
* fix(ci): we must authenticate to GCP to find disks
* fix(ci): add needed permissions for google auth
* fix(ci): the output needs to be echoed
* imp(ci): reduce diff with main
* fix(ci): remove redundant dependency
Co-authored-by: teor <teor@riseup.net>
* fix(ci): also add `false` to the JSON object output
* fix(ci): hasty copy/paste
* fix(ci): standardize comments
* fix(ci): run disk rebuilds if no disk was found
* fix(ci): build on any event if a cached disk is not found
* fix(ci): reduce diff with main
* docs(ci): reduce main diff
* fix(ci): sync .patch file with changes on the workflow
* fix(ci): consider network changes in new get-available-disks
* force GHA trigger
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Make jobs that use cached state wait for state rebuilds
* Run jobs that need cached state even if the rebuild was skipped
* Fix missing dependencies
And update a TODO
* refactor(ci): look for available disks instead of files changed
This ensure that if the constants.rs file was changed, we search for disks available in the whole repository with the same state.
If there's no disk available a rebuild is triggered depending the missing disk. And if there's a disk available, tests are run with this one.
* fix(ci): lwd syncs needs to wait for zebra disk rebuild
* docs(ci): use better comments on integration tests
* fix(ci): we must authenticate to GCP to find disks
* fix(ci): add needed permissions for google auth
* fix(ci): the output needs to be echoed
* imp(ci): reduce diff with main
* fix(ci): remove redundant dependency
Co-authored-by: teor <teor@riseup.net>
* fix(ci): also add `false` to the JSON object output
* fix(ci): hasty copy/paste
* fix(ci): standardize comments
* fix(ci): run disk rebuilds if no disk was found
* fix(ci): build on any event if a cached disk is not found
* fix(ci): sync .patch file with changes on the workflow
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Revert "Temporarily stop requiring cached lightwalletd state for the send transaction tests"
This reverts commit f6b29b151e.
* fix(ci): add a lightwalletd cached state to the test
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): lwd state condition
* fix(ci): differentiate tests that need a lwd cached state
* fix(ci): use the right state and save name for each test
* docs(ci): minor comment fixes
* docs(ci): better input description
* fix(ci): end `if` condition correctly
* fix(images): pass the state version to following steps
* fix(ci): $needs_lwd_state condition was inverted
* fix(ci): reduce disk selection code
* docs(ci): better disk search conditional explanation
* fix(ci): end if condition correctly
* fix(ci): evaluate $needs_zebra_state correctly
* fix(ci): use nested condition for readability
* fix(ci): disk search was using the wrong variable
* Temporarily use an earlier lightwalletd version
This checks if commit
e146dbf5c2
contains a mempool refresh deadlock bug.
* Actually rebuild the lightwalletd image
* Delete an unfinished comment
* Remove duplicate test in entrypoint.sh
* Keep a recent change to make tests consistent
* fix(ci): remove not used variable `lwd_state_dir`
* fix(ci): state wast not being added to the image name
* fix(ci): mount a docker volume with lightwalletd dir
If the volume doesn't mount this lwd cached state dir, the content won't be saved to the mounted disk in the VM
* fix(ci): lwd state condition
* docs(ci): explain disk mounting logic
* docs(ci): explain disk mounting decision better
* docs(ci): add a description for confusing input names
Co-authored-by: teor <teor@riseup.net>
* Require a cached state rebuild if the state version changes
* Find cached state disks with the same state version
And prefer `main` to other branches.
* Tweak filters to make them more specific
* Try adding inner quotes
* Try brackets instead
* Try two filters, rather than three
* Use Mainnet as the default network, remove duplicate env var
* Match the exact disk name format in one regular expression
* Log the exact expected disk name, including the network
* Consistently use CACHED_DISK_NAME as the env var name
* Temporary allow missing $NETWORK in disk names
* Print the exact search string
* Debug log the search string
* Use a generic alphabetical pattern rather than a regex group
Google Cloud doesn't seem to support regex groups.
* Add network name to disk match docs
* Fix the logged network name
* imp(ci): remove gcp verbose log
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* Drop sentry dependencies when enable-sentry feature is disabled
* Make lightwalletd gRPC tests depend on a new lightwalletd-grpc-tests feature
* fix(ci): remove enable-sentry feature from tests
* Add lightwalletd-grpc-tests feature for functionality or efficiency
And document where it is just used to stop re-compilations.
* Remove redundant `cmake` and `protobuf-compiler` dependencies
* Document Zebra's optional production and test feature flags
* Minimise dependencies in zcash-params/Dockerfile
* Minimise dependencies in docker/Dockerfile
* Add a workflow TODO
* Catch more errors in entrypoint.sh
Also makes entrypoint.sh compatible with more distributions
* Remove unnecessary quoting in entrypoint.sh
* Use exactly the same arguments to call CI tests
* Remove a redundant CI build
* Rename Cargo.lock check job
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* Drop sentry dependencies when enable-sentry feature is disabled
* Make lightwalletd gRPC tests depend on a new lightwalletd-grpc-tests feature
* fix(ci): remove enable-sentry feature from tests
* Add lightwalletd-grpc-tests feature for functionality or efficiency
And document where it is just used to stop re-compilations.
* Remove redundant `cmake` and `protobuf-compiler` dependencies
* Document Zebra's optional production and test feature flags
* Minimise dependencies in zcash-params/Dockerfile
* Minimise dependencies in docker/Dockerfile
* Add a workflow TODO
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* refactor(build): use better stage naming and document it
* refactort(build): use multiple cache sources
* docs(build): add a comment for cache
* fix(build): remove gcr.io as it does not supports OCI images
* feat(ci): add lightwalletd_*_sync tests to CI
* feat(ci): add lightwalletd RPC call test
* feat(ci): add send transactions test with lwd to CI
* fix(ci): create a variable to run transactions test
* refactor(ci): use docker in docker
This is a workaround for an issue related to disk partitioning, caused by a GCP service called Konlet, while mounting the cached disks to the VM and then to the container
* fix(build): persist docker login credentials
* fix(ci): get sync height from docker logs instead of gcp
* try: use gha cache for faster building
* fix(ci): mount disk in container to make it available in vm
* fix(build): do not invalidate cache between images
* try(docker): invalidate cache as less as possible
* fix(ci): GHA terminal is not a TTY
* fix(build): do not ignore entrypoint.sh
* fix
* fix(ci): mount using root priveleges
* fix(ci): use existing disk as cached state
* fix(ci): wait for disks to get mounted
* force rebuild
* fix failed force
* fix failed commit
* WIP
* fix(ci): some tests does not use a cached state
* wip
* refactor(ci): disk names and job segregation
* fix(ci): do not name boot and attached disk the same
* fix(ci): attach a disk to full sync, to snapshot the state
* fix(ci): use correct disk implementations
* fix(ci): use different disk name to allow test concurrency
* feat(ci): add lightwalledt send transaction test
* cleanup(ci): remove extra tests
* fix(ci): allow disk concurrency with tests
* fix(ci): add considerations for different tests
* fix(reusable): last fixes
* feat(ci): use reusable workflow for tests
* fix(rw): remove nested worflow
* fix(rw): minor fixes
* force rebuild
* fix(rw): do not use an input as job name
* fix(rw): remove variable id
* fix(ci): remove explicit conditions and id
* fix(ci): docker does not need the variable sign ($) to work
* fix(ci): mount typo
* fix(ci): if a sync fails, always delete the instance
This also reduces the amount of jobs needed.
* refactor(ci): make all test depend on the same build
* fix(ci): some tests require multiple variables
* fix(docker): variable substitution
* fix(ci): allow to run multiple commits from a PR at once
* fix(docker): lower the NETWORK env var for test names
* reduce uneeded diff
* imp(keys): use better naming for builds_disks
* imp(ci): use input defaults
* imp(ci): remove test_name in favor of test_id
* fix(ci): better key naming
* fix(ci): long disk names breaks GCP naming convention
* feat(ci): validate local state version with cached state
* fix(ci): add condition to run tests
* fix: typo
* fix: app_name should not be required
* fix: zebra_state_path shouldn't be required
* fix: reduce diff
* fix(ci): checkout to grep local state version
* Update .github/workflows/test.yml
Co-authored-by: teor <teor@riseup.net>
* revert: merge all tests into a single workflow
* Remove unused STATE_VERSION env var
* fix: minor fixes
* fix(ci): make test.patch the same as test
* fix(ci): negate the input value
* imp(ci): better cached state conditional handling
* imp(ci): exit code is captured by `docker run`
* fix(deploy): mount disks with better write performance
* fix(ci): change sync id to a broader id name
* fix(ci): use correct input validation
* fix(ci): do not make test with cached state dependant on other
* imp(ci): organiza keys better
* fix(ci): use appropiate naming
* fix(ci): create docker volume before mounting
* fix(lint): do not fail on all new changes
* imp(ci): do not report in pr review
* fix(ci): partition clean disks
* fix: typo
* fix: test called the wrong way
* fix(build): stop using gha cache
* ref(ci): validate run condition before calling reusable workflow
* fix(ci): use a better filesystem dir and fix other values
* fix: linting errors
* fix(ci): typo
* Revert "fix(build): stop using gha cache"
This reverts commit a8fbc5f416.
Cache expiration is a lesser evil than not using caching at all and then failing with a 401
* imp(ci): do not set a default for needs_zebra_state
* Update .github/workflows/test.yml
Co-authored-by: teor <teor@riseup.net>
* fix(deps): remove dependencies
* force build
* Update .github/workflows/test.yml
Co-authored-by: teor <teor@riseup.net>
* fix(docker): add RUST_LOG as an ARG and ENV
* fix(test): add `#[ignore]` to send transactions test
This test needs state then it should be marked as #[ignore]
* fix(ci): differentiate between root cache path and its dir
* Remove extra `state` directory
That was a workaround for an issue that has been fixed.
* imp(docs): use better test descriptions
Co-authored-by: teor <teor@riseup.net>
* fix: reduce unwanted diff with main
* fix(ci): make lwd conditions consistent
* Remove another extra `state` directory
Was also part of a workaround for an issue that has been fixed.
* fix(ci): use better conditionals to run test jobs
Co-authored-by: teor <teor@riseup.net>
* Tweak to support different lightwalletd versions
Some versions print `Waiting for block`, and some versions print
`Ingestor waiting for block`.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* refactor(lint): check specific files for each job
* refactor(lint): use an approach which requires less code
* fix(lint): validate against true string not boolean
* imp(build): reduce docker cache invalidation
Use scoped caching and more file ignores to reduce cache invalidation
* fix(build): add entrypoint.sh as a required file
* fix(build): do not logout if the build takes too long
* Add 'doc comment' about .dockerignore
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* refactor(ci): use docker in docker
This is a workaround for an issue related to disk partitioning, caused by a GCP service called Konlet, while mounting the cached disks to the VM and then to the container
* fix(build): persist docker login credentials
* fix(ci): get sync height from docker logs instead of gcp
* try: use gha cache for faster building
* fix(ci): mount disk in container to make it available in vm
* fix(build): do not invalidate cache between images
* try(docker): invalidate cache as less as possible
* fix(ci): GHA terminal is not a TTY
* fix(build): do not ignore entrypoint.sh
* fix
* fix(ci): mount using root priveleges
* fix(ci): use existing disk as cached state
* fix(ci): wait for disks to get mounted
* force rebuild
* fix failed force
* fix(ci): some tests does not use a cached state
* fix(ci): do not name boot and attached disk the same
* fix(ci): attach a disk to full sync, to snapshot the state
* fix(ci): use appropiate grep text depending on the test
* reduce diff
* fix(ci): use correct GCP disk source attribute
* imp(ci): reduce diff
* fix(ci): revert wrong deletion
* fix: revert uneeded changes
* fix: reduce main diff
* fix
* fix(ci): reduce diff
* fix(ci): garbage collect instances no matter the status
As we're not going to reuse test instances, the safest method to apply is to always delete this instances if they fail, get skipped or succeed running a workflow
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* docs(ci): imrpove comment
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* refactor(ci): test building in a separate workflow
* force a change
* force a change
* fix(ci): send the correct variables to the reusable build
* fix(ci): variables are not allowed
* fix(ci): conditions are not allowed as input
* fix(ci): use expected value
* refactor(build): simplify the use of other dockerfiles
* fix(cd): depend on docker build yml
* fix(cd): use main branch as image name
* imp(actions): remove uneeded variable repetition
* imp(build): remove unused variables
* imp(actions): rename the image building workflow
Not all images are for zebra execution as we also have one for zcash-params
* fix(ci): add dependable workflow in paths filters
* docs(ci): remove TODO as this won't be needed at least an issue arises
* docs(ci): CARGO_INCREMENTAL can decrease build time when running from a cache
* fix: revert forced changes
* fix(build): remove unused build inputs in zcash-params
* imp(cd): as this is the production image, use the executable name
* imp(ci): reduce log level to improve speed
Co-authored-by: teor <teor@riseup.net>
* imp(ci): use the correct name for the workflow
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
Arya
dependabot[bot]
teor
Gustavo Valverde
Alfredo Garcia
Conrado Gouvea
Dimitris Apostolou
Deirdre Connolly