* refactor(ci): keep tests jobs under the 6 hour timeout
When running a full sync or any other test which takes almost 5 hours, having those jobs running with other actions that might take several minutes, also reduces the overall time from the job_id.
We use a separate job for image creation and deletion to handle this cases.
* fix(ci): instance deletion can't run on non finished tests
* fix(ci): tests without a cached state might save to disk
* fix(ci): ignore failures when deleting an instance
* fix(ci): remove delete step `needs` redundancy
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feat(ci): add a codespell linting action
* fix(ci): run this job if the lint workflow is changed
* ci(codespell): add configuration file
* ci(codespell): exclude mermaid.min.js
* fix: wrong mermaid.min.js location
* ci(codespell): Sur from "Big Sur" is being considered as misspelled
* ci(codespell): make warning the max level
This won't restrict PRs from merging
* ci(codespell): lint on every push
* test: create a misspelling
* Revert "test: create a misspelling"
This reverts commit a2c91cda1e.
* fix(ci): allow for the lightwalletd-full-sync to mount the lwd-cache dir
* fix(ci): compare with a string
* imp(ci): run a lightwalletd tip if there's no lwd tip disk available
* docs(ci): add TODO explaining this is a temporal condition
* Require a cached state rebuild if the state version changes
* Find cached state disks with the same state version
And prefer `main` to other branches.
* Tweak filters to make them more specific
* Try adding inner quotes
* Try brackets instead
* Try two filters, rather than three
* Use Mainnet as the default network, remove duplicate env var
* Match the exact disk name format in one regular expression
* Log the exact expected disk name, including the network
* Consistently use CACHED_DISK_NAME as the env var name
* Temporary allow missing $NETWORK in disk names
* Print the exact search string
* Debug log the search string
* Use a generic alphabetical pattern rather than a regex group
Google Cloud doesn't seem to support regex groups.
* Add network name to disk match docs
* Fix the logged network name
* Make jobs that use cached state wait for state rebuilds
* Run jobs that need cached state even if the rebuild was skipped
* Fix missing dependencies
And update a TODO
* Revert "Use a generic alphabetical pattern rather than a regex group"
This reverts commit 970afe7b17.
* Revert "Temporary allow missing $NETWORK in disk names"
This reverts commit f1f66500c3.
* Make jobs that use cached state wait for state rebuilds
* Run jobs that need cached state even if the rebuild was skipped
* Fix missing dependencies
And update a TODO
* refactor(ci): look for available disks instead of files changed
This ensure that if the constants.rs file was changed, we search for disks available in the whole repository with the same state.
If there's no disk available a rebuild is triggered depending the missing disk. And if there's a disk available, tests are run with this one.
* fix(ci): lwd syncs needs to wait for zebra disk rebuild
* docs(ci): use better comments on integration tests
* fix(ci): we must authenticate to GCP to find disks
* fix(ci): add needed permissions for google auth
* fix(ci): the output needs to be echoed
* imp(ci): reduce diff with main
* fix(ci): remove redundant dependency
Co-authored-by: teor <teor@riseup.net>
* fix(ci): also add `false` to the JSON object output
* fix(ci): hasty copy/paste
* fix(ci): standardize comments
* fix(ci): run disk rebuilds if no disk was found
* fix(ci): build on any event if a cached disk is not found
* fix(ci): reduce diff with main
* docs(ci): reduce main diff
* fix(ci): sync .patch file with changes on the workflow
* fix(ci): consider network changes in new get-available-disks
* force GHA trigger
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Make jobs that use cached state wait for state rebuilds
* Run jobs that need cached state even if the rebuild was skipped
* Fix missing dependencies
And update a TODO
* refactor(ci): look for available disks instead of files changed
This ensure that if the constants.rs file was changed, we search for disks available in the whole repository with the same state.
If there's no disk available a rebuild is triggered depending the missing disk. And if there's a disk available, tests are run with this one.
* fix(ci): lwd syncs needs to wait for zebra disk rebuild
* docs(ci): use better comments on integration tests
* fix(ci): we must authenticate to GCP to find disks
* fix(ci): add needed permissions for google auth
* fix(ci): the output needs to be echoed
* imp(ci): reduce diff with main
* fix(ci): remove redundant dependency
Co-authored-by: teor <teor@riseup.net>
* fix(ci): also add `false` to the JSON object output
* fix(ci): hasty copy/paste
* fix(ci): standardize comments
* fix(ci): run disk rebuilds if no disk was found
* fix(ci): build on any event if a cached disk is not found
* fix(ci): sync .patch file with changes on the workflow
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Revert "Temporarily stop requiring cached lightwalletd state for the send transaction tests"
This reverts commit f6b29b151e.
* fix(ci): add a lightwalletd cached state to the test
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): lwd state condition
* fix(ci): differentiate tests that need a lwd cached state
* fix(ci): use the right state and save name for each test
* docs(ci): minor comment fixes
* docs(ci): better input description
* fix(ci): end `if` condition correctly
* fix(images): pass the state version to following steps
* fix(ci): $needs_lwd_state condition was inverted
* fix(ci): reduce disk selection code
* docs(ci): better disk search conditional explanation
* fix(ci): end if condition correctly
* fix(ci): evaluate $needs_zebra_state correctly
* fix(ci): use nested condition for readability
* fix(ci): disk search was using the wrong variable
* Temporarily use an earlier lightwalletd version
This checks if commit
e146dbf5c2
contains a mempool refresh deadlock bug.
* Actually rebuild the lightwalletd image
* Delete an unfinished comment
* Remove duplicate test in entrypoint.sh
* Keep a recent change to make tests consistent
* fix(ci): remove not used variable `lwd_state_dir`
* fix(ci): state wast not being added to the image name
* fix(ci): mount a docker volume with lightwalletd dir
If the volume doesn't mount this lwd cached state dir, the content won't be saved to the mounted disk in the VM
* fix(ci): lwd state condition
* docs(ci): explain disk mounting logic
* docs(ci): explain disk mounting decision better
* docs(ci): add a description for confusing input names
Co-authored-by: teor <teor@riseup.net>
* Require a cached state rebuild if the state version changes
* Find cached state disks with the same state version
And prefer `main` to other branches.
* Tweak filters to make them more specific
* Try adding inner quotes
* Try brackets instead
* Try two filters, rather than three
* Use Mainnet as the default network, remove duplicate env var
* Match the exact disk name format in one regular expression
* Log the exact expected disk name, including the network
* Consistently use CACHED_DISK_NAME as the env var name
* Temporary allow missing $NETWORK in disk names
* Print the exact search string
* Debug log the search string
* Use a generic alphabetical pattern rather than a regex group
Google Cloud doesn't seem to support regex groups.
* Add network name to disk match docs
* Fix the logged network name
* imp(ci): remove gcp verbose log
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* Drop sentry dependencies when enable-sentry feature is disabled
* Make lightwalletd gRPC tests depend on a new lightwalletd-grpc-tests feature
* fix(ci): remove enable-sentry feature from tests
* Add lightwalletd-grpc-tests feature for functionality or efficiency
And document where it is just used to stop re-compilations.
* Remove redundant `cmake` and `protobuf-compiler` dependencies
* Document Zebra's optional production and test feature flags
* Minimise dependencies in zcash-params/Dockerfile
* Minimise dependencies in docker/Dockerfile
* Add a workflow TODO
* Catch more errors in entrypoint.sh
Also makes entrypoint.sh compatible with more distributions
* Remove unnecessary quoting in entrypoint.sh
* Use exactly the same arguments to call CI tests
* Remove a redundant CI build
* Rename Cargo.lock check job
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* Drop sentry dependencies when enable-sentry feature is disabled
* Make lightwalletd gRPC tests depend on a new lightwalletd-grpc-tests feature
* fix(ci): remove enable-sentry feature from tests
* Add lightwalletd-grpc-tests feature for functionality or efficiency
And document where it is just used to stop re-compilations.
* Remove redundant `cmake` and `protobuf-compiler` dependencies
* Document Zebra's optional production and test feature flags
* Minimise dependencies in zcash-params/Dockerfile
* Minimise dependencies in docker/Dockerfile
* Add a workflow TODO
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* refactor(build): use better stage naming and document it
* refactort(build): use multiple cache sources
* docs(build): add a comment for cache
* fix(build): remove gcr.io as it does not supports OCI images
* feat(ci): add lightwalletd_*_sync tests to CI
* feat(ci): add lightwalletd RPC call test
* feat(ci): add send transactions test with lwd to CI
* fix(ci): create a variable to run transactions test
* refactor(ci): use docker in docker
This is a workaround for an issue related to disk partitioning, caused by a GCP service called Konlet, while mounting the cached disks to the VM and then to the container
* fix(build): persist docker login credentials
* fix(ci): get sync height from docker logs instead of gcp
* try: use gha cache for faster building
* fix(ci): mount disk in container to make it available in vm
* fix(build): do not invalidate cache between images
* try(docker): invalidate cache as less as possible
* fix(ci): GHA terminal is not a TTY
* fix(build): do not ignore entrypoint.sh
* fix
* fix(ci): mount using root priveleges
* fix(ci): use existing disk as cached state
* fix(ci): wait for disks to get mounted
* force rebuild
* fix failed force
* fix failed commit
* WIP
* fix(ci): some tests does not use a cached state
* wip
* refactor(ci): disk names and job segregation
* fix(ci): do not name boot and attached disk the same
* fix(ci): attach a disk to full sync, to snapshot the state
* fix(ci): use correct disk implementations
* fix(ci): use different disk name to allow test concurrency
* feat(ci): add lightwalledt send transaction test
* cleanup(ci): remove extra tests
* fix(ci): allow disk concurrency with tests
* fix(ci): add considerations for different tests
* fix(reusable): last fixes
* feat(ci): use reusable workflow for tests
* fix(rw): remove nested worflow
* fix(rw): minor fixes
* force rebuild
* fix(rw): do not use an input as job name
* fix(rw): remove variable id
* fix(ci): remove explicit conditions and id
* fix(ci): docker does not need the variable sign ($) to work
* fix(ci): mount typo
* fix(ci): if a sync fails, always delete the instance
This also reduces the amount of jobs needed.
* refactor(ci): make all test depend on the same build
* fix(ci): some tests require multiple variables
* fix(docker): variable substitution
* fix(ci): allow to run multiple commits from a PR at once
* fix(docker): lower the NETWORK env var for test names
* reduce uneeded diff
* imp(keys): use better naming for builds_disks
* imp(ci): use input defaults
* imp(ci): remove test_name in favor of test_id
* fix(ci): better key naming
* fix(ci): long disk names breaks GCP naming convention
* feat(ci): validate local state version with cached state
* fix(ci): add condition to run tests
* fix: typo
* fix: app_name should not be required
* fix: zebra_state_path shouldn't be required
* fix: reduce diff
* fix(ci): checkout to grep local state version
* Update .github/workflows/test.yml
Co-authored-by: teor <teor@riseup.net>
* revert: merge all tests into a single workflow
* Remove unused STATE_VERSION env var
* fix: minor fixes
* fix(ci): make test.patch the same as test
* fix(ci): negate the input value
* imp(ci): better cached state conditional handling
* imp(ci): exit code is captured by `docker run`
* fix(deploy): mount disks with better write performance
* fix(ci): change sync id to a broader id name
* fix(ci): use correct input validation
* fix(ci): do not make test with cached state dependant on other
* imp(ci): organiza keys better
* fix(ci): use appropiate naming
* fix(ci): create docker volume before mounting
* fix(lint): do not fail on all new changes
* imp(ci): do not report in pr review
* fix(ci): partition clean disks
* fix: typo
* fix: test called the wrong way
* fix(build): stop using gha cache
* ref(ci): validate run condition before calling reusable workflow
* fix(ci): use a better filesystem dir and fix other values
* fix: linting errors
* fix(ci): typo
* Revert "fix(build): stop using gha cache"
This reverts commit a8fbc5f416.
Cache expiration is a lesser evil than not using caching at all and then failing with a 401
* imp(ci): do not set a default for needs_zebra_state
* Update .github/workflows/test.yml
Co-authored-by: teor <teor@riseup.net>
* fix(deps): remove dependencies
* force build
* Update .github/workflows/test.yml
Co-authored-by: teor <teor@riseup.net>
* fix(docker): add RUST_LOG as an ARG and ENV
* fix(test): add `#[ignore]` to send transactions test
This test needs state then it should be marked as #[ignore]
* fix(ci): differentiate between root cache path and its dir
* Remove extra `state` directory
That was a workaround for an issue that has been fixed.
* imp(docs): use better test descriptions
Co-authored-by: teor <teor@riseup.net>
* fix: reduce unwanted diff with main
* fix(ci): make lwd conditions consistent
* Remove another extra `state` directory
Was also part of a workaround for an issue that has been fixed.
* fix(ci): use better conditionals to run test jobs
Co-authored-by: teor <teor@riseup.net>
* Tweak to support different lightwalletd versions
Some versions print `Waiting for block`, and some versions print
`Ingestor waiting for block`.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* refactor(lint): check specific files for each job
* refactor(lint): use an approach which requires less code
* fix(lint): validate against true string not boolean
* imp(build): reduce docker cache invalidation
Use scoped caching and more file ignores to reduce cache invalidation
* fix(build): add entrypoint.sh as a required file
* fix(build): do not logout if the build takes too long
* Add 'doc comment' about .dockerignore
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* refactor(ci): use docker in docker
This is a workaround for an issue related to disk partitioning, caused by a GCP service called Konlet, while mounting the cached disks to the VM and then to the container
* fix(build): persist docker login credentials
* fix(ci): get sync height from docker logs instead of gcp
* try: use gha cache for faster building
* fix(ci): mount disk in container to make it available in vm
* fix(build): do not invalidate cache between images
* try(docker): invalidate cache as less as possible
* fix(ci): GHA terminal is not a TTY
* fix(build): do not ignore entrypoint.sh
* fix
* fix(ci): mount using root priveleges
* fix(ci): use existing disk as cached state
* fix(ci): wait for disks to get mounted
* force rebuild
* fix failed force
* fix(ci): some tests does not use a cached state
* fix(ci): do not name boot and attached disk the same
* fix(ci): attach a disk to full sync, to snapshot the state
* fix(ci): use appropiate grep text depending on the test
* reduce diff
* fix(ci): use correct GCP disk source attribute
* imp(ci): reduce diff
* fix(ci): revert wrong deletion
* fix: revert uneeded changes
* fix: reduce main diff
* fix
* fix(ci): reduce diff
* fix(ci): garbage collect instances no matter the status
As we're not going to reuse test instances, the safest method to apply is to always delete this instances if they fail, get skipped or succeed running a workflow
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* docs(ci): imrpove comment
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* refactor(ci): test building in a separate workflow
* force a change
* force a change
* fix(ci): send the correct variables to the reusable build
* fix(ci): variables are not allowed
* fix(ci): conditions are not allowed as input
* fix(ci): use expected value
* refactor(build): simplify the use of other dockerfiles
* fix(cd): depend on docker build yml
* fix(cd): use main branch as image name
* imp(actions): remove uneeded variable repetition
* imp(build): remove unused variables
* imp(actions): rename the image building workflow
Not all images are for zebra execution as we also have one for zcash-params
* fix(ci): add dependable workflow in paths filters
* docs(ci): remove TODO as this won't be needed at least an issue arises
* docs(ci): CARGO_INCREMENTAL can decrease build time when running from a cache
* fix: revert forced changes
* fix(build): remove unused build inputs in zcash-params
* imp(cd): as this is the production image, use the executable name
* imp(ci): reduce log level to improve speed
Co-authored-by: teor <teor@riseup.net>
* imp(ci): use the correct name for the workflow
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
* fix(ci): do not delete instances from `main` branch on merge
* fix(ci): do not delete instances on merge
This was creating an unintended behavior, and so far instances are being cleaned up in its corresponding workflow.
* feat(ci): run cached state rebuilds in main branch
* fix(ci): allow the PR/branch name in the disk name
Move the hight information to the disk description, to reduce the name length
* fix(ci): add missing SHA
* fix(ci): regenerate chekpoint cached state on main
This will automatically regenerate the disk when a merge is completed on main
* tmp(ci): do not duplicate sync test at/after merge
This temporarily ensure the test just runs in the main branch, ensuring we can track it easier
* fix(ci): correctly use lowered network caps
In the Test workflow we were using a different approach than the one being used in the Full sync test.
Also, in the Full sync test the variable was LOWER_NET_NAME, but NETWORK was being used in the disk name, with caps.
* imp(ci): get state version from local constants.rs
* imp(ci): use the same get name approach
* fix(ci): use the correct name for state version variable
* imp(ci)!: use different disk names for cached states
Disk states synced to canopy and synced to the chain tip should have different names to reference correctly on actual and coming tests the needed disk.
* imp(ci): test-stateful-sync no longer depends on regenerate-stateful-disks
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* fix(ci): use a better name for network string conversion
* Revert "Apply suggestions from code review"
This reverts commit cbbfaf4e9c.
* fix: do not get log information if sync was skipped
* fix(ci): do not lower the variable name
* fix(ci): use the same lowering case for network everywhere
* test: more .dockerignore conditions
* fix: use the right approach to lower caps
* remove extra .dockerignore
* trigger a change for stateful disk regeneration
* imp(ci): use `checkpoint` as the disk reference
* revert wrong delete
* fix(ci): add INSTANCE_ID and correct logging message
* imp(ci): add `v` prefix to state version number
* fix(ci): remove typo from logging message to get the height
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* style(ci): comply with https://json.schemastore.org/github-workflow.json
Some substituions were harder to make as files were not standardized
* fix(mergify): use correct name for macos
* style(actions): revert to single quotes
* style: lint dependabot and mergify conf files
* style: remove conditions with missing context
* imp(lint): automate GH Actions linting
* fix(lint): some actions need to be triggered by PR event
* fix(lint): consider all workflow YAMLs
* Use the same paths in the patch file
* revert: keep condition as is
* add TODO
* fix: add missing checkpoint_sync input
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Change OutputLocation to contain a TransactionLocation
* Change OutputLocation reads from the database
* Update some doc comments
* Update some TODOs
* Change deleting spent UTXOs and updating spent balances
* Change adding new UTXOs and adding their values to balances
* Disable dead code warnings
* Update snapshot test code
* Update round-trip tests for OutputLocations
* Update snapshot test data
* Increment the database format version
* Remove a redundant try_into()
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Refactor redundant code
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* ci: attempt at fixing 'Regenerate stateful disks'
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* fix(ci): validate exit code
* fix(ci): use a single step
* fix: run from gcp command
* fix: remove variable substitution
* fix: escaping characters
* fix: handle bash issues with variable expansion
* fix: seggregate execution into multiple steps
* fix: the command is a string
* fix: move docker ignore to use caching
* revert panic
* fix: add panic and fix exit code command
* revert: remove panic
* fix: apply the exit code to all gcp tests
* imp(ci): use single line exit
* Clean up the CODEOWNERS file
We moved the dockerignore file into the docker directory,
so it doesn't need a separate entry any more.
Co-authored-by: teor <teor@riseup.net>
* Create disk image after a successful full sync test
* Extract full sync height and name zebrad cached state with it
* Read 500 lines to extract sync height
* Restrict log query to just the container output and fix regex syntax for ubuntu
* Explicitly search logs in descending time
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
Only run the state rebuild job if the database format version has (likely) changed.
If we have accidentally changed the format, but not changed the version,
we want to run with the old cached state, so this job fails.
If we change the state path without changing the version,
this job will take a few hours, because it will do a full rebuild.
When a PR is created and an image is built in a branch, the cache is also pushed to a `buildcache` tag.
As all PRs are using the same tag, sometimes the cache for a specific branch gets invalidated and makes it take longer on further pushes.
This fix might make the first commit take longer, but further ones will be faster if no changes in the code are applied.
* Update test.patch.yml with lightwalletd job
* Remove a workflow condition that will always be false
In general, patch workflows need the
opposite conditions to the original workflow.
But in this case, we know the result of the
condition will always be true, so we can just delete it.
Co-authored-by: teor <teor@riseup.net>
* fix(actions): use a specific shortening length for SHAs
The rlespinasse/github-slug-action now works without checking out the code, reduce time and improving security with following actions.
This requires to specify the GITHUB_SHA_SHORT variable length, as git uses 8 by default, but docker uses 7 by default.
* fix(actions): target correct rlespinasse/github-slug-action version
* fix(actions): just use major version
* fix(actions): github-slug-action is not being correctly referenced
* refactor(ci): use improved OIDC authentication
* fix(ci): standardize OIDC on all required jobs
* fix: wrong indentation
* fix(ci): remove non existing depency in clean job