* refactor(ci): use docker in docker
This is a workaround for an issue related to disk partitioning, caused by a GCP service called Konlet, while mounting the cached disks to the VM and then to the container
* fix(build): persist docker login credentials
* fix(ci): get sync height from docker logs instead of gcp
* try: use gha cache for faster building
* fix(ci): mount disk in container to make it available in vm
* fix(build): do not invalidate cache between images
* try(docker): invalidate cache as less as possible
* fix(ci): GHA terminal is not a TTY
* fix(build): do not ignore entrypoint.sh
* fix
* fix(ci): mount using root priveleges
* fix(ci): use existing disk as cached state
* fix(ci): wait for disks to get mounted
* force rebuild
* fix failed force
* fix(ci): some tests does not use a cached state
* fix(ci): do not name boot and attached disk the same
* fix(ci): attach a disk to full sync, to snapshot the state
* fix(ci): use appropiate grep text depending on the test
* reduce diff
* fix(ci): use correct GCP disk source attribute
* imp(ci): reduce diff
* fix(ci): revert wrong deletion
* fix: revert uneeded changes
* fix: reduce main diff
* fix
* fix(ci): reduce diff
* fix(ci): garbage collect instances no matter the status
As we're not going to reuse test instances, the safest method to apply is to always delete this instances if they fail, get skipped or succeed running a workflow
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* docs(ci): imrpove comment
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* refactor(ci): test building in a separate workflow
* force a change
* force a change
* fix(ci): send the correct variables to the reusable build
* fix(ci): variables are not allowed
* fix(ci): conditions are not allowed as input
* fix(ci): use expected value
* refactor(build): simplify the use of other dockerfiles
* fix(cd): depend on docker build yml
* fix(cd): use main branch as image name
* imp(actions): remove uneeded variable repetition
* imp(build): remove unused variables
* imp(actions): rename the image building workflow
Not all images are for zebra execution as we also have one for zcash-params
* fix(ci): add dependable workflow in paths filters
* docs(ci): remove TODO as this won't be needed at least an issue arises
* docs(ci): CARGO_INCREMENTAL can decrease build time when running from a cache
* fix: revert forced changes
* fix(build): remove unused build inputs in zcash-params
* imp(cd): as this is the production image, use the executable name
* imp(ci): reduce log level to improve speed
Co-authored-by: teor <teor@riseup.net>
* imp(ci): use the correct name for the workflow
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
* feat(ci): run cached state rebuilds in main branch
* fix(ci): allow the PR/branch name in the disk name
Move the hight information to the disk description, to reduce the name length
* fix(ci): add missing SHA
* fix(ci): regenerate chekpoint cached state on main
This will automatically regenerate the disk when a merge is completed on main
* tmp(ci): do not duplicate sync test at/after merge
This temporarily ensure the test just runs in the main branch, ensuring we can track it easier
* fix(ci): correctly use lowered network caps
In the Test workflow we were using a different approach than the one being used in the Full sync test.
Also, in the Full sync test the variable was LOWER_NET_NAME, but NETWORK was being used in the disk name, with caps.
* imp(ci): get state version from local constants.rs
* imp(ci): use the same get name approach
* fix(ci): use the correct name for state version variable
* imp(ci)!: use different disk names for cached states
Disk states synced to canopy and synced to the chain tip should have different names to reference correctly on actual and coming tests the needed disk.
* imp(ci): test-stateful-sync no longer depends on regenerate-stateful-disks
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* fix(ci): use a better name for network string conversion
* Revert "Apply suggestions from code review"
This reverts commit cbbfaf4e9c.
* fix: do not get log information if sync was skipped
* fix(ci): do not lower the variable name
* fix(ci): use the same lowering case for network everywhere
* test: more .dockerignore conditions
* fix: use the right approach to lower caps
* remove extra .dockerignore
* trigger a change for stateful disk regeneration
* imp(ci): use `checkpoint` as the disk reference
* revert wrong delete
* fix(ci): add INSTANCE_ID and correct logging message
* imp(ci): add `v` prefix to state version number
* fix(ci): remove typo from logging message to get the height
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* style(ci): comply with https://json.schemastore.org/github-workflow.json
Some substituions were harder to make as files were not standardized
* fix(mergify): use correct name for macos
* style(actions): revert to single quotes
* style: lint dependabot and mergify conf files
* style: remove conditions with missing context
* imp(lint): automate GH Actions linting
* fix(lint): some actions need to be triggered by PR event
* fix(lint): consider all workflow YAMLs
* Use the same paths in the patch file
* revert: keep condition as is
* add TODO
* fix: add missing checkpoint_sync input
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Change OutputLocation to contain a TransactionLocation
* Change OutputLocation reads from the database
* Update some doc comments
* Update some TODOs
* Change deleting spent UTXOs and updating spent balances
* Change adding new UTXOs and adding their values to balances
* Disable dead code warnings
* Update snapshot test code
* Update round-trip tests for OutputLocations
* Update snapshot test data
* Increment the database format version
* Remove a redundant try_into()
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Refactor redundant code
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* ci: attempt at fixing 'Regenerate stateful disks'
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* fix(ci): validate exit code
* fix(ci): use a single step
* fix: run from gcp command
* fix: remove variable substitution
* fix: escaping characters
* fix: handle bash issues with variable expansion
* fix: seggregate execution into multiple steps
* fix: the command is a string
* fix: move docker ignore to use caching
* revert panic
* fix: add panic and fix exit code command
* revert: remove panic
* fix: apply the exit code to all gcp tests
* imp(ci): use single line exit
* Clean up the CODEOWNERS file
We moved the dockerignore file into the docker directory,
so it doesn't need a separate entry any more.
Co-authored-by: teor <teor@riseup.net>
Only run the state rebuild job if the database format version has (likely) changed.
If we have accidentally changed the format, but not changed the version,
we want to run with the old cached state, so this job fails.
If we change the state path without changing the version,
this job will take a few hours, because it will do a full rebuild.
When a PR is created and an image is built in a branch, the cache is also pushed to a `buildcache` tag.
As all PRs are using the same tag, sometimes the cache for a specific branch gets invalidated and makes it take longer on further pushes.
This fix might make the first commit take longer, but further ones will be faster if no changes in the code are applied.
* fix(actions): use a specific shortening length for SHAs
The rlespinasse/github-slug-action now works without checking out the code, reduce time and improving security with following actions.
This requires to specify the GITHUB_SHA_SHORT variable length, as git uses 8 by default, but docker uses 7 by default.
* fix(actions): target correct rlespinasse/github-slug-action version
* fix(actions): just use major version
* fix(actions): github-slug-action is not being correctly referenced
* refactor(ci): use improved OIDC authentication
* fix(ci): standardize OIDC on all required jobs
* fix: wrong indentation
* fix(ci): remove non existing depency in clean job
GitHub's GHA cache gets invalidated at 10Gb, which is very easy to hit when we're building multiple times a day with several commits.
Instead use the registry, which won't get invalidated until a change is identified in the build process.
* refactor(test): reuse same GCP instance on a single PR
This also ensures the deployments are faster, and we only delete the instance when merging or closing the PR, instead of doing it on each push to the PR
* fix(deploy): add zone to updates
* fix: typo
* fix(ci): improve conditions for updates
* fix(deploy): delete old deployments instead of reusing it
* fix(deploy): keep delete command after run
* fix(deploy): always create an instance
* fix(deploy): delete disks on every delete command.
* imp(ci): use better id name
* Update .github/workflows/test.yml
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* imp: handle errors correctly on deletion
* fix: do not hide valid errors
* fix: edge case where the container is not ready yet
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* doc(db): fix some comments
* refactor(db): split disk serialization types into their own module
* refactor(db): split the disk format into modules
* doc(db/test): explain the RON serialization format
* fix: check TEST_FAKE_ACTIVATION_HEIGHTS at runtime
* fix(tests): add TEST_FAKE_ACTIVATION_HEIGHTS variable
This variable ensures the test is activated in the `test-fake-activation-heights` step
* fix(docker): do not run specific tests by default in entrypoint.sh
* fix(test): remove extra TEST_FULL_SYNC argument
* imp(timeout): wait for an average build time
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix: add missing job key
* fix(arm64): bump timeout to build without cache
* fix(deployment): apply changes made in 5004c4d3a1
* fix: remove uneeded condition
* refactor(tests): make vm names refer the test name
This also adds a build step for full sync, as there won't be a reference image when using workflow_dispatch
* fix(deployment): testing depends on the built image
* refactor(test): decouple full sync from other tests
As the full sync requires to be run just once and isolated, we're running this test in a separate workflow, after a PR has been approved.
* fix: revert to previous conditions in job regenerate-stateful-disks
* fix(condition): get disk sha if regeneration is not executed
* fix: typo
* Update .github/workflows/test-full-sync.yml
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* fix(build): bump build time for arm64
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* refactor(build): use OCI Image Format Specification for labels
This should also fix when an image gets built multiple times using the cache, as each image differs in labels
* refactor(tags): use PR context sha and ref
Remove the needed of PR Head SHA and Ref, as those can cause conflict depending on how the branch name has been established
* add(actions): lightwalletd continous integrations
* refactor(actions): build lightwalletd and reuse it in zebra
- Download lightwalletd source code
- Create a new Dockerfile for lightwalletd
- Use lightwalletd binary in Zebra's image
- Create a specific step to build/update lightwalletd
- Add lightwalletd integration test to the test suite
- Remove lightwalletd.yml, as it was harder to control
* refactor(docker): organize Dockerfiles and remove unused
Fixes: #3344
* feat(build): add arrm64 support
* fix(build): do not install google-compute-engine in arm64
This package is not available for this platform
* fix(build): do not build arm64 for tests
* fix(condition): indent for better visibility
* fix(condition): wrong use of operators
* ci(test): re-run tests when snapshot data changes
* fix(ci): rebuild state when disk format changes
* fix(ci): rebuild rust docs when code or dependencies change
* doc(ci): explain why we run jobs when files change
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* add(tests): full sync test
* fix(test): add build
* fix(deploy): escape double dashes '--' correctly
* fix(test): remove unexpected --no-capture arg
error: Found argument '--nocapture' which wasn't expected, or isn't valid in this context
* refactor(docker): use default executable as entrypoint
* refactor(startup): add a custom entrypoint
* fix(test): add missing TEST_FULL_SYNC variable
* test(timeout): use the biggest machine
* fix
* fix(deploy): use latest successful image
* typo
* refactor(docker): generate config file at startup
* revert(build): changes were made to docker
* fix(docker): send variables correctly to the entrypoint
* test different conf file approach
* fix(env): add RUN_TEST env variable
* ref: use previous approach
* fix(color): use environment variable
* fix(resources): use our normal machine size
* fix(ci): double CPU and RAM for full sync test
* fix(test): check for zebrad test output in the correct order
The mempool is only activated once, so we must check for that log first.
After mempool activation, the stop regex is logged at least once.
(It might be logged before as well, but we can't rely on that.)
When checking that the mempool didn't activate,
wait for the `zebrad` command to exit,
then check the entire log.
* fix(ci): run full sync test with full compiler optimisations
* fix(tests): reintroduce tests and run full sync on approval
* fix(tests): reduce the changelog
Co-authored-by: teor <teor@riseup.net>
Gustavo Valverde
dependabot[bot]
teor
Janito Vaqueiro Ferreira Filho
Conrado Gouvea