* refactor(ci): use docker in docker
This is a workaround for an issue related to disk partitioning, caused by a GCP service called Konlet, while mounting the cached disks to the VM and then to the container
* fix(build): persist docker login credentials
* fix(ci): get sync height from docker logs instead of gcp
* try: use gha cache for faster building
* fix(ci): mount disk in container to make it available in vm
* fix(build): do not invalidate cache between images
* try(docker): invalidate cache as less as possible
* fix(ci): GHA terminal is not a TTY
* fix(build): do not ignore entrypoint.sh
* fix
* fix(ci): mount using root priveleges
* fix(ci): use existing disk as cached state
* fix(ci): wait for disks to get mounted
* force rebuild
* fix failed force
* fix(ci): some tests does not use a cached state
* fix(ci): do not name boot and attached disk the same
* fix(ci): attach a disk to full sync, to snapshot the state
* fix(ci): use appropiate grep text depending on the test
* reduce diff
* fix(ci): use correct GCP disk source attribute
* imp(ci): reduce diff
* fix(ci): revert wrong deletion
* fix: revert uneeded changes
* fix: reduce main diff
* fix
* fix(ci): reduce diff
* fix(ci): garbage collect instances no matter the status
As we're not going to reuse test instances, the safest method to apply is to always delete this instances if they fail, get skipped or succeed running a workflow
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* docs(ci): imrpove comment
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* refactor(ci): test building in a separate workflow
* force a change
* force a change
* fix(ci): send the correct variables to the reusable build
* fix(ci): variables are not allowed
* fix(ci): conditions are not allowed as input
* fix(ci): use expected value
* refactor(build): simplify the use of other dockerfiles
* fix(cd): depend on docker build yml
* fix(cd): use main branch as image name
* imp(actions): remove uneeded variable repetition
* imp(build): remove unused variables
* imp(actions): rename the image building workflow
Not all images are for zebra execution as we also have one for zcash-params
* fix(ci): add dependable workflow in paths filters
* docs(ci): remove TODO as this won't be needed at least an issue arises
* docs(ci): CARGO_INCREMENTAL can decrease build time when running from a cache
* fix: revert forced changes
* fix(build): remove unused build inputs in zcash-params
* imp(cd): as this is the production image, use the executable name
* imp(ci): reduce log level to improve speed
Co-authored-by: teor <teor@riseup.net>
* imp(ci): use the correct name for the workflow
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
* fix(ci): do not delete instances from `main` branch on merge
* fix(ci): do not delete instances on merge
This was creating an unintended behavior, and so far instances are being cleaned up in its corresponding workflow.
* feat(ci): run cached state rebuilds in main branch
* fix(ci): allow the PR/branch name in the disk name
Move the hight information to the disk description, to reduce the name length
* fix(ci): add missing SHA
* fix(ci): regenerate chekpoint cached state on main
This will automatically regenerate the disk when a merge is completed on main
* tmp(ci): do not duplicate sync test at/after merge
This temporarily ensure the test just runs in the main branch, ensuring we can track it easier
* fix(ci): correctly use lowered network caps
In the Test workflow we were using a different approach than the one being used in the Full sync test.
Also, in the Full sync test the variable was LOWER_NET_NAME, but NETWORK was being used in the disk name, with caps.
* imp(ci): get state version from local constants.rs
* imp(ci): use the same get name approach
* fix(ci): use the correct name for state version variable
* imp(ci)!: use different disk names for cached states
Disk states synced to canopy and synced to the chain tip should have different names to reference correctly on actual and coming tests the needed disk.
* imp(ci): test-stateful-sync no longer depends on regenerate-stateful-disks
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* fix(ci): use a better name for network string conversion
* Revert "Apply suggestions from code review"
This reverts commit cbbfaf4e9c.
* fix: do not get log information if sync was skipped
* fix(ci): do not lower the variable name
* fix(ci): use the same lowering case for network everywhere
* test: more .dockerignore conditions
* fix: use the right approach to lower caps
* remove extra .dockerignore
* trigger a change for stateful disk regeneration
* imp(ci): use `checkpoint` as the disk reference
* revert wrong delete
* fix(ci): add INSTANCE_ID and correct logging message
* imp(ci): add `v` prefix to state version number
* fix(ci): remove typo from logging message to get the height
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* style(ci): comply with https://json.schemastore.org/github-workflow.json
Some substituions were harder to make as files were not standardized
* fix(mergify): use correct name for macos
* style(actions): revert to single quotes
* style: lint dependabot and mergify conf files
* style: remove conditions with missing context
* imp(lint): automate GH Actions linting
* fix(lint): some actions need to be triggered by PR event
* fix(lint): consider all workflow YAMLs
* Use the same paths in the patch file
* revert: keep condition as is
* add TODO
* fix: add missing checkpoint_sync input
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Change OutputLocation to contain a TransactionLocation
* Change OutputLocation reads from the database
* Update some doc comments
* Update some TODOs
* Change deleting spent UTXOs and updating spent balances
* Change adding new UTXOs and adding their values to balances
* Disable dead code warnings
* Update snapshot test code
* Update round-trip tests for OutputLocations
* Update snapshot test data
* Increment the database format version
* Remove a redundant try_into()
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Refactor redundant code
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* ci: attempt at fixing 'Regenerate stateful disks'
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* fix(ci): validate exit code
* fix(ci): use a single step
* fix: run from gcp command
* fix: remove variable substitution
* fix: escaping characters
* fix: handle bash issues with variable expansion
* fix: seggregate execution into multiple steps
* fix: the command is a string
* fix: move docker ignore to use caching
* revert panic
* fix: add panic and fix exit code command
* revert: remove panic
* fix: apply the exit code to all gcp tests
* imp(ci): use single line exit
* Clean up the CODEOWNERS file
We moved the dockerignore file into the docker directory,
so it doesn't need a separate entry any more.
Co-authored-by: teor <teor@riseup.net>
* Create disk image after a successful full sync test
* Extract full sync height and name zebrad cached state with it
* Read 500 lines to extract sync height
* Restrict log query to just the container output and fix regex syntax for ubuntu
* Explicitly search logs in descending time
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Update changelog for v1.0.0-beta.7
* Increment all crate versions
* Remove redundant release test that is now covered by CI
* Remove completed NU5 README check task from the release template
* Add Merge Freeze tool to the release checklist
* Simplify release checklist by removing unused steps
Only run the state rebuild job if the database format version has (likely) changed.
If we have accidentally changed the format, but not changed the version,
we want to run with the old cached state, so this job fails.
If we change the state path without changing the version,
this job will take a few hours, because it will do a full rebuild.
When a PR is created and an image is built in a branch, the cache is also pushed to a `buildcache` tag.
As all PRs are using the same tag, sometimes the cache for a specific branch gets invalidated and makes it take longer on further pushes.
This fix might make the first commit take longer, but further ones will be faster if no changes in the code are applied.
* Update test.patch.yml with lightwalletd job
* Remove a workflow condition that will always be false
In general, patch workflows need the
opposite conditions to the original workflow.
But in this case, we know the result of the
condition will always be true, so we can just delete it.
Co-authored-by: teor <teor@riseup.net>
* fix(actions): use a specific shortening length for SHAs
The rlespinasse/github-slug-action now works without checking out the code, reduce time and improving security with following actions.
This requires to specify the GITHUB_SHA_SHORT variable length, as git uses 8 by default, but docker uses 7 by default.
* fix(actions): target correct rlespinasse/github-slug-action version
* fix(actions): just use major version
* fix(actions): github-slug-action is not being correctly referenced
* refactor(ci): use improved OIDC authentication
* fix(ci): standardize OIDC on all required jobs
* fix: wrong indentation
* fix(ci): remove non existing depency in clean job
GitHub's GHA cache gets invalidated at 10Gb, which is very easy to hit when we're building multiple times a day with several commits.
Instead use the registry, which won't get invalidated until a change is identified in the build process.
* refactor(test): reuse same GCP instance on a single PR
This also ensures the deployments are faster, and we only delete the instance when merging or closing the PR, instead of doing it on each push to the PR
* fix(deploy): add zone to updates
* fix: typo
* fix(ci): improve conditions for updates
* fix(deploy): delete old deployments instead of reusing it
* fix(deploy): keep delete command after run
* fix(deploy): always create an instance
* fix(deploy): delete disks on every delete command.
* imp(ci): use better id name
* Update .github/workflows/test.yml
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* imp: handle errors correctly on deletion
* fix: do not hide valid errors
* fix: edge case where the container is not ready yet
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* doc(db): fix some comments
* refactor(db): split disk serialization types into their own module
* refactor(db): split the disk format into modules
* doc(db/test): explain the RON serialization format
* fix(ci): only run the full sync test on mergify queue PRs
* fix inconsistent indenting and syntax
* Update .github/workflows/test-full-sync.yml
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix: check TEST_FAKE_ACTIVATION_HEIGHTS at runtime
* fix(tests): add TEST_FAKE_ACTIVATION_HEIGHTS variable
This variable ensures the test is activated in the `test-fake-activation-heights` step
* fix(docker): do not run specific tests by default in entrypoint.sh
* fix(test): remove extra TEST_FULL_SYNC argument
* imp(timeout): wait for an average build time
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix: add missing job key
* fix(arm64): bump timeout to build without cache
* fix(deployment): apply changes made in 5004c4d3a1
* fix: remove uneeded condition
* refactor(tests): make vm names refer the test name
This also adds a build step for full sync, as there won't be a reference image when using workflow_dispatch
* fix(deployment): testing depends on the built image
* refactor(test): decouple full sync from other tests
As the full sync requires to be run just once and isolated, we're running this test in a separate workflow, after a PR has been approved.
* fix: revert to previous conditions in job regenerate-stateful-disks
* fix(condition): get disk sha if regeneration is not executed
* fix: typo
* Update .github/workflows/test-full-sync.yml
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* fix(build): bump build time for arm64
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* refactor(build): use OCI Image Format Specification for labels
This should also fix when an image gets built multiple times using the cache, as each image differs in labels
* refactor(tags): use PR context sha and ref
Remove the needed of PR Head SHA and Ref, as those can cause conflict depending on how the branch name has been established
* fix(ci): remove an unused trigger path
* doc(ci): explain lightwalletd trigger paths
* fix(test): check for adityapk00/lightwalletd behaviour in test harness
* fix(ci): work around buildx command error
* fix(ci): revert the workaround
* add(actions): lightwalletd continous integrations
* refactor(actions): build lightwalletd and reuse it in zebra
- Download lightwalletd source code
- Create a new Dockerfile for lightwalletd
- Use lightwalletd binary in Zebra's image
- Create a specific step to build/update lightwalletd
- Add lightwalletd integration test to the test suite
- Remove lightwalletd.yml, as it was harder to control
* refactor(docker): organize Dockerfiles and remove unused
Fixes: #3344
* feat(build): add arrm64 support
* fix(build): do not install google-compute-engine in arm64
This package is not available for this platform
* fix(build): do not build arm64 for tests
* fix(condition): indent for better visibility
* fix(condition): wrong use of operators
* ci(test): re-run tests when snapshot data changes
* fix(ci): rebuild state when disk format changes
* fix(ci): rebuild rust docs when code or dependencies change
* doc(ci): explain why we run jobs when files change
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* add(tests): full sync test
* fix(test): add build
* fix(deploy): escape double dashes '--' correctly
* fix(test): remove unexpected --no-capture arg
error: Found argument '--nocapture' which wasn't expected, or isn't valid in this context
* refactor(docker): use default executable as entrypoint
* refactor(startup): add a custom entrypoint
* fix(test): add missing TEST_FULL_SYNC variable
* test(timeout): use the biggest machine
* fix
* fix(deploy): use latest successful image
* typo
* refactor(docker): generate config file at startup
* revert(build): changes were made to docker
* fix(docker): send variables correctly to the entrypoint
* test different conf file approach
* fix(env): add RUN_TEST env variable
* ref: use previous approach
* fix(color): use environment variable
* fix(resources): use our normal machine size
* fix(ci): double CPU and RAM for full sync test
* fix(test): check for zebrad test output in the correct order
The mempool is only activated once, so we must check for that log first.
After mempool activation, the stop regex is logged at least once.
(It might be logged before as well, but we can't rely on that.)
When checking that the mempool didn't activate,
wait for the `zebrad` command to exit,
then check the entire log.
* fix(ci): run full sync test with full compiler optimisations
* fix(tests): reintroduce tests and run full sync on approval
* fix(tests): reduce the changelog
Co-authored-by: teor <teor@riseup.net>
* add(actions): lightwalletd continous integrations
* refactor(actions): build lightwalletd and reuse it in zebra
- Download lightwalletd source code
- Create a new Dockerfile for lightwalletd
- Use lightwalletd binary in Zebra's image
- Create a specific step to build/update lightwalletd
- Add lightwalletd integration test to the test suite
- Remove lightwalletd.yml, as it was harder to control
* fix(build): remove extra port being exposed
* fix(lightwalletd): test should be after `--` in cargo test
* revert(lint): do not lint external code as it can be confusing
* fix(test): lightwalletd_integration test is not ignored
* docs(docker): clarify the addition of unused args
* refactor(docker): organize Dockerfiles and remove unused
Fixes: #3344
* fix(actions): activate workflows on correct path changes
* test
* revert previous commit
* feat(build): add arm64 support with cross-compilation (#3659)
* feat(build): add arrm64 support
* fix(build): do not install google-compute-engine in arm64
This package is not available for this platform
* fix(build): do not build arm64 for tests
* fix(changes): reduce changelog
* Revert "feat(build): add arm64 support with cross-compilation (#3659)"
This reverts commit 291e00c405.
* feat(codeowners): add code owners in repository
* fix(path): recently split out crate
Co-authored-by: teor <teor@riseup.net>
* fix(teams): use reviewers instead of owners name
* fix(teams): wrong team name
* docs: use correct default explanation
* fix(path): add extra paths to devops team
Co-authored-by: teor <teor@riseup.net>
* refactor(state): move disk_db reads to a new zebra_db module
* refactor(state): make finalized value pool method names consistent
* refactor(state): split database writes into the zebra_db module
* refactor(state): move the block batch method to DiskWriteBatch
* refactor(state): actually add the zebra_db module
Unfortunately, I've lost the interim changes to this file,
so this commit might be the only one that compiles.
* refactor(state): add a newly created file to the cached state CI job
* Run Coverage collection on main
Resolves#3533
* fix(coverage): just run coverage on specific file changes to main
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(dependencies): update an unused duplicate dependency exception
This duplicate was removed by PR #3572, but other duplicates still exist.
* feat(ci): check for duplicate dependencies with optional features off
* fix(mergify, actions): use better names and require tests
* feat(queue): do not update the actual PR, create a draft
Do not allow to update/rebase the original pull request to check its mergeability. Create a draft pull request instead.
This doesn't add Mergify as a co-author
* feat(queue): do not interrupt already running queues
Our queues might take more than 5 hours even if the priority is low.
Do not allow interrupting the ongoing speculative checks when a pull request with higher priority enters in the queue.
* fix(mergify): move 'allow' attributes to queue_rules
* fix(mergify): attributes are not conditions
* refactor(state): split the disk_format module
* refactor(ci): add the new disk_db file to the state CI list
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(ci): clarify ignored test name
`--include-ignored` runs all tests, including tests
that would normally be ignored.
`-Zunstable-options` enables all unstable options,
but it doesn't do anything by itself.
There is a lot of overlap with "test-all" in this job,
which we might want to fix in a future PR.
* fix(ci): remove unused -Zunstable-options
`--include-ignored` is now stable, so `unstable-options` is not needed.
* fix(test): delete a redundant test
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(test): use the short SHA from actual run if valid
* fix(test): if condition must evaluate to a single false
* fix(test): do not run logs and upload if not needed
* imp(test): allow test stateful sync after disk regeneration
This takes is fast enough, so it shouldn't do any harm if run just after a ~3 hours test
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Dependabot creates branches with versions using a dot notation, and some tests fails because of this
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Gustavo Valverde
teor
dependabot[bot]
Deirdre Connolly
Conrado Gouvea
Janito Vaqueiro Ferreira Filho