* Add the `scan_start_where_left` test to CI
* Fix passing of features
Co-authored-by: Arya <aryasolhi@gmail.com>
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* ref(workflow): move most scripts to their own executable
* debug: JSON value
* fix(scripts): move remaining script to its own file
* fix(script): revert to the correct disk search logic
* fix(scripts)
* fix(scripts): use correct NETWORK with lowercase
* fix: typo
* fix(script): wrong variable assignment
* fix(script): use correct return values inside a function
* fix(script): fix value assigment
* test: debug
* fix(script): make disk conditions simpler
* fix(script): export variables to the `shell` executing the script
* fix(script): do not fail on expected unbound variables
* test: output
* fix(scripts): do not `echo` a variable more than once
* fix(scripts): typo
* docs(workflow): adds a description at the top of each file (#8009)
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Marek <mail@marek.onl>
* Run clippy and build all targets on all crates individually
* Fix prod and test features for scanner deps
* Standardise dependency order
* Remove unnecessary async in tests
* Fix an unused import in a test
* Work around a no space left on device error
* Actually just use a larger runner
* Skip external branches in ci-unit-tests-docker.yml
* Patch external repositories in ci-unit-tests-docker.patch.yml
* Revert ci-unit-tests-docker.patch.yml
* Add a separate external PR patch in ci-unit-tests-docker.patch-external.yml
* Document ci-unit-tests-docker.patch.yml
* Document ci-unit-tests-docker.yml
* Skip dependabot config validation for external PRs
* Trivial Rust code modification to make Rust workflows run
* Use correct method for checking source branch refs
* Fix the check condition
* Simplify job structure
* Use correct skip condition
* Make the start docs visible so we can link to them
* Skip integration tests on external PRs
* Allow external PRs to pass CD Docker tests
* Allow external PRs to pass Docs build and deploy
* Update PR template checklists
* Add a testing section
* Move close issues to the top section
* Change checklist phrasing so the answer is yes when the PR is complete
* Explain how to handle irrelevant checkboxes
* Another yes answer is good
* Fix indent
* Ask if PRs can be split
* Remove DockerHub tags for v1.x.y and edge
* Make the latest tag point to the production image, not experimental
* Remove docker image file change trigger from the OS CI workflow
* Disable latest tag by default
* Enable the latest tag for the production build only
* Restore `edge` tag, it is used in CI and it is CI only
* Add Docker tag removal to CHANGELOG.md
* Actually enforce job order
* Revert tag publishing changes, but keep extra docs and tag order changes
* Update the changelog
* Simple replacements of doc.zebra.zfnd.org with docs.rs
* Manual fixes for specific main/internal/external docs
* Point developer docs to doc-internal.zebra.zfnd.org
* fastmod --glob '\!.git' -- doc.zebra.zfnd.org/zebrad docs.rs/zebrad/latest/zebrad
* Manually remove any remaining doc.zfnd.zebra.org links
* Remove the external docs job
* Add changelog entry and fix links
* Fix links that were broken before this PR
* Remove the `zebrad download` command
* Reorder command.rs imports
* Remove zcash-params/Dockerfile and the main Dockerfile commands that use it
* Stop building zcash-params Docker images in CI
* Update CHANGELOG for `zebrad download` removal
* Clarify why the image is smaller
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* Remove duplicate CI docker job and make features consistent
* Remove duplicate OS job
* Rename experimental release build
* Make Rust features into GitHub repository variables
* Remove redundant features in entrypoint.sh
* Remove a dependency on a deleted job
* Fix syntax of array
* Another fix attempt
* Undo some accidental merge overwrites
* Add missing space
* Explain how default is implemented
* Fix missing --features and quoting
* We can fix this later
* Use vars directly in with: blocks
* Use correct features for fake activation heights test
* ref(workflows): consolidate workflows based on their purpose
This also renames the workflows to make their naming more consistent and adding a naming convention
Fixes: #6166Fixes: #6167
* fix(workflows): use correct name for patch
* fix(workflow): docker unit tests
* fix(release): validation error
Error:
```
Validation Failed: {"resource":"Release","code":"invalid","field":"target_commitish"}
```
Fixes: https://github.com/release-drafter/release-drafter/issues/1125
* fix(workflows): reference correct name
* fix: remove extra workflow
* fix(workflows): use larger runners
* fix(workflow): remove code already in docker unit-test
* fix(unit-tests): start zebra the right way
* fix: typo in patch name
* chore: move job to logical order
* imp(workflows): use better name for gcp tests
* ref(workflow): use a single job to run GCP tests
* fix(ci): do not format existing cached states if required
* test: wait for the instance to be fully ready
* fix(ci): use correct logic before formating
* fix: use correct condition
* fix: get more information
* fix(ci): use better shell handling and upgrade OS
* fix(ci): use better approach
* fix: `$DISK_OPTION` is not being correctly passed
* fix typo
* fix: more typos
* fix: use busybox
* fix: mount Docker volume at run and not before
* fix: use correct condition and simpler while
* add: missing merge changes
* chore: use better name for find-disks
* fix(ci): use the `entrypoint.sh` to change the Network
* fix(ci): add missing `ZEBRA_CONF_PATH` variable
* fix(ci): considerate new updates to jobs
* fix(ci): allow to build the entrypoint file with testnet
* fix(entrypoint): allow to create a dir and file with a single variable
* fix(ci): add missing jobs to `failure-issue`
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* fix(ci): use better comment
* refactor: test config file in CI and CD with a reusable workflow
* fix(ci): wrong name used
* fix(ci): use checkout
* fix(ci): improve docker config tests
* fix(ci): use better name for protection rules
* chore: missing merge conflicts
---------
Co-authored-by: teor <teor@riseup.net>
* ref(workflows): consolidate workflows based on their purpose
This also renames the workflows to make their naming more consistent and adding a naming convention
Fixes: #6166Fixes: #6167
* fix(workflows): use correct name for patch
* fix(workflow): docker unit tests
* fix(release): validation error
Error:
```
Validation Failed: {"resource":"Release","code":"invalid","field":"target_commitish"}
```
Fixes: https://github.com/release-drafter/release-drafter/issues/1125
* fix(workflows): reference correct name
* fix: remove extra workflow
* fix(workflows): use larger runners
* fix(workflow): remove code already in docker unit-test
* fix(unit-tests): start zebra the right way
* fix: typo in patch name
* chore: move job to logical order
* imp(workflows): use better name for gcp tests
* add: missing merge changes
* chore: use better name for find-disks
* fix(ci): use the `entrypoint.sh` to change the Network
* fix(ci): add missing `ZEBRA_CONF_PATH` variable
* fix(ci): allow to build the entrypoint file with testnet
* fix(entrypoint): allow to create a dir and file with a single variable
* refactor: test config file in CI and CD with a reusable workflow
* fix(ci): wrong name used
* fix(ci): use checkout
* fix(ci): improve docker config tests
* fix(ci): use better name for protection rules
* Fix changed workflow file name in docs patch file
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Arya <aryasolhi@gmail.com>
* fix(cd): depend on file tests
* fix(docs): adapt to new workflow name
* fix: revert test coverage on CD
* chore: reduce diff
* fix(ci): allow using variable images for reusable workflows
* fix(dockerfile): use variables or default for config path and file
* fix(entrypoint): if `$ZEBRA_CONF_PATH` is set, do not override it
* Fix patch job names and remove failure job testnet dependencies
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Arya <aryasolhi@gmail.com>
* ref(docker): combine test and prod entrypoints into one
This makes the following changes
- Uses `-x` instead of echoing the variables values
- Sets default values where required
- Create a function to list directories
- Create a function to run cargo tests
- Use a better approach to handle different options in the `case` manegement for tests and production
- Replaces all instances of `runtime-entrypoint.sh` with `entrypoint.sh`
* chore: use 2 spaces insted of 4 for indentation
* fix: add missing `test-threads` to `fully_synced_rpc_`
* fix: allow 3 arguments instead of 2 to handle `fully_synced_rpc_` correctly
* chore: remove extra file
* chore: add comments to main parts of the file
* fix(ci): just create the `$ZEBRA_CONF_PATH` if `$ENTRYPOINT_FEATURES` is not set
* fix(entrypoint): just print ${ZEBRA_CONF_PATH} if exists
* fix: missing condition
* ref: handle tests better if `$ENTRYPOINT_FEATURES` is set
* fix(ci): We just want `ZEBRA_CONF_PATH` to be set in the `release` image
* fix(entrypoint): fix the overall `case` logic
* fix(ci): allos to run with custom config in CI image
* fix(ci): more edgecases
* fix: we don't need to find files, but subdirectories
* fix(ci): handle Signal Forwarding and exit codes for `cargo`
* fix(ci): parse `fully_synced_rpc_` test correctly
* chore: add missing cache dir variable for LWD
* fix(entrypoint): handle an arbitrary number of arguments
* fix(entrypoint): handle features list
* fix(entrypoint): typo
* chore: typo
* fix: use `exit-nopipe` with consistent `shell` usage
Temporarily disabled the `set -e` option around the docker logs command to handle the broken pipe error gracefully.
Handle more complex scenarios in our `Result of ${{ inputs.test_id }} test` job
* fix: Use single quotes for the outer command
* fix: use same approach for CD
* test: check launch failure logs
* fix: revert CD changes
* fix: do not try to increase the disk size and wait mounting
* fix: increase GB a bit more
* fix: do not fail on pipe failure
* fix: use plain `tee /dev/stderr`
If this does not work try `(tee … || true)`
* fix: `tee` not stoping on cd config tests
* fix: match logic with GCP tests
* fix(cd): handle pipe and other errors correctly
* try `tee --output-error=exit-nopipe`
* fix: TRAP without pipefail
* test: pipefail with exit and trap
* fix: use a subshell
* fix(ci): wait for mounting and show system logs if fail
* fix(ci): GCP is not always mounting disks in the same order
* fix: use `grep` instead of `awk`
* fix: typo
* fix: use simpler `grep` command
* fix: do not sleep if not require
* chore: reduce diff
* add buld failures ticket generation to most important workflows
* add missing newline at the end of the file
* doc fixes
* add missing testnet build check
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: teor <teor@riseup.net>
* fix(ci): disk validation for docker volume mount
* Use a symlink for lightwalletd cached state rather than mounting the same volume twice
* Avoid "sdb seems to be busy" errors from docker by adding extra sleeps
* Add a missing backslash
* Remove symlink from workflow
* Symlink lightwalletd path in entrypoint.sh
* Retry on failure and check Docker logs
* End ssh shell lines with explicit terminators
* Delete Docker containers if Docker mount fails
* Revert symlink changes in entrypoint.sh
* Debug using lsof
* Use correct lsof commands
* Use correct syntax for lsof +D
* fix(ci): make multiple validations before mounting
Loop and checks for three conditions:
The device `/dev/sdb` exists.
No process is using the device `/dev/sdb`.
No process is using the Docker volume directory.
* fix: do not pre-mount docker volume
The Docker version available with the newer `cos-stable` OS (https://cloud.google.com/release-notes#cos-109-17800-0-45) allows to mount the image when running it.
Mounting it before makes the disk unavailable.
* fix: remove extra `;`
* fix: just confirm with `lsof` and show it's output
* chore: reduce diff
---------
Co-authored-by: teor <teor@riseup.net>
* Add a state full sync to the release-checklist.md
* Add state test and freeze to release.md
* Make testing requirement clearer
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Use `trap '' PIPE` as a command in the pipeline might exit early and we want the script to continue executing, as using `grep --max-count=1` might exit after finding the first match, causing a broken pipe error with `tee`.
This is being combined with `set -o pipefail` to have strict error handling and we want the script to fail if any command in the pipeline fails.
* Add instructions for changing crate owners
* Explain new owner invites
* Remove duplicate `cargo login` instructions from the release checklist
* Remove owners as well
* Missing log line
* feat(ci): add self-hosted runners for long tests
* fix(workflow): use correct runner name
* test: add runner to more jobs
* fix: remove custom packages which are not available in self-hosted runner
This step was mostly required as we did SSH several times during PR, Queue and Push to `main`, but this hasn been reduced considerably.
* Revert "fix: remove custom packages which are not available in self-hosted runner"
This reverts commit 5b5d5464fe.
* fix: do not generate a public key
* fix: install `openssh`in new Ubuntu image
* fix: update OS before installing
* fix: allow silent install
* ref(workflow): use a single job for sync tests
* ref(workflow): use a single job for setup and launch
* fix: apply review recommendations
* chore: comment removal
* obtain lightwalletd from the logs
* revert #7332
* temporarily stop checking that the update sync has a good cached state
* Revert "temporarily stop checking that the update sync has a good cached state"
This reverts commit 0fb10938d9.
* add `save_to_disk` to workflow
* Fix lightwalletd height_grep_text for ECC fork
* Add a TODO about blocking the async executor
---------
Co-authored-by: teor <teor@riseup.net>
Marek
dependabot[bot]
Gustavo Valverde
Alfredo Garcia
teor
Arya
Pili Guerra