* Remove duplicate CI docker job and make features consistent
* Remove duplicate OS job
* Rename experimental release build
* Make Rust features into GitHub repository variables
* Remove redundant features in entrypoint.sh
* Remove a dependency on a deleted job
* Fix syntax of array
* Another fix attempt
* Undo some accidental merge overwrites
* Add missing space
* Explain how default is implemented
* Fix missing --features and quoting
* We can fix this later
* Use vars directly in with: blocks
* Use correct features for fake activation heights test
* ref(workflows): consolidate workflows based on their purpose
This also renames the workflows to make their naming more consistent and adding a naming convention
Fixes: #6166Fixes: #6167
* fix(workflows): use correct name for patch
* fix(workflow): docker unit tests
* fix(release): validation error
Error:
```
Validation Failed: {"resource":"Release","code":"invalid","field":"target_commitish"}
```
Fixes: https://github.com/release-drafter/release-drafter/issues/1125
* fix(workflows): reference correct name
* fix: remove extra workflow
* fix(workflows): use larger runners
* fix(workflow): remove code already in docker unit-test
* fix(unit-tests): start zebra the right way
* fix: typo in patch name
* chore: move job to logical order
* imp(workflows): use better name for gcp tests
* ref(workflow): use a single job to run GCP tests
* fix(ci): do not format existing cached states if required
* test: wait for the instance to be fully ready
* fix(ci): use correct logic before formating
* fix: use correct condition
* fix: get more information
* fix(ci): use better shell handling and upgrade OS
* fix(ci): use better approach
* fix: `$DISK_OPTION` is not being correctly passed
* fix typo
* fix: more typos
* fix: use busybox
* fix: mount Docker volume at run and not before
* fix: use correct condition and simpler while
* add: missing merge changes
* chore: use better name for find-disks
* fix(ci): use the `entrypoint.sh` to change the Network
* fix(ci): add missing `ZEBRA_CONF_PATH` variable
* fix(ci): considerate new updates to jobs
* fix(ci): allow to build the entrypoint file with testnet
* fix(entrypoint): allow to create a dir and file with a single variable
* fix(ci): add missing jobs to `failure-issue`
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* fix(ci): use better comment
* refactor: test config file in CI and CD with a reusable workflow
* fix(ci): wrong name used
* fix(ci): use checkout
* fix(ci): improve docker config tests
* fix(ci): use better name for protection rules
* chore: missing merge conflicts
---------
Co-authored-by: teor <teor@riseup.net>
* ref(workflows): consolidate workflows based on their purpose
This also renames the workflows to make their naming more consistent and adding a naming convention
Fixes: #6166Fixes: #6167
* fix(workflows): use correct name for patch
* fix(workflow): docker unit tests
* fix(release): validation error
Error:
```
Validation Failed: {"resource":"Release","code":"invalid","field":"target_commitish"}
```
Fixes: https://github.com/release-drafter/release-drafter/issues/1125
* fix(workflows): reference correct name
* fix: remove extra workflow
* fix(workflows): use larger runners
* fix(workflow): remove code already in docker unit-test
* fix(unit-tests): start zebra the right way
* fix: typo in patch name
* chore: move job to logical order
* imp(workflows): use better name for gcp tests
* add: missing merge changes
* chore: use better name for find-disks
* fix(ci): use the `entrypoint.sh` to change the Network
* fix(ci): add missing `ZEBRA_CONF_PATH` variable
* fix(ci): allow to build the entrypoint file with testnet
* fix(entrypoint): allow to create a dir and file with a single variable
* refactor: test config file in CI and CD with a reusable workflow
* fix(ci): wrong name used
* fix(ci): use checkout
* fix(ci): improve docker config tests
* fix(ci): use better name for protection rules
* Fix changed workflow file name in docs patch file
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Arya <aryasolhi@gmail.com>
* fix(cd): depend on file tests
* fix(docs): adapt to new workflow name
* fix: revert test coverage on CD
* chore: reduce diff
* fix(ci): allow using variable images for reusable workflows
* fix(dockerfile): use variables or default for config path and file
* fix(entrypoint): if `$ZEBRA_CONF_PATH` is set, do not override it
* Fix patch job names and remove failure job testnet dependencies
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Arya <aryasolhi@gmail.com>
* ref(docker): combine test and prod entrypoints into one
This makes the following changes
- Uses `-x` instead of echoing the variables values
- Sets default values where required
- Create a function to list directories
- Create a function to run cargo tests
- Use a better approach to handle different options in the `case` manegement for tests and production
- Replaces all instances of `runtime-entrypoint.sh` with `entrypoint.sh`
* chore: use 2 spaces insted of 4 for indentation
* fix: add missing `test-threads` to `fully_synced_rpc_`
* fix: allow 3 arguments instead of 2 to handle `fully_synced_rpc_` correctly
* chore: remove extra file
* chore: add comments to main parts of the file
* fix(ci): just create the `$ZEBRA_CONF_PATH` if `$ENTRYPOINT_FEATURES` is not set
* fix(entrypoint): just print ${ZEBRA_CONF_PATH} if exists
* fix: missing condition
* ref: handle tests better if `$ENTRYPOINT_FEATURES` is set
* fix(ci): We just want `ZEBRA_CONF_PATH` to be set in the `release` image
* fix(entrypoint): fix the overall `case` logic
* fix(ci): allos to run with custom config in CI image
* fix(ci): more edgecases
* fix: we don't need to find files, but subdirectories
* fix(ci): handle Signal Forwarding and exit codes for `cargo`
* fix(ci): parse `fully_synced_rpc_` test correctly
* chore: add missing cache dir variable for LWD
* fix(entrypoint): handle an arbitrary number of arguments
* fix(entrypoint): handle features list
* fix(entrypoint): typo
* chore: typo
* fix: use `exit-nopipe` with consistent `shell` usage
Temporarily disabled the `set -e` option around the docker logs command to handle the broken pipe error gracefully.
Handle more complex scenarios in our `Result of ${{ inputs.test_id }} test` job
* fix: Use single quotes for the outer command
* fix: use same approach for CD
* test: check launch failure logs
* fix: revert CD changes
* fix: do not try to increase the disk size and wait mounting
* fix: increase GB a bit more
* fix: do not fail on pipe failure
* fix: use plain `tee /dev/stderr`
If this does not work try `(tee … || true)`
* fix: `tee` not stoping on cd config tests
* fix: match logic with GCP tests
* fix(cd): handle pipe and other errors correctly
* try `tee --output-error=exit-nopipe`
* fix: TRAP without pipefail
* test: pipefail with exit and trap
* fix: use a subshell
* fix(ci): wait for mounting and show system logs if fail
* fix(ci): GCP is not always mounting disks in the same order
* fix: use `grep` instead of `awk`
* fix: typo
* fix: use simpler `grep` command
* fix: do not sleep if not require
* chore: reduce diff
* add buld failures ticket generation to most important workflows
* add missing newline at the end of the file
* doc fixes
* add missing testnet build check
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: teor <teor@riseup.net>
* fix(ci): disk validation for docker volume mount
* Use a symlink for lightwalletd cached state rather than mounting the same volume twice
* Avoid "sdb seems to be busy" errors from docker by adding extra sleeps
* Add a missing backslash
* Remove symlink from workflow
* Symlink lightwalletd path in entrypoint.sh
* Retry on failure and check Docker logs
* End ssh shell lines with explicit terminators
* Delete Docker containers if Docker mount fails
* Revert symlink changes in entrypoint.sh
* Debug using lsof
* Use correct lsof commands
* Use correct syntax for lsof +D
* fix(ci): make multiple validations before mounting
Loop and checks for three conditions:
The device `/dev/sdb` exists.
No process is using the device `/dev/sdb`.
No process is using the Docker volume directory.
* fix: do not pre-mount docker volume
The Docker version available with the newer `cos-stable` OS (https://cloud.google.com/release-notes#cos-109-17800-0-45) allows to mount the image when running it.
Mounting it before makes the disk unavailable.
* fix: remove extra `;`
* fix: just confirm with `lsof` and show it's output
* chore: reduce diff
---------
Co-authored-by: teor <teor@riseup.net>
* Add a state full sync to the release-checklist.md
* Add state test and freeze to release.md
* Make testing requirement clearer
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Use `trap '' PIPE` as a command in the pipeline might exit early and we want the script to continue executing, as using `grep --max-count=1` might exit after finding the first match, causing a broken pipe error with `tee`.
This is being combined with `set -o pipefail` to have strict error handling and we want the script to fail if any command in the pipeline fails.
* Add instructions for changing crate owners
* Explain new owner invites
* Remove duplicate `cargo login` instructions from the release checklist
* Remove owners as well
* Missing log line
* feat(ci): add self-hosted runners for long tests
* fix(workflow): use correct runner name
* test: add runner to more jobs
* fix: remove custom packages which are not available in self-hosted runner
This step was mostly required as we did SSH several times during PR, Queue and Push to `main`, but this hasn been reduced considerably.
* Revert "fix: remove custom packages which are not available in self-hosted runner"
This reverts commit 5b5d5464fe.
* fix: do not generate a public key
* fix: install `openssh`in new Ubuntu image
* fix: update OS before installing
* fix: allow silent install
* ref(workflow): use a single job for sync tests
* ref(workflow): use a single job for setup and launch
* fix: apply review recommendations
* chore: comment removal
* obtain lightwalletd from the logs
* revert #7332
* temporarily stop checking that the update sync has a good cached state
* Revert "temporarily stop checking that the update sync has a good cached state"
This reverts commit 0fb10938d9.
* add `save_to_disk` to workflow
* Fix lightwalletd height_grep_text for ECC fork
* Add a TODO about blocking the async executor
---------
Co-authored-by: teor <teor@riseup.net>
* Update issue templates to add a custom blank template
* revert changes to other templates
* Update custom.md
Add details about intended template usage
* fix(docs): use specific variables to avoid using an `environment`
* fix(docs): build and publish the correct paths
* fix(docs): remove all environments
* ref(docs): consolidate docs linting and publishing
* chore: minimum lint
* fix: cross-reference
* fix(docs): remove preview URLs from deployments
* fix(docs): use specific variables to avoid using an `environment`
* fix(docs): build and publish the correct paths
* fix(docs): remove all environments
* fix(doc): consolidate firebase hosting in prod project
* fix(deploy): use correct environment and permissions
* ref(firebase): use a single firebase configuration
* fix(firebase): entrypoint no longer needed
* ref(docs): deploy docs on PR for preview
* fix(docs): allow to comment on PR
* fix(docs): allow enough time to build docs
* fix(docs): cancel previous docs build
* fix(docs): use a more specific group
* fix(firebase): extra permissions
* fix: commit `.firebaserc`
* fix(mdbook): use different actual directory
* fix(deploy): increase Rust build timeout
* chore: more timeout for building and publishing
* fix(deploy): login to GCP before deploying
This is to avoid timeout issues with the GCP credentials
* chore: lint code
* chore: revert SC2002 fix try
* ref(docker): remove all unrequired docker arguments
* fix(ci): use correct `$NETWORK` approach for tests
* fix(release): do not change default `$NETWORK` for experimental image
* Update .github/workflows/continous-integration-docker.yml
Co-authored-by: Marek <mail@marek.onl>
* Revert "fix(release): do not change default `$NETWORK` for experimental image"
This reverts commit bd5b6c831b.
* fix: typo
---------
Co-authored-by: Marek <mail@marek.onl>
* ref(issues): use newer template where applicable
This also fixes emojis, indenting and issues with GitHub YAML schema for issues template
* fix(issues): do not require testing or related work
* fix(issues): textarea required
* fix(issues): do not be strict on issues structure
* Move format upgrades to their own module and enum
* Launch a format change thread if needed, and shut it down during shutdown
* Add some TODOs and remove a redundant timer
* Regularly check for panics in the state upgrade task
* Only run example upgrade once, change version field names
* Increment database format to 25.0.2: add format change task
* Log the running and initial disk database format versions on startup
* Add initial disk and running state versions to cached state images in CI
* Fix missing imports
* Fix typo in logs workflow command
* Add a force_save_to_disk argument to the CI workflow
* Move use_internet_connection into zebrad_config()
* fastmod can_spawn_zebrad_for_rpc can_spawn_zebrad_for_test_type zebra*
* Add a spawn_zebrad_without_rpc() function
* Remove unused copy_state() test code
* Assert that upgrades and downgrades happen with the correct versions
* Add a kill_and_return_output() method for tests
* Add a test for new_state_format() versions (no upgrades or downgrades)
* Add use_internet_connection to can_spawn_zebrad_for_test_type()
* Fix workflow parameter passing
* Check that reopening a new database doesn't upgrade (or downgrade) the format
* Allow ephemeral to be set to false even if we don't have a cached state
* Add a test type that will accept any kind of state
* When re-using a directory, configure the state test config with that path
* Actually mark newly created databases with their format versions
* Wait for the state to be opened before testing the format
* Run state format tests on mainnet and testnet configs (no network access)
* run multiple reopens in tests
* Test upgrades run correctly
* Test that version downgrades work as expected (best effort)
* Add a TODO for testing partial updates
* Fix missing test arguments
* clippy if chain
* Fix typo
* another typo
* Pass a database instance to the format upgrade task
* Fix a timing issue in the tests
* Fix version matching in CI
* Use correct env var reference
* Use correct github env file
* Wait for the database to be written before killing Zebra
* Use correct workflow syntax
* Version changes aren't always upgrades
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Reduce number of dependabot PRs
* Validate dependabot config before merging
* Fix indentation
* Fix spacing again
* Split large group
* Expand comments, fix typos
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Activate production features on docs.rs
* Make version requirements more flexible to work around unpublished versions
* Remove a redundant feature requirement that causes publishing issues
* fix release auto-replacement format
* cargo release replace --verbose --execute --package zebrad
* Add a missing 1.0.1 changelog entry
* Fix incorrect commands in the release checklist
* Check the crates.io release script in CI, to avoid release failures
* Fix release date in changelog
* Fix workflow typo
* Add --no-confirm to the workflow
* Fix outdated version in release check command
* After the release, add a "remove do-not-merge" step
* Set git identity before committing
* Don't overwrite tweaked versions on upgrade
* cargo release version doesn't support --dependent-version
* Update the same commands in the release checklist
* Commit to git after each change
* And in the checklist
* Add cargo release config
* In CI, allow any branch and provide a fake previous tag name
* Remove redundant `cargo release commit`
* Use a simpler release version process
* Start simplifying the release dry run in CI
* Fix a cargo dependency lint
* just skip the failing crates for now
* Fix the release checklist to match CI
* Use the latest versions of all Zebra dependencies
dependabot[bot]
teor
Gustavo Valverde
Arya
Alfredo Garcia
Pili Guerra
Marek
Deirdre Connolly