Commit Graph

117 Commits

Author SHA1 Message Date
Gustavo Valverde aa7205d660
refactor(docker): Enhance Zebra configuration options and entrypoint logic (#9344)
* refactor(docker): Enhance Zebra configuration options and entrypoint logic

- Introduced multiple methods for configuring Zebra, including custom config file paths and automatic generation from environment variables.
- Updated entrypoint script to handle configuration more robustly, checking for existing files and generating defaults as needed.
- Improved documentation in `docker.md` to clarify configuration options and their precedence.
- Adjusted environment variable handling in `docker/.env` for better clarity and functionality.
- Refactored `prepare_conf_file` to create a complete configuration based on environment variables, streamlining the setup process.

* fix(entrypoint): use correct `ZEBRA_RPC_PORT` variable

This also removes the logic to set the default port for RPC as this was introducing a buggy logic

* fix(docker): remove non-essential variables and set correct defaults

* chore(docker): use the default config for docker-compose as the default approach

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

* chore: better comment for user understanding

* fix(entrypoint): set permissions correctly for parent directories

* chore: align documentation and implementation

* chore: clearer and aligned comments with docker-compose

* fix(entrypoint): chown the zebra config file when `ZEBRA_CONF_PATH` is not set

---------

Co-authored-by: Marek <mail@marek.onl>
2025-04-16 12:27:32 +00:00
Gustavo Valverde db48daea05
fix(docker): add default CMD to test stage in Dockerfile (#9334)
* fix(docker): set `HOME` environment variable explicitly in Dockerfile

The HOME environment variable was defaulting to /root when the container started, causing cache directories to be incorrectly set up under /root/.cache/zebra instead of /home/zebra/.cache/zebra. This explicit setting ensures the HOME environment variable is correctly set to the zebra user's home directory.

* fix(docker): add default `CMD` to test stage in Dockerfile

When running the container using docker-compose without explicitly
providing a command, the entrypoint.sh script was attempting to execute
`exec_as_user` with no arguments, resulting in a gosu error:

```
exec_as_user
exec gosu 10001:10001
Usage: gosu user-spec command [args]
```

By adding `CMD ["cargo", "test"]` to the test stage in the Dockerfile,
we ensure a default command is available for the entrypoint script to
execute, preventing the gosu error when no command is explicitly provided.

This fix allows `docker-compose.test.yml` to run successfully without
needing to specify a command in the service definition.
2025-03-31 21:28:25 +00:00
Gustavo Valverde 32de8ef5fd
fix(docker): set `HOME` environment variable explicitly in Dockerfile (#9333)
The HOME environment variable was defaulting to /root when the container started, causing cache directories to be incorrectly set up under /root/.cache/zebra instead of /home/zebra/.cache/zebra. This explicit setting ensures the HOME environment variable is correctly set to the zebra user's home directory.
2025-03-22 15:25:30 +00:00
Marek f0140a4cd8
fix(ci): Better permission and cache dirs handling in Docker (#9323)
* Use gosu only once

* Remove `COLORBT_SHOW_HIDDEN`

* Simplify Dockerfile

* Remove `check_directory_files` from entrypoint

* Remove check for `ZEBRA_CONF_PATH` in entrypoint

* Simplify ownership setting for `ZEBRA_CACHE_DIR`

* Simplify ownership setting for `LOG_FILE`

* Refactor Dockerfile & entrypoint

* Refactor vars in Dockerfile

* fmt

* Use `chown` for `ZEBRA_CONF_PATH`

* `run_cargo_test` -> `run_test`

* Make `run_test` runnable with gosu

* Cosmetics

* Don't pre-compile Zebra

* Revert: "Don't pre-compile Zebra"

* Fix the custom conf test

* Reintroduce `CARGO_HOME` in Dockerfile

* Pass `FEATURES` as env var to entrypoint

* Fix ARGs in Dockerfile

* Revert "Remove `COLORBT_SHOW_HIDDEN`"

This reverts commit 960d5ca308.

* Specify cache state dir in CI

* Specify lwd cache dir in CI

* refactor: reorganize variables and avoid running entrypoint commands in subshell (#9326)

* refactor(docker): improve container configuration and security

- Optimize Dockerfile build stages and environment variables
- Improve file operations with proper ownership
- Streamline entrypoint script privilege management

* refactor(docker): enhance user management and directory ownership

- Add HOME argument back to ensure proper user home directory setup
- Implement ownership change for the user's home directory

* refactor(docker): remove redundant cache directory setup

- Eliminate explicit creation and ownership setting for LWD and Zebra cache directories in Dockerfile.
- Introduce default values for cache directories in entrypoint script, allowing for environment variable overrides.

* fix: run all cargo commands as user

* chore: reduce diff

* fix: revert to more robust command array

---------

Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
Co-authored-by: Gustavo Valverde <g.valverde02@gmail.com>
2025-03-18 08:58:19 +00:00
Gustavo Valverde 49741e8b47
fix(ci): Add Rust environment variables to Docker test workflows (#9318)
* feat(ci): Add Rust environment variables to Docker test workflows

Enhance test workflows by adding Rust-specific environment variables:
- Include RUST_LOG for logging configuration
- Add RUST_BACKTRACE and RUST_LIB_BACKTRACE for improved error tracing
- Include COLORBT_SHOW_HIDDEN for detailed backtraces
- Add CARGO_INCREMENTAL for build performance optimization

These changes improve debugging capabilities and provide more flexibility in test environments across dockerized CI jobs.

* feat(ci): Add CARGO_INCREMENTAL build argument to Docker workflow

Enable configurable Rust incremental compilation in Docker builds by:
- Adding CARGO_INCREMENTAL as a build argument in GitHub Actions workflow
- Setting a default value of 0 in Dockerfile to control build performance
- Integrating with existing Rust environment variable configuration

This change provides more granular control over Rust compilation strategies in containerized builds.

* fix(docker): resolve user creation issues in Dockerfile

- Move WORKDIR after user creation to prevent home directory ownership issues
- Properly set environment variables for UID, GID, HOME, and USER in each stage
- Reorganize Dockerfile to ensure home directory is created after user setup
- Fix interactive prompts during adduser by ensuring proper directory ownership

* fix(docs): Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2025-03-12 10:43:31 +00:00
Marek de7e5b547f
refactor(docker): allow r/w access in mounted volumes (#9281)
* Switch to a non-privileged user in tests

* Change test env setup

* Remove unneeded ARGs

* Simplify UID & GID handling in `runtime` target

* Simplify docs

* refactor(docker): Improve user and permission handling in Dockerfiles

- Add gosu for flexible non-root user execution
- Enhance user and group creation with configurable UID/GID
- Modify entrypoint script to support dynamic user switching
- Improve cache and log directory permission management
- Update comments to clarify user and permission strategies

* refactor(docker): Improve Zebra config file handling in entrypoint script

- Enhance error handling for missing config file (now exits with error)
- Simplify config preparation logic by removing redundant file copying
- Update comments to reflect new config file handling approach
- Ensure consistent use of ZEBRA_CONF_PATH throughout the script

* refactor(docker): Enhance container user security and configuration

- Increase UID/GID to 10001 to minimize host system user conflicts
- Remove `--system` flag from user and group creation to prevent potential issues
- Add detailed comments explaining UID/GID selection rationale
- Improve security by using high UID/GID values to reduce namespace collision risks
- Remove redundant `chmod` for entrypoint script

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Gustavo Valverde <g.valverde02@gmail.com>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2025-03-03 19:21:03 +01:00
Marek 38b7313ea1
chore: bump Rust from 1.84.0 to 1.85.0 in Docker (#9290) 2025-02-24 14:35:39 +00:00
Marek ac25192afc
docs(docker): update examples for running Zebra in Docker (#9269)
* Rm `.env` files

* Update `mining-docker.md`

* Revert "Rm `.env` files"

This reverts commit caaa4559c3.

* Add `enable_cookie_auth` to default Zebra conf

* Rename `default_zebra_config.toml`

* fmt `prometheus.yaml`

* Update `docker/test.env`

* Update `docker/.env`

* Refactor `docker compose` for lwd

* Enable disabling cookie authentication

* Update `docker compose` for tests

* Update general `docker compose`

* Update docs for running Zebra in Docker

* Add example `docker compose` file for Grafana

* Fix a bug in an example command

* Refactor test execution logic in entrypoint

* Rename `v2.1.0.toml` conf to `custom-conf.toml`

* Fix CI tests for loading of custom conf files

* Use the new conf file name in CI checks

* Use an extended regexp for custom conf CI check
2025-02-19 17:44:06 +00:00
Marek b2fb1a7d56
fix(tests): Check if a cached state is present in scanner tests (#9251)
* Fix `scan_task_commands` test

* Fix `scan_start_where_left` test

* fmt

* fmt

* Fix var for scanning tests
2025-02-13 18:15:32 +00:00
Marek 4132c0e4e9
ci: Refactor Dockerfile & entrypoint (#8923)
* Refactor formatting & docs

* Refactor the `runtime` stage in Dockerfile

* Remove unused code from `entrypoint.sh`

* Simplify `entrypoint.sh` setup

* Revise docs & formatting

* Adjust default values for env vars

* Bump Rust v from 1.79 to 1.81 in Dockerfile

* Refactor `entrypoint.sh`

* Refactor `Dockerfile`

* Add TODOs for monitoring stage to Dockerfile

* Refactor `Dockerfile`

* Add TODOs for monitoring stage to Dockerfile

* Fix a typo

* Allow running `zebrad` in test mode

* Allow custom config for `zebrad` in test mode

* Remove `curl` from the `runtime` Docker image

* Remove redundant echos

* Remove a malfunctioning CD test

The test was using a custom config file set in `test_variables`.
However, the file was not included in the Docker image, and the
entrypoint script created a new, default one under the original file's
path. Zebra then loaded this new file, and the test passed because the
pattern in `grep_patterns` matched Zebra's output containing the
original path, even though the config file was different.

* Remove a redundant CI test

* Remove all packages from the `runtime` stage

* Docs cosmetics

* Clarify docs

* Bump Rust version

* Remove a security note

* Explicitly specify network cache dir

* Explicitly specify cookie dir

* Set UID, GID and home dir for the `zebra` user

* Set a working dir for the `zebra` user

* Don't remove `FEATURES`

* Try re-introducing the `testnet-conf` check

* `ZEBRA_CACHED_STATE_DIR` -> `ZEBRA_CACHE_DIR`

This dir doesn't hold only the state cache anymore, but also the cache
for network peers, and the cookie file.

* Refactor the dir structure

* Check that `ZEBRA_CONF_PATH` exists in the image

* Improve the check for `ZEBRA_CONF_PATH`

* Use different flag in the `ZEBRA_CONF_PATH` check

* Simplify the `ZEBRA_CONF_PATH` check

* Fix spelling

* Comment out the `testnet-conf` CI check

* Add commented out `test-zebra-conf-path` CI check

* Reintroduce `testnet-conf` CI check

* Update the `custom-conf` CI check

* Add `v2.1.0.toml` conf file

* Refine the `v2.1.0.toml` conf file

* Remove `ZEBRA_LISTEN_ADDR` from the entrypoint

* Remove `ZEBRA_CHECKPOINT_SYNC` from the entrypoint

* Stop supporting configuration of the RPC port

* Add default conf file

* Prepare Zebra's config in the entrypoint script

* Remove unneeded packages from the `deps` target

* Docs cosmetics

* Use only `$FEATURES` in entrypoint

* Simplify handling of Rust features

* Add a TODO

* Add CI debug statements

* Don't require test vars in conf test

* Reintroduce `protoc`

* Remove `-e NETWORK`

* Remove `ZEBRA_FORCE_USE_COLOR=1`

* Remove `ZEBRA_CACHE_DIR=/var/cache/zebrad-cache`

* Reintroduce the "custom-conf" test

* Set up test env the same way as prod

* Don't repeatedly check for conf file in entrypoint

* Simplify file ownership in Dockerfile

* Fix checkpoint tests in entrypoint

* Fix Zebra config CI tests

* `LIGHTWALLETD_DATA_DIR` -> `LWD_CACHE_DIR`

* Add config for `LWD_CACHE_DIR` to Dockerfile

* `/var/cache/zebrad-cache` -> `~/.cache/zebra`

* `var/cache/lwd-cache` -> `/home/zebra/.cache/lwd`

* Remove `LOG_COLOR=false` from GCP setup

* Don't specify `LWD_CACHE_DIR` in CI tests

* Don't switch to `zebra` user for tests in Docker

* Join "experimental" and "all" tests in CI

* Remove outdated docs

* Refactor tests with fake activation heights

* Fix tests for scanner
2025-02-13 09:15:32 +00:00
Arya 8cfb61f52c
add(ci): Check that dependencies have all been published to crates.io on release PRs (#8992)
* Adds a test to check for crates in the Cargo.lock file that are being pulled in from a git source.

* add `check_no_git_refs_in_cargo_lock` to CI

* try skip instead of exclude

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2024-11-27 15:49:12 +00:00
Gustavo Valverde 3983428ac4
fix(ci): run most lighwalletd tests correctly (#9038)
A LWD test was expecting the `ZEBRA_TEST_LIGHTWALLETD` to be set, but this variable is needed for all LWD tests and not specifically for `lightwalletd_integration`.

We had to rename this variable on a buggy `elif` statement in our Docker entrypoint.

This was avoiding most LWD tests to run correctly.
2024-11-19 11:41:57 +00:00
Alfredo Garcia 17648cc6e1
remove `scan-start-where-left-test` from CI (#9026) 2024-11-19 10:25:37 +01:00
Arya 47b7614ea8
updates minimum supported Rust version in Zebrad and rust verison in Dockerfile (#8956) 2024-10-22 00:41:34 +00:00
Skylar Saveland 3220520592
Update docker-compose.yml - tiny typo (#8893) 2024-10-07 22:14:03 +00:00
Gustavo Valverde 6951988456
chore(docker): remove debugging output by default (#8870) 2024-09-19 17:25:40 +00:00
Gustavo Valverde aec07f24ff
ref(ci): consolidate cached states workflows and scripts (#8865)
* ref(ci): consolidate cached states workflows and scripts

We've been using multiple approaches to locate and retrieve cached states in GCP. However, this has made it difficult to reuse the same methods across new workflows or different scenarios.

To address this, we've streamlined the process to make it more reusable in other contexts. This change will support deploying instances from both the `main` branch and `release`, simplifying future implementations and speeding up the process.

Changes:
- Use a single bash script (`gcp-get-cached-disks.sh`) to get cached states names and availability
- Move script logic from `sub-find-cached-disks.yml` to `gcp-get-cached-disks.sh` and adapt `sub-find-cached-disks.yml` to allow to output available disks and disks names.
- Simplify parameters usage in `sub-deploy-integration-tests-gcp.yml` and convert the `Find ${{ inputs.test_id }} cached state disk` step into an independent job, to be able to use the `sub-find-cached-disks.yml` reusable workflow
- Remove repetition in `sub-ci-integration-tests-gcp.yml`

* ref(tests): Use the `ZEBRA_CACHED_STATE_DIR` env var across tests

We had a technical debt with some tests using a hardcoded value for the cache directory (`/zebrad-cache`), which generated inconsistency across disks and cached states directories.

Changes:
- Allow sync tests to use the `ZEBRA_CACHED_STATE_DIR` as the cache directory, if specified
- Update the `entrypoint.sh` to reflect this change
- Add the `ZEBRA_CACHED_STATE_DIR` variable to the missing tests in `sub-ci-integration-tests-gcp.yml`, and remove extra parameters to call reusable workflows.
2024-09-19 12:31:33 +00:00
Gustavo Valverde d31eea5f64
ref(docker): use cache mounts for build cache (#8796)
* ref(docker): leverage cache mount with bind mounts

This update eliminates the need for external tools like `cargo-chef` to leverage caching layers, resulting in an average build time reduction of 4m30s (~36% improvement).

While this solution doesn't fully resolve the issues mentioned in https://github.com/ZcashFoundation/zebra/issues/6169#issuecomment-1712776391, it represents the best possible approach without resorting to custom solutions, which we'd prefer to avoid.

* chore: remove extra `WORKDIR` and imp comments

* chore: improve comment legibility

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
Co-authored-by: Arya <aryasolhi@gmail.com>
2024-09-05 13:29:22 +00:00
Marek 5541b27a12
Remove `shielded-scan` from experimental features (#8827) 2024-09-03 10:04:36 +00:00
Gustavo Valverde cdb9efdb27
fix(docker): allow the `zebra` user access to relevant dirs (#8817)
* fix(docker): allow the `zebra` user access to relevant dirs

When runnning a Zebra node using Docker without a privileged user, you won't be able to modify some files and directories, not even the ones in the current directory, as the `zebra` user has no permission to `/`.

The best way to solve this is making the `/opt/zebrad` the current `WORKDIR`. This also requires moving the `entrypoint.sh` from the root `/` directory to `/etc/zebrad` as this directory is used to save configuration, and other files.

An `APP_HOME` ARG is used as not all platforms where a Docker container is deployed allows writting permissions to the `/opt` directory. This allow some users to re-build the image with a custom `WORKDIR`

* fix(docker): allow starting the container without a `zebrad` command

As `gosu` is just required and available in our `runtime` image, trying to run `docker run -it --rm  --name tests -t zfnd/zebra:<pr> /bin/bash` in other stages will fail, as `gosu` is not available.
2024-08-29 19:57:17 +00:00
Gustavo Valverde ec85aa8a48
fix(docker): add `gosu` and remove unsupported flag in `adduser` (#8808)
* fix(docker): typo and uknown option in debian

* fix(docker): use `gosu` for rootless execution

Some of our entrypoint commands requires creating directories and files in places a non-privileged user can't access.

So we use `gosu` to step down from `root` to a non-privileged user during container startup, right at our application execution.
2024-08-27 21:29:50 +00:00
Gustavo Valverde 0d36681d8f
fix(docker): do not run the Zebra nodes with the root user (#8803) 2024-08-27 11:55:24 +00:00
Alfredo Garcia 1238ec0c63
tests(scanner): Move zebra scanner tests to binary (#8659)
* readd the scanner tests in the new binary context

* remove commented out tests from zebrad

* add Cargo.lock

* add local copy of `ZECPAGES_SAPLING_VIEWING_KEY` to binary scanning tests

* moves scanner-results-reader to zebra-scan, adds zebrad bin to zebra-scan, adds a short sleep before trying to connect to gRPC server in test

* always include zebrad dependency in zebra-scan

* renames zebrad bin in zebra-scan to zebrad-for-scanner

* skip test when a cached state is missing

* remove scanning config from zebrad

* move `scan_task_commands` out of zebrad acceptance tests

* remove scanner config from zebrad

* do not panic if env var is not there

* re enable start_scan_where left, remove shielded-scan feature foir scanning tests in CI

* change app names

* uncomment test

* fix the scanner tests in CI by changing the test command

* add log needed by CI?

* move test

* rename trait, docs fix

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2024-07-15 21:15:04 +00:00
Arya e64f3dcddd
add(test): Check that scan task removes keys, unify scan task acceptance tests, and add test to CI (#8308)
* merges scan task tests and checks that the scan task removes keys

* Adds test to CI

* Skip to expected block height and assert that the result is from that height

* Updates test docs and uses zebrad state path for scan cache

* Log "Zcash network: {network}" from scan task commands test
2024-03-04 21:25:50 +00:00
Gustavo Valverde 56fca2c9ee
ref(build)!: use official lightwallted image for CI (#8323) 2024-02-29 15:50:50 +00:00
kai 6c0aa042ad
fix(docker): set log rotation to avoid docker bugs (#8269)
Containers can generate a significant amount of logs,
consuming disk space over time. Here I added configuration
for maximum log file size (max-size) and the number of retained
log files (max-file) in the logging driver options
to control disk usage and prevent log files from
growing indefinitely.
2024-02-20 10:40:38 +00:00
Marek 584bd23718
Update Debian from Bullseye to Bookworm (#8273) 2024-02-14 23:02:06 +00:00
Gustavo Valverde 5feb40e0b8
feat(docker): allow users to use Zebra + LWD with persistent states (#8215)
* feat(docker): allow users to use Zebra + LWD with persistent states

* fix(docs): better documentation and compose execution

* chore(docs): better context box

* chore: do not commit further user changes to the `.env`

This can always be overriden by forcing this pushes if it was intended

* fix(compose): do not add extra volumes and ports

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2024-02-01 21:54:17 +00:00
Gustavo Valverde 18477ecbbe
feat(docker): add `docker-compose` file to run CI locally (#8209)
* ref(docker): Consolidate all tests in the `entrypoint.sh` script

* fix(ci): missing docker argument

* fix(ci): do not blank `ENTRYPOINT_FEATURES` vairable

* fix(docker): do not add `shielded-scan` as part of `TEST_FEATURES`

* chore: remove misleading comment

* feat(docker): add `docker-compose` file to run CI locally

* fix(docs): do not remove previous sections until we have a substitute for users

* fix(docs): revert some changes

* fix: typos

* ref(docker): Consolidate all tests in the `entrypoint.sh` script

* fix(ci): missing docker argument

* fix(ci): do not blank `ENTRYPOINT_FEATURES` vairable

* fix(docker): do not add `shielded-scan` as part of `TEST_FEATURES`

* chore: remove misleading comment

* Update docker/Dockerfile

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2024-01-30 19:19:44 +00:00
Gustavo Valverde 184c30e0fc
ref(docker): Consolidate all tests in the `entrypoint.sh` script (#8175)
* ref(docker): Consolidate all tests in the `entrypoint.sh` script

* fix(ci): missing docker argument

* fix(ci): do not blank `ENTRYPOINT_FEATURES` vairable

* fix(docker): do not add `shielded-scan` as part of `TEST_FEATURES`

* chore: remove misleading comment

* Update docker/Dockerfile

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2024-01-30 17:47:05 +00:00
Marek 34a3ba8c69
add(scan): Add the `scan_start_where_left` test to CI (#8172)
* Add the `scan_start_where_left` test to CI

* Fix passing of features

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2024-01-23 23:07:53 +00:00
teor 5dd33d7265
Add `ldb` RocksDB query tool to the Dockerfile (#8074)
* Add `ldb` RocksDB query tool to the Dockerfile

* Add shielded scanning link to run.md

* Add `ldb` tool instructions to install.md

* Add a rough framework for shielded-scan.md

* Add security warning and feature limitations to shielded-scan.md
2023-12-08 00:14:54 +00:00
teor 71a9865b6c
remove(cmd): Remove `zebrad download` command, because it no longer does anything (#7819)
* Remove the `zebrad download` command

* Reorder command.rs imports

* Remove zcash-params/Dockerfile and the main Dockerfile commands that use it

* Stop building zcash-params Docker images in CI

* Update CHANGELOG for `zebrad download` removal

* Clarify why the image is smaller

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2023-10-25 23:42:27 +00:00
teor ae10eafb57
Update Dockerfile features comment (#7815) 2023-10-25 02:38:34 +00:00
teor 60b447bef2
change(ci): Remove duplicate CI jobs for getblocktemplate-rpcs (#7753)
* Remove duplicate CI docker job and make features consistent

* Remove duplicate OS job

* Rename experimental release build

* Make Rust features into GitHub repository variables

* Remove redundant features in entrypoint.sh

* Remove a dependency on a deleted job

* Fix syntax of array

* Another fix attempt

* Undo some accidental merge overwrites

* Add missing space

* Explain how default is implemented

* Fix missing --features and quoting

* We can fix this later

* Use vars directly in with: blocks

* Use correct features for fake activation heights test
2023-10-22 15:40:10 +00:00
Gustavo Valverde fc0133e886
ref(workflows): consolidate workflows based on their purpose (#7616)
* ref(workflows): consolidate workflows based on their purpose

This also renames the workflows to make their naming more consistent and adding a naming convention

Fixes: #6166
Fixes: #6167

* fix(workflows): use correct name for patch

* fix(workflow): docker unit tests

* fix(release): validation error

Error:
```
Validation Failed: {"resource":"Release","code":"invalid","field":"target_commitish"}
```

Fixes: https://github.com/release-drafter/release-drafter/issues/1125

* fix(workflows): reference correct name

* fix: remove extra workflow

* fix(workflows): use larger runners

* fix(workflow): remove code already in docker unit-test

* fix(unit-tests): start zebra the right way

* fix: typo in patch name

* chore: move job to logical order

* imp(workflows): use better name for gcp tests

* add: missing merge changes

* chore: use better name for find-disks

* fix(ci): use the `entrypoint.sh` to change the Network

* fix(ci): add missing `ZEBRA_CONF_PATH` variable

* fix(ci): allow to build the entrypoint file with testnet

* fix(entrypoint): allow to create a dir and file with a single variable

* refactor: test config file in CI and CD with a reusable workflow

* fix(ci): wrong name used

* fix(ci): use checkout

* fix(ci): improve docker config tests

* fix(ci): use better name for protection rules

* Fix changed workflow file name in docs patch file

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Arya <aryasolhi@gmail.com>

* fix(cd): depend on file tests

* fix(docs): adapt to new workflow name

* fix: revert test coverage on CD

* chore: reduce diff

* fix(ci): allow using variable images for reusable workflows

* fix(dockerfile): use variables or default for config path and file

* fix(entrypoint): if `$ZEBRA_CONF_PATH` is set, do not override it

* Fix patch job names and remove failure job testnet dependencies

---------

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Arya <aryasolhi@gmail.com>
2023-10-18 06:16:02 +00:00
Gustavo Valverde fbd6b0f8b3
ref(docker): combine test and prod entrypoints into one (#7660)
* ref(docker): combine test and prod entrypoints into one

This makes the following changes
- Uses `-x` instead of echoing the variables values
- Sets default values where required
- Create a function to list directories
- Create a function to run cargo tests
- Use a better approach to handle different options in the `case` manegement for tests and production
- Replaces all instances of `runtime-entrypoint.sh` with `entrypoint.sh`

* chore: use 2 spaces insted of 4 for indentation

* fix: add missing `test-threads` to `fully_synced_rpc_`

* fix: allow 3 arguments instead of 2 to handle `fully_synced_rpc_` correctly

* chore: remove extra file

* chore: add comments to main parts of the file

* fix(ci): just create the `$ZEBRA_CONF_PATH` if `$ENTRYPOINT_FEATURES` is not set

* fix(entrypoint): just print ${ZEBRA_CONF_PATH} if exists

* fix: missing condition

* ref: handle tests better if `$ENTRYPOINT_FEATURES` is set

* fix(ci): We just want `ZEBRA_CONF_PATH` to be set in the `release` image

* fix(entrypoint): fix the overall `case` logic

* fix(ci): allos to run with custom config in CI image

* fix(ci): more edgecases

* fix: we don't need to find files, but subdirectories

* fix(ci): handle Signal Forwarding and exit codes for `cargo`

* fix(ci): parse `fully_synced_rpc_` test correctly

* chore: add missing cache dir variable for LWD

* fix(entrypoint): handle an arbitrary number of arguments

* fix(entrypoint): handle features list

* fix(entrypoint): typo

* chore: typo
2023-10-11 19:04:45 +00:00
Alfredo Garcia 80da28b3e3
tests(rpc): Add fixed test vectors for `z_getsubtreesbyindex` from zcashd to zebra (#7515)
* add fixed test vectors for `z_getsubtreesbyindex`

* change to snapshots

* add test to docker

* remove assert lines from snapshots

* add more tests

* change test description

* change test name

* run both tests together

* wait for state version update in test

* Run one test at a time to avoid state locking issues

---------

Co-authored-by: teor <teor@riseup.net>
2023-09-20 23:40:21 +00:00
Gustavo Valverde ff60260464
fix(build): reintroduce `RUST_LOG` info (#7404)
Not having the default `info` level causes some tests to not output the needed logs to debug information.
2023-08-29 07:45:26 +00:00
Gustavo Valverde bf6832f53a
fix: remove reference to old `zealous-zebra` project (#7303)
* fix: remove reference to old `zealous-zebra` project

* fix(build): use `edge` tag from our repositories
2023-08-08 10:24:48 +00:00
teor a61d464d4d
Fix Docker git commit handling (#7265) 2023-07-20 11:50:13 +00:00
Gustavo Valverde 3faef29d23
ref(docker): organize `ENV` and `ARG` values based on their usage (#7200)
* ref(docker): use a single variable for test features

* ref(docker): scope `ARG`s and `ENV`s correctly

* fix(docker): use variables as expected on test build

* fix(docker): use correct `$RPC_PORT` validation

* revert(docker): revert to using extra `ENTRYPOINT_FEATURES`

* fix(rust): missing features replacements

* fix(docker): enable backtraces for errors and panics

This is a costly function!!

* ref(docker): remove `$NETWORK` as an `ARG`

* fix typo

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2023-07-17 20:03:17 +00:00
Marek 32dd6180de
add(Docker): Enable miners to set their address for mining rewards. (#7178)
* Enable RPC port in general scenarios

* Add `mining.miner_address` to runtime entrypoint
2023-07-10 10:46:52 +00:00
Marek 776c7c7cd2
Enable RPC port in general scenarios (#7177) 2023-07-10 06:07:32 +00:00
Marek 1014e3c785
Add default RPC port depending on Zcash network (#7162)
The `runtime-entrypoint.sh` uses the `RPC_PORT` env var when the user
specifies the `getblocktemplate-rpc` feature, but this env var is unset
unless the user sets it. This commit sets the default values for
`RPC_PORT` depending on `NETWORK`.
2023-07-06 23:23:47 +00:00
Marek 00dd110265
Use standard syntax for Dockerfile ENV instruction (#7155) 2023-07-05 23:01:22 +00:00
teor aa8489373e
Use release images for zcash-params downloads (#7097) 2023-06-29 20:19:15 +00:00
Gustavo Valverde 8c90f65391
refactor(docker): allow more flexible zebra configuration (#7045)
* fix(docker): use `entrypoint.sh` as default for users

* ref(entrypoint): allow more flexible configurations

This changes allow users to:
- Mount their own configuration file
- Allow for Zebra to be exposed outside the container or not
- Allow the user to turn off sync
- Allow to enable `metrics` and `tracing`, exposing them or not

Having the `-x` option prints variable expasions, so we don't have to echo each value.

* chore(docker): remove unused ARGs from the Dockerfile

ARGs are not available at build time, so we don't require this ARGs as their ENV variables counterparts are being set in the `entrypoint`, at runtime.

* revert: keep old naming

* fix: renaming mistake :)

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* fix(docker): revert some breaking changes

* imp(docker): allow more flexibility with FEATURES config

* chore(docker): remove confusing port on `EXPOSE`

* chore(docker): remove unused command

* fix(docker): handle quotes while building the conf file

---------

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-06-28 02:36:07 +00:00
teor 62b4fa21a2
change(docker): Replace the zcash-params Dockerfile build with a zebrad binary copy (#7054)
* Replace the zcash-params Dockerfile build with a zebrad binary copy

* Update zcash-params workflow conditions
2023-06-28 02:34:46 +00:00
teor 76a7ff45a9
fix(deps): Replace openssl with rustls in tests and experimental features (#7047)
* Remove openssl dependency and prevent it coming back

* Put the arguments in the right place

* Put comment in the right place

* Add a default-docker feature to zebrad and use it in workflows and Docker files

* Fix a comment typo

* Make sure that Docker production builds don't use openssl

* Rename feature to default-release-binaries
2023-06-26 05:44:19 +00:00