Commit Graph

60 Commits

Author SHA1 Message Date
Gustavo Valverde db48daea05
fix(docker): add default CMD to test stage in Dockerfile (#9334)
* fix(docker): set `HOME` environment variable explicitly in Dockerfile

The HOME environment variable was defaulting to /root when the container started, causing cache directories to be incorrectly set up under /root/.cache/zebra instead of /home/zebra/.cache/zebra. This explicit setting ensures the HOME environment variable is correctly set to the zebra user's home directory.

* fix(docker): add default `CMD` to test stage in Dockerfile

When running the container using docker-compose without explicitly
providing a command, the entrypoint.sh script was attempting to execute
`exec_as_user` with no arguments, resulting in a gosu error:

```
exec_as_user
exec gosu 10001:10001
Usage: gosu user-spec command [args]
```

By adding `CMD ["cargo", "test"]` to the test stage in the Dockerfile,
we ensure a default command is available for the entrypoint script to
execute, preventing the gosu error when no command is explicitly provided.

This fix allows `docker-compose.test.yml` to run successfully without
needing to specify a command in the service definition.
2025-03-31 21:28:25 +00:00
Gustavo Valverde 32de8ef5fd
fix(docker): set `HOME` environment variable explicitly in Dockerfile (#9333)
The HOME environment variable was defaulting to /root when the container started, causing cache directories to be incorrectly set up under /root/.cache/zebra instead of /home/zebra/.cache/zebra. This explicit setting ensures the HOME environment variable is correctly set to the zebra user's home directory.
2025-03-22 15:25:30 +00:00
Marek f0140a4cd8
fix(ci): Better permission and cache dirs handling in Docker (#9323)
* Use gosu only once

* Remove `COLORBT_SHOW_HIDDEN`

* Simplify Dockerfile

* Remove `check_directory_files` from entrypoint

* Remove check for `ZEBRA_CONF_PATH` in entrypoint

* Simplify ownership setting for `ZEBRA_CACHE_DIR`

* Simplify ownership setting for `LOG_FILE`

* Refactor Dockerfile & entrypoint

* Refactor vars in Dockerfile

* fmt

* Use `chown` for `ZEBRA_CONF_PATH`

* `run_cargo_test` -> `run_test`

* Make `run_test` runnable with gosu

* Cosmetics

* Don't pre-compile Zebra

* Revert: "Don't pre-compile Zebra"

* Fix the custom conf test

* Reintroduce `CARGO_HOME` in Dockerfile

* Pass `FEATURES` as env var to entrypoint

* Fix ARGs in Dockerfile

* Revert "Remove `COLORBT_SHOW_HIDDEN`"

This reverts commit 960d5ca308.

* Specify cache state dir in CI

* Specify lwd cache dir in CI

* refactor: reorganize variables and avoid running entrypoint commands in subshell (#9326)

* refactor(docker): improve container configuration and security

- Optimize Dockerfile build stages and environment variables
- Improve file operations with proper ownership
- Streamline entrypoint script privilege management

* refactor(docker): enhance user management and directory ownership

- Add HOME argument back to ensure proper user home directory setup
- Implement ownership change for the user's home directory

* refactor(docker): remove redundant cache directory setup

- Eliminate explicit creation and ownership setting for LWD and Zebra cache directories in Dockerfile.
- Introduce default values for cache directories in entrypoint script, allowing for environment variable overrides.

* fix: run all cargo commands as user

* chore: reduce diff

* fix: revert to more robust command array

---------

Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
Co-authored-by: Gustavo Valverde <g.valverde02@gmail.com>
2025-03-18 08:58:19 +00:00
Gustavo Valverde 49741e8b47
fix(ci): Add Rust environment variables to Docker test workflows (#9318)
* feat(ci): Add Rust environment variables to Docker test workflows

Enhance test workflows by adding Rust-specific environment variables:
- Include RUST_LOG for logging configuration
- Add RUST_BACKTRACE and RUST_LIB_BACKTRACE for improved error tracing
- Include COLORBT_SHOW_HIDDEN for detailed backtraces
- Add CARGO_INCREMENTAL for build performance optimization

These changes improve debugging capabilities and provide more flexibility in test environments across dockerized CI jobs.

* feat(ci): Add CARGO_INCREMENTAL build argument to Docker workflow

Enable configurable Rust incremental compilation in Docker builds by:
- Adding CARGO_INCREMENTAL as a build argument in GitHub Actions workflow
- Setting a default value of 0 in Dockerfile to control build performance
- Integrating with existing Rust environment variable configuration

This change provides more granular control over Rust compilation strategies in containerized builds.

* fix(docker): resolve user creation issues in Dockerfile

- Move WORKDIR after user creation to prevent home directory ownership issues
- Properly set environment variables for UID, GID, HOME, and USER in each stage
- Reorganize Dockerfile to ensure home directory is created after user setup
- Fix interactive prompts during adduser by ensuring proper directory ownership

* fix(docs): Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2025-03-12 10:43:31 +00:00
Marek de7e5b547f
refactor(docker): allow r/w access in mounted volumes (#9281)
* Switch to a non-privileged user in tests

* Change test env setup

* Remove unneeded ARGs

* Simplify UID & GID handling in `runtime` target

* Simplify docs

* refactor(docker): Improve user and permission handling in Dockerfiles

- Add gosu for flexible non-root user execution
- Enhance user and group creation with configurable UID/GID
- Modify entrypoint script to support dynamic user switching
- Improve cache and log directory permission management
- Update comments to clarify user and permission strategies

* refactor(docker): Improve Zebra config file handling in entrypoint script

- Enhance error handling for missing config file (now exits with error)
- Simplify config preparation logic by removing redundant file copying
- Update comments to reflect new config file handling approach
- Ensure consistent use of ZEBRA_CONF_PATH throughout the script

* refactor(docker): Enhance container user security and configuration

- Increase UID/GID to 10001 to minimize host system user conflicts
- Remove `--system` flag from user and group creation to prevent potential issues
- Add detailed comments explaining UID/GID selection rationale
- Improve security by using high UID/GID values to reduce namespace collision risks
- Remove redundant `chmod` for entrypoint script

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Gustavo Valverde <g.valverde02@gmail.com>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2025-03-03 19:21:03 +01:00
Marek 38b7313ea1
chore: bump Rust from 1.84.0 to 1.85.0 in Docker (#9290) 2025-02-24 14:35:39 +00:00
Marek ac25192afc
docs(docker): update examples for running Zebra in Docker (#9269)
* Rm `.env` files

* Update `mining-docker.md`

* Revert "Rm `.env` files"

This reverts commit caaa4559c3.

* Add `enable_cookie_auth` to default Zebra conf

* Rename `default_zebra_config.toml`

* fmt `prometheus.yaml`

* Update `docker/test.env`

* Update `docker/.env`

* Refactor `docker compose` for lwd

* Enable disabling cookie authentication

* Update `docker compose` for tests

* Update general `docker compose`

* Update docs for running Zebra in Docker

* Add example `docker compose` file for Grafana

* Fix a bug in an example command

* Refactor test execution logic in entrypoint

* Rename `v2.1.0.toml` conf to `custom-conf.toml`

* Fix CI tests for loading of custom conf files

* Use the new conf file name in CI checks

* Use an extended regexp for custom conf CI check
2025-02-19 17:44:06 +00:00
Marek 4132c0e4e9
ci: Refactor Dockerfile & entrypoint (#8923)
* Refactor formatting & docs

* Refactor the `runtime` stage in Dockerfile

* Remove unused code from `entrypoint.sh`

* Simplify `entrypoint.sh` setup

* Revise docs & formatting

* Adjust default values for env vars

* Bump Rust v from 1.79 to 1.81 in Dockerfile

* Refactor `entrypoint.sh`

* Refactor `Dockerfile`

* Add TODOs for monitoring stage to Dockerfile

* Refactor `Dockerfile`

* Add TODOs for monitoring stage to Dockerfile

* Fix a typo

* Allow running `zebrad` in test mode

* Allow custom config for `zebrad` in test mode

* Remove `curl` from the `runtime` Docker image

* Remove redundant echos

* Remove a malfunctioning CD test

The test was using a custom config file set in `test_variables`.
However, the file was not included in the Docker image, and the
entrypoint script created a new, default one under the original file's
path. Zebra then loaded this new file, and the test passed because the
pattern in `grep_patterns` matched Zebra's output containing the
original path, even though the config file was different.

* Remove a redundant CI test

* Remove all packages from the `runtime` stage

* Docs cosmetics

* Clarify docs

* Bump Rust version

* Remove a security note

* Explicitly specify network cache dir

* Explicitly specify cookie dir

* Set UID, GID and home dir for the `zebra` user

* Set a working dir for the `zebra` user

* Don't remove `FEATURES`

* Try re-introducing the `testnet-conf` check

* `ZEBRA_CACHED_STATE_DIR` -> `ZEBRA_CACHE_DIR`

This dir doesn't hold only the state cache anymore, but also the cache
for network peers, and the cookie file.

* Refactor the dir structure

* Check that `ZEBRA_CONF_PATH` exists in the image

* Improve the check for `ZEBRA_CONF_PATH`

* Use different flag in the `ZEBRA_CONF_PATH` check

* Simplify the `ZEBRA_CONF_PATH` check

* Fix spelling

* Comment out the `testnet-conf` CI check

* Add commented out `test-zebra-conf-path` CI check

* Reintroduce `testnet-conf` CI check

* Update the `custom-conf` CI check

* Add `v2.1.0.toml` conf file

* Refine the `v2.1.0.toml` conf file

* Remove `ZEBRA_LISTEN_ADDR` from the entrypoint

* Remove `ZEBRA_CHECKPOINT_SYNC` from the entrypoint

* Stop supporting configuration of the RPC port

* Add default conf file

* Prepare Zebra's config in the entrypoint script

* Remove unneeded packages from the `deps` target

* Docs cosmetics

* Use only `$FEATURES` in entrypoint

* Simplify handling of Rust features

* Add a TODO

* Add CI debug statements

* Don't require test vars in conf test

* Reintroduce `protoc`

* Remove `-e NETWORK`

* Remove `ZEBRA_FORCE_USE_COLOR=1`

* Remove `ZEBRA_CACHE_DIR=/var/cache/zebrad-cache`

* Reintroduce the "custom-conf" test

* Set up test env the same way as prod

* Don't repeatedly check for conf file in entrypoint

* Simplify file ownership in Dockerfile

* Fix checkpoint tests in entrypoint

* Fix Zebra config CI tests

* `LIGHTWALLETD_DATA_DIR` -> `LWD_CACHE_DIR`

* Add config for `LWD_CACHE_DIR` to Dockerfile

* `/var/cache/zebrad-cache` -> `~/.cache/zebra`

* `var/cache/lwd-cache` -> `/home/zebra/.cache/lwd`

* Remove `LOG_COLOR=false` from GCP setup

* Don't specify `LWD_CACHE_DIR` in CI tests

* Don't switch to `zebra` user for tests in Docker

* Join "experimental" and "all" tests in CI

* Remove outdated docs

* Refactor tests with fake activation heights

* Fix tests for scanner
2025-02-13 09:15:32 +00:00
Arya 47b7614ea8
updates minimum supported Rust version in Zebrad and rust verison in Dockerfile (#8956) 2024-10-22 00:41:34 +00:00
Gustavo Valverde d31eea5f64
ref(docker): use cache mounts for build cache (#8796)
* ref(docker): leverage cache mount with bind mounts

This update eliminates the need for external tools like `cargo-chef` to leverage caching layers, resulting in an average build time reduction of 4m30s (~36% improvement).

While this solution doesn't fully resolve the issues mentioned in https://github.com/ZcashFoundation/zebra/issues/6169#issuecomment-1712776391, it represents the best possible approach without resorting to custom solutions, which we'd prefer to avoid.

* chore: remove extra `WORKDIR` and imp comments

* chore: improve comment legibility

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
Co-authored-by: Arya <aryasolhi@gmail.com>
2024-09-05 13:29:22 +00:00
Marek 5541b27a12
Remove `shielded-scan` from experimental features (#8827) 2024-09-03 10:04:36 +00:00
Gustavo Valverde cdb9efdb27
fix(docker): allow the `zebra` user access to relevant dirs (#8817)
* fix(docker): allow the `zebra` user access to relevant dirs

When runnning a Zebra node using Docker without a privileged user, you won't be able to modify some files and directories, not even the ones in the current directory, as the `zebra` user has no permission to `/`.

The best way to solve this is making the `/opt/zebrad` the current `WORKDIR`. This also requires moving the `entrypoint.sh` from the root `/` directory to `/etc/zebrad` as this directory is used to save configuration, and other files.

An `APP_HOME` ARG is used as not all platforms where a Docker container is deployed allows writting permissions to the `/opt` directory. This allow some users to re-build the image with a custom `WORKDIR`

* fix(docker): allow starting the container without a `zebrad` command

As `gosu` is just required and available in our `runtime` image, trying to run `docker run -it --rm  --name tests -t zfnd/zebra:<pr> /bin/bash` in other stages will fail, as `gosu` is not available.
2024-08-29 19:57:17 +00:00
Gustavo Valverde ec85aa8a48
fix(docker): add `gosu` and remove unsupported flag in `adduser` (#8808)
* fix(docker): typo and uknown option in debian

* fix(docker): use `gosu` for rootless execution

Some of our entrypoint commands requires creating directories and files in places a non-privileged user can't access.

So we use `gosu` to step down from `root` to a non-privileged user during container startup, right at our application execution.
2024-08-27 21:29:50 +00:00
Gustavo Valverde 0d36681d8f
fix(docker): do not run the Zebra nodes with the root user (#8803) 2024-08-27 11:55:24 +00:00
Gustavo Valverde 56fca2c9ee
ref(build)!: use official lightwallted image for CI (#8323) 2024-02-29 15:50:50 +00:00
Marek 584bd23718
Update Debian from Bullseye to Bookworm (#8273) 2024-02-14 23:02:06 +00:00
Gustavo Valverde 5feb40e0b8
feat(docker): allow users to use Zebra + LWD with persistent states (#8215)
* feat(docker): allow users to use Zebra + LWD with persistent states

* fix(docs): better documentation and compose execution

* chore(docs): better context box

* chore: do not commit further user changes to the `.env`

This can always be overriden by forcing this pushes if it was intended

* fix(compose): do not add extra volumes and ports

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2024-02-01 21:54:17 +00:00
Gustavo Valverde 184c30e0fc
ref(docker): Consolidate all tests in the `entrypoint.sh` script (#8175)
* ref(docker): Consolidate all tests in the `entrypoint.sh` script

* fix(ci): missing docker argument

* fix(ci): do not blank `ENTRYPOINT_FEATURES` vairable

* fix(docker): do not add `shielded-scan` as part of `TEST_FEATURES`

* chore: remove misleading comment

* Update docker/Dockerfile

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2024-01-30 17:47:05 +00:00
teor 5dd33d7265
Add `ldb` RocksDB query tool to the Dockerfile (#8074)
* Add `ldb` RocksDB query tool to the Dockerfile

* Add shielded scanning link to run.md

* Add `ldb` tool instructions to install.md

* Add a rough framework for shielded-scan.md

* Add security warning and feature limitations to shielded-scan.md
2023-12-08 00:14:54 +00:00
teor 71a9865b6c
remove(cmd): Remove `zebrad download` command, because it no longer does anything (#7819)
* Remove the `zebrad download` command

* Reorder command.rs imports

* Remove zcash-params/Dockerfile and the main Dockerfile commands that use it

* Stop building zcash-params Docker images in CI

* Update CHANGELOG for `zebrad download` removal

* Clarify why the image is smaller

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2023-10-25 23:42:27 +00:00
teor ae10eafb57
Update Dockerfile features comment (#7815) 2023-10-25 02:38:34 +00:00
Gustavo Valverde fc0133e886
ref(workflows): consolidate workflows based on their purpose (#7616)
* ref(workflows): consolidate workflows based on their purpose

This also renames the workflows to make their naming more consistent and adding a naming convention

Fixes: #6166
Fixes: #6167

* fix(workflows): use correct name for patch

* fix(workflow): docker unit tests

* fix(release): validation error

Error:
```
Validation Failed: {"resource":"Release","code":"invalid","field":"target_commitish"}
```

Fixes: https://github.com/release-drafter/release-drafter/issues/1125

* fix(workflows): reference correct name

* fix: remove extra workflow

* fix(workflows): use larger runners

* fix(workflow): remove code already in docker unit-test

* fix(unit-tests): start zebra the right way

* fix: typo in patch name

* chore: move job to logical order

* imp(workflows): use better name for gcp tests

* add: missing merge changes

* chore: use better name for find-disks

* fix(ci): use the `entrypoint.sh` to change the Network

* fix(ci): add missing `ZEBRA_CONF_PATH` variable

* fix(ci): allow to build the entrypoint file with testnet

* fix(entrypoint): allow to create a dir and file with a single variable

* refactor: test config file in CI and CD with a reusable workflow

* fix(ci): wrong name used

* fix(ci): use checkout

* fix(ci): improve docker config tests

* fix(ci): use better name for protection rules

* Fix changed workflow file name in docs patch file

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Arya <aryasolhi@gmail.com>

* fix(cd): depend on file tests

* fix(docs): adapt to new workflow name

* fix: revert test coverage on CD

* chore: reduce diff

* fix(ci): allow using variable images for reusable workflows

* fix(dockerfile): use variables or default for config path and file

* fix(entrypoint): if `$ZEBRA_CONF_PATH` is set, do not override it

* Fix patch job names and remove failure job testnet dependencies

---------

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Arya <aryasolhi@gmail.com>
2023-10-18 06:16:02 +00:00
Gustavo Valverde fbd6b0f8b3
ref(docker): combine test and prod entrypoints into one (#7660)
* ref(docker): combine test and prod entrypoints into one

This makes the following changes
- Uses `-x` instead of echoing the variables values
- Sets default values where required
- Create a function to list directories
- Create a function to run cargo tests
- Use a better approach to handle different options in the `case` manegement for tests and production
- Replaces all instances of `runtime-entrypoint.sh` with `entrypoint.sh`

* chore: use 2 spaces insted of 4 for indentation

* fix: add missing `test-threads` to `fully_synced_rpc_`

* fix: allow 3 arguments instead of 2 to handle `fully_synced_rpc_` correctly

* chore: remove extra file

* chore: add comments to main parts of the file

* fix(ci): just create the `$ZEBRA_CONF_PATH` if `$ENTRYPOINT_FEATURES` is not set

* fix(entrypoint): just print ${ZEBRA_CONF_PATH} if exists

* fix: missing condition

* ref: handle tests better if `$ENTRYPOINT_FEATURES` is set

* fix(ci): We just want `ZEBRA_CONF_PATH` to be set in the `release` image

* fix(entrypoint): fix the overall `case` logic

* fix(ci): allos to run with custom config in CI image

* fix(ci): more edgecases

* fix: we don't need to find files, but subdirectories

* fix(ci): handle Signal Forwarding and exit codes for `cargo`

* fix(ci): parse `fully_synced_rpc_` test correctly

* chore: add missing cache dir variable for LWD

* fix(entrypoint): handle an arbitrary number of arguments

* fix(entrypoint): handle features list

* fix(entrypoint): typo

* chore: typo
2023-10-11 19:04:45 +00:00
Gustavo Valverde ff60260464
fix(build): reintroduce `RUST_LOG` info (#7404)
Not having the default `info` level causes some tests to not output the needed logs to debug information.
2023-08-29 07:45:26 +00:00
Gustavo Valverde bf6832f53a
fix: remove reference to old `zealous-zebra` project (#7303)
* fix: remove reference to old `zealous-zebra` project

* fix(build): use `edge` tag from our repositories
2023-08-08 10:24:48 +00:00
teor a61d464d4d
Fix Docker git commit handling (#7265) 2023-07-20 11:50:13 +00:00
Gustavo Valverde 3faef29d23
ref(docker): organize `ENV` and `ARG` values based on their usage (#7200)
* ref(docker): use a single variable for test features

* ref(docker): scope `ARG`s and `ENV`s correctly

* fix(docker): use variables as expected on test build

* fix(docker): use correct `$RPC_PORT` validation

* revert(docker): revert to using extra `ENTRYPOINT_FEATURES`

* fix(rust): missing features replacements

* fix(docker): enable backtraces for errors and panics

This is a costly function!!

* ref(docker): remove `$NETWORK` as an `ARG`

* fix typo

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2023-07-17 20:03:17 +00:00
Marek 00dd110265
Use standard syntax for Dockerfile ENV instruction (#7155) 2023-07-05 23:01:22 +00:00
Gustavo Valverde 8c90f65391
refactor(docker): allow more flexible zebra configuration (#7045)
* fix(docker): use `entrypoint.sh` as default for users

* ref(entrypoint): allow more flexible configurations

This changes allow users to:
- Mount their own configuration file
- Allow for Zebra to be exposed outside the container or not
- Allow the user to turn off sync
- Allow to enable `metrics` and `tracing`, exposing them or not

Having the `-x` option prints variable expasions, so we don't have to echo each value.

* chore(docker): remove unused ARGs from the Dockerfile

ARGs are not available at build time, so we don't require this ARGs as their ENV variables counterparts are being set in the `entrypoint`, at runtime.

* revert: keep old naming

* fix: renaming mistake :)

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* fix(docker): revert some breaking changes

* imp(docker): allow more flexibility with FEATURES config

* chore(docker): remove confusing port on `EXPOSE`

* chore(docker): remove unused command

* fix(docker): handle quotes while building the conf file

---------

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-06-28 02:36:07 +00:00
teor 76a7ff45a9
fix(deps): Replace openssl with rustls in tests and experimental features (#7047)
* Remove openssl dependency and prevent it coming back

* Put the arguments in the right place

* Put comment in the right place

* Add a default-docker feature to zebrad and use it in workflows and Docker files

* Fix a comment typo

* Make sure that Docker production builds don't use openssl

* Rename feature to default-release-binaries
2023-06-26 05:44:19 +00:00
teor 8861de6a7c
fix(docker): Stop resetting the `cargo-chef` cache in the Dockerfile (#6934)
* Fix Dockerfile cache use

* Remove cache-breaking COPY commands

* Use git to only reset files modified by cargo-chef

* Copy .git and sources before cargo chef cook

* Update .dockerignore to include .git

* Don't use .git

* Use rsync instead of git

* Maybe COPY is needed

* Actually copy changed files using rsync

* Actually copy the files using the correct rsync syntax

* Remove ls commands from Dockerfile
2023-06-22 04:16:50 +00:00
teor 912693bf0a
Fix Dockerfile cache use (#6933) 2023-06-14 22:15:51 +00:00
Gustavo Valverde 17d36ffc7a
fix(deploy): allow the container to raise in MIGs (#6893)
* fix(deploy): allow the container to raise in MIGs

* fix(docker): add the `ZEBRA_CACHED_STATE_DIR` as a default `ENV`

This no longer requires the env variable to be defined in other places, unless we're changing the default configuration
2023-06-09 19:16:39 +00:00
teor 89bf875744
fix(deployment): Fix log file path and log colour (#6890)
* Fix CD log file path handling

* Variables don't substitute

* Disable color escapes in Google Cloud logs

* Use correct elif syntax
2023-06-09 07:41:09 +00:00
teor d9add4a01f
change(cd): Deploy testnet instances for every main branch push and release (#6842) 2023-06-08 15:44:30 +10:00
Marek 5421447804
Listen on `0.0.0.0` instead of `127.0.0.1` (#6755)
Binding `127.0.0.1` means that Zebra will accept inbound connections
coming only from the loopback network interface. This is desirable as
long as Zebra runs on a native machine.

When Zebra runs inside a Docker container, incoming connections coming
from the host machine don't come from the container's loopback
interface. In order to be able to connect to Zebra from the host
machine, we can listen on `0.0.0.0` so Zebra accepts inbound connections
coming from any interface. Users then can limit inbound connection to
the loopback of their host by

```bash
docker run -p 127.0.0.1:8232:8232 zfnd/zebra:1.0.0-rc.8
```
2023-05-24 16:56:22 +00:00
teor 1461c912f9
change(ci): Generate mainnet checkpoints in CI (#6550)
* Add extra test type modes to support zebra-checkpoints

* Add Mainnet and Testnet zebra-checkpoints test harnesses

* Add zebra-checkpoints to test docker images

* Add zebra-checkpoints test entrypoints

* Add Mainnet CI workflow for zebra-checkpoints

* Enable zebra-checkpoints feature in the test image

* Use the same features for (almost) all the docker tests

* Make workflow features match Docker features

* Add a feature note

* Add a zebra-checkpoints test feature to zebrad

* Remove the "no cached state" testnet code

* Log a startup message to standard error when launching zebra-checkpoints

* Rename tests to avoid partial name conflicts

* Fix log formatting

* Add sentry feature to experimental docker image build

* Explain what ENTRYPOINT_FEATURES is used for

* Use the correct zebra-checkpoints path

* Silence zebrad logs while generating checkpoints

* Fix zebra-checkpoints log handling

* Re-enable waiting for zebrad to fully sync

* Add documentation for how to run these tests individually

* Start generating checkpoints from the last compiled-in checkpoint

* Fix clippy lints

* Revert changes to TestType

* Wait for all the checkpoints before finishing

* Add more stderr debugging to zebra-checkpoints

* Fix an outdated module comment

* Add a workaround for zebra-checkpoints launch/run issues

* Use temp dir and log what it is

* Log extra metadata about the zebra-checkpoints binary

* Add note about unstable feature -Z bindeps

* Temporarily make the test run faster and with debug info

* Log the original test command name when showing stdout and stderr

* Try zebra-checkpoints in the system path first, then the cargo path

* Fix slow thread close bug in dual process test harness

* If the logs are shown, don't say they are hidden

* Run `zebra-checkpoints --help` to work out what's going on in CI

* Build `zebra-utils` binaries for `zebrad` integration tests

* Revert temporary debugging changes

* Revert changes that were moved to another PR
2023-04-27 04:39:43 +00:00
Arya 671420bd84
feat(release): create Docker hub binary with mining enabled on release (#6228)
* Duplicates Dockerfile

* updates mining-testnet Dockerfile with getblocktemplate-rpcs feature, Testnet by default, and an RPC port

* renames mining-testnet.Dockerfile and adds workflow for publishing images on release

* replaces space-seperated features with commas

* Adds .experimental tag suffix, removes new dockerfile, makes lightwalletd tests conditional

* updates build-args to pass on features directly

* adds "lightwalletd-grpc-tests" as default test_features in build-docker-image

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* adds tag suffix to cache keys

---------

Co-authored-by: teor <teor@riseup.net>
2023-03-24 07:10:40 +00:00
Gustavo Valverde 85bcbbdf16
ci: add a test to check that the Docker image config works (#5968)
* ci: add a test to validate Zebra's config file and path

* fix: use `ZEBRA_CONF_PATH` as single variable locating the conf

* fix: do not remove the containers

* fix: use extended regex

* fix: use different steps to validate the conf tests

* fix: do not specify a default CMD for running Docker in test builds

* fix: use actual starting commands for entrypoint

* fix: do not add cargo twice if cargo is in $1

* fix: allow to run `zebrad` in the `tests` stage of Dockerfile

* fix: new entrypoint does not allow an empty CMD

* fix: do not duplicate the `zebrad` command

* fix: segregate configuration jobs

* refactor(entrypoint): handle better parameters conditions

* fix: make `zebrad` an executable command in `tests` stage

* Show the commands that are being executed in the new docker test

* Show full logs without tee or grep

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* fix: use the actual path inside docker

* fix: use `grep` with exit code

If the container is logging to stderr, piping works only for stdout, so we're adding `2>&1`

* fix: use `grep -q` to get an exit code

* fix: fail if any error is detected

* fix: fail if this test takes more than 5 minutes

* fix: update patch workflows

* feat: test Dockerfile `runtime` config

* fix: depend on the configuration test to continue

Co-authored-by: teor <teor@riseup.net>
2023-01-23 06:41:59 +00:00
Gustavo Valverde bbbd56d7ce
fix(build): restore Rust stable in builds and beta for CI (#5515)
* Revert "fix(ci): Build zebrad with Rust 1.63 to avoid Zcash parameter download hangs (#5251)"

This reverts commit 9cb6c559f4.

* Revert "Require Rust 1.63 in the README (#5359)"

This reverts commit ee0edef857.

* Revert "Revert "ci(build): re-enable Rust beta tests in ubuntu (#5024)" (#5090)"

This reverts commit 579d6be4e9.
2022-11-01 10:51:29 +10:00
teor 469c471d92
Use correct TOML syntax in Docker zebrad.toml (#5320) 2022-10-04 02:51:43 +00:00
teor f71bb74951
fix(docker): Make default command work in docker images, disable optional listener ports (#5313)
* Disable optional listener ports so the default config is secure

* Fix Zebra config file path in Dockerfile
2022-10-03 17:50:27 +00:00
teor 9cb6c559f4
fix(ci): Build zebrad with Rust 1.63 to avoid Zcash parameter download hangs (#5251)
* Try running coverage with Rust 1.63

* Run GitHub Actions tests with Rust 1.63

* Change from stable to 1.63 in the patch file

* Use Rust 1.63 to download Zcash parameters

* Use Rust 1.63 to build Docker zebrad images

* Make Rust 1.63 a supported platform, and make stable temporarily unsupported
2022-09-26 12:37:23 +00:00
Gustavo Valverde 726f732640
fix(build): a path must exist before creating a file in it (#5177)
* fix(build): a path must exist before creating a file in it

Other directories in the Dockerfile were automatically created with the `WORKDIR` and  `COPY` commands. In this case it has to be explicitly created

* fix(build): use a variable for the conf path and another for the file

Add some pending actions to consider, for a better user experience.

* fix(runtime): run with the correct path
2022-09-15 22:26:32 +00:00
Gustavo Valverde e48e8e5572
chore(build): allow a custom `zebrad` config file path (#5163)
* chore(build): allow a custom `zebrad` config file path

Previous behavior:
The zebrad config file was hardcoded into the Dockerfile, allowing to
change specific values, but not the config file as a whole

Expected behavior:
Allow the user to specify the config file themselves, for example with
the nginx image you can pass -v /host/path/nginx.conf:/etc/nginx/nginx.conf:ro

Solution:
Use and ARG, with a default value, to allow the user to change the location
of the zebrad configuration file. This also sets the location under /etc,
honoring The Filesystem Hierarchy Standard (FHS) for linux

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2022-09-15 09:05:24 +00:00
teor f804ff6238
fix(docker): Copy lightwalletd from the correct path during Docker builds (#4886)
* Copy lightwalletd from the updated Docker path

* Try the build path instead

* Fix the build stage entrypoint path
2022-08-04 06:29:49 +00:00
Gustavo Valverde 6ff0a42318
refactor(build): use FHS for deployments and artifacts (#4786)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-08-03 23:43:32 +00:00
teor 29e73b3f3e
breaking(diagnostics): make zebrad diagnostics into optional compile-time features (#4539)
* Disable the flamegraph feature by default at compile time

* Disable the journald feature by default at compile time

* Also disable inferno dependency, and rearrange features

* Disable the prometheus feature by default at compile time

* Disable the tracing filter reload feature by default at compile time

* Disable tests when corresponding features are disabled

* Add compile-time tracing features to user docs

* Add compile-time features to the metrics user docs

* Document diagnostics as part of the start command tasks and services

* breaking(diagnostics): rename "enable-sentry" feature to "sentry" (#4623)

* Also skip conflict tests when those ports are disabled

* breaking(diagnostics): rename "enable-sentry" feature to "sentry"

This is mostly:
```sh
fastmod enable-sentry sentry
```

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-16 19:56:40 +00:00
Dimitris Apostolou 59ae77d04b
Fix typos (#4397) 2022-05-16 05:33:08 +00:00
teor f789f8aaa7
fix(ci): make test selection and logging consistent (#4375)
* fix(ci): sentry is not longer being activated in test builds

This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.

* fix(build): workaround the failed to fetch oauth token error

* Drop sentry dependencies when enable-sentry feature is disabled

* Make lightwalletd gRPC tests depend on a new lightwalletd-grpc-tests feature

* fix(ci): remove enable-sentry feature from tests

* Add lightwalletd-grpc-tests feature for functionality or efficiency

And document where it is just used to stop re-compilations.

* Remove redundant `cmake` and `protobuf-compiler` dependencies

* Document Zebra's optional production and test feature flags

* Minimise dependencies in zcash-params/Dockerfile

* Minimise dependencies in docker/Dockerfile

* Add a workflow TODO

* Catch more errors in entrypoint.sh

Also makes entrypoint.sh compatible with more distributions

* Remove unnecessary quoting in entrypoint.sh

* Use exactly the same arguments to call CI tests

* Remove a redundant CI build

* Rename Cargo.lock check job

Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-05-11 23:41:33 +00:00