* refactor!: remove experimental features from release builds
BREAKING CHANGE: Experimental features are no longer built into release images.
Users must now build custom images with specific features if needed.
- Remove experimental build job from release workflow
- Remove experimental features from CI matrix
- Update documentation to reflect new feature strategy
- Simplify Dockerfile by removing experimental feature args
- Keep RUN_ALL_EXPERIMENTAL_TESTS for testing purposes only
Migration: Users who need specific features should:
1. Build custom images using --build-arg FEATURES="feature1 feature2"
2. Use the documented feature flags in their builds
3. Refer to the new documentation for building with custom features
Closes #9210Closes#9204
Partially #7415
* revert(ci): Add support for experimental features in test runs
Enable passing experimental features to Docker test runs by:
- Updating workflow configuration to pass EXPERIMENTAL_FEATURES
- Modifying entrypoint script to include experimental features during test execution
- Documenting new experimental test category in workflow README
* refactor(docs): streamline Docker documentation and feature testing strategy
- Reorganized and clarified advanced usage instructions for building Docker images with custom features, including metrics support.
- Enhanced the presentation of alternatives and configuration details for Docker usage.
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
* chore: revert deleted sentence
* Update book/src/user/docker.md
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
This also updates to the latest `zcash_client_backend` dependency, and
fixes a few imports to use base types instead of reexported versions
that may be deprecated in the future.
* fix(docker): set `HOME` environment variable explicitly in Dockerfile
The HOME environment variable was defaulting to /root when the container started, causing cache directories to be incorrectly set up under /root/.cache/zebra instead of /home/zebra/.cache/zebra. This explicit setting ensures the HOME environment variable is correctly set to the zebra user's home directory.
* fix(docker): add default `CMD` to test stage in Dockerfile
When running the container using docker-compose without explicitly
providing a command, the entrypoint.sh script was attempting to execute
`exec_as_user` with no arguments, resulting in a gosu error:
```
exec_as_user
exec gosu 10001:10001
Usage: gosu user-spec command [args]
```
By adding `CMD ["cargo", "test"]` to the test stage in the Dockerfile,
we ensure a default command is available for the entrypoint script to
execute, preventing the gosu error when no command is explicitly provided.
This fix allows `docker-compose.test.yml` to run successfully without
needing to specify a command in the service definition.
* Adds a new trait for disk format upgrades, implements in on a new struct, `PruneTrees`, and moves the logic for tree deduplication to the trait impl
* refactors add subtrees format upgrade to use new trait
* refactors fix tree keys, cache genesis roots, and value balance upgrades to use new trait
* Applies suggestions from code review:
- Avoids duplicate validation of format upgrades at startup when db is already upgraded,
- Minor refactors
- Doc fixes and cleanups
* Applies suggestions from code review
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Adds reconsider_block method and appropriate test cases
* Refactors invalidated_blocks state to use IndexMap. Returns error from validate_and_commit if a candidate block's hash is in the map of invalidated blocks. Stores invalidated_blocks by height and clears when finalizing. Checks against non finalized tip hash to create a new chain if parnt_chain doesn't exist. Renames ReconsiderError variant NonPreviouslyInvalidatedBlock to MissingInvalidatedBlock.
* Moves MAX_INVALIDATED_BLOCKS constant to constants.rs
* Maintains invalidate_blocks cleanup in reconsider_block and finalize(). Removes unused ReconsiderError variant. Opts to refuse block consideration if parent_chain does not exist. Adds db handle to reconsider_block function. Edits max blocks constant documentation
* Checks the finalized state first to create a new chain from non finalized blocks only before checking parent_chain.
The HOME environment variable was defaulting to /root when the container started, causing cache directories to be incorrectly set up under /root/.cache/zebra instead of /home/zebra/.cache/zebra. This explicit setting ensures the HOME environment variable is correctly set to the zebra user's home directory.
* Use gosu only once
* Remove `COLORBT_SHOW_HIDDEN`
* Simplify Dockerfile
* Remove `check_directory_files` from entrypoint
* Remove check for `ZEBRA_CONF_PATH` in entrypoint
* Simplify ownership setting for `ZEBRA_CACHE_DIR`
* Simplify ownership setting for `LOG_FILE`
* Refactor Dockerfile & entrypoint
* Refactor vars in Dockerfile
* fmt
* Use `chown` for `ZEBRA_CONF_PATH`
* `run_cargo_test` -> `run_test`
* Make `run_test` runnable with gosu
* Cosmetics
* Don't pre-compile Zebra
* Revert: "Don't pre-compile Zebra"
* Fix the custom conf test
* Reintroduce `CARGO_HOME` in Dockerfile
* Pass `FEATURES` as env var to entrypoint
* Fix ARGs in Dockerfile
* Revert "Remove `COLORBT_SHOW_HIDDEN`"
This reverts commit 960d5ca308.
* Specify cache state dir in CI
* Specify lwd cache dir in CI
* refactor: reorganize variables and avoid running entrypoint commands in subshell (#9326)
* refactor(docker): improve container configuration and security
- Optimize Dockerfile build stages and environment variables
- Improve file operations with proper ownership
- Streamline entrypoint script privilege management
* refactor(docker): enhance user management and directory ownership
- Add HOME argument back to ensure proper user home directory setup
- Implement ownership change for the user's home directory
* refactor(docker): remove redundant cache directory setup
- Eliminate explicit creation and ownership setting for LWD and Zebra cache directories in Dockerfile.
- Introduce default values for cache directories in entrypoint script, allowing for environment variable overrides.
* fix: run all cargo commands as user
* chore: reduce diff
* fix: revert to more robust command array
---------
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
Co-authored-by: Gustavo Valverde <g.valverde02@gmail.com>
* feat(ci): Add Rust environment variables to Docker test workflows
Enhance test workflows by adding Rust-specific environment variables:
- Include RUST_LOG for logging configuration
- Add RUST_BACKTRACE and RUST_LIB_BACKTRACE for improved error tracing
- Include COLORBT_SHOW_HIDDEN for detailed backtraces
- Add CARGO_INCREMENTAL for build performance optimization
These changes improve debugging capabilities and provide more flexibility in test environments across dockerized CI jobs.
* feat(ci): Add CARGO_INCREMENTAL build argument to Docker workflow
Enable configurable Rust incremental compilation in Docker builds by:
- Adding CARGO_INCREMENTAL as a build argument in GitHub Actions workflow
- Setting a default value of 0 in Dockerfile to control build performance
- Integrating with existing Rust environment variable configuration
This change provides more granular control over Rust compilation strategies in containerized builds.
* fix(docker): resolve user creation issues in Dockerfile
- Move WORKDIR after user creation to prevent home directory ownership issues
- Properly set environment variables for UID, GID, HOME, and USER in each stage
- Reorganize Dockerfile to ensure home directory is created after user setup
- Fix interactive prompts during adduser by ensuring proper directory ownership
* fix(docs): Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* docs(decisions): add architectural decision records structure
Create a structured decision records system to document important technical choices across multiple domains (DevOps, Network, Consensus, etc.).
This implements a modified MADR template approach for preserving context, trade-offs, and reasoning behind significant architectural decisions.
* fix(docs): suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* Switch to a non-privileged user in tests
* Change test env setup
* Remove unneeded ARGs
* Simplify UID & GID handling in `runtime` target
* Simplify docs
* refactor(docker): Improve user and permission handling in Dockerfiles
- Add gosu for flexible non-root user execution
- Enhance user and group creation with configurable UID/GID
- Modify entrypoint script to support dynamic user switching
- Improve cache and log directory permission management
- Update comments to clarify user and permission strategies
* refactor(docker): Improve Zebra config file handling in entrypoint script
- Enhance error handling for missing config file (now exits with error)
- Simplify config preparation logic by removing redundant file copying
- Update comments to reflect new config file handling approach
- Ensure consistent use of ZEBRA_CONF_PATH throughout the script
* refactor(docker): Enhance container user security and configuration
- Increase UID/GID to 10001 to minimize host system user conflicts
- Remove `--system` flag from user and group creation to prevent potential issues
- Add detailed comments explaining UID/GID selection rationale
- Improve security by using high UID/GID values to reduce namespace collision risks
- Remove redundant `chmod` for entrypoint script
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Gustavo Valverde <g.valverde02@gmail.com>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Update PR template
* Don't automatically assign the `C-trivial` label
* Use `C-exclude-from-changelog` label
* Simplify the PR template
* Simplify the PR template
* Simplify the PR template
* Update .github/pull_request_template.md
Co-authored-by: Pili Guerra <1311133+mpguerra@users.noreply.github.com>
* grammar
---------
Co-authored-by: Pili Guerra <1311133+mpguerra@users.noreply.github.com>
* fix(ci): Improve workflow conditions for cached disk jobs
Refactor GitHub workflow conditions to:
- Handle workflow dispatch events more precisely
- Prevent running cached disk jobs on forked PRs
- Ensure consistent behavior across different deployment workflows
- Avoid skipping main branch deployments
- Updated the if condition for the deploy-nodes job to ensure it only runs when the build job runs successfully and is not skipped.
* fix(ci): disable custom Zebra config test and fix registry condition
- Comment out the `test-zebra-conf-path` job in the deployment workflow
- Fix a syntax error in the GCP resource deletion workflow's condition
- Remove an extra single quote in the `if` condition for the `clean-registries` job
* Rm `.env` files
* Update `mining-docker.md`
* Revert "Rm `.env` files"
This reverts commit caaa4559c3.
* Add `enable_cookie_auth` to default Zebra conf
* Rename `default_zebra_config.toml`
* fmt `prometheus.yaml`
* Update `docker/test.env`
* Update `docker/.env`
* Refactor `docker compose` for lwd
* Enable disabling cookie authentication
* Update `docker compose` for tests
* Update general `docker compose`
* Update docs for running Zebra in Docker
* Add example `docker compose` file for Grafana
* Fix a bug in an example command
* Refactor test execution logic in entrypoint
* Rename `v2.1.0.toml` conf to `custom-conf.toml`
* Fix CI tests for loading of custom conf files
* Use the new conf file name in CI checks
* Use an extended regexp for custom conf CI check
* Fix GitHub workflows
There are two workflows that have bugs in them:
- cd-deploy-nodes-gcp is missing a job name, which is then depended-on
later
- sub-ci-integration-tests-gcp changed the name of a job without
changing its dependents
* Stop certain CI jobs from running on forks
There are various GitHub CI jobs that won’t work on forks. E.g., some
need credentials for ZF’s DockerHub or GCP accounts.
Unfortunately, you can’t prevent entire workflows from running this way,
but this disables a minimal number of jobs to keep forks from failing
whenever `main` is pushed to.
* introduce new fields to `GetInfo`
* add address book to methods for getinfo connections field
* add the version field to getinfo
* track last node error or warning and display it in getinfo rpc method
* add the rest of the fields, minor cleanup
* fix the tests
* clippy
* feat(network): Add misbehavior tracking for peers in the address book
* - Add a `misbehavior_score` field to `MetaAddr`
- Add a `bans_by_ip` field to `AddressBook`
- Update the `AddressBook::update()` method to:
- increment misbehavior scores in its entries,
- add addr ips to bans_by_ip if the score is excessive,
- remove any addrs at the banned ip
- Avoid responding to `GetAddr` requests with addresses of misbehaving peers (return None from `sanitized()`),
- Avoid new inbound or outbound connections to banned ips
* Drops banned peer connections in peer set's `poll_ready()` method
* Adds rudimentary misbehavior score tracking
* fixes some proptests, moves and removes some TODOs
* fixes lint
* Removes outdated TODO
* Adds stub for acceptance test
* updates call to updated fn
* Stores likely inbound peer connection addresses in address book and return their IPs with the default port instead of the transient port when responding to GetAddr requests
* Avoids gossiping peer addrs from inbound connections
* updates test to check that sanitize won't return inbound peer addrs or addrs with non-zero misbehaviour scores.
updated misbehaviour score for potentially unavoidable errors.
* Updates `generate` RPC to support any network where PoW is disabled.
Updates acceptance test to check that zebrad instances disconnect once one of them advertises a block with an invalid PoW
* minor tangential cleanup
* Finishes acceptance test, sends misbehavior updates from syncer, and always updates address book entries if the update is to their misbehaviour score
* skip test on windows (to minimize risk of port conflict)
* Applies suggestions from code review
This change enhances the GCP deployment workflows with several improvements:
- Add static IP address support for long-running nodes
- Implement dynamic network selection matrix
- Add manual deployment trigger support
- Update GCP configuration syntax for better compatibility
- Remove redundant deploy-instance job
- Improve environment labeling and log file handling
The static IP addresses will help maintain stable network connectivity
for long-running nodes, particularly important for testnet peer discovery.
Fixes#8763
Gustavo Valverde
Alfredo Garcia
Kris Nuttycombe
dependabot[bot]
VolodymyrBg
Jack Grigg
Marek
Arya
crStiv
Conrado Gouvea
Elijah Hampton
Dmitry
Hazel OHearn
Greg Pfeil