* fix(docker): set `HOME` environment variable explicitly in Dockerfile
The HOME environment variable was defaulting to /root when the container started, causing cache directories to be incorrectly set up under /root/.cache/zebra instead of /home/zebra/.cache/zebra. This explicit setting ensures the HOME environment variable is correctly set to the zebra user's home directory.
* fix(docker): add default `CMD` to test stage in Dockerfile
When running the container using docker-compose without explicitly
providing a command, the entrypoint.sh script was attempting to execute
`exec_as_user` with no arguments, resulting in a gosu error:
```
exec_as_user
exec gosu 10001:10001
Usage: gosu user-spec command [args]
```
By adding `CMD ["cargo", "test"]` to the test stage in the Dockerfile,
we ensure a default command is available for the entrypoint script to
execute, preventing the gosu error when no command is explicitly provided.
This fix allows `docker-compose.test.yml` to run successfully without
needing to specify a command in the service definition.
* Adds a new trait for disk format upgrades, implements in on a new struct, `PruneTrees`, and moves the logic for tree deduplication to the trait impl
* refactors add subtrees format upgrade to use new trait
* refactors fix tree keys, cache genesis roots, and value balance upgrades to use new trait
* Applies suggestions from code review:
- Avoids duplicate validation of format upgrades at startup when db is already upgraded,
- Minor refactors
- Doc fixes and cleanups
* Applies suggestions from code review
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Adds reconsider_block method and appropriate test cases
* Refactors invalidated_blocks state to use IndexMap. Returns error from validate_and_commit if a candidate block's hash is in the map of invalidated blocks. Stores invalidated_blocks by height and clears when finalizing. Checks against non finalized tip hash to create a new chain if parnt_chain doesn't exist. Renames ReconsiderError variant NonPreviouslyInvalidatedBlock to MissingInvalidatedBlock.
* Moves MAX_INVALIDATED_BLOCKS constant to constants.rs
* Maintains invalidate_blocks cleanup in reconsider_block and finalize(). Removes unused ReconsiderError variant. Opts to refuse block consideration if parent_chain does not exist. Adds db handle to reconsider_block function. Edits max blocks constant documentation
* Checks the finalized state first to create a new chain from non finalized blocks only before checking parent_chain.
The HOME environment variable was defaulting to /root when the container started, causing cache directories to be incorrectly set up under /root/.cache/zebra instead of /home/zebra/.cache/zebra. This explicit setting ensures the HOME environment variable is correctly set to the zebra user's home directory.
* Use gosu only once
* Remove `COLORBT_SHOW_HIDDEN`
* Simplify Dockerfile
* Remove `check_directory_files` from entrypoint
* Remove check for `ZEBRA_CONF_PATH` in entrypoint
* Simplify ownership setting for `ZEBRA_CACHE_DIR`
* Simplify ownership setting for `LOG_FILE`
* Refactor Dockerfile & entrypoint
* Refactor vars in Dockerfile
* fmt
* Use `chown` for `ZEBRA_CONF_PATH`
* `run_cargo_test` -> `run_test`
* Make `run_test` runnable with gosu
* Cosmetics
* Don't pre-compile Zebra
* Revert: "Don't pre-compile Zebra"
* Fix the custom conf test
* Reintroduce `CARGO_HOME` in Dockerfile
* Pass `FEATURES` as env var to entrypoint
* Fix ARGs in Dockerfile
* Revert "Remove `COLORBT_SHOW_HIDDEN`"
This reverts commit 960d5ca308.
* Specify cache state dir in CI
* Specify lwd cache dir in CI
* refactor: reorganize variables and avoid running entrypoint commands in subshell (#9326)
* refactor(docker): improve container configuration and security
- Optimize Dockerfile build stages and environment variables
- Improve file operations with proper ownership
- Streamline entrypoint script privilege management
* refactor(docker): enhance user management and directory ownership
- Add HOME argument back to ensure proper user home directory setup
- Implement ownership change for the user's home directory
* refactor(docker): remove redundant cache directory setup
- Eliminate explicit creation and ownership setting for LWD and Zebra cache directories in Dockerfile.
- Introduce default values for cache directories in entrypoint script, allowing for environment variable overrides.
* fix: run all cargo commands as user
* chore: reduce diff
* fix: revert to more robust command array
---------
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
Co-authored-by: Gustavo Valverde <g.valverde02@gmail.com>
* feat(ci): Add Rust environment variables to Docker test workflows
Enhance test workflows by adding Rust-specific environment variables:
- Include RUST_LOG for logging configuration
- Add RUST_BACKTRACE and RUST_LIB_BACKTRACE for improved error tracing
- Include COLORBT_SHOW_HIDDEN for detailed backtraces
- Add CARGO_INCREMENTAL for build performance optimization
These changes improve debugging capabilities and provide more flexibility in test environments across dockerized CI jobs.
* feat(ci): Add CARGO_INCREMENTAL build argument to Docker workflow
Enable configurable Rust incremental compilation in Docker builds by:
- Adding CARGO_INCREMENTAL as a build argument in GitHub Actions workflow
- Setting a default value of 0 in Dockerfile to control build performance
- Integrating with existing Rust environment variable configuration
This change provides more granular control over Rust compilation strategies in containerized builds.
* fix(docker): resolve user creation issues in Dockerfile
- Move WORKDIR after user creation to prevent home directory ownership issues
- Properly set environment variables for UID, GID, HOME, and USER in each stage
- Reorganize Dockerfile to ensure home directory is created after user setup
- Fix interactive prompts during adduser by ensuring proper directory ownership
* fix(docs): Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* docs(decisions): add architectural decision records structure
Create a structured decision records system to document important technical choices across multiple domains (DevOps, Network, Consensus, etc.).
This implements a modified MADR template approach for preserving context, trade-offs, and reasoning behind significant architectural decisions.
* fix(docs): suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* Switch to a non-privileged user in tests
* Change test env setup
* Remove unneeded ARGs
* Simplify UID & GID handling in `runtime` target
* Simplify docs
* refactor(docker): Improve user and permission handling in Dockerfiles
- Add gosu for flexible non-root user execution
- Enhance user and group creation with configurable UID/GID
- Modify entrypoint script to support dynamic user switching
- Improve cache and log directory permission management
- Update comments to clarify user and permission strategies
* refactor(docker): Improve Zebra config file handling in entrypoint script
- Enhance error handling for missing config file (now exits with error)
- Simplify config preparation logic by removing redundant file copying
- Update comments to reflect new config file handling approach
- Ensure consistent use of ZEBRA_CONF_PATH throughout the script
* refactor(docker): Enhance container user security and configuration
- Increase UID/GID to 10001 to minimize host system user conflicts
- Remove `--system` flag from user and group creation to prevent potential issues
- Add detailed comments explaining UID/GID selection rationale
- Improve security by using high UID/GID values to reduce namespace collision risks
- Remove redundant `chmod` for entrypoint script
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Gustavo Valverde <g.valverde02@gmail.com>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Update PR template
* Don't automatically assign the `C-trivial` label
* Use `C-exclude-from-changelog` label
* Simplify the PR template
* Simplify the PR template
* Simplify the PR template
* Update .github/pull_request_template.md
Co-authored-by: Pili Guerra <1311133+mpguerra@users.noreply.github.com>
* grammar
---------
Co-authored-by: Pili Guerra <1311133+mpguerra@users.noreply.github.com>
* fix(ci): Improve workflow conditions for cached disk jobs
Refactor GitHub workflow conditions to:
- Handle workflow dispatch events more precisely
- Prevent running cached disk jobs on forked PRs
- Ensure consistent behavior across different deployment workflows
- Avoid skipping main branch deployments
- Updated the if condition for the deploy-nodes job to ensure it only runs when the build job runs successfully and is not skipped.
* fix(ci): disable custom Zebra config test and fix registry condition
- Comment out the `test-zebra-conf-path` job in the deployment workflow
- Fix a syntax error in the GCP resource deletion workflow's condition
- Remove an extra single quote in the `if` condition for the `clean-registries` job
* Rm `.env` files
* Update `mining-docker.md`
* Revert "Rm `.env` files"
This reverts commit caaa4559c3.
* Add `enable_cookie_auth` to default Zebra conf
* Rename `default_zebra_config.toml`
* fmt `prometheus.yaml`
* Update `docker/test.env`
* Update `docker/.env`
* Refactor `docker compose` for lwd
* Enable disabling cookie authentication
* Update `docker compose` for tests
* Update general `docker compose`
* Update docs for running Zebra in Docker
* Add example `docker compose` file for Grafana
* Fix a bug in an example command
* Refactor test execution logic in entrypoint
* Rename `v2.1.0.toml` conf to `custom-conf.toml`
* Fix CI tests for loading of custom conf files
* Use the new conf file name in CI checks
* Use an extended regexp for custom conf CI check
* Fix GitHub workflows
There are two workflows that have bugs in them:
- cd-deploy-nodes-gcp is missing a job name, which is then depended-on
later
- sub-ci-integration-tests-gcp changed the name of a job without
changing its dependents
* Stop certain CI jobs from running on forks
There are various GitHub CI jobs that won’t work on forks. E.g., some
need credentials for ZF’s DockerHub or GCP accounts.
Unfortunately, you can’t prevent entire workflows from running this way,
but this disables a minimal number of jobs to keep forks from failing
whenever `main` is pushed to.
* introduce new fields to `GetInfo`
* add address book to methods for getinfo connections field
* add the version field to getinfo
* track last node error or warning and display it in getinfo rpc method
* add the rest of the fields, minor cleanup
* fix the tests
* clippy
* feat(network): Add misbehavior tracking for peers in the address book
* - Add a `misbehavior_score` field to `MetaAddr`
- Add a `bans_by_ip` field to `AddressBook`
- Update the `AddressBook::update()` method to:
- increment misbehavior scores in its entries,
- add addr ips to bans_by_ip if the score is excessive,
- remove any addrs at the banned ip
- Avoid responding to `GetAddr` requests with addresses of misbehaving peers (return None from `sanitized()`),
- Avoid new inbound or outbound connections to banned ips
* Drops banned peer connections in peer set's `poll_ready()` method
* Adds rudimentary misbehavior score tracking
* fixes some proptests, moves and removes some TODOs
* fixes lint
* Removes outdated TODO
* Adds stub for acceptance test
* updates call to updated fn
* Stores likely inbound peer connection addresses in address book and return their IPs with the default port instead of the transient port when responding to GetAddr requests
* Avoids gossiping peer addrs from inbound connections
* updates test to check that sanitize won't return inbound peer addrs or addrs with non-zero misbehaviour scores.
updated misbehaviour score for potentially unavoidable errors.
* Updates `generate` RPC to support any network where PoW is disabled.
Updates acceptance test to check that zebrad instances disconnect once one of them advertises a block with an invalid PoW
* minor tangential cleanup
* Finishes acceptance test, sends misbehavior updates from syncer, and always updates address book entries if the update is to their misbehaviour score
* skip test on windows (to minimize risk of port conflict)
* Applies suggestions from code review
This change enhances the GCP deployment workflows with several improvements:
- Add static IP address support for long-running nodes
- Implement dynamic network selection matrix
- Add manual deployment trigger support
- Update GCP configuration syntax for better compatibility
- Remove redundant deploy-instance job
- Improve environment labeling and log file handling
The static IP addresses will help maintain stable network connectivity
for long-running nodes, particularly important for testnet peer discovery.
Fixes#8763
Release Drafter v6.1.0 has a regression that creates multiple duplicate draft
releases instead of updating existing ones. This change:
- Disables pre-release flag for draft releases
- Excludes pre-releases from being included in drafts
- Explicitly defines PR event types in workflow
See: release-drafter/release-drafter#1425
* Refactor formatting & docs
* Refactor the `runtime` stage in Dockerfile
* Remove unused code from `entrypoint.sh`
* Simplify `entrypoint.sh` setup
* Revise docs & formatting
* Adjust default values for env vars
* Bump Rust v from 1.79 to 1.81 in Dockerfile
* Refactor `entrypoint.sh`
* Refactor `Dockerfile`
* Add TODOs for monitoring stage to Dockerfile
* Refactor `Dockerfile`
* Add TODOs for monitoring stage to Dockerfile
* Fix a typo
* Allow running `zebrad` in test mode
* Allow custom config for `zebrad` in test mode
* Remove `curl` from the `runtime` Docker image
* Remove redundant echos
* Remove a malfunctioning CD test
The test was using a custom config file set in `test_variables`.
However, the file was not included in the Docker image, and the
entrypoint script created a new, default one under the original file's
path. Zebra then loaded this new file, and the test passed because the
pattern in `grep_patterns` matched Zebra's output containing the
original path, even though the config file was different.
* Remove a redundant CI test
* Remove all packages from the `runtime` stage
* Docs cosmetics
* Clarify docs
* Bump Rust version
* Remove a security note
* Explicitly specify network cache dir
* Explicitly specify cookie dir
* Set UID, GID and home dir for the `zebra` user
* Set a working dir for the `zebra` user
* Don't remove `FEATURES`
* Try re-introducing the `testnet-conf` check
* `ZEBRA_CACHED_STATE_DIR` -> `ZEBRA_CACHE_DIR`
This dir doesn't hold only the state cache anymore, but also the cache
for network peers, and the cookie file.
* Refactor the dir structure
* Check that `ZEBRA_CONF_PATH` exists in the image
* Improve the check for `ZEBRA_CONF_PATH`
* Use different flag in the `ZEBRA_CONF_PATH` check
* Simplify the `ZEBRA_CONF_PATH` check
* Fix spelling
* Comment out the `testnet-conf` CI check
* Add commented out `test-zebra-conf-path` CI check
* Reintroduce `testnet-conf` CI check
* Update the `custom-conf` CI check
* Add `v2.1.0.toml` conf file
* Refine the `v2.1.0.toml` conf file
* Remove `ZEBRA_LISTEN_ADDR` from the entrypoint
* Remove `ZEBRA_CHECKPOINT_SYNC` from the entrypoint
* Stop supporting configuration of the RPC port
* Add default conf file
* Prepare Zebra's config in the entrypoint script
* Remove unneeded packages from the `deps` target
* Docs cosmetics
* Use only `$FEATURES` in entrypoint
* Simplify handling of Rust features
* Add a TODO
* Add CI debug statements
* Don't require test vars in conf test
* Reintroduce `protoc`
* Remove `-e NETWORK`
* Remove `ZEBRA_FORCE_USE_COLOR=1`
* Remove `ZEBRA_CACHE_DIR=/var/cache/zebrad-cache`
* Reintroduce the "custom-conf" test
* Set up test env the same way as prod
* Don't repeatedly check for conf file in entrypoint
* Simplify file ownership in Dockerfile
* Fix checkpoint tests in entrypoint
* Fix Zebra config CI tests
* `LIGHTWALLETD_DATA_DIR` -> `LWD_CACHE_DIR`
* Add config for `LWD_CACHE_DIR` to Dockerfile
* `/var/cache/zebrad-cache` -> `~/.cache/zebra`
* `var/cache/lwd-cache` -> `/home/zebra/.cache/lwd`
* Remove `LOG_COLOR=false` from GCP setup
* Don't specify `LWD_CACHE_DIR` in CI tests
* Don't switch to `zebra` user for tests in Docker
* Join "experimental" and "all" tests in CI
* Remove outdated docs
* Refactor tests with fake activation heights
* Fix tests for scanner
* zebra-rpc: Correctly map JSON-RPC to/from 2.0
The existing code was only handling the `lightwalletd` client format
which includes a `"jsonrpc": "1.0"` entry, and it was returning
responses as JSON-RPC 2.0 regardless of what the client expected.
* Update zebra-rpc/src/server/http_request_compatibility.rs
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* Adds some of the required fields on `getblockchaininfo` output.
* Adds state request/response variants for querying disk usage
* Adds `size_on_disk`, `chain_supply`, and `monitored` fields.
* Updates snapshots
* fixes prop tests
* fixes doc lints
* Adds missing `size()` method
* Fixes lwd integration test issue by updating get_blockchain_info to fallback on default values instead of returning an error if the state is empty.
Related: Runs state queries in parallel from getblockchaininfo RPC and removes the BlockHeader query by getting the tip block time from the latest chain tip channel.
* Updates failing proptests
* fixes lint
* Adds new invalidate_block method to non finalized state. Adds test case in vectors.rs. Updates non finalized state to track invalidated_blocks
* Removes InvalidatedBlockData struct. Wraps invalidated_blocks HashMap values with Arc. Optimizies invalidate_block fn to return early if root has hash of desired block.
* Update metrics whenever chain_set is modified
* Adds child_blocks method and refactors invalidate_block in chain.rs. Refactors invalidate_block in NonFinalizedState.
---------
Co-authored-by: Elijah Hampton <elijahhampton@pop-os.lan>
Alfredo Garcia
Arya
dependabot[bot]
crStiv
Gustavo Valverde
Jack Grigg
Conrado Gouvea
Elijah Hampton
Marek
Dmitry
Hazel OHearn
Greg Pfeil
Metalcape
cryptoraph