* ref(docker): leverage cache mount with bind mounts
This update eliminates the need for external tools like `cargo-chef` to leverage caching layers, resulting in an average build time reduction of 4m30s (~36% improvement).
While this solution doesn't fully resolve the issues mentioned in https://github.com/ZcashFoundation/zebra/issues/6169#issuecomment-1712776391, it represents the best possible approach without resorting to custom solutions, which we'd prefer to avoid.
* chore: remove extra `WORKDIR` and imp comments
* chore: improve comment legibility
Co-authored-by: Arya <aryasolhi@gmail.com>
---------
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
Co-authored-by: Arya <aryasolhi@gmail.com>
* fix(docker): allow the `zebra` user access to relevant dirs
When runnning a Zebra node using Docker without a privileged user, you won't be able to modify some files and directories, not even the ones in the current directory, as the `zebra` user has no permission to `/`.
The best way to solve this is making the `/opt/zebrad` the current `WORKDIR`. This also requires moving the `entrypoint.sh` from the root `/` directory to `/etc/zebrad` as this directory is used to save configuration, and other files.
An `APP_HOME` ARG is used as not all platforms where a Docker container is deployed allows writting permissions to the `/opt` directory. This allow some users to re-build the image with a custom `WORKDIR`
* fix(docker): allow starting the container without a `zebrad` command
As `gosu` is just required and available in our `runtime` image, trying to run `docker run -it --rm --name tests -t zfnd/zebra:<pr> /bin/bash` in other stages will fail, as `gosu` is not available.
* fix(docker): typo and uknown option in debian
* fix(docker): use `gosu` for rootless execution
Some of our entrypoint commands requires creating directories and files in places a non-privileged user can't access.
So we use `gosu` to step down from `root` to a non-privileged user during container startup, right at our application execution.
* readd the scanner tests in the new binary context
* remove commented out tests from zebrad
* add Cargo.lock
* add local copy of `ZECPAGES_SAPLING_VIEWING_KEY` to binary scanning tests
* moves scanner-results-reader to zebra-scan, adds zebrad bin to zebra-scan, adds a short sleep before trying to connect to gRPC server in test
* always include zebrad dependency in zebra-scan
* renames zebrad bin in zebra-scan to zebrad-for-scanner
* skip test when a cached state is missing
* remove scanning config from zebrad
* move `scan_task_commands` out of zebrad acceptance tests
* remove scanner config from zebrad
* do not panic if env var is not there
* re enable start_scan_where left, remove shielded-scan feature foir scanning tests in CI
* change app names
* uncomment test
* fix the scanner tests in CI by changing the test command
* add log needed by CI?
* move test
* rename trait, docs fix
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* merges scan task tests and checks that the scan task removes keys
* Adds test to CI
* Skip to expected block height and assert that the result is from that height
* Updates test docs and uses zebrad state path for scan cache
* Log "Zcash network: {network}" from scan task commands test
Containers can generate a significant amount of logs,
consuming disk space over time. Here I added configuration
for maximum log file size (max-size) and the number of retained
log files (max-file) in the logging driver options
to control disk usage and prevent log files from
growing indefinitely.
* feat(docker): allow users to use Zebra + LWD with persistent states
* fix(docs): better documentation and compose execution
* chore(docs): better context box
* chore: do not commit further user changes to the `.env`
This can always be overriden by forcing this pushes if it was intended
* fix(compose): do not add extra volumes and ports
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* ref(docker): Consolidate all tests in the `entrypoint.sh` script
* fix(ci): missing docker argument
* fix(ci): do not blank `ENTRYPOINT_FEATURES` vairable
* fix(docker): do not add `shielded-scan` as part of `TEST_FEATURES`
* chore: remove misleading comment
* feat(docker): add `docker-compose` file to run CI locally
* fix(docs): do not remove previous sections until we have a substitute for users
* fix(docs): revert some changes
* fix: typos
* ref(docker): Consolidate all tests in the `entrypoint.sh` script
* fix(ci): missing docker argument
* fix(ci): do not blank `ENTRYPOINT_FEATURES` vairable
* fix(docker): do not add `shielded-scan` as part of `TEST_FEATURES`
* chore: remove misleading comment
* Update docker/Dockerfile
Co-authored-by: Arya <aryasolhi@gmail.com>
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* ref(docker): Consolidate all tests in the `entrypoint.sh` script
* fix(ci): missing docker argument
* fix(ci): do not blank `ENTRYPOINT_FEATURES` vairable
* fix(docker): do not add `shielded-scan` as part of `TEST_FEATURES`
* chore: remove misleading comment
* Update docker/Dockerfile
Co-authored-by: Arya <aryasolhi@gmail.com>
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* Add the `scan_start_where_left` test to CI
* Fix passing of features
Co-authored-by: Arya <aryasolhi@gmail.com>
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* Add `ldb` RocksDB query tool to the Dockerfile
* Add shielded scanning link to run.md
* Add `ldb` tool instructions to install.md
* Add a rough framework for shielded-scan.md
* Add security warning and feature limitations to shielded-scan.md
* Remove the `zebrad download` command
* Reorder command.rs imports
* Remove zcash-params/Dockerfile and the main Dockerfile commands that use it
* Stop building zcash-params Docker images in CI
* Update CHANGELOG for `zebrad download` removal
* Clarify why the image is smaller
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* Remove duplicate CI docker job and make features consistent
* Remove duplicate OS job
* Rename experimental release build
* Make Rust features into GitHub repository variables
* Remove redundant features in entrypoint.sh
* Remove a dependency on a deleted job
* Fix syntax of array
* Another fix attempt
* Undo some accidental merge overwrites
* Add missing space
* Explain how default is implemented
* Fix missing --features and quoting
* We can fix this later
* Use vars directly in with: blocks
* Use correct features for fake activation heights test
* ref(workflows): consolidate workflows based on their purpose
This also renames the workflows to make their naming more consistent and adding a naming convention
Fixes: #6166Fixes: #6167
* fix(workflows): use correct name for patch
* fix(workflow): docker unit tests
* fix(release): validation error
Error:
```
Validation Failed: {"resource":"Release","code":"invalid","field":"target_commitish"}
```
Fixes: https://github.com/release-drafter/release-drafter/issues/1125
* fix(workflows): reference correct name
* fix: remove extra workflow
* fix(workflows): use larger runners
* fix(workflow): remove code already in docker unit-test
* fix(unit-tests): start zebra the right way
* fix: typo in patch name
* chore: move job to logical order
* imp(workflows): use better name for gcp tests
* add: missing merge changes
* chore: use better name for find-disks
* fix(ci): use the `entrypoint.sh` to change the Network
* fix(ci): add missing `ZEBRA_CONF_PATH` variable
* fix(ci): allow to build the entrypoint file with testnet
* fix(entrypoint): allow to create a dir and file with a single variable
* refactor: test config file in CI and CD with a reusable workflow
* fix(ci): wrong name used
* fix(ci): use checkout
* fix(ci): improve docker config tests
* fix(ci): use better name for protection rules
* Fix changed workflow file name in docs patch file
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Arya <aryasolhi@gmail.com>
* fix(cd): depend on file tests
* fix(docs): adapt to new workflow name
* fix: revert test coverage on CD
* chore: reduce diff
* fix(ci): allow using variable images for reusable workflows
* fix(dockerfile): use variables or default for config path and file
* fix(entrypoint): if `$ZEBRA_CONF_PATH` is set, do not override it
* Fix patch job names and remove failure job testnet dependencies
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Arya <aryasolhi@gmail.com>
* ref(docker): combine test and prod entrypoints into one
This makes the following changes
- Uses `-x` instead of echoing the variables values
- Sets default values where required
- Create a function to list directories
- Create a function to run cargo tests
- Use a better approach to handle different options in the `case` manegement for tests and production
- Replaces all instances of `runtime-entrypoint.sh` with `entrypoint.sh`
* chore: use 2 spaces insted of 4 for indentation
* fix: add missing `test-threads` to `fully_synced_rpc_`
* fix: allow 3 arguments instead of 2 to handle `fully_synced_rpc_` correctly
* chore: remove extra file
* chore: add comments to main parts of the file
* fix(ci): just create the `$ZEBRA_CONF_PATH` if `$ENTRYPOINT_FEATURES` is not set
* fix(entrypoint): just print ${ZEBRA_CONF_PATH} if exists
* fix: missing condition
* ref: handle tests better if `$ENTRYPOINT_FEATURES` is set
* fix(ci): We just want `ZEBRA_CONF_PATH` to be set in the `release` image
* fix(entrypoint): fix the overall `case` logic
* fix(ci): allos to run with custom config in CI image
* fix(ci): more edgecases
* fix: we don't need to find files, but subdirectories
* fix(ci): handle Signal Forwarding and exit codes for `cargo`
* fix(ci): parse `fully_synced_rpc_` test correctly
* chore: add missing cache dir variable for LWD
* fix(entrypoint): handle an arbitrary number of arguments
* fix(entrypoint): handle features list
* fix(entrypoint): typo
* chore: typo
* add fixed test vectors for `z_getsubtreesbyindex`
* change to snapshots
* add test to docker
* remove assert lines from snapshots
* add more tests
* change test description
* change test name
* run both tests together
* wait for state version update in test
* Run one test at a time to avoid state locking issues
---------
Co-authored-by: teor <teor@riseup.net>
* ref(docker): use a single variable for test features
* ref(docker): scope `ARG`s and `ENV`s correctly
* fix(docker): use variables as expected on test build
* fix(docker): use correct `$RPC_PORT` validation
* revert(docker): revert to using extra `ENTRYPOINT_FEATURES`
* fix(rust): missing features replacements
* fix(docker): enable backtraces for errors and panics
This is a costly function!!
* ref(docker): remove `$NETWORK` as an `ARG`
* fix typo
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
The `runtime-entrypoint.sh` uses the `RPC_PORT` env var when the user
specifies the `getblocktemplate-rpc` feature, but this env var is unset
unless the user sets it. This commit sets the default values for
`RPC_PORT` depending on `NETWORK`.
* fix(docker): use `entrypoint.sh` as default for users
* ref(entrypoint): allow more flexible configurations
This changes allow users to:
- Mount their own configuration file
- Allow for Zebra to be exposed outside the container or not
- Allow the user to turn off sync
- Allow to enable `metrics` and `tracing`, exposing them or not
Having the `-x` option prints variable expasions, so we don't have to echo each value.
* chore(docker): remove unused ARGs from the Dockerfile
ARGs are not available at build time, so we don't require this ARGs as their ENV variables counterparts are being set in the `entrypoint`, at runtime.
* revert: keep old naming
* fix: renaming mistake :)
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* fix(docker): revert some breaking changes
* imp(docker): allow more flexibility with FEATURES config
* chore(docker): remove confusing port on `EXPOSE`
* chore(docker): remove unused command
* fix(docker): handle quotes while building the conf file
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Remove openssl dependency and prevent it coming back
* Put the arguments in the right place
* Put comment in the right place
* Add a default-docker feature to zebrad and use it in workflows and Docker files
* Fix a comment typo
* Make sure that Docker production builds don't use openssl
* Rename feature to default-release-binaries
* Fix Dockerfile cache use
* Remove cache-breaking COPY commands
* Use git to only reset files modified by cargo-chef
* Copy .git and sources before cargo chef cook
* Update .dockerignore to include .git
* Don't use .git
* Use rsync instead of git
* Maybe COPY is needed
* Actually copy changed files using rsync
* Actually copy the files using the correct rsync syntax
* Remove ls commands from Dockerfile
* fix(deploy): allow the container to raise in MIGs
* fix(docker): add the `ZEBRA_CACHED_STATE_DIR` as a default `ENV`
This no longer requires the env variable to be defined in other places, unless we're changing the default configuration
Binding `127.0.0.1` means that Zebra will accept inbound connections
coming only from the loopback network interface. This is desirable as
long as Zebra runs on a native machine.
When Zebra runs inside a Docker container, incoming connections coming
from the host machine don't come from the container's loopback
interface. In order to be able to connect to Zebra from the host
machine, we can listen on `0.0.0.0` so Zebra accepts inbound connections
coming from any interface. Users then can limit inbound connection to
the loopback of their host by
```bash
docker run -p 127.0.0.1:8232:8232 zfnd/zebra:1.0.0-rc.8
```
* Split checking for cached state disks into its own workflow
* Fix workflow field order
* Run the top-level workflow when the reusable workflow changes
* And run dependent workflows for pull requests as well
* Remove redundant output names
* Document the existing and new workflow jobs
* Add the network to the "no disk found" message
* Tweak existing docs and descriptions
* Generate Zebra checkpoints on testnet
* Add a full sync testnet entrypoint, and simplify mainnet env vars
* Only run the full testnet sync on the main branch
* Deduplicate and update the zebra-checkpoints docs
* Add instructions for automatic checkpoint generation
* Hide some details in the release checklist
* Update release checkpoint instructions to use CI
* Only update the cache in one job on mainnet
* Add extra test type modes to support zebra-checkpoints
* Add Mainnet and Testnet zebra-checkpoints test harnesses
* Add zebra-checkpoints to test docker images
* Add zebra-checkpoints test entrypoints
* Add Mainnet CI workflow for zebra-checkpoints
* Enable zebra-checkpoints feature in the test image
* Use the same features for (almost) all the docker tests
* Make workflow features match Docker features
* Add a feature note
* Add a zebra-checkpoints test feature to zebrad
* Remove the "no cached state" testnet code
* Log a startup message to standard error when launching zebra-checkpoints
* Rename tests to avoid partial name conflicts
* Fix log formatting
* Add sentry feature to experimental docker image build
* Explain what ENTRYPOINT_FEATURES is used for
* Use the correct zebra-checkpoints path
* Silence zebrad logs while generating checkpoints
* Fix zebra-checkpoints log handling
* Re-enable waiting for zebrad to fully sync
* Add documentation for how to run these tests individually
* Start generating checkpoints from the last compiled-in checkpoint
* Fix clippy lints
* Revert changes to TestType
* Wait for all the checkpoints before finishing
* Add more stderr debugging to zebra-checkpoints
* Fix an outdated module comment
* Add a workaround for zebra-checkpoints launch/run issues
* Use temp dir and log what it is
* Log extra metadata about the zebra-checkpoints binary
* Add note about unstable feature -Z bindeps
* Temporarily make the test run faster and with debug info
* Log the original test command name when showing stdout and stderr
* Try zebra-checkpoints in the system path first, then the cargo path
* Fix slow thread close bug in dual process test harness
* If the logs are shown, don't say they are hidden
* Run `zebra-checkpoints --help` to work out what's going on in CI
* Build `zebra-utils` binaries for `zebrad` integration tests
* Revert temporary debugging changes
* Revert changes that were moved to another PR
* Duplicates Dockerfile
* updates mining-testnet Dockerfile with getblocktemplate-rpcs feature, Testnet by default, and an RPC port
* renames mining-testnet.Dockerfile and adds workflow for publishing images on release
* replaces space-seperated features with commas
* Adds .experimental tag suffix, removes new dockerfile, makes lightwalletd tests conditional
* updates build-args to pass on features directly
* adds "lightwalletd-grpc-tests" as default test_features in build-docker-image
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* adds tag suffix to cache keys
---------
Co-authored-by: teor <teor@riseup.net>
* ci: add a test to validate Zebra's config file and path
* fix: use `ZEBRA_CONF_PATH` as single variable locating the conf
* fix: do not remove the containers
* fix: use extended regex
* fix: use different steps to validate the conf tests
* fix: do not specify a default CMD for running Docker in test builds
* fix: use actual starting commands for entrypoint
* fix: do not add cargo twice if cargo is in $1
* fix: allow to run `zebrad` in the `tests` stage of Dockerfile
* fix: new entrypoint does not allow an empty CMD
* fix: do not duplicate the `zebrad` command
* fix: segregate configuration jobs
* refactor(entrypoint): handle better parameters conditions
* fix: make `zebrad` an executable command in `tests` stage
* Show the commands that are being executed in the new docker test
* Show full logs without tee or grep
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* fix: use the actual path inside docker
* fix: use `grep` with exit code
If the container is logging to stderr, piping works only for stdout, so we're adding `2>&1`
* fix: use `grep -q` to get an exit code
* fix: fail if any error is detected
* fix: fail if this test takes more than 5 minutes
* fix: update patch workflows
* feat: test Dockerfile `runtime` config
* fix: depend on the configuration test to continue
Co-authored-by: teor <teor@riseup.net>
* Re-apply: add acceptance test for getblocktemplate method in CI (#5653)
Revert "Revert "change(tests): add acceptance test for getblocktemplate method in CI (#5653)" (#5672)"
This reverts commit 6446e0ec1b.
* Fix incorrect MAX_CONTEXT_BLOCKS assertion in state
* Actually negate the miner fee for the RPC output
* Try the RPC again after waiting for transactions to verify
* Log before the test waits for the mempool to verify transactions
* Use the new ssh key secrets in CI
* adds test for getblocktemplate rpc method
* adds the new test to CI
* adds a couple logs
* Adds example for running the test in acceptance.rs
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* updates mod docs for tests that use future blocks
* updates submitblock test to use TestType methods
* prunes redundant code
* adds check_sync_logs_until
* adds assertion for needs cached state & rpc server
* updates get_raw_future_blocks fn with rpc calls
* updates to get_raw_future_blocks fn and submit_block test
* Rename LightwalletdTestType to TestType
* moves TestType and random_known_rpc_port_config to test_type.rs and config.rs
* moves get_raw_future_blocks to cached_state.rs
* updates ci workflows to include submit block test
* adds get_future_blocks fn and uses it in load_transactions_from_future_blocks
* updates CI docker
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Applies suggestions from code review
* Updates misnamed closure param
* updates mod docs for test_type.rs
Co-authored-by: teor <teor@riseup.net>
Gustavo Valverde
Marek
Alfredo Garcia
Arya
kai
teor