* ref(docker): combine test and prod entrypoints into one
This makes the following changes
- Uses `-x` instead of echoing the variables values
- Sets default values where required
- Create a function to list directories
- Create a function to run cargo tests
- Use a better approach to handle different options in the `case` manegement for tests and production
- Replaces all instances of `runtime-entrypoint.sh` with `entrypoint.sh`
* chore: use 2 spaces insted of 4 for indentation
* fix: add missing `test-threads` to `fully_synced_rpc_`
* fix: allow 3 arguments instead of 2 to handle `fully_synced_rpc_` correctly
* chore: remove extra file
* chore: add comments to main parts of the file
* fix(ci): just create the `$ZEBRA_CONF_PATH` if `$ENTRYPOINT_FEATURES` is not set
* fix(entrypoint): just print ${ZEBRA_CONF_PATH} if exists
* fix: missing condition
* ref: handle tests better if `$ENTRYPOINT_FEATURES` is set
* fix(ci): We just want `ZEBRA_CONF_PATH` to be set in the `release` image
* fix(entrypoint): fix the overall `case` logic
* fix(ci): allos to run with custom config in CI image
* fix(ci): more edgecases
* fix: we don't need to find files, but subdirectories
* fix(ci): handle Signal Forwarding and exit codes for `cargo`
* fix(ci): parse `fully_synced_rpc_` test correctly
* chore: add missing cache dir variable for LWD
* fix(entrypoint): handle an arbitrary number of arguments
* fix(entrypoint): handle features list
* fix(entrypoint): typo
* chore: typo
* add fixed test vectors for `z_getsubtreesbyindex`
* change to snapshots
* add test to docker
* remove assert lines from snapshots
* add more tests
* change test description
* change test name
* run both tests together
* wait for state version update in test
* Run one test at a time to avoid state locking issues
---------
Co-authored-by: teor <teor@riseup.net>
* ref(docker): use a single variable for test features
* ref(docker): scope `ARG`s and `ENV`s correctly
* fix(docker): use variables as expected on test build
* fix(docker): use correct `$RPC_PORT` validation
* revert(docker): revert to using extra `ENTRYPOINT_FEATURES`
* fix(rust): missing features replacements
* fix(docker): enable backtraces for errors and panics
This is a costly function!!
* ref(docker): remove `$NETWORK` as an `ARG`
* fix typo
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
The `runtime-entrypoint.sh` uses the `RPC_PORT` env var when the user
specifies the `getblocktemplate-rpc` feature, but this env var is unset
unless the user sets it. This commit sets the default values for
`RPC_PORT` depending on `NETWORK`.
* fix(docker): use `entrypoint.sh` as default for users
* ref(entrypoint): allow more flexible configurations
This changes allow users to:
- Mount their own configuration file
- Allow for Zebra to be exposed outside the container or not
- Allow the user to turn off sync
- Allow to enable `metrics` and `tracing`, exposing them or not
Having the `-x` option prints variable expasions, so we don't have to echo each value.
* chore(docker): remove unused ARGs from the Dockerfile
ARGs are not available at build time, so we don't require this ARGs as their ENV variables counterparts are being set in the `entrypoint`, at runtime.
* revert: keep old naming
* fix: renaming mistake :)
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* fix(docker): revert some breaking changes
* imp(docker): allow more flexibility with FEATURES config
* chore(docker): remove confusing port on `EXPOSE`
* chore(docker): remove unused command
* fix(docker): handle quotes while building the conf file
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Remove openssl dependency and prevent it coming back
* Put the arguments in the right place
* Put comment in the right place
* Add a default-docker feature to zebrad and use it in workflows and Docker files
* Fix a comment typo
* Make sure that Docker production builds don't use openssl
* Rename feature to default-release-binaries
* Fix Dockerfile cache use
* Remove cache-breaking COPY commands
* Use git to only reset files modified by cargo-chef
* Copy .git and sources before cargo chef cook
* Update .dockerignore to include .git
* Don't use .git
* Use rsync instead of git
* Maybe COPY is needed
* Actually copy changed files using rsync
* Actually copy the files using the correct rsync syntax
* Remove ls commands from Dockerfile
* fix(deploy): allow the container to raise in MIGs
* fix(docker): add the `ZEBRA_CACHED_STATE_DIR` as a default `ENV`
This no longer requires the env variable to be defined in other places, unless we're changing the default configuration
Binding `127.0.0.1` means that Zebra will accept inbound connections
coming only from the loopback network interface. This is desirable as
long as Zebra runs on a native machine.
When Zebra runs inside a Docker container, incoming connections coming
from the host machine don't come from the container's loopback
interface. In order to be able to connect to Zebra from the host
machine, we can listen on `0.0.0.0` so Zebra accepts inbound connections
coming from any interface. Users then can limit inbound connection to
the loopback of their host by
```bash
docker run -p 127.0.0.1:8232:8232 zfnd/zebra:1.0.0-rc.8
```
* Split checking for cached state disks into its own workflow
* Fix workflow field order
* Run the top-level workflow when the reusable workflow changes
* And run dependent workflows for pull requests as well
* Remove redundant output names
* Document the existing and new workflow jobs
* Add the network to the "no disk found" message
* Tweak existing docs and descriptions
* Generate Zebra checkpoints on testnet
* Add a full sync testnet entrypoint, and simplify mainnet env vars
* Only run the full testnet sync on the main branch
* Deduplicate and update the zebra-checkpoints docs
* Add instructions for automatic checkpoint generation
* Hide some details in the release checklist
* Update release checkpoint instructions to use CI
* Only update the cache in one job on mainnet
* Add extra test type modes to support zebra-checkpoints
* Add Mainnet and Testnet zebra-checkpoints test harnesses
* Add zebra-checkpoints to test docker images
* Add zebra-checkpoints test entrypoints
* Add Mainnet CI workflow for zebra-checkpoints
* Enable zebra-checkpoints feature in the test image
* Use the same features for (almost) all the docker tests
* Make workflow features match Docker features
* Add a feature note
* Add a zebra-checkpoints test feature to zebrad
* Remove the "no cached state" testnet code
* Log a startup message to standard error when launching zebra-checkpoints
* Rename tests to avoid partial name conflicts
* Fix log formatting
* Add sentry feature to experimental docker image build
* Explain what ENTRYPOINT_FEATURES is used for
* Use the correct zebra-checkpoints path
* Silence zebrad logs while generating checkpoints
* Fix zebra-checkpoints log handling
* Re-enable waiting for zebrad to fully sync
* Add documentation for how to run these tests individually
* Start generating checkpoints from the last compiled-in checkpoint
* Fix clippy lints
* Revert changes to TestType
* Wait for all the checkpoints before finishing
* Add more stderr debugging to zebra-checkpoints
* Fix an outdated module comment
* Add a workaround for zebra-checkpoints launch/run issues
* Use temp dir and log what it is
* Log extra metadata about the zebra-checkpoints binary
* Add note about unstable feature -Z bindeps
* Temporarily make the test run faster and with debug info
* Log the original test command name when showing stdout and stderr
* Try zebra-checkpoints in the system path first, then the cargo path
* Fix slow thread close bug in dual process test harness
* If the logs are shown, don't say they are hidden
* Run `zebra-checkpoints --help` to work out what's going on in CI
* Build `zebra-utils` binaries for `zebrad` integration tests
* Revert temporary debugging changes
* Revert changes that were moved to another PR
* Duplicates Dockerfile
* updates mining-testnet Dockerfile with getblocktemplate-rpcs feature, Testnet by default, and an RPC port
* renames mining-testnet.Dockerfile and adds workflow for publishing images on release
* replaces space-seperated features with commas
* Adds .experimental tag suffix, removes new dockerfile, makes lightwalletd tests conditional
* updates build-args to pass on features directly
* adds "lightwalletd-grpc-tests" as default test_features in build-docker-image
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* adds tag suffix to cache keys
---------
Co-authored-by: teor <teor@riseup.net>
* ci: add a test to validate Zebra's config file and path
* fix: use `ZEBRA_CONF_PATH` as single variable locating the conf
* fix: do not remove the containers
* fix: use extended regex
* fix: use different steps to validate the conf tests
* fix: do not specify a default CMD for running Docker in test builds
* fix: use actual starting commands for entrypoint
* fix: do not add cargo twice if cargo is in $1
* fix: allow to run `zebrad` in the `tests` stage of Dockerfile
* fix: new entrypoint does not allow an empty CMD
* fix: do not duplicate the `zebrad` command
* fix: segregate configuration jobs
* refactor(entrypoint): handle better parameters conditions
* fix: make `zebrad` an executable command in `tests` stage
* Show the commands that are being executed in the new docker test
* Show full logs without tee or grep
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* fix: use the actual path inside docker
* fix: use `grep` with exit code
If the container is logging to stderr, piping works only for stdout, so we're adding `2>&1`
* fix: use `grep -q` to get an exit code
* fix: fail if any error is detected
* fix: fail if this test takes more than 5 minutes
* fix: update patch workflows
* feat: test Dockerfile `runtime` config
* fix: depend on the configuration test to continue
Co-authored-by: teor <teor@riseup.net>
* Re-apply: add acceptance test for getblocktemplate method in CI (#5653)
Revert "Revert "change(tests): add acceptance test for getblocktemplate method in CI (#5653)" (#5672)"
This reverts commit 6446e0ec1b.
* Fix incorrect MAX_CONTEXT_BLOCKS assertion in state
* Actually negate the miner fee for the RPC output
* Try the RPC again after waiting for transactions to verify
* Log before the test waits for the mempool to verify transactions
* Use the new ssh key secrets in CI
* adds test for getblocktemplate rpc method
* adds the new test to CI
* adds a couple logs
* Adds example for running the test in acceptance.rs
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* updates mod docs for tests that use future blocks
* updates submitblock test to use TestType methods
* prunes redundant code
* adds check_sync_logs_until
* adds assertion for needs cached state & rpc server
* updates get_raw_future_blocks fn with rpc calls
* updates to get_raw_future_blocks fn and submit_block test
* Rename LightwalletdTestType to TestType
* moves TestType and random_known_rpc_port_config to test_type.rs and config.rs
* moves get_raw_future_blocks to cached_state.rs
* updates ci workflows to include submit block test
* adds get_future_blocks fn and uses it in load_transactions_from_future_blocks
* updates CI docker
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Applies suggestions from code review
* Updates misnamed closure param
* updates mod docs for test_type.rs
Co-authored-by: teor <teor@riseup.net>
* Try running coverage with Rust 1.63
* Run GitHub Actions tests with Rust 1.63
* Change from stable to 1.63 in the patch file
* Use Rust 1.63 to download Zcash parameters
* Use Rust 1.63 to build Docker zebrad images
* Make Rust 1.63 a supported platform, and make stable temporarily unsupported
* fix(build): a path must exist before creating a file in it
Other directories in the Dockerfile were automatically created with the `WORKDIR` and `COPY` commands. In this case it has to be explicitly created
* fix(build): use a variable for the conf path and another for the file
Add some pending actions to consider, for a better user experience.
* fix(runtime): run with the correct path
* chore(build): allow a custom `zebrad` config file path
Previous behavior:
The zebrad config file was hardcoded into the Dockerfile, allowing to
change specific values, but not the config file as a whole
Expected behavior:
Allow the user to specify the config file themselves, for example with
the nginx image you can pass -v /host/path/nginx.conf:/etc/nginx/nginx.conf:ro
Solution:
Use and ARG, with a default value, to allow the user to change the location
of the zebrad configuration file. This also sets the location under /etc,
honoring The Filesystem Hierarchy Standard (FHS) for linux
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* Fix clippy::let_and_return
* Increase lightwalletd test timeouts for zebrad slowness
* Add a `zebrad_update_sync()` test, that update syncs Zebra without lightwalletd
* Run the zebrad-update-sync test in CI
* Add extra zebrad time to workaround lightwalletd bugs
* Disable the flamegraph feature by default at compile time
* Disable the journald feature by default at compile time
* Also disable inferno dependency, and rearrange features
* Disable the prometheus feature by default at compile time
* Disable the tracing filter reload feature by default at compile time
* Disable tests when corresponding features are disabled
* Add compile-time tracing features to user docs
* Add compile-time features to the metrics user docs
* Document diagnostics as part of the start command tasks and services
* breaking(diagnostics): rename "enable-sentry" feature to "sentry" (#4623)
* Also skip conflict tests when those ports are disabled
* breaking(diagnostics): rename "enable-sentry" feature to "sentry"
This is mostly:
```sh
fastmod enable-sentry sentry
```
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(ci): lwd state condition
* fix(ci): differentiate tests that need a lwd cached state
* fix(ci): use the right state and save name for each test
* docs(ci): minor comment fixes
* docs(ci): better input description
* fix(ci): end `if` condition correctly
* fix(images): pass the state version to following steps
* fix(ci): $needs_lwd_state condition was inverted
* fix(ci): reduce disk selection code
* docs(ci): better disk search conditional explanation
* fix(ci): end if condition correctly
* fix(ci): evaluate $needs_zebra_state correctly
* fix(ci): use nested condition for readability
* fix(ci): disk search was using the wrong variable
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* Drop sentry dependencies when enable-sentry feature is disabled
* Make lightwalletd gRPC tests depend on a new lightwalletd-grpc-tests feature
* fix(ci): remove enable-sentry feature from tests
* Add lightwalletd-grpc-tests feature for functionality or efficiency
And document where it is just used to stop re-compilations.
* Remove redundant `cmake` and `protobuf-compiler` dependencies
* Document Zebra's optional production and test feature flags
* Minimise dependencies in zcash-params/Dockerfile
* Minimise dependencies in docker/Dockerfile
* Add a workflow TODO
* Catch more errors in entrypoint.sh
Also makes entrypoint.sh compatible with more distributions
* Remove unnecessary quoting in entrypoint.sh
* Use exactly the same arguments to call CI tests
* Remove a redundant CI build
* Rename Cargo.lock check job
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* Drop sentry dependencies when enable-sentry feature is disabled
* Make lightwalletd gRPC tests depend on a new lightwalletd-grpc-tests feature
* fix(ci): remove enable-sentry feature from tests
* Add lightwalletd-grpc-tests feature for functionality or efficiency
And document where it is just used to stop re-compilations.
* Remove redundant `cmake` and `protobuf-compiler` dependencies
* Document Zebra's optional production and test feature flags
* Minimise dependencies in zcash-params/Dockerfile
* Minimise dependencies in docker/Dockerfile
* Add a workflow TODO
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* refactor(build): use better stage naming and document it
* refactort(build): use multiple cache sources
* docs(build): add a comment for cache
* fix(build): remove gcr.io as it does not supports OCI images
* feat(ci): add lightwalletd_*_sync tests to CI
* feat(ci): add lightwalletd RPC call test
* feat(ci): add send transactions test with lwd to CI
* fix(ci): create a variable to run transactions test
* refactor(ci): use docker in docker
This is a workaround for an issue related to disk partitioning, caused by a GCP service called Konlet, while mounting the cached disks to the VM and then to the container
* fix(build): persist docker login credentials
* fix(ci): get sync height from docker logs instead of gcp
* try: use gha cache for faster building
* fix(ci): mount disk in container to make it available in vm
* fix(build): do not invalidate cache between images
* try(docker): invalidate cache as less as possible
* fix(ci): GHA terminal is not a TTY
* fix(build): do not ignore entrypoint.sh
* fix
* fix(ci): mount using root priveleges
* fix(ci): use existing disk as cached state
* fix(ci): wait for disks to get mounted
* force rebuild
* fix failed force
* fix failed commit
* WIP
* fix(ci): some tests does not use a cached state
* wip
* refactor(ci): disk names and job segregation
* fix(ci): do not name boot and attached disk the same
* fix(ci): attach a disk to full sync, to snapshot the state
* fix(ci): use correct disk implementations
* fix(ci): use different disk name to allow test concurrency
* feat(ci): add lightwalledt send transaction test
* cleanup(ci): remove extra tests
* fix(ci): allow disk concurrency with tests
* fix(ci): add considerations for different tests
* fix(reusable): last fixes
* feat(ci): use reusable workflow for tests
* fix(rw): remove nested worflow
* fix(rw): minor fixes
* force rebuild
* fix(rw): do not use an input as job name
* fix(rw): remove variable id
* fix(ci): remove explicit conditions and id
* fix(ci): docker does not need the variable sign ($) to work
* fix(ci): mount typo
* fix(ci): if a sync fails, always delete the instance
This also reduces the amount of jobs needed.
* refactor(ci): make all test depend on the same build
* fix(ci): some tests require multiple variables
* fix(docker): variable substitution
* fix(ci): allow to run multiple commits from a PR at once
* fix(docker): lower the NETWORK env var for test names
* reduce uneeded diff
* imp(keys): use better naming for builds_disks
* imp(ci): use input defaults
* imp(ci): remove test_name in favor of test_id
* fix(ci): better key naming
* fix(ci): long disk names breaks GCP naming convention
* feat(ci): validate local state version with cached state
* fix(ci): add condition to run tests
* fix: typo
* fix: app_name should not be required
* fix: zebra_state_path shouldn't be required
* fix: reduce diff
* fix(ci): checkout to grep local state version
* Update .github/workflows/test.yml
Co-authored-by: teor <teor@riseup.net>
* revert: merge all tests into a single workflow
* Remove unused STATE_VERSION env var
* fix: minor fixes
* fix(ci): make test.patch the same as test
* fix(ci): negate the input value
* imp(ci): better cached state conditional handling
* imp(ci): exit code is captured by `docker run`
* fix(deploy): mount disks with better write performance
* fix(ci): change sync id to a broader id name
* fix(ci): use correct input validation
* fix(ci): do not make test with cached state dependant on other
* imp(ci): organiza keys better
* fix(ci): use appropiate naming
* fix(ci): create docker volume before mounting
* fix(lint): do not fail on all new changes
* imp(ci): do not report in pr review
* fix(ci): partition clean disks
* fix: typo
* fix: test called the wrong way
* fix(build): stop using gha cache
* ref(ci): validate run condition before calling reusable workflow
* fix(ci): use a better filesystem dir and fix other values
* fix: linting errors
* fix(ci): typo
* Revert "fix(build): stop using gha cache"
This reverts commit a8fbc5f416.
Cache expiration is a lesser evil than not using caching at all and then failing with a 401
* imp(ci): do not set a default for needs_zebra_state
* Update .github/workflows/test.yml
Co-authored-by: teor <teor@riseup.net>
* fix(deps): remove dependencies
* force build
* Update .github/workflows/test.yml
Co-authored-by: teor <teor@riseup.net>
* fix(docker): add RUST_LOG as an ARG and ENV
* fix(test): add `#[ignore]` to send transactions test
This test needs state then it should be marked as #[ignore]
* fix(ci): differentiate between root cache path and its dir
* Remove extra `state` directory
That was a workaround for an issue that has been fixed.
* imp(docs): use better test descriptions
Co-authored-by: teor <teor@riseup.net>
* fix: reduce unwanted diff with main
* fix(ci): make lwd conditions consistent
* Remove another extra `state` directory
Was also part of a workaround for an issue that has been fixed.
* fix(ci): use better conditionals to run test jobs
Co-authored-by: teor <teor@riseup.net>
* Tweak to support different lightwalletd versions
Some versions print `Waiting for block`, and some versions print
`Ingestor waiting for block`.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Gustavo Valverde
Alfredo Garcia
teor
Marek
Arya
Dimitris Apostolou