* Add `ldb` RocksDB query tool to the Dockerfile
* Add shielded scanning link to run.md
* Add `ldb` tool instructions to install.md
* Add a rough framework for shielded-scan.md
* Add security warning and feature limitations to shielded-scan.md
* Remove the `zebrad download` command
* Reorder command.rs imports
* Remove zcash-params/Dockerfile and the main Dockerfile commands that use it
* Stop building zcash-params Docker images in CI
* Update CHANGELOG for `zebrad download` removal
* Clarify why the image is smaller
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* Remove duplicate CI docker job and make features consistent
* Remove duplicate OS job
* Rename experimental release build
* Make Rust features into GitHub repository variables
* Remove redundant features in entrypoint.sh
* Remove a dependency on a deleted job
* Fix syntax of array
* Another fix attempt
* Undo some accidental merge overwrites
* Add missing space
* Explain how default is implemented
* Fix missing --features and quoting
* We can fix this later
* Use vars directly in with: blocks
* Use correct features for fake activation heights test
* ref(workflows): consolidate workflows based on their purpose
This also renames the workflows to make their naming more consistent and adding a naming convention
Fixes: #6166Fixes: #6167
* fix(workflows): use correct name for patch
* fix(workflow): docker unit tests
* fix(release): validation error
Error:
```
Validation Failed: {"resource":"Release","code":"invalid","field":"target_commitish"}
```
Fixes: https://github.com/release-drafter/release-drafter/issues/1125
* fix(workflows): reference correct name
* fix: remove extra workflow
* fix(workflows): use larger runners
* fix(workflow): remove code already in docker unit-test
* fix(unit-tests): start zebra the right way
* fix: typo in patch name
* chore: move job to logical order
* imp(workflows): use better name for gcp tests
* add: missing merge changes
* chore: use better name for find-disks
* fix(ci): use the `entrypoint.sh` to change the Network
* fix(ci): add missing `ZEBRA_CONF_PATH` variable
* fix(ci): allow to build the entrypoint file with testnet
* fix(entrypoint): allow to create a dir and file with a single variable
* refactor: test config file in CI and CD with a reusable workflow
* fix(ci): wrong name used
* fix(ci): use checkout
* fix(ci): improve docker config tests
* fix(ci): use better name for protection rules
* Fix changed workflow file name in docs patch file
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Arya <aryasolhi@gmail.com>
* fix(cd): depend on file tests
* fix(docs): adapt to new workflow name
* fix: revert test coverage on CD
* chore: reduce diff
* fix(ci): allow using variable images for reusable workflows
* fix(dockerfile): use variables or default for config path and file
* fix(entrypoint): if `$ZEBRA_CONF_PATH` is set, do not override it
* Fix patch job names and remove failure job testnet dependencies
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Arya <aryasolhi@gmail.com>
* ref(docker): combine test and prod entrypoints into one
This makes the following changes
- Uses `-x` instead of echoing the variables values
- Sets default values where required
- Create a function to list directories
- Create a function to run cargo tests
- Use a better approach to handle different options in the `case` manegement for tests and production
- Replaces all instances of `runtime-entrypoint.sh` with `entrypoint.sh`
* chore: use 2 spaces insted of 4 for indentation
* fix: add missing `test-threads` to `fully_synced_rpc_`
* fix: allow 3 arguments instead of 2 to handle `fully_synced_rpc_` correctly
* chore: remove extra file
* chore: add comments to main parts of the file
* fix(ci): just create the `$ZEBRA_CONF_PATH` if `$ENTRYPOINT_FEATURES` is not set
* fix(entrypoint): just print ${ZEBRA_CONF_PATH} if exists
* fix: missing condition
* ref: handle tests better if `$ENTRYPOINT_FEATURES` is set
* fix(ci): We just want `ZEBRA_CONF_PATH` to be set in the `release` image
* fix(entrypoint): fix the overall `case` logic
* fix(ci): allos to run with custom config in CI image
* fix(ci): more edgecases
* fix: we don't need to find files, but subdirectories
* fix(ci): handle Signal Forwarding and exit codes for `cargo`
* fix(ci): parse `fully_synced_rpc_` test correctly
* chore: add missing cache dir variable for LWD
* fix(entrypoint): handle an arbitrary number of arguments
* fix(entrypoint): handle features list
* fix(entrypoint): typo
* chore: typo
* add fixed test vectors for `z_getsubtreesbyindex`
* change to snapshots
* add test to docker
* remove assert lines from snapshots
* add more tests
* change test description
* change test name
* run both tests together
* wait for state version update in test
* Run one test at a time to avoid state locking issues
---------
Co-authored-by: teor <teor@riseup.net>
* ref(docker): use a single variable for test features
* ref(docker): scope `ARG`s and `ENV`s correctly
* fix(docker): use variables as expected on test build
* fix(docker): use correct `$RPC_PORT` validation
* revert(docker): revert to using extra `ENTRYPOINT_FEATURES`
* fix(rust): missing features replacements
* fix(docker): enable backtraces for errors and panics
This is a costly function!!
* ref(docker): remove `$NETWORK` as an `ARG`
* fix typo
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
The `runtime-entrypoint.sh` uses the `RPC_PORT` env var when the user
specifies the `getblocktemplate-rpc` feature, but this env var is unset
unless the user sets it. This commit sets the default values for
`RPC_PORT` depending on `NETWORK`.
* fix(docker): use `entrypoint.sh` as default for users
* ref(entrypoint): allow more flexible configurations
This changes allow users to:
- Mount their own configuration file
- Allow for Zebra to be exposed outside the container or not
- Allow the user to turn off sync
- Allow to enable `metrics` and `tracing`, exposing them or not
Having the `-x` option prints variable expasions, so we don't have to echo each value.
* chore(docker): remove unused ARGs from the Dockerfile
ARGs are not available at build time, so we don't require this ARGs as their ENV variables counterparts are being set in the `entrypoint`, at runtime.
* revert: keep old naming
* fix: renaming mistake :)
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* fix(docker): revert some breaking changes
* imp(docker): allow more flexibility with FEATURES config
* chore(docker): remove confusing port on `EXPOSE`
* chore(docker): remove unused command
* fix(docker): handle quotes while building the conf file
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Remove openssl dependency and prevent it coming back
* Put the arguments in the right place
* Put comment in the right place
* Add a default-docker feature to zebrad and use it in workflows and Docker files
* Fix a comment typo
* Make sure that Docker production builds don't use openssl
* Rename feature to default-release-binaries
* Fix Dockerfile cache use
* Remove cache-breaking COPY commands
* Use git to only reset files modified by cargo-chef
* Copy .git and sources before cargo chef cook
* Update .dockerignore to include .git
* Don't use .git
* Use rsync instead of git
* Maybe COPY is needed
* Actually copy changed files using rsync
* Actually copy the files using the correct rsync syntax
* Remove ls commands from Dockerfile
* fix(deploy): allow the container to raise in MIGs
* fix(docker): add the `ZEBRA_CACHED_STATE_DIR` as a default `ENV`
This no longer requires the env variable to be defined in other places, unless we're changing the default configuration
Binding `127.0.0.1` means that Zebra will accept inbound connections
coming only from the loopback network interface. This is desirable as
long as Zebra runs on a native machine.
When Zebra runs inside a Docker container, incoming connections coming
from the host machine don't come from the container's loopback
interface. In order to be able to connect to Zebra from the host
machine, we can listen on `0.0.0.0` so Zebra accepts inbound connections
coming from any interface. Users then can limit inbound connection to
the loopback of their host by
```bash
docker run -p 127.0.0.1:8232:8232 zfnd/zebra:1.0.0-rc.8
```
* Split checking for cached state disks into its own workflow
* Fix workflow field order
* Run the top-level workflow when the reusable workflow changes
* And run dependent workflows for pull requests as well
* Remove redundant output names
* Document the existing and new workflow jobs
* Add the network to the "no disk found" message
* Tweak existing docs and descriptions
* Generate Zebra checkpoints on testnet
* Add a full sync testnet entrypoint, and simplify mainnet env vars
* Only run the full testnet sync on the main branch
* Deduplicate and update the zebra-checkpoints docs
* Add instructions for automatic checkpoint generation
* Hide some details in the release checklist
* Update release checkpoint instructions to use CI
* Only update the cache in one job on mainnet
* Add extra test type modes to support zebra-checkpoints
* Add Mainnet and Testnet zebra-checkpoints test harnesses
* Add zebra-checkpoints to test docker images
* Add zebra-checkpoints test entrypoints
* Add Mainnet CI workflow for zebra-checkpoints
* Enable zebra-checkpoints feature in the test image
* Use the same features for (almost) all the docker tests
* Make workflow features match Docker features
* Add a feature note
* Add a zebra-checkpoints test feature to zebrad
* Remove the "no cached state" testnet code
* Log a startup message to standard error when launching zebra-checkpoints
* Rename tests to avoid partial name conflicts
* Fix log formatting
* Add sentry feature to experimental docker image build
* Explain what ENTRYPOINT_FEATURES is used for
* Use the correct zebra-checkpoints path
* Silence zebrad logs while generating checkpoints
* Fix zebra-checkpoints log handling
* Re-enable waiting for zebrad to fully sync
* Add documentation for how to run these tests individually
* Start generating checkpoints from the last compiled-in checkpoint
* Fix clippy lints
* Revert changes to TestType
* Wait for all the checkpoints before finishing
* Add more stderr debugging to zebra-checkpoints
* Fix an outdated module comment
* Add a workaround for zebra-checkpoints launch/run issues
* Use temp dir and log what it is
* Log extra metadata about the zebra-checkpoints binary
* Add note about unstable feature -Z bindeps
* Temporarily make the test run faster and with debug info
* Log the original test command name when showing stdout and stderr
* Try zebra-checkpoints in the system path first, then the cargo path
* Fix slow thread close bug in dual process test harness
* If the logs are shown, don't say they are hidden
* Run `zebra-checkpoints --help` to work out what's going on in CI
* Build `zebra-utils` binaries for `zebrad` integration tests
* Revert temporary debugging changes
* Revert changes that were moved to another PR
* Duplicates Dockerfile
* updates mining-testnet Dockerfile with getblocktemplate-rpcs feature, Testnet by default, and an RPC port
* renames mining-testnet.Dockerfile and adds workflow for publishing images on release
* replaces space-seperated features with commas
* Adds .experimental tag suffix, removes new dockerfile, makes lightwalletd tests conditional
* updates build-args to pass on features directly
* adds "lightwalletd-grpc-tests" as default test_features in build-docker-image
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* adds tag suffix to cache keys
---------
Co-authored-by: teor <teor@riseup.net>
* ci: add a test to validate Zebra's config file and path
* fix: use `ZEBRA_CONF_PATH` as single variable locating the conf
* fix: do not remove the containers
* fix: use extended regex
* fix: use different steps to validate the conf tests
* fix: do not specify a default CMD for running Docker in test builds
* fix: use actual starting commands for entrypoint
* fix: do not add cargo twice if cargo is in $1
* fix: allow to run `zebrad` in the `tests` stage of Dockerfile
* fix: new entrypoint does not allow an empty CMD
* fix: do not duplicate the `zebrad` command
* fix: segregate configuration jobs
* refactor(entrypoint): handle better parameters conditions
* fix: make `zebrad` an executable command in `tests` stage
* Show the commands that are being executed in the new docker test
* Show full logs without tee or grep
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* fix: use the actual path inside docker
* fix: use `grep` with exit code
If the container is logging to stderr, piping works only for stdout, so we're adding `2>&1`
* fix: use `grep -q` to get an exit code
* fix: fail if any error is detected
* fix: fail if this test takes more than 5 minutes
* fix: update patch workflows
* feat: test Dockerfile `runtime` config
* fix: depend on the configuration test to continue
Co-authored-by: teor <teor@riseup.net>
* Re-apply: add acceptance test for getblocktemplate method in CI (#5653)
Revert "Revert "change(tests): add acceptance test for getblocktemplate method in CI (#5653)" (#5672)"
This reverts commit 6446e0ec1b.
* Fix incorrect MAX_CONTEXT_BLOCKS assertion in state
* Actually negate the miner fee for the RPC output
* Try the RPC again after waiting for transactions to verify
* Log before the test waits for the mempool to verify transactions
* Use the new ssh key secrets in CI
* adds test for getblocktemplate rpc method
* adds the new test to CI
* adds a couple logs
* Adds example for running the test in acceptance.rs
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* updates mod docs for tests that use future blocks
* updates submitblock test to use TestType methods
* prunes redundant code
* adds check_sync_logs_until
* adds assertion for needs cached state & rpc server
* updates get_raw_future_blocks fn with rpc calls
* updates to get_raw_future_blocks fn and submit_block test
* Rename LightwalletdTestType to TestType
* moves TestType and random_known_rpc_port_config to test_type.rs and config.rs
* moves get_raw_future_blocks to cached_state.rs
* updates ci workflows to include submit block test
* adds get_future_blocks fn and uses it in load_transactions_from_future_blocks
* updates CI docker
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Applies suggestions from code review
* Updates misnamed closure param
* updates mod docs for test_type.rs
Co-authored-by: teor <teor@riseup.net>
* Try running coverage with Rust 1.63
* Run GitHub Actions tests with Rust 1.63
* Change from stable to 1.63 in the patch file
* Use Rust 1.63 to download Zcash parameters
* Use Rust 1.63 to build Docker zebrad images
* Make Rust 1.63 a supported platform, and make stable temporarily unsupported
* fix(build): a path must exist before creating a file in it
Other directories in the Dockerfile were automatically created with the `WORKDIR` and `COPY` commands. In this case it has to be explicitly created
* fix(build): use a variable for the conf path and another for the file
Add some pending actions to consider, for a better user experience.
* fix(runtime): run with the correct path
* chore(build): allow a custom `zebrad` config file path
Previous behavior:
The zebrad config file was hardcoded into the Dockerfile, allowing to
change specific values, but not the config file as a whole
Expected behavior:
Allow the user to specify the config file themselves, for example with
the nginx image you can pass -v /host/path/nginx.conf:/etc/nginx/nginx.conf:ro
Solution:
Use and ARG, with a default value, to allow the user to change the location
of the zebrad configuration file. This also sets the location under /etc,
honoring The Filesystem Hierarchy Standard (FHS) for linux
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* Fix clippy::let_and_return
* Increase lightwalletd test timeouts for zebrad slowness
* Add a `zebrad_update_sync()` test, that update syncs Zebra without lightwalletd
* Run the zebrad-update-sync test in CI
* Add extra zebrad time to workaround lightwalletd bugs
* Disable the flamegraph feature by default at compile time
* Disable the journald feature by default at compile time
* Also disable inferno dependency, and rearrange features
* Disable the prometheus feature by default at compile time
* Disable the tracing filter reload feature by default at compile time
* Disable tests when corresponding features are disabled
* Add compile-time tracing features to user docs
* Add compile-time features to the metrics user docs
* Document diagnostics as part of the start command tasks and services
* breaking(diagnostics): rename "enable-sentry" feature to "sentry" (#4623)
* Also skip conflict tests when those ports are disabled
* breaking(diagnostics): rename "enable-sentry" feature to "sentry"
This is mostly:
```sh
fastmod enable-sentry sentry
```
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(ci): lwd state condition
* fix(ci): differentiate tests that need a lwd cached state
* fix(ci): use the right state and save name for each test
* docs(ci): minor comment fixes
* docs(ci): better input description
* fix(ci): end `if` condition correctly
* fix(images): pass the state version to following steps
* fix(ci): $needs_lwd_state condition was inverted
* fix(ci): reduce disk selection code
* docs(ci): better disk search conditional explanation
* fix(ci): end if condition correctly
* fix(ci): evaluate $needs_zebra_state correctly
* fix(ci): use nested condition for readability
* fix(ci): disk search was using the wrong variable
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* Drop sentry dependencies when enable-sentry feature is disabled
* Make lightwalletd gRPC tests depend on a new lightwalletd-grpc-tests feature
* fix(ci): remove enable-sentry feature from tests
* Add lightwalletd-grpc-tests feature for functionality or efficiency
And document where it is just used to stop re-compilations.
* Remove redundant `cmake` and `protobuf-compiler` dependencies
* Document Zebra's optional production and test feature flags
* Minimise dependencies in zcash-params/Dockerfile
* Minimise dependencies in docker/Dockerfile
* Add a workflow TODO
* Catch more errors in entrypoint.sh
Also makes entrypoint.sh compatible with more distributions
* Remove unnecessary quoting in entrypoint.sh
* Use exactly the same arguments to call CI tests
* Remove a redundant CI build
* Rename Cargo.lock check job
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(ci): sentry is not longer being activated in test builds
This removes sentry from all the test execution, as some tests might fail as sentry wasn't initially built, or it might take more time to build as it will have to build with sentry.
* fix(build): workaround the failed to fetch oauth token error
* Drop sentry dependencies when enable-sentry feature is disabled
* Make lightwalletd gRPC tests depend on a new lightwalletd-grpc-tests feature
* fix(ci): remove enable-sentry feature from tests
* Add lightwalletd-grpc-tests feature for functionality or efficiency
And document where it is just used to stop re-compilations.
* Remove redundant `cmake` and `protobuf-compiler` dependencies
* Document Zebra's optional production and test feature flags
* Minimise dependencies in zcash-params/Dockerfile
* Minimise dependencies in docker/Dockerfile
* Add a workflow TODO
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
teor
Gustavo Valverde
Alfredo Garcia
Marek
Arya
Dimitris Apostolou