zebra/docker/Dockerfile

229 lines
9.0 KiB
Docker
Raw Normal View History

# syntax=docker/dockerfile:1
# check=skip=UndefinedVar
# If you want to include a file in the Docker image, add it to .dockerignore.
#
# We are using 4 stages:
# - deps: install build dependencies and sets the needed variables
# - tests: builds tests binaries
# - release: builds release binaries
# - runtime: runs the release binaries
#
# We first set default values for build arguments used across the stages.
# Each stage must define the build arguments (ARGs) it uses.
#
# Build zebrad with these features
#
# Keep these argument defaults in sync with GitHub vars.RUST_PROD_FEATURES and vars.RUST_TEST_FEATURES
# https://github.com/ZcashFoundation/zebra/settings/variables/actions
ARG FEATURES="default-release-binaries"
ARG TEST_FEATURES="lightwalletd-grpc-tests zebra-checkpoints"
ARG EXPERIMENTAL_FEATURES=""
ARG APP_HOME="/opt/zebrad"
ARG RUST_VERSION=1.79.0
# In this stage we download all system requirements to build the project
#
# It also captures all the build arguments to be used as environment variables.
# We set defaults for the arguments, in case the build does not include this information.
FROM rust:${RUST_VERSION}-bookworm AS deps
SHELL ["/bin/bash", "-xo", "pipefail", "-c"]
# Set the default path for the zebrad binary
ARG APP_HOME
ENV APP_HOME=${APP_HOME}
WORKDIR ${APP_HOME}
refactor(cd): improve Docker and gcloud usage without Cloud Build (#3431) * refactor (cd): overall pipeline improvement - Use a more ENV configurable Dockerfile - Remove cloudbuild dependency - Use compute optimized machine types - Use SSD instead of normal hard drives - Move Sentry endpoint to secrets - Use a single yml for auto & manual deploy - Migrate to Google Artifact Registry * refactor (cd): overall pipeline improvement - Use a more ENV configurable Dockerfile - Remove cloudbuild dependency - Use compute optimized machine types - Use SSD instead of normal hard drives - Move Sentry endpoint to secrets - Use a single yml for auto & manual deploy - Migrate to Google Artifact Registry * refactor (cd): use newer google auth action * fix (cd): use newer secret as gcp credential * fix (docker): do not create extra directories * fix (docker): ignore .github for caching purposes * fix (docker): use latest rust * fix: use a better name for manual deployment * refactor (docker): use standard directories for executable * fix (cd): most systems expect a "latest" tag Caching from the latest image is one of the main reasons to add this extra tag. Before this commit, the inline cache was not being used. * fix (cd): push the build image and the cache separately The inline cache exporter only supports `min` cache mode. To enable `max` cache mode, push the image and the cache separately by using the registry cache exporter. This also allows for smaller release images. * fix (cd): remove unused GHA cache We're leveraging the registry to cache the actions, instead of using the 10GB limits from Github Actions cache storage * refactor (cd): use cargo-chef for caching rust deps * fix (release): use newer debian to reduce vulnerabilities * fix (cd): use same zone, region and service accounts * fix (cd): use same disk size and type for all deployments * refactor (cd): activate interactive shells Use interactive shells for manual and test deployments. This allow greater flexibility if troubleshooting is needed inside the machines * fix (docker): do not build with different settings Compiling might be slow because different steps are compiling the same code 2-4 times because of the variations * fix(cd): use Mainnet instead of mainnet * fix(docker): remove tests as a runtime dependency * fix(cd): use default service account with cloud-platform scope * fix(cd): keep compatibility with gcr.io To prevent conflicts between registries, and migrate when the time is right, we'll keep pushing to both registries and use github actions cache to prevent conflicts between artifacts. * fix(docker): do not download zcash params twice * feat(docker): add google OS Config agent Use a separate step to have better flexibility in case a better approach is available * fix(docker): allow to use zebrad as a command * feat: add an image to inherit from with zcash params * refactor(docker): use cached zcash params from previous build * imp(cd): add double safety measure for production
2022-02-08 16:50:13 -08:00
# Install zebra build deps and Dockerfile deps
refactor(cd): improve Docker and gcloud usage without Cloud Build (#3431) * refactor (cd): overall pipeline improvement - Use a more ENV configurable Dockerfile - Remove cloudbuild dependency - Use compute optimized machine types - Use SSD instead of normal hard drives - Move Sentry endpoint to secrets - Use a single yml for auto & manual deploy - Migrate to Google Artifact Registry * refactor (cd): overall pipeline improvement - Use a more ENV configurable Dockerfile - Remove cloudbuild dependency - Use compute optimized machine types - Use SSD instead of normal hard drives - Move Sentry endpoint to secrets - Use a single yml for auto & manual deploy - Migrate to Google Artifact Registry * refactor (cd): use newer google auth action * fix (cd): use newer secret as gcp credential * fix (docker): do not create extra directories * fix (docker): ignore .github for caching purposes * fix (docker): use latest rust * fix: use a better name for manual deployment * refactor (docker): use standard directories for executable * fix (cd): most systems expect a "latest" tag Caching from the latest image is one of the main reasons to add this extra tag. Before this commit, the inline cache was not being used. * fix (cd): push the build image and the cache separately The inline cache exporter only supports `min` cache mode. To enable `max` cache mode, push the image and the cache separately by using the registry cache exporter. This also allows for smaller release images. * fix (cd): remove unused GHA cache We're leveraging the registry to cache the actions, instead of using the 10GB limits from Github Actions cache storage * refactor (cd): use cargo-chef for caching rust deps * fix (release): use newer debian to reduce vulnerabilities * fix (cd): use same zone, region and service accounts * fix (cd): use same disk size and type for all deployments * refactor (cd): activate interactive shells Use interactive shells for manual and test deployments. This allow greater flexibility if troubleshooting is needed inside the machines * fix (docker): do not build with different settings Compiling might be slow because different steps are compiling the same code 2-4 times because of the variations * fix(cd): use Mainnet instead of mainnet * fix(docker): remove tests as a runtime dependency * fix(cd): use default service account with cloud-platform scope * fix(cd): keep compatibility with gcr.io To prevent conflicts between registries, and migrate when the time is right, we'll keep pushing to both registries and use github actions cache to prevent conflicts between artifacts. * fix(docker): do not download zcash params twice * feat(docker): add google OS Config agent Use a separate step to have better flexibility in case a better approach is available * fix(docker): allow to use zebrad as a command * feat: add an image to inherit from with zcash params * refactor(docker): use cached zcash params from previous build * imp(cd): add double safety measure for production
2022-02-08 16:50:13 -08:00
RUN apt-get -qq update && \
apt-get -qq install -y --no-install-recommends \
llvm \
libclang-dev \
clang \
ca-certificates \
add(test): Integration test to send transactions using lightwalletd (#4068) * Export the `zebra_state::Config::db_path` method Make it easier for tests to discover the sub-directory used to store the chain state data. * Generate code for interfacing with lightwalletd Use the `tonic-build` crate to generate Rust code for communicating with lightwalletd using gRPC. The `*.proto` files were obtained from the Zcash lightwalletd repository. * Use `block::Height` instead of `Height` Import the `block` instead to make it slightly clearer. * Add helper function to remove a file if it exists Try to remove it and ignore an error if it says that the file doesn't exist. This will be used later to remove the lock file from a copied chain state directory. * Add helper function to copy chain state dirs Copy an existing chain state directory into a new temporary directory. * Add a `BoxStateService` type alias Make it easier to write and read a boxed version of a state service. * Add a helper function to start the state service Make it easier to specify the state service to use an existing state cache directory. * Import `eyre!` macro at the module level Allow it to be used in different places without having to repeat the imports. * Add `load_tip_height_from_state_directory` helper A function to discover the current chain tip height stored in a state cache. * Add helper function to prepare partial sync. state Loads a partially synchronized cached state directory into a temporary directory that can be used by a zebrad instance, and also returns the chain tip block height of that state. * Add `perform_full_sync_starting_from` helper Runs a zebrad with an existing partially synchronized state, and finishes synchronizing it to the network chain tip. * Add function to load transactions from a block Use a provided state service to load all transactions from a block at a specified height. The state service is a generic type parameter, because `zebra_state::service::ReadStateService` is not exported publicly. Using a generic type parameter also allows the service to be wrapped in layers if needed in the future. * Add `load_transactions_from_block_after` helper A function to load transactions from a block stored in a cached state directory. The cached state must be synchronized to a chain tip higher than the requested height. * Add helper function to load some test transactions Given a partially synchronized chain state, it will extend that chain by performing a full synchronization, and obtain some transactions from one of the newly added blocks. * Update `spawn_zebrad_for_rpc_without_initial_peers` Wait until the mempool is activated. * Add method to start lightwalletd with RPC server Returns the lightwalletd instance and the port that it's listening for RPC connections. The instance can reuse an existing cached lightwalletd state if the `LIGHTWALLETD_DATA_DIR` environment variable is set. * Add a `LightwalletdRpcClient` type alias To make it easier to identify the type generated from the Protobuf files. * Add helper function to connect to lightwalletd Prepare an RPC client to send requests to a lightwalletd instance. * Add a `prepare_send_transaction_request` helper Creates a request message for lightwalletd to send a transaction. * Add test to send transactions using lightwalletd Obtain some valid transactions from future blocks and try to send them to a lightwalletd instance connected to a zebrad instance that hasn't seen those transactions yet. The transactions should be successfully queued in Zebra's mempool. * Make `zebra_directory` parameter generic Allow using a `TempDir` or a `PathBuf`. * Move lightwalletd protobuf files Place them closer to the module directory, so that it's clearer that they specify the RPC protocol for lightwalletd, and not Zebra itself. * Don't use coinbase transactions in the test Coinbase transactions are rejected by the mempool. * Don't remove state lock file It is removed automatically by Zebra when it shuts down, so if it exists it should be reported as a bug. * Force mempool to be enabled in Zebrad instance Speed up the initialization of the Zebrad instance used for lightwalletd to connect to. * Refactor to create `LIGHTWALLETD_DATA_DIR_VAR` Document how the environment variable can be used to speed up the test. * Check for process errors in spawned Zebra instance Enable checking for known process failure messages. * Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant Document why it exists and how the choice of the value affects the test. * Add `LIGHTWALLETD_TEST_TIMEOUT` constant And use it for the Zebrad and the Lightwalletd instances used in the send transaction integration test. * Check `lightwalletd` process for errors Enable checking the lightwalletd process for known failure messages. * Update `tonic` and `prost` dependencies Use the latest version and fix CI failures because `rustfmt` isn't installed in the build environment. * Create `send_transaction_test` module Move the send transaction using lightwalletd test and its helper functions into a new module. * Move `LIGHTWALLETD_TEST_TIMEOUT` constant Place it in the parent `lightwalletd` module. * Move gRPC helper functions and types to `rpc` mod. Make them more accessible so that they can be used by other tests. * Create a `cached_state` module Move the test utility functions related to using a cached Zebra state into the module. * Move `perform_full_sync_starting_from` to `sync` Keep to closer to the synchronization utility functions. * Move Zebra cached state path variable constant Place it in the `cached_state` module. * Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set Make it part of the set of tests ignored as a whole if no lightwalletd tests should be executed. * Move `spawn_zebrad_for_rpc_without_initial_peers` Place it in the `launch` sub-module. * Rename `rpc` module into `wallet_grpc` Avoid any potential misunderstandings when the name is seen out of context. * Allow duplicate `heck` dependency At least until `structopt` is updated or `zebra-utils` is updated to use `clap` 3. * Fix a deny.toml typo * fix(build): CMake is required by `prost` crate Co-authored-by: teor <teor@riseup.net> Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-04-27 16:06:11 -07:00
protobuf-compiler \
rocksdb-tools \
&& rm -rf /var/lib/apt/lists/* /tmp/*
refactor(cd): improve Docker and gcloud usage without Cloud Build (#3431) * refactor (cd): overall pipeline improvement - Use a more ENV configurable Dockerfile - Remove cloudbuild dependency - Use compute optimized machine types - Use SSD instead of normal hard drives - Move Sentry endpoint to secrets - Use a single yml for auto & manual deploy - Migrate to Google Artifact Registry * refactor (cd): overall pipeline improvement - Use a more ENV configurable Dockerfile - Remove cloudbuild dependency - Use compute optimized machine types - Use SSD instead of normal hard drives - Move Sentry endpoint to secrets - Use a single yml for auto & manual deploy - Migrate to Google Artifact Registry * refactor (cd): use newer google auth action * fix (cd): use newer secret as gcp credential * fix (docker): do not create extra directories * fix (docker): ignore .github for caching purposes * fix (docker): use latest rust * fix: use a better name for manual deployment * refactor (docker): use standard directories for executable * fix (cd): most systems expect a "latest" tag Caching from the latest image is one of the main reasons to add this extra tag. Before this commit, the inline cache was not being used. * fix (cd): push the build image and the cache separately The inline cache exporter only supports `min` cache mode. To enable `max` cache mode, push the image and the cache separately by using the registry cache exporter. This also allows for smaller release images. * fix (cd): remove unused GHA cache We're leveraging the registry to cache the actions, instead of using the 10GB limits from Github Actions cache storage * refactor (cd): use cargo-chef for caching rust deps * fix (release): use newer debian to reduce vulnerabilities * fix (cd): use same zone, region and service accounts * fix (cd): use same disk size and type for all deployments * refactor (cd): activate interactive shells Use interactive shells for manual and test deployments. This allow greater flexibility if troubleshooting is needed inside the machines * fix (docker): do not build with different settings Compiling might be slow because different steps are compiling the same code 2-4 times because of the variations * fix(cd): use Mainnet instead of mainnet * fix(docker): remove tests as a runtime dependency * fix(cd): use default service account with cloud-platform scope * fix(cd): keep compatibility with gcr.io To prevent conflicts between registries, and migrate when the time is right, we'll keep pushing to both registries and use github actions cache to prevent conflicts between artifacts. * fix(docker): do not download zcash params twice * feat(docker): add google OS Config agent Use a separate step to have better flexibility in case a better approach is available * fix(docker): allow to use zebrad as a command * feat: add an image to inherit from with zcash params * refactor(docker): use cached zcash params from previous build * imp(cd): add double safety measure for production
2022-02-08 16:50:13 -08:00
# Build arguments and variables set for tracelog levels and debug information
#
# We set defaults to all variables.
ARG RUST_LOG
ENV RUST_LOG=${RUST_LOG:-info}
feat(actions)!: add full sync test (#3582) * add(tests): full sync test * fix(test): add build * fix(deploy): escape double dashes '--' correctly * fix(test): remove unexpected --no-capture arg error: Found argument '--nocapture' which wasn't expected, or isn't valid in this context * refactor(docker): use default executable as entrypoint * refactor(startup): add a custom entrypoint * fix(test): add missing TEST_FULL_SYNC variable * test(timeout): use the biggest machine * fix * fix(deploy): use latest successful image * typo * refactor(docker): generate config file at startup * revert(build): changes were made to docker * fix(docker): send variables correctly to the entrypoint * test different conf file approach * fix(env): add RUN_TEST env variable * ref: use previous approach * fix(color): use environment variable * fix(resources): use our normal machine size * fix(ci): double CPU and RAM for full sync test * fix(test): check for zebrad test output in the correct order The mempool is only activated once, so we must check for that log first. After mempool activation, the stop regex is logged at least once. (It might be logged before as well, but we can't rely on that.) When checking that the mempool didn't activate, wait for the `zebrad` command to exit, then check the entire log. * fix(ci): run full sync test with full compiler optimisations * fix(tests): reintroduce tests and run full sync on approval * fix(tests): reduce the changelog Co-authored-by: teor <teor@riseup.net>
2022-03-02 06:15:24 -08:00
ARG RUST_BACKTRACE
ENV RUST_BACKTRACE=${RUST_BACKTRACE:-1}
ARG RUST_LIB_BACKTRACE
ENV RUST_LIB_BACKTRACE=${RUST_LIB_BACKTRACE:-1}
ARG COLORBT_SHOW_HIDDEN
ENV COLORBT_SHOW_HIDDEN=${COLORBT_SHOW_HIDDEN:-1}
ARG SHORT_SHA
# If this is not set, it must be an empty string, so Zebra can try an alternative git commit source:
2023-07-20 04:50:13 -07:00
# https://github.com/ZcashFoundation/zebra/blob/9ebd56092bcdfc1a09062e15a0574c94af37f389/zebrad/src/application.rs#L179-L182
ENV SHORT_SHA=${SHORT_SHA:-}
feat(actions)!: add full sync test (#3582) * add(tests): full sync test * fix(test): add build * fix(deploy): escape double dashes '--' correctly * fix(test): remove unexpected --no-capture arg error: Found argument '--nocapture' which wasn't expected, or isn't valid in this context * refactor(docker): use default executable as entrypoint * refactor(startup): add a custom entrypoint * fix(test): add missing TEST_FULL_SYNC variable * test(timeout): use the biggest machine * fix * fix(deploy): use latest successful image * typo * refactor(docker): generate config file at startup * revert(build): changes were made to docker * fix(docker): send variables correctly to the entrypoint * test different conf file approach * fix(env): add RUN_TEST env variable * ref: use previous approach * fix(color): use environment variable * fix(resources): use our normal machine size * fix(ci): double CPU and RAM for full sync test * fix(test): check for zebrad test output in the correct order The mempool is only activated once, so we must check for that log first. After mempool activation, the stop regex is logged at least once. (It might be logged before as well, but we can't rely on that.) When checking that the mempool didn't activate, wait for the `zebrad` command to exit, then check the entire log. * fix(ci): run full sync test with full compiler optimisations * fix(tests): reintroduce tests and run full sync on approval * fix(tests): reduce the changelog Co-authored-by: teor <teor@riseup.net>
2022-03-02 06:15:24 -08:00
ENV CARGO_HOME="${APP_HOME}/.cargo/"
# Copy the entrypoint script to be used on both images
COPY ./docker/entrypoint.sh /etc/zebrad/entrypoint.sh
# In this stage we build tests (without running then)
#
# We also download needed dependencies for tests to work, from other images.
# An entrypoint.sh is only available in this step for easier test handling with variables.
FROM deps AS tests
# Skip IPv6 tests by default, as some CI environment don't have IPv6 available
ARG ZEBRA_SKIP_IPV6_TESTS
ENV ZEBRA_SKIP_IPV6_TESTS=${ZEBRA_SKIP_IPV6_TESTS:-1}
# Use ENTRYPOINT_FEATURES to override the specific features used to run tests in entrypoint.sh,
# separately from the test and production image builds.
ARG FEATURES
ARG TEST_FEATURES
ARG EXPERIMENTAL_FEATURES
# TODO: add empty $EXPERIMENTAL_FEATURES when we can avoid adding an extra space to the end of the string
ARG ENTRYPOINT_FEATURES="${FEATURES} ${TEST_FEATURES}"
# Build Zebra test binaries, but don't run them
# Leverage a cache mount to /usr/local/cargo/registry/
# for downloaded dependencies, a cache mount to /usr/local/cargo/git/db
# for git repository dependencies, and a cache mount to ${APP_HOME}/target/ for
# compiled dependencies which will speed up subsequent builds.
# Leverage a bind mount to each crate directory to avoid having to copy the
# source code into the container. Once built, copy the executable to an
# output directory before the cache mounted ${APP_HOME}/target/ is unmounted.
RUN --mount=type=bind,source=zebrad,target=zebrad \
--mount=type=bind,source=zebra-chain,target=zebra-chain \
--mount=type=bind,source=zebra-network,target=zebra-network \
--mount=type=bind,source=zebra-state,target=zebra-state \
--mount=type=bind,source=zebra-script,target=zebra-script \
--mount=type=bind,source=zebra-consensus,target=zebra-consensus \
--mount=type=bind,source=zebra-rpc,target=zebra-rpc \
--mount=type=bind,source=zebra-node-services,target=zebra-node-services \
--mount=type=bind,source=zebra-test,target=zebra-test \
--mount=type=bind,source=zebra-utils,target=zebra-utils \
--mount=type=bind,source=zebra-scan,target=zebra-scan \
--mount=type=bind,source=zebra-grpc,target=zebra-grpc \
--mount=type=bind,source=tower-batch-control,target=tower-batch-control \
--mount=type=bind,source=tower-fallback,target=tower-fallback \
--mount=type=bind,source=Cargo.toml,target=Cargo.toml \
--mount=type=bind,source=Cargo.lock,target=Cargo.lock \
--mount=type=cache,target=${APP_HOME}/target/ \
--mount=type=cache,target=/usr/local/cargo/git/db \
--mount=type=cache,target=/usr/local/cargo/registry/ \
cargo test --locked --release --features "${ENTRYPOINT_FEATURES}" --workspace --no-run && \
cp ${APP_HOME}/target/release/zebrad /usr/local/bin && \
cp ${APP_HOME}/target/release/zebra-checkpoints /usr/local/bin
# Copy the lightwalletd binary and source files to be able to run tests
COPY --from=electriccoinco/lightwalletd:latest /usr/local/bin/lightwalletd /usr/local/bin/
COPY ./ ./
feat(actions)!: add full sync test (#3582) * add(tests): full sync test * fix(test): add build * fix(deploy): escape double dashes '--' correctly * fix(test): remove unexpected --no-capture arg error: Found argument '--nocapture' which wasn't expected, or isn't valid in this context * refactor(docker): use default executable as entrypoint * refactor(startup): add a custom entrypoint * fix(test): add missing TEST_FULL_SYNC variable * test(timeout): use the biggest machine * fix * fix(deploy): use latest successful image * typo * refactor(docker): generate config file at startup * revert(build): changes were made to docker * fix(docker): send variables correctly to the entrypoint * test different conf file approach * fix(env): add RUN_TEST env variable * ref: use previous approach * fix(color): use environment variable * fix(resources): use our normal machine size * fix(ci): double CPU and RAM for full sync test * fix(test): check for zebrad test output in the correct order The mempool is only activated once, so we must check for that log first. After mempool activation, the stop regex is logged at least once. (It might be logged before as well, but we can't rely on that.) When checking that the mempool didn't activate, wait for the `zebrad` command to exit, then check the entire log. * fix(ci): run full sync test with full compiler optimisations * fix(tests): reintroduce tests and run full sync on approval * fix(tests): reduce the changelog Co-authored-by: teor <teor@riseup.net>
2022-03-02 06:15:24 -08:00
# Entrypoint environment variables
ENV ENTRYPOINT_FEATURES=${ENTRYPOINT_FEATURES}
# We repeat the ARGs here, so they are available in the entrypoint.sh script for $RUN_ALL_EXPERIMENTAL_TESTS
ARG EXPERIMENTAL_FEATURES="journald prometheus filter-reload"
ENV ENTRYPOINT_FEATURES_EXPERIMENTAL="${ENTRYPOINT_FEATURES} ${EXPERIMENTAL_FEATURES}"
# By default, runs the entrypoint tests specified by the environmental variables (if any are set)
ENTRYPOINT [ "/etc/zebrad/entrypoint.sh" ]
# In this stage we build a release (generate the zebrad binary)
#
# This step also adds `cache mounts` as this stage is completely independent from the
# `test` stage. This step is a dependency for the `runtime` stage, which uses the resulting
# zebrad binary from this step.
FROM deps AS release
2023-06-14 15:15:51 -07:00
ARG FEATURES
RUN --mount=type=bind,source=tower-batch-control,target=tower-batch-control \
--mount=type=bind,source=tower-fallback,target=tower-fallback \
--mount=type=bind,source=zebra-chain,target=zebra-chain \
--mount=type=bind,source=zebra-consensus,target=zebra-consensus \
--mount=type=bind,source=zebra-grpc,target=zebra-grpc \
--mount=type=bind,source=zebra-network,target=zebra-network \
--mount=type=bind,source=zebra-node-services,target=zebra-node-services \
--mount=type=bind,source=zebra-rpc,target=zebra-rpc \
--mount=type=bind,source=zebra-scan,target=zebra-scan \
--mount=type=bind,source=zebra-script,target=zebra-script \
--mount=type=bind,source=zebra-state,target=zebra-state \
--mount=type=bind,source=zebra-test,target=zebra-test \
--mount=type=bind,source=zebra-utils,target=zebra-utils \
--mount=type=bind,source=zebrad,target=zebrad \
--mount=type=bind,source=Cargo.toml,target=Cargo.toml \
--mount=type=bind,source=Cargo.lock,target=Cargo.lock \
--mount=type=cache,target=${APP_HOME}/target/ \
--mount=type=cache,target=/usr/local/cargo/git/db \
--mount=type=cache,target=/usr/local/cargo/registry/ \
cargo build --locked --release --features "${FEATURES}" --package zebrad --bin zebrad && \
cp ${APP_HOME}/target/release/zebrad /usr/local/bin
# This stage is only used when deploying nodes or when only the resulting zebrad binary is needed
#
# To save space, this step starts from scratch using debian, and only adds the resulting
# binary from the `release` stage
FROM debian:bookworm-slim AS runtime
# Set the default path for the zebrad binary
ARG APP_HOME
ENV APP_HOME=${APP_HOME}
WORKDIR ${APP_HOME}
RUN apt-get update && \
refactor(cd): improve Docker and gcloud usage without Cloud Build (#3431) * refactor (cd): overall pipeline improvement - Use a more ENV configurable Dockerfile - Remove cloudbuild dependency - Use compute optimized machine types - Use SSD instead of normal hard drives - Move Sentry endpoint to secrets - Use a single yml for auto & manual deploy - Migrate to Google Artifact Registry * refactor (cd): overall pipeline improvement - Use a more ENV configurable Dockerfile - Remove cloudbuild dependency - Use compute optimized machine types - Use SSD instead of normal hard drives - Move Sentry endpoint to secrets - Use a single yml for auto & manual deploy - Migrate to Google Artifact Registry * refactor (cd): use newer google auth action * fix (cd): use newer secret as gcp credential * fix (docker): do not create extra directories * fix (docker): ignore .github for caching purposes * fix (docker): use latest rust * fix: use a better name for manual deployment * refactor (docker): use standard directories for executable * fix (cd): most systems expect a "latest" tag Caching from the latest image is one of the main reasons to add this extra tag. Before this commit, the inline cache was not being used. * fix (cd): push the build image and the cache separately The inline cache exporter only supports `min` cache mode. To enable `max` cache mode, push the image and the cache separately by using the registry cache exporter. This also allows for smaller release images. * fix (cd): remove unused GHA cache We're leveraging the registry to cache the actions, instead of using the 10GB limits from Github Actions cache storage * refactor (cd): use cargo-chef for caching rust deps * fix (release): use newer debian to reduce vulnerabilities * fix (cd): use same zone, region and service accounts * fix (cd): use same disk size and type for all deployments * refactor (cd): activate interactive shells Use interactive shells for manual and test deployments. This allow greater flexibility if troubleshooting is needed inside the machines * fix (docker): do not build with different settings Compiling might be slow because different steps are compiling the same code 2-4 times because of the variations * fix(cd): use Mainnet instead of mainnet * fix(docker): remove tests as a runtime dependency * fix(cd): use default service account with cloud-platform scope * fix(cd): keep compatibility with gcr.io To prevent conflicts between registries, and migrate when the time is right, we'll keep pushing to both registries and use github actions cache to prevent conflicts between artifacts. * fix(docker): do not download zcash params twice * feat(docker): add google OS Config agent Use a separate step to have better flexibility in case a better approach is available * fix(docker): allow to use zebrad as a command * feat: add an image to inherit from with zcash params * refactor(docker): use cached zcash params from previous build * imp(cd): add double safety measure for production
2022-02-08 16:50:13 -08:00
apt-get install -y --no-install-recommends \
ca-certificates \
curl \
rocksdb-tools \
gosu \
&& rm -rf /var/lib/apt/lists/* /tmp/*
# Create a non-privileged user that the app will run under.
# Running as root inside the container is running as root in the Docker host
# If an attacker manages to break out of the container, they will have root access to the host
# See https://docs.docker.com/go/dockerfile-user-best-practices/
ARG USER=zebra
ENV USER=${USER}
ARG UID=10001
ENV UID=${UID}
ARG GID=10001
ENV GID=${GID}
RUN addgroup --system --gid ${GID} ${USER} \
&& adduser \
--system \
--disabled-login \
--shell /bin/bash \
--home ${APP_HOME} \
--uid "${UID}" \
--gid "${GID}" \
${USER}
# Config settings for zebrad
ARG FEATURES
ENV FEATURES=${FEATURES}
ref(docker): combine test and prod entrypoints into one (#7660) * ref(docker): combine test and prod entrypoints into one This makes the following changes - Uses `-x` instead of echoing the variables values - Sets default values where required - Create a function to list directories - Create a function to run cargo tests - Use a better approach to handle different options in the `case` manegement for tests and production - Replaces all instances of `runtime-entrypoint.sh` with `entrypoint.sh` * chore: use 2 spaces insted of 4 for indentation * fix: add missing `test-threads` to `fully_synced_rpc_` * fix: allow 3 arguments instead of 2 to handle `fully_synced_rpc_` correctly * chore: remove extra file * chore: add comments to main parts of the file * fix(ci): just create the `$ZEBRA_CONF_PATH` if `$ENTRYPOINT_FEATURES` is not set * fix(entrypoint): just print ${ZEBRA_CONF_PATH} if exists * fix: missing condition * ref: handle tests better if `$ENTRYPOINT_FEATURES` is set * fix(ci): We just want `ZEBRA_CONF_PATH` to be set in the `release` image * fix(entrypoint): fix the overall `case` logic * fix(ci): allos to run with custom config in CI image * fix(ci): more edgecases * fix: we don't need to find files, but subdirectories * fix(ci): handle Signal Forwarding and exit codes for `cargo` * fix(ci): parse `fully_synced_rpc_` test correctly * chore: add missing cache dir variable for LWD * fix(entrypoint): handle an arbitrary number of arguments * fix(entrypoint): handle features list * fix(entrypoint): typo * chore: typo
2023-10-11 12:04:45 -07:00
# Path and name of the config file
# These are set to a default value when not defined in the environment
ref(workflows): consolidate workflows based on their purpose (#7616) * ref(workflows): consolidate workflows based on their purpose This also renames the workflows to make their naming more consistent and adding a naming convention Fixes: #6166 Fixes: #6167 * fix(workflows): use correct name for patch * fix(workflow): docker unit tests * fix(release): validation error Error: ``` Validation Failed: {"resource":"Release","code":"invalid","field":"target_commitish"} ``` Fixes: https://github.com/release-drafter/release-drafter/issues/1125 * fix(workflows): reference correct name * fix: remove extra workflow * fix(workflows): use larger runners * fix(workflow): remove code already in docker unit-test * fix(unit-tests): start zebra the right way * fix: typo in patch name * chore: move job to logical order * imp(workflows): use better name for gcp tests * add: missing merge changes * chore: use better name for find-disks * fix(ci): use the `entrypoint.sh` to change the Network * fix(ci): add missing `ZEBRA_CONF_PATH` variable * fix(ci): allow to build the entrypoint file with testnet * fix(entrypoint): allow to create a dir and file with a single variable * refactor: test config file in CI and CD with a reusable workflow * fix(ci): wrong name used * fix(ci): use checkout * fix(ci): improve docker config tests * fix(ci): use better name for protection rules * Fix changed workflow file name in docs patch file * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> Co-authored-by: Arya <aryasolhi@gmail.com> * fix(cd): depend on file tests * fix(docs): adapt to new workflow name * fix: revert test coverage on CD * chore: reduce diff * fix(ci): allow using variable images for reusable workflows * fix(dockerfile): use variables or default for config path and file * fix(entrypoint): if `$ZEBRA_CONF_PATH` is set, do not override it * Fix patch job names and remove failure job testnet dependencies --------- Co-authored-by: teor <teor@riseup.net> Co-authored-by: Arya <aryasolhi@gmail.com>
2023-10-17 23:16:02 -07:00
ENV ZEBRA_CONF_DIR=${ZEBRA_CONF_DIR:-/etc/zebrad}
ENV ZEBRA_CONF_FILE=${ZEBRA_CONF_FILE:-zebrad.toml}
ref(docker): combine test and prod entrypoints into one (#7660) * ref(docker): combine test and prod entrypoints into one This makes the following changes - Uses `-x` instead of echoing the variables values - Sets default values where required - Create a function to list directories - Create a function to run cargo tests - Use a better approach to handle different options in the `case` manegement for tests and production - Replaces all instances of `runtime-entrypoint.sh` with `entrypoint.sh` * chore: use 2 spaces insted of 4 for indentation * fix: add missing `test-threads` to `fully_synced_rpc_` * fix: allow 3 arguments instead of 2 to handle `fully_synced_rpc_` correctly * chore: remove extra file * chore: add comments to main parts of the file * fix(ci): just create the `$ZEBRA_CONF_PATH` if `$ENTRYPOINT_FEATURES` is not set * fix(entrypoint): just print ${ZEBRA_CONF_PATH} if exists * fix: missing condition * ref: handle tests better if `$ENTRYPOINT_FEATURES` is set * fix(ci): We just want `ZEBRA_CONF_PATH` to be set in the `release` image * fix(entrypoint): fix the overall `case` logic * fix(ci): allos to run with custom config in CI image * fix(ci): more edgecases * fix: we don't need to find files, but subdirectories * fix(ci): handle Signal Forwarding and exit codes for `cargo` * fix(ci): parse `fully_synced_rpc_` test correctly * chore: add missing cache dir variable for LWD * fix(entrypoint): handle an arbitrary number of arguments * fix(entrypoint): handle features list * fix(entrypoint): typo * chore: typo
2023-10-11 12:04:45 -07:00
RUN mkdir -p ${ZEBRA_CONF_DIR} && chown ${UID}:${UID} ${ZEBRA_CONF_DIR} \
&& chown ${UID}:${UID} ${APP_HOME}
COPY --from=release /usr/local/bin/zebrad /usr/local/bin
COPY --from=release /etc/zebrad/entrypoint.sh /etc/zebrad
# Expose configured ports
EXPOSE 8233 18233
# Update the config file based on the Docker run variables,
# and launch zebrad with it
ENTRYPOINT [ "/etc/zebrad/entrypoint.sh" ]
ref(docker): combine test and prod entrypoints into one (#7660) * ref(docker): combine test and prod entrypoints into one This makes the following changes - Uses `-x` instead of echoing the variables values - Sets default values where required - Create a function to list directories - Create a function to run cargo tests - Use a better approach to handle different options in the `case` manegement for tests and production - Replaces all instances of `runtime-entrypoint.sh` with `entrypoint.sh` * chore: use 2 spaces insted of 4 for indentation * fix: add missing `test-threads` to `fully_synced_rpc_` * fix: allow 3 arguments instead of 2 to handle `fully_synced_rpc_` correctly * chore: remove extra file * chore: add comments to main parts of the file * fix(ci): just create the `$ZEBRA_CONF_PATH` if `$ENTRYPOINT_FEATURES` is not set * fix(entrypoint): just print ${ZEBRA_CONF_PATH} if exists * fix: missing condition * ref: handle tests better if `$ENTRYPOINT_FEATURES` is set * fix(ci): We just want `ZEBRA_CONF_PATH` to be set in the `release` image * fix(entrypoint): fix the overall `case` logic * fix(ci): allos to run with custom config in CI image * fix(ci): more edgecases * fix: we don't need to find files, but subdirectories * fix(ci): handle Signal Forwarding and exit codes for `cargo` * fix(ci): parse `fully_synced_rpc_` test correctly * chore: add missing cache dir variable for LWD * fix(entrypoint): handle an arbitrary number of arguments * fix(entrypoint): handle features list * fix(entrypoint): typo * chore: typo
2023-10-11 12:04:45 -07:00
CMD ["zebrad"]