zebra/.github/workflows/ci.yml

248 lines
7.5 KiB
YAML
Raw Normal View History

name: CI
on:
workflow_dispatch:
pull_request:
path:
- '**/*.rs'
- '**/*.txt'
- '**/Cargo.toml'
- '**/Cargo.lock'
- '**/deny.toml'
- 'docker/**'
- '.github/workflows/ci.yml'
env:
CARGO_INCREMENTAL: 0
RUST_BACKTRACE: full
jobs:
test:
name: Test (+${{ matrix.rust }}) on ${{ matrix.os }}
# The large timeout is to accommodate:
# - Windows builds (75 minutes, typically 30-50 minutes)
# - parameter downloads (40 minutes, but only when the cache expires)
timeout-minutes: 115
runs-on: ${{ matrix.os }}
strategy:
2021-02-18 15:54:35 -08:00
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macOS-latest]
rust: [stable]
steps:
- uses: actions/checkout@v2.4.0
with:
persist-credentials: false
2021-06-14 18:36:33 -07:00
- uses: actions-rs/toolchain@v1
with:
2021-02-11 07:06:54 -08:00
toolchain: ${{ matrix.rust }}
2021-04-29 13:47:15 -07:00
profile: minimal
override: true
2021-06-14 18:36:33 -07:00
- uses: Swatinem/rust-cache@v1
- name: cargo fetch
uses: actions-rs/cargo@v1.0.3
with:
command: fetch
2021-06-14 18:36:33 -07:00
- name: Install LLVM on Windows
if: matrix.os == 'windows-latest'
run: |
choco install llvm -y
echo "C:\Program Files\LLVM\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
echo "LIBCLANG_PATH=C:\Program Files\LLVM\bin" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
2021-06-14 18:36:33 -07:00
Download Zcash Sapling parameters and load them from cached files (#3057) * Replace Zcash parameters crates with pre-downloaded local parameter files * Download Zcash parameters using the `zcashd` script in CI and Docker * Add a zcash_proofs dependency to zebra-consensus * Download Sapling parameters using zcash_proofs, rather than fetch-params.sh * Add a new `zebrad download` subcommand This command isn't required for nomrmal usage. But it's useful when testing, or launching multiple Zebra instances. * Use `zebrad download` in CI to pre-download parameters * Log a helpful hint if downloading fails * Allow some duplicate dependencies currently hidden by orchard * Spawn a separate task to download Groth16 parameters * Run the parameter download with code coverage This avoids re-compining Zebra with and without coverage. * Update Cargo.lock after rebase * Try to pass `download` as an argument to `zebrad` in coverage CI * Fix copy and paste comment typos * Add path and download examples, like zcash_proofs * Download params in CI just like zcash_proofs does * Delete a redundant build step * Implement graceful shutdown for zebrad start * Send coverage summary to /dev/null when getting the params path * Use the correct parameters path and download commands in CI * Explain pre-downloads * Avoid calling params_folder twice * Rename parameter types and methods for consistency ```sh fastmod SaplingParams SaplingParameters zebra* fastmod Groth16Params Groth16Parameters zebra* fastmod PARAMS GROTH16_PARAMETERS zebra* fastmod params_folder directory zebra* ``` And a manual variable name tweak. * rustfmt * Remove a redundant coverage step Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 15:02:56 -08:00
- name: Skip network tests on Ubuntu and Windows
# Ubuntu runners don't have reliable network or DNS during test steps.
Download Zcash Sapling parameters and load them from cached files (#3057) * Replace Zcash parameters crates with pre-downloaded local parameter files * Download Zcash parameters using the `zcashd` script in CI and Docker * Add a zcash_proofs dependency to zebra-consensus * Download Sapling parameters using zcash_proofs, rather than fetch-params.sh * Add a new `zebrad download` subcommand This command isn't required for nomrmal usage. But it's useful when testing, or launching multiple Zebra instances. * Use `zebrad download` in CI to pre-download parameters * Log a helpful hint if downloading fails * Allow some duplicate dependencies currently hidden by orchard * Spawn a separate task to download Groth16 parameters * Run the parameter download with code coverage This avoids re-compining Zebra with and without coverage. * Update Cargo.lock after rebase * Try to pass `download` as an argument to `zebrad` in coverage CI * Fix copy and paste comment typos * Add path and download examples, like zcash_proofs * Download params in CI just like zcash_proofs does * Delete a redundant build step * Implement graceful shutdown for zebrad start * Send coverage summary to /dev/null when getting the params path * Use the correct parameters path and download commands in CI * Explain pre-downloads * Avoid calling params_folder twice * Rename parameter types and methods for consistency ```sh fastmod SaplingParams SaplingParameters zebra* fastmod Groth16Params Groth16Parameters zebra* fastmod PARAMS GROTH16_PARAMETERS zebra* fastmod params_folder directory zebra* ``` And a manual variable name tweak. * rustfmt * Remove a redundant coverage step Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 15:02:56 -08:00
# Windows runners have an unreliable network.
shell: bash
if: matrix.os != 'macOS-latest'
run: echo "ZEBRA_SKIP_NETWORK_TESTS=1" >> $GITHUB_ENV
2021-06-14 18:36:33 -07:00
- name: Minimise proptest cases on macOS and Windows
# We set cases to 1, because some tests already run 1 case by default.
# We keep maximum shrink iterations at the default value, because it only happens on failure.
#
# Windows compilation and tests are slower than other platforms.
# macOS runners do extra network tests, so they take longer.
shell: bash
if: matrix.os != 'ubuntu-latest'
run: |
echo "PROPTEST_CASES=1" >> $GITHUB_ENV
echo "PROPTEST_MAX_SHRINK_ITERS=1024" >> $GITHUB_ENV
Update to Tokio 1.13.0 (#2994) * Update `tower` to version `0.4.9` Update to latest version to add support for Tokio version 1. * Replace usage of `ServiceExt::ready_and` It was deprecated in favor of `ServiceExt::ready`. * Update Tokio dependency to version `1.13.0` This will break the build because the code isn't ready for the update, but future commits will fix the issues. * Replace import of `tokio::stream::StreamExt` Use `futures::stream::StreamExt` instead, because newer versions of Tokio don't have the `stream` feature. * Use `IntervalStream` in `zebra-network` In newer versions of Tokio `Interval` doesn't implement `Stream`, so the wrapper types from `tokio-stream` have to be used instead. * Use `IntervalStream` in `inventory_registry` In newer versions of Tokio the `Interval` type doesn't implement `Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used instead. * Use `BroadcastStream` in `inventory_registry` In newer versions of Tokio `broadcast::Receiver` doesn't implement `Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This also requires changing the error type that is used. * Handle `Semaphore::acquire` error in `tower-batch` Newer versions of Tokio can return an error if the semaphore is closed. This shouldn't happen in `tower-batch` because the semaphore is never closed. * Handle `Semaphore::acquire` error in `zebrad` test On newer versions of Tokio `Semaphore::acquire` can return an error if the semaphore is closed. This shouldn't happen in the test because the semaphore is never closed. * Update some `zebra-network` dependencies Use versions compatible with Tokio version 1. * Upgrade Hyper to version 0.14 Use a version that supports Tokio version 1. * Update `metrics` dependency to version 0.17 And also update the `metrics-exporter-prometheus` to version 0.6.1. These updates are to make sure Tokio 1 is supported. * Use `f64` as the histogram data type `u64` isn't supported as the histogram data type in newer versions of `metrics`. * Update the initialization of the metrics component Make it compatible with the new version of `metrics`. * Simplify build version counter Remove all constants and use the new `metrics::incement_counter!` macro. * Change metrics output line to match on The snapshot string isn't included in the newer version of `metrics-exporter-prometheus`. * Update `sentry` to version 0.23.0 Use a version compatible with Tokio version 1. * Remove usage of `TracingIntegration` This seems to not be available from `sentry-tracing` anymore, so it needs to be replaced. * Add sentry layer to tracing initialization This seems like the replacement for `TracingIntegration`. * Remove unnecessary conversion Suggested by a Clippy lint. * Update Cargo lock file Apply all of the updates to dependencies. * Ban duplicate tokio dependencies Also ban git sources for tokio dependencies. * Stop allowing sentry-tracing git repository in `deny.toml` * Allow remaining duplicates after the tokio upgrade * Use C: drive for CI build output on Windows GitHub Actions uses a Windows image with two disk drives, and the default D: drive is smaller than the C: drive. Zebra currently uses a lot of space to build, so it has to use the C: drive to avoid CI build failures because of insufficient space. Co-authored-by: teor <teor@riseup.net>
2021-11-02 11:46:57 -07:00
- name: Change target output directory on Windows
# Windows doesn't have enough space on the D: drive, so we redirect the build output to the
# larger C: drive.
# TODO: Remove this workaround once the build is more efficient (#3005).
if: matrix.os == 'windows-latest'
run: |
mkdir C:\zebra-target
echo "CARGO_TARGET_DIR=C:\zebra-target" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
Download Zcash Sapling parameters and load them from cached files (#3057) * Replace Zcash parameters crates with pre-downloaded local parameter files * Download Zcash parameters using the `zcashd` script in CI and Docker * Add a zcash_proofs dependency to zebra-consensus * Download Sapling parameters using zcash_proofs, rather than fetch-params.sh * Add a new `zebrad download` subcommand This command isn't required for nomrmal usage. But it's useful when testing, or launching multiple Zebra instances. * Use `zebrad download` in CI to pre-download parameters * Log a helpful hint if downloading fails * Allow some duplicate dependencies currently hidden by orchard * Spawn a separate task to download Groth16 parameters * Run the parameter download with code coverage This avoids re-compining Zebra with and without coverage. * Update Cargo.lock after rebase * Try to pass `download` as an argument to `zebrad` in coverage CI * Fix copy and paste comment typos * Add path and download examples, like zcash_proofs * Download params in CI just like zcash_proofs does * Delete a redundant build step * Implement graceful shutdown for zebrad start * Send coverage summary to /dev/null when getting the params path * Use the correct parameters path and download commands in CI * Explain pre-downloads * Avoid calling params_folder twice * Rename parameter types and methods for consistency ```sh fastmod SaplingParams SaplingParameters zebra* fastmod Groth16Params Groth16Parameters zebra* fastmod PARAMS GROTH16_PARAMETERS zebra* fastmod params_folder directory zebra* ``` And a manual variable name tweak. * rustfmt * Remove a redundant coverage step Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 15:02:56 -08:00
# Modified from:
# https://github.com/zcash/librustzcash/blob/c48bb4def2e122289843ddb3cb2984c325c03ca0/.github/workflows/ci.yml#L20-L33
- name: Fetch path to Zcash parameters
working-directory: ./zebra-consensus
shell: bash
run: echo "ZCASH_PARAMS=$(cargo run --example get-params-path)" >> $GITHUB_ENV
- name: Cache Zcash parameters
id: cache-params
uses: actions/cache@v2
with:
path: ${{ env.ZCASH_PARAMS }}
key: ${{ runner.os }}-sprout-and-sapling-params
Download Zcash Sapling parameters and load them from cached files (#3057) * Replace Zcash parameters crates with pre-downloaded local parameter files * Download Zcash parameters using the `zcashd` script in CI and Docker * Add a zcash_proofs dependency to zebra-consensus * Download Sapling parameters using zcash_proofs, rather than fetch-params.sh * Add a new `zebrad download` subcommand This command isn't required for nomrmal usage. But it's useful when testing, or launching multiple Zebra instances. * Use `zebrad download` in CI to pre-download parameters * Log a helpful hint if downloading fails * Allow some duplicate dependencies currently hidden by orchard * Spawn a separate task to download Groth16 parameters * Run the parameter download with code coverage This avoids re-compining Zebra with and without coverage. * Update Cargo.lock after rebase * Try to pass `download` as an argument to `zebrad` in coverage CI * Fix copy and paste comment typos * Add path and download examples, like zcash_proofs * Download params in CI just like zcash_proofs does * Delete a redundant build step * Implement graceful shutdown for zebrad start * Send coverage summary to /dev/null when getting the params path * Use the correct parameters path and download commands in CI * Explain pre-downloads * Avoid calling params_folder twice * Rename parameter types and methods for consistency ```sh fastmod SaplingParams SaplingParameters zebra* fastmod Groth16Params Groth16Parameters zebra* fastmod PARAMS GROTH16_PARAMETERS zebra* fastmod params_folder directory zebra* ``` And a manual variable name tweak. * rustfmt * Remove a redundant coverage step Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 15:02:56 -08:00
- name: Fetch Zcash parameters
if: steps.cache-params.outputs.cache-hit != 'true'
working-directory: ./zebra-consensus
run: cargo run --example download-params
- name: Run tests
uses: actions-rs/cargo@v1.0.3
with:
command: test
args: --verbose --all
# Explicitly run any tests that are usually #[ignored]
2021-06-14 18:36:33 -07:00
- name: Run zebrad large sync tests
# Skip the entire step on Ubuntu and Windows, because the test would be skipped anyway due to ZEBRA_SKIP_NETWORK_TESTS
if: matrix.os == 'macOS-latest'
uses: actions-rs/cargo@v1.0.3
with:
command: test
# Note: this only runs the zebrad acceptance tests, because re-running all the test binaries is slow on Windows
args: --verbose --package zebrad --test acceptance sync_large_checkpoints_ -- --ignored
test-fake-activation-heights:
name: Build (+${{ matrix.rust }}) zebra-state with fake activation heights on ubuntu-latest
timeout-minutes: 60
runs-on: ubuntu-latest
strategy:
matrix:
rust: [stable]
steps:
- uses: actions/checkout@v2.4.0
with:
persist-credentials: false
- uses: actions-rs/toolchain@v1
with:
toolchain: ${{ matrix.rust }}
profile: minimal
override: true
- uses: Swatinem/rust-cache@v1
- name: cargo fetch
uses: actions-rs/cargo@v1.0.3
with:
command: fetch
# This test changes zebra-chain's activation heights,
# which can recompile all the Zebra crates,
# so we want its build products to be cached separately.
#
# Also, we don't want to accidentally use the fake heights in other tests.
- name: Run tests with fake activation heights
uses: actions-rs/cargo@v1.0.3
env:
TEST_FAKE_ACTIVATION_HEIGHTS: ""
with:
command: test
# Note: this only runs the zebra-state crate tests,
# because re-running all the test binaries can be slow
args: --verbose --package zebra-state --lib -- with_fake_activation_heights
build-chain-no-features:
2021-02-11 08:44:06 -08:00
name: Build (+${{ matrix.rust }}) zebra-chain w/o features on ubuntu-latest
timeout-minutes: 60
runs-on: ubuntu-latest
strategy:
matrix:
rust: [stable, beta]
steps:
- uses: actions/checkout@v2.4.0
with:
persist-credentials: false
2021-06-14 18:36:33 -07:00
- uses: actions-rs/toolchain@v1
with:
toolchain: ${{ matrix.rust }}
2021-04-29 13:47:15 -07:00
profile: minimal
override: true
2021-06-14 18:36:33 -07:00
- uses: Swatinem/rust-cache@v1
- name: cargo fetch
uses: actions-rs/cargo@v1.0.3
with:
command: fetch
2021-06-14 18:36:33 -07:00
- name: Run build without features enabled
working-directory: ./zebra-chain
run: cargo build --verbose --no-default-features
build:
name: Build (+stable) on ubuntu-latest
timeout-minutes: 60
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2.4.0
with:
persist-credentials: false
2021-06-14 18:36:33 -07:00
- uses: actions-rs/toolchain@v1
with:
toolchain: stable
2021-04-29 13:47:15 -07:00
profile: minimal
override: true
2021-06-14 18:36:33 -07:00
- uses: Swatinem/rust-cache@v1
# This check makes sure the crate dependency check is accurate
- name: Check Cargo.lock is up to date
uses: actions-rs/cargo@v1.0.3
with:
command: check
args: --locked --all-features --all-targets
- name: cargo fetch
uses: actions-rs/cargo@v1.0.3
with:
command: fetch
2021-06-14 18:36:33 -07:00
- name: Build
uses: actions-rs/cargo@v1.0.3
with:
command: build
args: --verbose --release
2020-01-23 13:40:32 -08:00
cargo-deny:
name: Check deny.toml crate dependencies and validate licenses
runs-on: ubuntu-latest
strategy:
matrix:
checks:
- bans
- sources
# Prevent sudden announcement of a new advisory from failing ci:
continue-on-error: ${{ matrix.checks == 'advisories' }}
steps:
- uses: actions/checkout@v2
with:
persist-credentials: false
- uses: EmbarkStudios/cargo-deny-action@v1
with:
command: check ${{ matrix.checks }}
args: --all-features --workspace