change(ci): Generate testnet checkpoints in CI (#6581)

* Split checking for cached state disks into its own workflow

* Fix workflow field order

* Run the top-level workflow when the reusable workflow changes

* And run dependent workflows for pull requests as well

* Remove redundant output names

* Document the existing and new workflow jobs

* Add the network to the "no disk found" message

* Tweak existing docs and descriptions

* Generate Zebra checkpoints on testnet

* Add a full sync testnet entrypoint, and simplify mainnet env vars

* Only run the full testnet sync on the main branch

* Deduplicate and update the zebra-checkpoints docs

* Add instructions for automatic checkpoint generation

* Hide some details in the release checklist

* Update release checkpoint instructions to use CI

* Only update the cache in one job on mainnet
This commit is contained in:
teor 2023-05-04 09:14:22 +10:00 committed by GitHub
parent aea3833c07
commit 937d704e69
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 185 additions and 82 deletions

View File

@ -15,7 +15,9 @@ Zebra follows [semantic versioning](https://semver.org). Semantic versions look
The [draft `zebrad` changelog](https://github.com/ZcashFoundation/zebra/releases) will have an automatic version bump. This version is based on [the labels on the PRs in the release](https://github.com/ZcashFoundation/zebra/blob/main/.github/release-drafter.yml).
Check that the automatic `zebrad` version increment is correct:
Check that the automatic `zebrad` version increment matches the changes in the release:
<details>
If we're releasing a mainnet network upgrade, it is a `major` release:
1. Increment the `major` version of _*all*_ the Zebra crates.
@ -33,6 +35,8 @@ Otherwise, it is a `patch` release:
Zebra's Rust API is not stable or supported, so we keep all the crates on the same beta `pre-release` version.
</details>
### Version Locations
Once you know which versions you want to increment, you can find them in the:
@ -81,9 +85,8 @@ You can use a command like:
## Checkpoints
With every release and for performance reasons, we want to update the Zebra checkpoints. More information on how to do this can be found in [the zebra-checkpoints README](https://github.com/ZcashFoundation/zebra/blob/main/zebra-consensus/src/checkpoint/README.md).
To do this you will need a synchronized `zcashd` node. You can request help from other zebra team members to submit this PR if you can't make it yourself at the moment of the release.
For performance and security, we want to update the Zebra checkpoints in every release.
You can copy the latest checkpoints from CI by following [the zebra-checkpoints README](https://github.com/ZcashFoundation/zebra/blob/main/zebra-utils/README.md#zebra-checkpoints).
## Missed Dependency Updates
@ -188,8 +191,13 @@ If the release contains new features (`major` or `minor`), or high-priority bug
## Release Failures
If building or running fails after tagging:
<details>
1. Fix the bug that caused the failure
2. Increment versions again, following these instructions from the start
3. Update the code and documentation with a **new** git tag
4. Update `CHANGELOG.md` with details about the fix
5. Tag a **new** release
</details>

View File

@ -26,7 +26,7 @@ on:
run-full-sync:
type: boolean
default: false
description: 'Just run a Zebra full sync and update tip disks'
description: 'Just run a Zebra full sync on `network`, and update tip disks'
required: true
run-lwd-sync:
type: boolean
@ -82,7 +82,7 @@ jobs:
# Check if the cached state disks used by the tests are available for the default network.
#
# The default network is Mainnet unless a manually triggered workflow or repository variable
# The default network is mainnet unless a manually triggered workflow or repository variable
# is configured differently.
#
# The outputs for this job have the same names as the workflow outputs in find-cached-disks.yml
@ -92,19 +92,19 @@ jobs:
with:
network: ${{ inputs.network || vars.ZCASH_NETWORK }}
# Check if the cached state disks used by the tests are available for Testnet.
# Check if the cached state disks used by the tests are available for testnet.
#
# The outputs for this job have the same names as the workflow outputs in find-cached-disks.yml
# Some outputs are ignored, because we don't run those jobs on Testnet.
# Some outputs are ignored, because we don't run those jobs on testnet.
get-available-disks-testnet:
name: Check if cached state disks exist for Testnet
name: Check if cached state disks exist for testnet
uses: ./.github/workflows/find-cached-disks.yml
with:
network: 'Testnet'
# Build the docker image used by the tests.
#
# The default network in the Zebra config in the image is Mainnet, unless a manually triggered
# The default network in the Zebra config in the image is mainnet, unless a manually triggered
# workflow or repository variable is configured differently. Testnet jobs change that config to
# testnet when running the image.
build:
@ -338,13 +338,15 @@ jobs:
# zebrad cached tip state tests
# Test that Zebra can run a full mainnet sync,
# Test that Zebra can run a full sync on mainnet,
# and regenerate chain tip Zebra cached state disks.
#
# This test always runs on mainnet.
#
# Runs:
# - after every PR is merged to `main`
# - on every PR update, but only if there's no available disk matching the actual state version from constants.rs
# - on request, using workflow_dispatch with run-full-sync
# - on every PR update, but only if the state version in constants.rs has no cached disk
# - in manual workflow runs, when run-full-sync is 'true' and network is 'Mainnet'
#
# Note: the output from get-available-disks should match with the caller workflow inputs
test-full-sync:
@ -354,13 +356,15 @@ jobs:
# to also run on Mergify head branches,
# add `|| (github.event_name == 'push' && startsWith(github.head_ref, 'mergify/merge-queue/'))`:
# https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#running-your-workflow-based-on-the-head-or-base-branch-of-a-pull-request-1
if: ${{ (github.event_name == 'push' && github.ref_name == 'main') || !fromJSON(needs.get-available-disks.outputs.zebra_tip_disk) || github.event.inputs.run-full-sync == 'true' }}
if: ${{ (github.event_name == 'push' && github.ref_name == 'main') || !fromJSON(needs.get-available-disks.outputs.zebra_tip_disk) || (github.event.inputs.run-full-sync == 'true' && github.event.inputs.network == 'Mainnet') }}
with:
app_name: zebrad
test_id: full-sync-to-tip
test_description: Test a full sync up to the tip
# The value of FULL_SYNC_MAINNET_TIMEOUT_MINUTES is currently ignored.
test_variables: '-e TEST_FULL_SYNC=1 -e ZEBRA_FORCE_USE_COLOR=1 -e FULL_SYNC_MAINNET_TIMEOUT_MINUTES=0'
# TODO: update the test to use {{ input.network }} instead?
test_variables: '-e FULL_SYNC_MAINNET_TIMEOUT_MINUTES=0 -e ZEBRA_FORCE_USE_COLOR=1'
network: 'Mainnet'
# This test runs for longer than 6 hours, so it needs multiple jobs
is_long_test: true
needs_zebra_state: false
@ -411,11 +415,13 @@ jobs:
height_grep_text: 'current_height.*=.*Height.*\('
secrets: inherit
# zebra checkpoint generation tests
# zebra mainnet checkpoint generation tests
# Test that Zebra can generate mainnet checkpoints after syncing to the chain tip,
# using a cached Zebra tip state,
#
# This test always runs on mainnet.
#
# Runs:
# - after every PR is merged to `main`
# - on every PR update
@ -431,17 +437,101 @@ jobs:
app_name: zebrad
test_id: generate-checkpoints-mainnet
test_description: Generate Zebra checkpoints on mainnet
# TODO: update the test to use {{ input.network }} instead?
test_variables: '-e GENERATE_CHECKPOINTS_MAINNET=1 -e ZEBRA_FORCE_USE_COLOR=1 -e ZEBRA_CACHED_STATE_DIR=/var/cache/zebrad-cache'
network: 'Mainnet'
needs_zebra_state: true
# update the disk on every PR, to increase CI speed
saves_to_disk: true
# test-update-sync updates the disk on every PR, so we don't need to do it here
saves_to_disk: false
disk_suffix: tip
root_state_path: '/var/cache'
zebra_state_dir: 'zebrad-cache'
height_grep_text: 'current_height.*=.*Height.*\('
secrets: inherit
# TODO: testnet checkpoints, test-full-sync, test-update-sync, get-available-disks (+ reusable)
# zebra testnet checkpoint generation tests
#
# These tests will fail when testnet is unstable, they should not be required to merge.
#
# TODO: ignore failures on testnet, so any failures don't appear in the GitHub interface.
# Test that Zebra can run a full testnet sync, and regenerate chain tip Zebra cached state disks.
# This job always runs on testnet, regardless of any inputs or variable settings.
#
# Runs:
# - after every PR is merged to `main`
# - on every PR update, but only if the state version in constants.rs has no cached disk
# - in manual workflow runs, when run-full-sync is 'true' and network is 'Testnet'
#
# Note: the output from get-available-disks-testnet should match with the caller workflow inputs
test-full-sync-testnet:
name: Zebra tip on testnet
needs: [ build, get-available-disks-testnet ]
uses: ./.github/workflows/deploy-gcp-tests.yml
# to also run on Mergify head branches,
# add `|| (github.event_name == 'push' && startsWith(github.head_ref, 'mergify/merge-queue/'))`:
# https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#running-your-workflow-based-on-the-head-or-base-branch-of-a-pull-request-1
if: ${{ (github.event_name == 'push' && github.ref_name == 'main') || !fromJSON(needs.get-available-disks-testnet.outputs.zebra_tip_disk) || (github.event.inputs.run-full-sync == 'true' && github.event.inputs.network == 'Testnet') }}
with:
app_name: zebrad
test_id: full-sync-to-tip-testnet
test_description: Test a full sync up to the tip on testnet
# The value of FULL_SYNC_TESTNET_TIMEOUT_MINUTES is currently ignored.
test_variables: '-e FULL_SYNC_TESTNET_TIMEOUT_MINUTES=0 -e ZEBRA_FORCE_USE_COLOR=1'
network: 'Testnet'
# A full testnet sync took about 2 hours in April 2023
is_long_test: false
needs_zebra_state: false
saves_to_disk: true
disk_suffix: tip
height_grep_text: 'current_height.*=.*Height.*\('
secrets: inherit
# We want to prevent multiple full zebrad syncs running at the same time,
# but we don't want to cancel running syncs on `main` if a new PR gets merged,
# because we might never get a finished sync.
#
# Instead, we let the first sync complete, then queue the latest pending sync, cancelling any syncs in between.
# (As the general workflow concurrency group just gets matched in Pull Requests,
# it has no impact on this job.)
#
# TODO:
# - allow multiple manual syncs on a branch, and isolate manual syncs from automatic syncs, by adding '-${{ github.run_id }}' when github.event.inputs.run-full-sync is true
# - stop multiple automatic full syncs across different PRs by removing '${{ github.ref }}' when needs.get-available-disks-testnet.outputs.zebra_tip_disk is true
concurrency:
group: ${{ github.workflow }}${{ github.ref }}-test-full-sync-testnet
cancel-in-progress: false
# Test that Zebra can generate testnet checkpoints after syncing to the chain tip,
# using a cached Zebra tip state.
#
# This test always runs on testnet.
#
# Runs:
# - after every PR is merged to `main`
# - on every PR update
#
# If the state version has changed, waits for the new cached state to be created.
# Otherwise, if the state rebuild was skipped, runs immediately after the build job.
generate-checkpoints-testnet:
name: Generate checkpoints testnet
needs: test-full-sync-testnet
uses: ./.github/workflows/deploy-gcp-tests.yml
if: ${{ !cancelled() && !failure() && github.event.inputs.regenerate-disks != 'true' && github.event.inputs.run-full-sync != 'true' && github.event.inputs.run-lwd-sync != 'true' }}
with:
app_name: zebrad
test_id: generate-checkpoints-testnet
test_description: Generate Zebra checkpoints on testnet
test_variables: '-e GENERATE_CHECKPOINTS_TESTNET=1 -e ZEBRA_FORCE_USE_COLOR=1 -e ZEBRA_CACHED_STATE_DIR=/var/cache/zebrad-cache'
network: 'Testnet'
needs_zebra_state: true
# update the disk on every PR, to increase CI speed
# we don't have a test-update-sync-testnet job, so we need to update the disk here
saves_to_disk: true
disk_suffix: tip
root_state_path: '/var/cache'
zebra_state_dir: 'zebrad-cache'
height_grep_text: 'current_height.*=.*Height.*\('
secrets: inherit
# lightwalletd cached tip state tests

View File

@ -29,12 +29,18 @@ case "$1" in
# For these tests, we activate the gRPC feature to avoid recompiling `zebrad`,
# but we don't actually run any gRPC tests.
elif [[ "$TEST_FULL_SYNC" -eq "1" ]]; then
# Run a Zebra full sync test.
elif [[ -n "$FULL_SYNC_MAINNET_TIMEOUT_MINUTES" ]]; then
# Run a Zebra full sync test on mainnet.
cargo test --locked --release --features "$ENTRYPOINT_FEATURES" --package zebrad --test acceptance -- --nocapture --include-ignored full_sync_mainnet
# List directory generated by test
# TODO: replace with $ZEBRA_CACHED_STATE_DIR in Rust and workflows
ls -lh "/zebrad-cache"/*/* || (echo "No /zebrad-cache/*/*"; ls -lhR "/zebrad-cache" | head -50 || echo "No /zebrad-cache directory")
elif [[ -n "$FULL_SYNC_TESTNET_TIMEOUT_MINUTES" ]]; then
# Run a Zebra full sync test on testnet.
cargo test --locked --release --features "$ENTRYPOINT_FEATURES" --package zebrad --test acceptance -- --nocapture --include-ignored full_sync_testnet
# List directory generated by test
# TODO: replace with $ZEBRA_CACHED_STATE_DIR in Rust and workflows
ls -lh "/zebrad-cache"/*/* || (echo "No /zebrad-cache/*/*"; ls -lhR "/zebrad-cache" | head -50 || echo "No /zebrad-cache directory")
elif [[ "$TEST_DISK_REBUILD" -eq "1" ]]; then
# Run a Zebra sync up to the mandatory checkpoint.
#

View File

@ -5,64 +5,8 @@ Zebra validates [settled network upgrades](https://zips.z.cash/protocol/protocol
- [Mainnet checkpoints](https://github.com/ZcashFoundation/zebra/blob/main/zebra-consensus/src/checkpoint/main-checkpoints.txt)
- [Testnet checkpoints](https://github.com/ZcashFoundation/zebra/blob/main/zebra-consensus/src/checkpoint/test-checkpoints.txt)
Some of these checkpoints can be disabled using the following config:
```sh
[consensus]
checkpoint_sync = false
```
(Zebra always uses checkpoints up to its mandatory checkpoint,
so that we do not have to implement obsolete consensus rules.)
Using these checkpoints increases Zebra's security against some attacks.
## Update checkpoints
Checkpoint lists are distributed with Zebra, maintainers should update them about every few months to get newer hashes. Here we explain how this process is done.
Or jump straight to [the exact commands for updating the lists](https://github.com/ZcashFoundation/zebra/tree/main/zebra-utils/README.md#zebra-checkpoints).
### Use the `zebra-checkpoints` utility
`zebra-checkpoints` is the program we use to collect checkpoints. Currently this program uses `zcash-cli` to get the hashes. `zcash-cli` must be available in your machine and it must be connected to a synchronized (Mainnet or Testnet) instance of `zebrad` or `zcashd` to get the most recent hashes. Make sure your `zebrad` or `zcashd` is [listening for RPC requests](https://github.com/ZcashFoundation/zebra/blob/c0849ad8cb3aefe927e1735e586ce3c72d96d16c/zebra-rpc/src/config.rs#L29). If you are on a Debian system, `zcash-cli` [can be installed as a package](https://zcash.readthedocs.io/en/latest/rtd_pages/install_debian_bin_packages.html).
First, [build the `zebra-checkpoints` binary](https://github.com/ZcashFoundation/zebra/tree/main/zebra-utils/README.md#zebra-checkpoints).
When updating the lists there is no need to start from the genesis block. The program option `--last-checkpoint` will let you specify at what block height you want to start. In general the maintainer will copy the last height from each list and use the mentioned option to start from there.
It is easier if `zcash-cli` is in your execution path however you can specify the location of it anywhere in the filesystem with option `--cli`.
By default, `zebra-checkpoints` will use a `zebrad` backend. If the running instance is `zcashd`, please add `-b zcashd` to your command.
To update the checkpoint list, run:
```sh
$ zebra-checkpoints --last-checkpoint $(tail -1 zebra-consensus/src/checkpoint/main-checkpoints.txt | cut -d" " -f1) | tee --append zebra-consensus/src/checkpoint/main-checkpoints.txt
1574231 000000000030874b90f23cba865355a2058c635a81396a1c6026f1d7ea89e035
1574631 000000000114d8c70ae8bc0bcc5b94b6b03e9086365b09dd9d3e77b0a2b6554e
1575031 00000000010074d017d6628ca2cf3327d68be11c993c0e4d3c961ebb81c741dd
1575431 000000000001060a02bb142d09de383591c2d5a6f589b982bd9ccf2ad8b72e7c
1575831 000000000094e82cca9af540721e6415097ffd5d70b6adefb8aef6b5898e9b08
1576231 0000000000710c69b4c8d851883e1ad7c14e74b9c79b0cafc0e6c64cf07a54ab
...
```
If we are looking to update the testnet hashes we must make sure the cli is connected with a testnet chain. If we are using `zcashd` as the backend and this is running locally, we can make this by starting with `zcashd -testnet`. If we are using `zebrad` as the backend, then we must start with a configuration file where the `network` field of the `[network]` section is `Testnet`.
Anything we add after `--` will pass through into the `zcash-cli` program so we can specify the testnet here.
To regenerate the list from the genesis block, run:
```sh
$ zebra-checkpoints -- -testnet
0 05a60a92d99d85997cce3b87616c089f6124d7342af37106edc76126334a2c38
400 000393fe014f5ff5de7c9f0aa669ee074c9a7743a6bdc1d1686149b4b36090d8
800 0003bef995cd09a4a2bcb96580fa435ed10c0b44239191fafc825477e84a325d
1200 00011c96be8ea3df2d8dfafd8c12a8e59b643e728d867b8a295e88ca8b420c6f
1600 0000b0e880a18a1c14feaedac1733003376bce05154979ce8aeae7d8d02718ec
2000 000033804e1bebd8ef6112938dc58d5939874bcb4536e203489eb75520f3c555
...
```
### Submit new hashes as pull request
- If you started from a block different than the genesis append the obtained list of hashes at the end of the existing files. If you started from genesis you can replace the entire list files.
- Open a pull request with the updated lists into the zebra `main` branch.
Checkpoint lists are distributed with Zebra, maintainers should update them about every few months to get newer hashes. Here are [the exact commands for updating the lists](https://github.com/ZcashFoundation/zebra/tree/main/zebra-utils/README.md#zebra-checkpoints).

View File

@ -15,7 +15,34 @@ Binaries are easier to use if they are located in your system execution path.
This command generates a list of zebra checkpoints, and writes them to standard output. Each checkpoint consists of a block height and hash.
To create checkpoints, you need a synchronized instance of `zebrad` or `zcashd`, and the `zcash-cli` RPC client.
#### Automatic Checkpoint Generation in CI
Zebra's GitHub workflows automatically generate checkpoints after every `main` branch update.
These checkpoints can be copied into the `main-checkpoints.txt` and `test-checkpoints.txt` files.
To find the latest checkpoints on the `main` branch:
1. Find the [latest completed `CI Docker` workflow run on `main`](https://github.com/ZcashFoundation/zebra/actions/workflows/continous-integration-docker.yml?query=branch%3Amain).
Due to GitHub UI issues, some runs will show as waiting, cancelled, or failed,
but the checkpoints have still been generated.
2. Go to the `Result of generate-checkpoints-mainnet` step in the
`Run generate-checkpoints-mainnet` job, in the `Generate checkpoints mainnet` job
3. Scroll down until you see the list of checkpoints, it should start around line 200
4. Add those checkpoints to the end of `zebra-consensus/src/checkpoint/main-checkpoints.txt`
5. Repeat steps 2 to 4 for `testnet`
6. Open a pull request at https://github.com/ZcashFoundation/zebra/pulls
#### Manual Checkpoint Generation
To create checkpoints, you need a synchronized instance of `zebrad` or `zcashd`.
`zebrad` can be queried directly or via an installed `zcash-cli` RPC client.
`zcashd` must be queried via `zcash-cli`, which performs the correct RPC authentication.
#### Checkpoint Generation Setup
Make sure your `zebrad` or `zcashd` is [listening for RPC requests](https://doc.zebra.zfnd.org/zebra_rpc/config/struct.Config.html#structfield.listen_addr),
and synced to the network tip.
If you are on a Debian system, `zcash-cli` [can be installed as a package](https://zcash.readthedocs.io/en/latest/rtd_pages/install_debian_bin_packages.html).
`zebra-checkpoints` is a standalone rust binary, you can compile it using:
@ -23,20 +50,48 @@ To create checkpoints, you need a synchronized instance of `zebrad` or `zcashd`,
cargo install --locked --features zebra-checkpoints --git https://github.com/ZcashFoundation/zebra zebra-utils
```
Then update the checkpoints using these commands:
#### Checkpoint Generation Commands
You can update the checkpoints using these commands:
```sh
zebra-checkpoints --last-checkpoint $(tail -1 zebra-consensus/src/checkpoint/main-checkpoints.txt | cut -d" " -f1) | tee --append zebra-consensus/src/checkpoint/main-checkpoints.txt &
zebra-checkpoints --last-checkpoint $(tail -1 zebra-consensus/src/checkpoint/test-checkpoints.txt | cut -d" " -f1) -- -testnet | tee --append zebra-consensus/src/checkpoint/test-checkpoints.txt &
wait
```
When updating the lists there is no need to start from the genesis block. The program option
`--last-checkpoint` will let you specify at what block height you want to start. Usually, the
maintainers will copy the last height from each list, and start from there.
Other useful options are:
- `--transport direct`: connect directly to a `zebrad` instance
- `--addr`: supply a custom RPC address and port for the node
- `-- -testnet`: connect the `zcash-cli` binary to a testnet node instance
You can see all the `zebra-checkpoints` options using:
```sh
target/release/zebra-checkpoints --help
```
For more details, see the [`zebra-checkpoints` README.](https://github.com/ZcashFoundation/zebra/tree/main/zebra-consensus/src/checkpoint/README.md)
For more details about checkpoint lists, see the [`zebra-checkpoints` README.](https://github.com/ZcashFoundation/zebra/tree/main/zebra-consensus/src/checkpoint/README.md)
#### Checkpoint Generation for Testnet
To update the testnet checkpoints, `zebra-checkpoints` needs to connect to a testnet node.
To launch a testnet node, you can either:
- start `zebrad` [with a `zebrad.toml` with `network.network` set to `Testnet`](https://doc.zebra.zfnd.org/zebra_network/struct.Config.html#structfield.network), or
- run `zcashd -testnet`.
Then use the commands above to renegerate the checkpoints.
#### Submit new checkpoints as pull request
- If you started from the last checkpoint in the current list, add the checkpoint list to the end
of the existing checkpoint file. If you started from genesis, replace the entire file.
- Open a pull request with the updated Mainnet and Testnet lists at:
https://github.com/ZcashFoundation/zebra/pulls
### zebrad-hash-lookup