zecfoundation
/
zebra
mirror of https://github.com/ZcashFoundation/zebra.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

3. Require network names in cached state disk names (#4392) 

* Require a cached state rebuild if the state version changes

* Find cached state disks with the same state version

And prefer `main` to other branches.

* Tweak filters to make them more specific

* Try adding inner quotes

* Try brackets instead

* Try two filters, rather than three

* Use Mainnet as the default network, remove duplicate env var

* Match the exact disk name format in one regular expression

* Log the exact expected disk name, including the network

* Consistently use CACHED_DISK_NAME as the env var name

* Temporary allow missing $NETWORK in disk names

* Print the exact search string

* Debug log the search string

* Use a generic alphabetical pattern rather than a regex group

Google Cloud doesn't seem to support regex groups.

* Add network name to disk match docs

* Fix the logged network name

* Make jobs that use cached state wait for state rebuilds

* Run jobs that need cached state even if the rebuild was skipped

* Fix missing dependencies

And update a TODO

* Revert "Use a generic alphabetical pattern rather than a regex group"

This reverts commit 970afe7b17.

* Revert "Temporary allow missing $NETWORK in disk names"

This reverts commit f1f66500c3.

* Make jobs that use cached state wait for state rebuilds

* Run jobs that need cached state even if the rebuild was skipped

* Fix missing dependencies

And update a TODO

* refactor(ci): look for available disks instead of files changed

This ensure that if the constants.rs file was changed, we search for disks available in the whole repository with the same state.

If there's no disk available a rebuild is triggered depending the missing disk. And if there's a disk available, tests are run with this one.

* fix(ci): lwd syncs needs to wait for zebra disk rebuild

* docs(ci): use better comments on integration tests

* fix(ci): we must authenticate to GCP to find disks

* fix(ci): add needed permissions for google auth

* fix(ci): the output needs to be echoed

* imp(ci): reduce diff with main

* fix(ci): remove redundant dependency

Co-authored-by: teor <teor@riseup.net>

* fix(ci): also add `false` to the JSON object output

* fix(ci): hasty copy/paste

* fix(ci): standardize comments

* fix(ci): run disk rebuilds if no disk was found

* fix(ci): build on any event if a cached disk is not found

* fix(ci): reduce diff with main

* docs(ci): reduce main diff

* fix(ci): sync .patch file with changes on the workflow

* fix(ci): consider network changes in new get-available-disks

* force GHA trigger

Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
This commit is contained in:
teor 2022-05-20 10:44:11 +10:00 committed by GitHub
parent 1c10f40b29
commit 712ef40438
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 6 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
.github/workflows/continous-integration-docker.yml vendored
View File

@ -106,7 +106,7 @@ jobs:
LOCAL_STATE_VERSION=$(grep -oE "DATABASE_FORMAT_VERSION: .* [0-9]+" "$GITHUB_WORKSPACE/zebra-state/src/constants.rs" | grep -oE "[0-9]+" | tail -n1)
echo "STATE_VERSION: $LOCAL_STATE_VERSION"
LWD_TIP_DISK=$(gcloud compute images list --filter="name~lwd-cache-.+-[0-9a-f]+-v${LOCAL_STATE_VERSION}-[a-z]*-tip" --format="value(NAME)" --sort-by=~creationTimestamp --limit=1)
LWD_TIP_DISK=$(gcloud compute images list --filter="name~lwd-cache-.+-[0-9a-f]+-v${LOCAL_STATE_VERSION}-${NETWORK}-tip" --format="value(NAME)" --sort-by=~creationTimestamp --limit=1)
if [[ -z "$LWD_TIP_DISK" ]]; then
echo "No TIP disk found for LWD"
echo "::set-output name=lwd_tip_disk::${{ toJSON(false) }}"
@ -115,7 +115,7 @@ jobs:
echo "::set-output name=lwd_tip_disk::${{ toJSON(true) }}"
fi
ZEBRA_TIP_DISK=$(gcloud compute images list --filter="name~zebrad-cache-.+-[0-9a-f]+-v${LOCAL_STATE_VERSION}-[a-z]*-tip" --format="value(NAME)" --sort-by=~creationTimestamp --limit=1)
ZEBRA_TIP_DISK=$(gcloud compute images list --filter="name~zebrad-cache-.+-[0-9a-f]+-v${LOCAL_STATE_VERSION}-${NETWORK}-tip" --format="value(NAME)" --sort-by=~creationTimestamp --limit=1)
if [[ -z "$ZEBRA_TIP_DISK" ]]; then
echo "No TIP disk found for ZEBRA"
echo "::set-output name=zebra_tip_disk::${{ toJSON(false) }}"
@ -124,7 +124,7 @@ jobs:
echo "::set-output name=zebra_tip_disk::${{ toJSON(true) }}"
fi
ZEBRA_CHECKPOINT_DISK=$(gcloud compute images list --filter="name~zebrad-cache-.+-[0-9a-f]+-v${LOCAL_STATE_VERSION}-[a-z]*-checkpoint" --format="value(NAME)" --sort-by=~creationTimestamp --limit=1)
ZEBRA_CHECKPOINT_DISK=$(gcloud compute images list --filter="name~zebrad-cache-.+-[0-9a-f]+-v${LOCAL_STATE_VERSION}-${NETWORK}-checkpoint" --format="value(NAME)" --sort-by=~creationTimestamp --limit=1)
if [[ -z "$ZEBRA_CHECKPOINT_DISK" ]]; then
echo "No CHECKPOINT found for ZEBRA"
echo "::set-output name=zebra_checkpoint_disk::${{ toJSON(false) }}"

12
.github/workflows/deploy-gcp-tests.yml vendored
View File

@ -241,24 +241,18 @@ jobs:
# Try to find an image generated from the main branch
# Fields are listed in the "Create image from state disk" step
#
# TODO: require ${NETWORK} in the name after PR #4391 merges to main, and runs a full sync
# network should replace [a-z]*
CACHED_DISK_NAME=$(gcloud compute images list --filter="name~${DISK_PREFIX}-main-[0-9a-f]+-v${LOCAL_STATE_VERSION}-[a-z]*-${{ inputs.disk_suffix }}" --format="value(NAME)" --sort-by=~creationTimestamp --limit=1)
CACHED_DISK_NAME=$(gcloud compute images list --filter="name~${DISK_PREFIX}-main-[0-9a-f]+-v${LOCAL_STATE_VERSION}-${NETWORK}-${{ inputs.disk_suffix }}" --format="value(NAME)" --sort-by=~creationTimestamp --limit=1)
echo "main Disk: $CACHED_DISK_NAME"
if [[ -z "$CACHED_DISK_NAME" ]]; then
# Try to find an image generated from any other branch
#
# TODO: require ${NETWORK} in the name after PRs #4391 and #4385 merge to main
# network should replace [a-z]*
CACHED_DISK_NAME=$(gcloud compute images list --filter="name~${DISK_PREFIX}-.+-[0-9a-f]+-v${LOCAL_STATE_VERSION}-[a-z]*-${{ inputs.disk_suffix }}" --format="value(NAME)" --sort-by=~creationTimestamp --limit=1)
CACHED_DISK_NAME=$(gcloud compute images list --filter="name~${DISK_PREFIX}-.+-[0-9a-f]+-v${LOCAL_STATE_VERSION}-${NETWORK}-${{ inputs.disk_suffix }}" --format="value(NAME)" --sort-by=~creationTimestamp --limit=1)
echo "Disk: $CACHED_DISK_NAME"
fi
if [[ -z "$CACHED_DISK_NAME" ]]; then
echo "No cached state disk available"
echo "Expected ${{ inputs.disk_prefix }}-(branch)-[0-9a-f]+-v${LOCAL_STATE_VERSION}-${NETWORK}-${{ inputs.disk_suffix }}"
echo "Expected ${DISK_PREFIX}-(branch)-[0-9a-f]+-v${LOCAL_STATE_VERSION}-${NETWORK}-${{ inputs.disk_suffix }}"
echo "Cached state test jobs must depend on the cached state rebuild job"
exit 1
fi