refactor(test): dockerize tests and run sync in detached mode (#3459)

* refactor (cd): overall pipeline improvement - Use a more ENV configurable Dockerfile - Remove cloudbuild dependency - Use compute optimized machine types - Use SSD instead of normal hard drives - Move Sentry endpoint to secrets - Use a single yml for auto & manual deploy - Migrate to Google Artifact Registry * refactor (cd): overall pipeline improvement - Use a more ENV configurable Dockerfile - Remove cloudbuild dependency - Use compute optimized machine types - Use SSD instead of normal hard drives - Move Sentry endpoint to secrets - Use a single yml for auto & manual deploy - Migrate to Google Artifact Registry * refactor (cd): use newer google auth action * fix (cd): use newer secret as gcp credential * fix (docker): do not create extra directories * fix (docker): ignore .github for caching purposes * fix (docker): use latest rust * fix (cd): bump build timeout * fix: use a better name for manual deployment * refactor (docker): use standard directories for executable * fix (cd): most systems expect a "latest" tag Caching from the latest image is one of the main reasons to add this extra tag. Before this commit, the inline cache was not being used. * fix (cd): push the build image and the cache separately The inline cache exporter only supports `min` cache mode. To enable `max` cache mode, push the image and the cache separately by using the registry cache exporter. This also allows for smaller release images. * fix (cd): remove unused GHA cache We're leveraging the registry to cache the actions, instead of using the 10GB limits from Github Actions cache storage * refactor (cd): use cargo-chef for caching rust deps * fix: move build system deps before cargo cheg cook * fix (release): use newer debian to reduce vulnerabilities * fix (cd): use same zone, region and service accounts * fix (cd): use same disk size and type for all deployments * refactor (cd): activate interactive shells Use interactive shells for manual and test deployments. This allow greater flexibility if troubleshooting is needed inside the machines * refactor (test): use docker artifact from registry Instead of using a VM to SSH into in to build and test. Build in GHA (to have the logs available), run the workspace tests in GHA, and just run the sync tests in GCP Use a cintainer VM with zebra's image directly on it, and pass the needed parameters to run the Sync past mandatory checkpoint. * tmp (cd): bump timeout for building from scratch * tmp (test): bump build time * fix (cd, test): bump build time-out to 210 minutes * fix (docker): do not build with different settings Compiling might be slow because different steps are compiling the same code 2-4 times because of the variations * revert (docker): do not fix the rust version * fix (docker): build on the root directory * refactor(docker): Use base image commands and tools * fix (cd): use correct variables & values, add build concurrency * fix(cd): use Mainnet instead of mainnet * imp: remove checkout as Buildkit uses the git context * fix (docker): just Buildkit uses a .dockerignore in a path * imp (cd): just use needed variables in the right place * imp (cd): do not checkout if not needed * test: run on push * refactor(docker): reduce build changes * fix(cd): not checking out was limiting some variables * refactor(test): add an multistage exclusive for testing * fix(cd): remove tests as a runtime dependency * fix(cd): use default service account with cloud-platform scope * fix(cd): revert checkout actions * fix: use GA c2 instead of Preview c2d machine types * fix(actions): remove workflow_dispatch from patched actions This causes GitHub confusion as it can't determined which of the actions using workflow_dispatch is the right one * fix(actions): remove patches from push actions * test: validate changes on each push * fix(test): wrong file syntax on test job * fix(test): add missing env parameters * fix(docker): Do not rebuild to download params and run tests * fix(test): setup gcloud and loginto artifact just when needed Try not to rebuild the tests * fix(test): use GCP container to sync past mandatory checkpoint * fix(test): missing separators * test * fix(test): mount the available disk * push * refactor(test): merge disk regeneration into test.yml * fix(cd): minor typo fixes * fix(docker): rebuild on .github changes * fix(cd): keep compatibility with gcr.io To prevent conflicts between registries, and migrate when the time is right, we'll keep pushing to both registries and use github actions cache to prevent conflicts between artifacts. * fix(cd): typo and scope * fix(cd): typos everywhere * refactor(test): use smarter docker wait and keep old registry * fix(cd): do not constraint the CPUs for bigger machines * revert(cd): reduce PR diff as there's a separate one for tests * fix(docker): add .github as it has no impact on caching * fix(test): run command correctly * fix(test): wiat and create image if previous step succeded * force rebuild * fix(test): do not restrict interdependant steps based on event * force push * feat(docker): add google OS Config agent Use a separate step to have better flexibility in case a better approach is available * fix(test): remove all hardoced values and increase disks * fix(test): use correct commands on deploy * fix(test): use args as required by google * fix(docker): try not to invalidate zebrad download cache * fix(test): minor typo * refactor(test): decouple jobs for better modularity This also allows faster tests as testing Zunstable won't be a dependency and it can't stop already started jobs if it fails. * fix(test): Do not try to execute ss and commands in one line * fix(test): do not show undeeded information in the terminal * fix(test): sleep befor/after machine creation/deletion * fix(docker): do not download zcash params twice * feat(docker): add google OS Config agent Use a separate step to have better flexibility in case a better approach is available * merge: docker-actions-refactor into docker-test-refactor * test docker wait scenarios * fix(docker): $HOME variables is not being expanded * fix(test): allow docker wait to work correctly * fix(docker): do not use variables while using COPY * fix(docker): allow to use zebrad as a command * fix(cd): use test .yml from main * fix(cd): Do not duplicate network values The Dockerfile has an ARG with a default value of 'Mainnet', if this value is changed it will be done manually on a workflow_dispatch, making the ENV option a uneeded duplicate in this workflow * fix(test): use bigger machine type for compute intensive tasks * refactor(test): add tests in CI file * fix(test): remove duplicated tests * fix(test): typo * test: build on .github changes temporarily * fix(test): bigger machines have no effect on sync times * feat: add an image to inherit from with zcash params * fix(cd): use the right image name and allow push to test * fix(cd): use the right docker target and remove extra builds * refactor(docker): use cached zcash params from previous build * fix(cd): finalize for merging * imp(cd): add double safety measure for production * fix(cd): use specific SHA for containers * fix(cd): use latest gcloud action version * fix(test): use the network as Mainnet and remove the uppercase from tests * fix(test): run disk regeneration on specific file change Just run this regeneration when changing the following files: https://github.com/ZcashFoundation/zebra/blob/main/zebra-state/src/service/finalized_state/disk_format.rs https://github.com/ZcashFoundation/zebra/blob/main/zebra-state/src/service/finalized_state.rs https://github.com/ZcashFoundation/zebra/blob/main/zebra-state/src/constants.rs * refactor(test): seggregate disks regeneration from tests Allow to regenerate disks without running tests, and to run tests from previous disk regeneration. Disk will be regenerated just if specific files were changed, or triggered manually. Tests will run just if a disk regeneration was not manually triggered. * fix(test): gcp disks require lower case conventions * fix(test): validate logs being emmited by docker GHA is transforming is somehow transforwing the variable to lowercase also, so we're changint it to adapt to it * test * fix(test): force tty terminal * fix(test): use a one line command to test terminal output * fix(test): always delete test instance * fix(test): use short SHA from the PR head Using the SHA from the base, creates confusion and it's not accurate with the SHA being shown and used on GitHub. We have to keep both as manual runs with `workflow_dispatch` does not have a PR SHA * fix(ci): do not trigger CI on docker changes There's no impact in this workflow when a change is done in the dockerfile * Instead of runing cargo test when the instance gets created, run this commands afterwards in a different step. As GHA TTY is not working as expected, and workarounds does not play nicely with `gcloud compute ssh` actions/runner#241 (comment) we decided to get the container name from the logs, log directly to the container and run the cargo command from there. * doc(test): document reasoning for new steps * fix(test): increase machine type and ssh timeout * fix(test): run tests on creation and follow container logs This allows to follow logs in Github Actions terminal, while the GCP container is still running. Just delete the instance when following the logs ends successfully or fails * finalize(test): do not rebuild image when changing actions * fix(test): run tests on creation and follow container logs This allows to follow logs in Github Actions terminal, while the GCP container is still running. Just delete the instance when following the logs ends successfully or fails Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-15 20:54:16 -04:00 · 2022-02-15 20:54:16 -04:00 · fe2edca1ba
parent 83827b9d03
commit fe2edca1ba
7 changed files with 366 additions and 155 deletions
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@ -9,7 +9,6 @@ on:
      - '**/Cargo.toml'
      - '**/Cargo.lock'
      - '**/deny.toml'
-      - 'docker/**'
      - '.github/workflows/ci.yml'

 env:
--- a/.github/workflows/coverage.patch.yml
+++ b/.github/workflows/coverage.patch.yml
@ -1,7 +1,6 @@
 name: Coverage

 on:
-  workflow_dispatch:
  pull_request:
    paths-ignore:
      - '**/*.rs'
--- a/.github/workflows/docs.patch.yml
+++ b/.github/workflows/docs.patch.yml
@ -1,21 +0,0 @@
-name: Docs
-
-on:
-  workflow_dispatch:
-  push:
-    branches:
-      - main
-    paths-ignore:
-      - 'book/**'
-      - '**/firebase.json'
-      - 'katex-header.html'
-      - '.github/workflows/docs.yml'
-
-jobs:
-  build:
-    name: Build and Deploy Docs (+beta)
-    timeout-minutes: 30
-    runs-on: ubuntu-latest
-
-    steps:
-      - run: 'echo "No build required"'
--- a/.github/workflows/regenerate-stateful-test-disks.yml
+++ b/.github/workflows/regenerate-stateful-test-disks.yml
@ -1,70 +0,0 @@
-name: Regenerate test state
-
-on:
-  workflow_dispatch:
-    inputs:
-      network:
-        default: 'mainnet'
-
-env:
-  PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
-  ZONE: europe-west1-b
-  MACHINE_TYPE: n2-standard-4
-  DEPLOY_SA: cos-vm@${{ secrets.GCP_PROJECT_ID }}.iam.gserviceaccount.com
-
-jobs:
-
-  regenerate:
-    name: Regenerate test state
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v2.4.0
-        with:
-          persist-credentials: false
-
-      - name: Inject slug/short variables
-        uses: rlespinasse/github-slug-action@v4
-
-      - name: Set up gcloud
-        uses: google-github-actions/setup-gcloud@v0.5.0
-        with:
-          project_id: ${{ env.PROJECT_ID }}
-          service_account_key: ${{ secrets.GCLOUD_AUTH }}
-
-      # Creates Compute Engine virtual machine instance w/ disks
-      - name: Create instance
-        run: |
-          gcloud compute instances create-with-container "zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_SHA_SHORT }}" \
-          --boot-disk-size 100GB \
-          --boot-disk-type pd-ssd \
-          --container-image rust:buster \
-          --container-mount-disk mount-path='/${{ github.event.inputs.network }}',name="zebrad-cache-${{ env.GITHUB_SHA_SHORT }}-${{ github.event.inputs.network }}-canopy" \
-          --container-restart-policy never \
-          --create-disk name="zebrad-cache-${{ env.GITHUB_SHA_SHORT }}-${{ github.event.inputs.network }}-canopy",size=100GB,type=pd-balanced \
-          --machine-type ${{ env.MACHINE_TYPE }} \
-          --service-account ${{ env.DEPLOY_SA }} \
-          --scopes cloud-platform \
-          --tags zebrad \
-          --zone "${{ env.ZONE }}"
-      # Build and run test container to sync up to activation and no further
-      - name: Regenerate state for tests
-        id: regenerate-state
-        run: |
-          gcloud compute ssh "zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_SHA_SHORT }}" --zone "${{ env.ZONE }}" --command \
-          "git clone -b ${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }} https://github.com/ZcashFoundation/zebra.git &&
-          cd zebra/ &&
-          docker build --build-arg SHORT_SHA=${{ env.GITHUB_SHA_SHORT }} -f docker/Dockerfile.test -t zebrad-test . &&
-          docker run -i -e "ZEBRA_SKIP_IPV6_TESTS=1" --mount type=bind,source=/mnt/disks/gce-containers-mounts/gce-persistent-disks/zebrad-cache-${{ env.GITHUB_SHA_SHORT }}-${{ github.event.inputs.network }}-canopy,target=/zebrad-cache zebrad-test:latest cargo test --verbose --features test_sync_to_mandatory_checkpoint_${{ github.event.inputs.network }} --manifest-path zebrad/Cargo.toml sync_to_mandatory_checkpoint_${{ github.event.inputs.network }};
-          "
-      # Create image from disk that will be used in test.yml workflow
-      - name: Create image from state disk
-        # Only run if the earlier step succeeds
-        if: steps.regenerate-state.outcome == 'success'
-        run: |
-          gcloud compute images create "zebrad-cache-${{ env.GITHUB_SHA_SHORT }}-${{ github.event.inputs.network }}-canopy" --source-disk="zebrad-cache-${{ env.GITHUB_SHA_SHORT }}-${{ github.event.inputs.network }}-canopy" --source-disk-zone="${{ env.ZONE }}"
-      # Clean up
-      - name: Delete test instance
-        # Always run even if the earlier step fails
-        if: ${{ always() }}
-        run: |
-          gcloud compute instances delete "zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_SHA_SHORT }}" --delete-disks all --zone "${{ env.ZONE }}"
--- a/.github/workflows/test.patch.yml
+++ b/.github/workflows/test.patch.yml
@ -1,22 +0,0 @@
-name: Test
-
-on:
-  workflow_dispatch:
-  push:
-    branches:
-      - main
-    paths-ignore:
-      - '**/*.rs'
-      - '**/*.txt'
-      - '**/Cargo.toml'
-      - '**/Cargo.lock'
-      - 'docker/**'
-      - '.github/workflows/test.yml'
-
-jobs:
-  test:
-    name: Run all tests
-    runs-on: ubuntu-latest
-
-    steps:
-      - run: 'echo "No build required"'
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@ -2,7 +2,14 @@ name: Test

 on:
  workflow_dispatch:
-  push:
+    inputs:
+      network:
+        default: 'Mainnet'
+      regenerate-disks:
+        type: boolean
+        default: false
+        description: Just update stateful disks
+  pull_request:
    branches:
      - main
    paths:
@ -14,15 +21,21 @@ on:
      - '.github/workflows/test.yml'

 env:
+  CARGO_INCREMENTAL: '1'
+  ZEBRA_SKIP_IPV6_TESTS: "1"
+  NETWORK: Mainnet
  PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
-  ZONE: europe-west1-b
-  MACHINE_TYPE: n2-standard-8
-  DEPLOY_SA: cos-vm@${{ secrets.GCP_PROJECT_ID }}.iam.gserviceaccount.com
+  GAR_BASE: us-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/zebra
+  GCR_BASE: gcr.io/${{ secrets.GCP_PROJECT_ID }}
+  REGION: us-central1
+  ZONE: us-central1-a
+  MACHINE_TYPE: c2-standard-8
+  IMAGE_NAME: zebrad-test

 jobs:
-
-  test:
-    name: Run all tests
+  build:
+    name: Build images
+    timeout-minutes: 210
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2.4.0
@ -32,41 +45,351 @@ jobs:
      - name: Inject slug/short variables
        uses: rlespinasse/github-slug-action@v4

-      - name: Set up gcloud
-        uses: google-github-actions/setup-gcloud@v0.5.0
+      # Setup Docker Buildx to allow use of docker cache layers from GH
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v1
        with:
-          project_id: ${{ env.PROJECT_ID }}
-          service_account_key: ${{ secrets.GCLOUD_AUTH }}
+          driver-opts: network=host

-      # Creates Compute Engine virtual machine instance w/ disks
-      - name: Create instance
+      - name: Login to Google Artifact Registry
+        uses: docker/login-action@v1.12.0
+        with:
+          registry: us-docker.pkg.dev
+          username: _json_key
+          password: ${{ secrets.GOOGLE_CREDENTIALS }}
+
+      - name: Login to Google Container Registry
+        uses: docker/login-action@v1.12.0
+        with:
+          registry: gcr.io
+          username: _json_key
+          password: ${{ secrets.GOOGLE_CREDENTIALS }}
+
+      # Build and push image to Google Artifact Registry
+      - name: Build & push
+        id: docker_build
+        uses: docker/build-push-action@v2.8.0
+        with:
+          target: tester
+          context: .
+          file: ./docker/Dockerfile.build
+          tags: |
+            ${{ env.GAR_BASE }}/${{ env.IMAGE_NAME }}:latest
+            ${{ env.GAR_BASE }}/${{ env.IMAGE_NAME }}:${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}
+            ${{ env.GCR_BASE }}/${{ env.GITHUB_REPOSITORY_SLUG_URL }}/${{ env.IMAGE_NAME }}:latest
+            ${{ env.GCR_BASE }}/${{ env.GITHUB_REPOSITORY_SLUG_URL }}/${{ env.IMAGE_NAME }}:${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}
+          build-args: |
+            NETWORK=${{ github.event.inputs.network || env.NETWORK }}
+            SHORT_SHA=${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}
+            RUST_BACKTRACE=full
+            ZEBRA_SKIP_NETWORK_TESTS="1"
+            CHECKPOINT_SYNC=${{ github.event.inputs.checkpoint_sync || true }}
+            RUST_LOG=debug
+            SENTRY_DSN=${{ secrets.SENTRY_ENDPOINT }}
+          push: true
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+  test-all:
+    name: Test all
+    runs-on: ubuntu-latest
+    needs: build
+    if: ${{ github.event.inputs.regenerate-disks != 'true' }}
+    steps:
+      - uses: actions/checkout@v2.4.0
+        with:
+          persist-credentials: false
+
+      - name: Inject slug/short variables
+        uses: rlespinasse/github-slug-action@v4
+
+      - name: Run all zebrad tests
        run: |
-          gcloud compute instances create-with-container "zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_SHA_SHORT }}" \
+          docker pull ${{ env.GAR_BASE }}/${{ env.IMAGE_NAME }}:${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}
+          docker run -e ZEBRA_SKIP_IPV6_TESTS --name zebrad-tests -t ${{ env.GAR_BASE }}/${{ env.IMAGE_NAME }}:${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }} cargo test --locked --release --features enable-sentry --workspace
+
+  test-fake-activation-heights:
+    name: Test with fake activation heights
+    runs-on: ubuntu-latest
+    needs: build
+    if: ${{ github.event.inputs.regenerate-disks != 'true' }}
+    steps:
+      - uses: actions/checkout@v2.4.0
+        with:
+          persist-credentials: false
+
+      - name: Inject slug/short variables
+        uses: rlespinasse/github-slug-action@v4
+
+      - name: Run tests with fake activation heights
+        run: |
+          docker pull ${{ env.GAR_BASE }}/${{ env.IMAGE_NAME }}:${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}
+          docker run -e ZEBRA_SKIP_IPV6_TESTS --name zebrad-tests -t ${{ env.GAR_BASE }}/${{ env.IMAGE_NAME }}:${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }} cargo test --locked --release --package zebra-state --lib -- with_fake_activation_heights
+
+  test-large-sync:
+    name: Test large sync
+    runs-on: ubuntu-latest
+    needs: build
+    if: ${{ github.event.inputs.regenerate-disks != 'true' }}
+    steps:
+      - uses: actions/checkout@v2.4.0
+        with:
+          persist-credentials: false
+
+      - name: Inject slug/short variables
+        uses: rlespinasse/github-slug-action@v4
+
+      - name: Run zebrad large sync tests
+        run: |
+          docker pull ${{ env.GAR_BASE }}/${{ env.IMAGE_NAME }}:${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}
+          docker run -e ZEBRA_SKIP_IPV6_TESTS --name zebrad-tests -t ${{ env.GAR_BASE }}/${{ env.IMAGE_NAME }}:${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }} cargo test --locked --release --features enable-sentry --test acceptance sync_large_checkpoints_ -- --ignored
+
+  test-zunstable:
+    name: Test with Zunstable options
+    runs-on: ubuntu-latest
+    needs: build
+    if: ${{ github.event.inputs.regenerate-disks != 'true' }}
+    steps:
+      - uses: actions/checkout@v2.4.0
+        with:
+          persist-credentials: false
+
+      - name: Inject slug/short variables
+        uses: rlespinasse/github-slug-action@v4
+
+      - name: Run Zunstable-options tests
+        run: |
+          docker pull ${{ env.GAR_BASE }}/${{ env.IMAGE_NAME }}:${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}
+          docker run -e ZEBRA_SKIP_IPV6_TESTS --name zebrad-tests -t ${{ env.GAR_BASE }}/${{ env.IMAGE_NAME }}:${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }} cargo test --locked --release --features enable-sentry --workspace -- -Zunstable-options --include-ignored
+
+  regenerate-stateful-disks:
+    name: Renerate state disks
+    runs-on: ubuntu-latest
+    needs: build
+    steps:
+      - uses: actions/checkout@v2.4.0
+        with:
+          persist-credentials: false
+          fetch-depth: '2'
+
+      - name: Get specific changed files
+        id: changed-files-specific
+        uses: tj-actions/changed-files@v14.4
+        with:
+          files: |
+            /zebra-state/**/disk_format.rs
+            /zebra-state/**/finalized_state.rs
+            /zebra-state/**/constants.rs
+
+      - name: Inject slug/short variables
+        uses: rlespinasse/github-slug-action@v4
+
+      - name: Downcase network name for disks
+        run: |
+          echo LOWER_NET_NAME="${{ github.event.inputs.network || env.NETWORK }}" | awk '{print tolower($0)}' >> $GITHUB_ENV
+
+      # Setup gcloud CLI
+      - name: Authenticate to Google Cloud
+        id: auth
+        uses: google-github-actions/auth@v0.5.0
+        with:
+          credentials_json: ${{ secrets.GOOGLE_CREDENTIALS }}
+
+      - name: Create GCP compute instance
+        id: create-instance
+        if: ${{ steps.changed-files-specific.outputs.any_changed == 'true' }} || ${{ github.event.inputs.regenerate-disks == 'true' }}
+        run: |
+          gcloud compute instances create-with-container "zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}" \
          --boot-disk-size 100GB \
          --boot-disk-type pd-ssd \
-          --container-image rust:buster \
-          --container-mount-disk mount-path='/mainnet',name="zebrad-cache-${{ env.GITHUB_SHA_SHORT }}-mainnet-canopy" \
-          --container-restart-policy never \
-          --create-disk name="zebrad-cache-${{ env.GITHUB_SHA_SHORT }}-mainnet-canopy",image=zebrad-cache-1558f3378-mainnet-canopy \
+          --create-disk name="zebrad-cache-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}-${{ env.lower_net_name }}-canopy",size=100GB,type=pd-ssd \
+          --container-mount-disk mount-path='/zebrad-cache',name="zebrad-cache-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}-${{ env.lower_net_name }}-canopy" \
+          --container-image ${{ env.GAR_BASE }}/${{ env.IMAGE_NAME }}:${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }} \
+          --container-restart-policy=never \
+          --container-stdin \
+          --container-tty \
+          --container-command="cargo" \
+          --container-arg="test" \
+          --container-arg="--locked" \
+          --container-arg="--release" \
+          --container-arg="--features" \
+          --container-arg="enable-sentry,test_sync_to_mandatory_checkpoint_${{ env.lower_net_name }}" \
+          --container-arg="--manifest-path" \
+          --container-arg="zebrad/Cargo.toml" \
+          --container-arg="sync_to_mandatory_checkpoint_${{ env.lower_net_name }}" \
+          --container-env=ZEBRA_SKIP_IPV6_TESTS=1 \
          --machine-type ${{ env.MACHINE_TYPE }} \
-          --service-account ${{ env.DEPLOY_SA }} \
          --scopes cloud-platform \
+          --metadata=google-monitoring-enabled=true,google-logging-enabled=true \
          --tags zebrad \
          --zone "${{ env.ZONE }}"

-      # Build and run test container
-      - name: Run all tests
+      # TODO: this approach is very mesy, but getting the just created container name is very error prone and GCP doesn't have a workaround for this without requiring a TTY
+      # This TODO relates to the following issues:
+      # https://github.com/actions/runner/issues/241
+      # https://www.googlecloudcommunity.com/gc/Infrastructure-Compute-Storage/SSH-into-Compute-Container-not-easily-possible/td-p/170915
+      - name: Get container name from logs
+        id: get-container-name
+        if: steps.create-instance.outcome == 'success'
        run: |
-          gcloud compute ssh "zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_SHA_SHORT }}" --ssh-flag="-o ServerAliveInterval=5" --zone "${{ env.ZONE }}" --command \
-          "git clone -b ${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }} https://github.com/ZcashFoundation/zebra.git &&
-          cd zebra/ &&
-          docker build --build-arg SHORT_SHA=${{ env.GITHUB_SHA_SHORT }} -f docker/Dockerfile.test -t zebrad-test . &&
-          docker run -t -e ZEBRA_SKIP_IPV6_TESTS=1 zebrad-test:latest cargo test --workspace --no-fail-fast -- -Zunstable-options --include-ignored &&
-          docker run -t -e ZEBRA_SKIP_IPV6_TESTS=1 --mount type=bind,source=/mnt/disks/gce-containers-mounts/gce-persistent-disks/zebrad-cache-${{ env.GITHUB_SHA_SHORT }}-mainnet-canopy,target=/zebrad-cache zebrad-test:latest cargo test --verbose --features test_sync_past_mandatory_checkpoint_mainnet --manifest-path zebrad/Cargo.toml sync_past_mandatory_checkpoint_mainnet
-          "
-      # Clean up
+          INSTANCE_ID=$(gcloud compute instances describe zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }} --zone ${{ env.ZONE }} --format='value(id)')
+          echo "Using instance: $INSTANCE_ID"
+          while [[ ${CONTAINER_NAME} != *"zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}"* ]]; do
+              CONTAINER_NAME=$(gcloud logging read 'log_name=projects/${{ env.PROJECT_ID }}/logs/cos_system AND jsonPayload.MESSAGE:zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}' --format='value(jsonPayload.MESSAGE)' --limit=1 | grep -o '...-zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}-....' | tr -d "'.")
+              echo "Using container: ${CONTAINER_NAME} from instance: ${INSTANCE_ID}"
+              sleep 10
+          done
+          CONTAINER_NAME=$(gcloud logging read 'log_name=projects/${{ env.PROJECT_ID }}/logs/cos_system AND jsonPayload.MESSAGE:zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}' --format='value(jsonPayload.MESSAGE)' --limit=1 | grep -o '...-zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}-....' | tr -d "'.")
+          echo "::set-output name=zebra_container::$CONTAINER_NAME"
+
+      - name: Regenerate stateful disks logs
+        id: sync-to-checkpoint
+        run: |
+          gcloud compute ssh \
+          zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }} \
+          --zone ${{ env.ZONE }} \
+          --quiet \
+          --ssh-flag="-o ServerAliveInterval=5" \
+          --command="docker logs --follow ${{ env.ZEBRA_CONTAINER }}"
+        env:
+          ZEBRA_CONTAINER: ${{ steps.get-container-name.outputs.zebra_container }}
+
+      # Create image from disk that will be used to sync past mandatory checkpoint test
+      # Force the image creation as the disk is still attached even though is not being used by the container
+      - name: Create image from state disk
+        # Only run if the earlier step succeeds
+        if: steps.sync-to-checkpoint.outcome == 'success'
+        run: |
+          gcloud compute images create zebrad-cache-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}-${{ env.lower_net_name }}-canopy \
+          --force \
+          --source-disk=zebrad-cache-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}-${{ env.lower_net_name }}-canopy \
+          --source-disk-zone=${{ env.ZONE }} \
+          --storage-location=us \
+          --description="Created from head branch ${{ env.GITHUB_HEAD_REF_SLUG }} targeting ${{ env.GITHUB_BASE_REF_SLUG }} from PR ${{ env.GITHUB_REF_SLUG }} with commit ${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA }}"
+
+      - name: Write the disk SHORT_SHA to a txt
+        if: steps.sync-to-checkpoint.outcome == 'success'
+        run: |
+          short_sha=$(echo "${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}")
+          echo "$short_sha" > latest-disk-state-sha.txt
+
+      - name: Upload the disk state txt
+        uses: actions/upload-artifact@v2.3.1
+        with:
+          name: latest-disk-state-sha
+          path: latest-disk-state-sha.txt
+          retention-days: 1095
+
      - name: Delete test instance
-        # Always run even if the earlier step fails
-        if: ${{ always() }}
+        # Do not delete the instance if the sync timeouts in GitHub
+        if: ${{ steps.sync-to-checkpoint.outcome == 'success' }} || ${{ steps.sync-to-checkpoint.outcome == 'failure' }}
+        continue-on-error: true
        run: |
-          gcloud compute instances delete "zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_SHA_SHORT }}" --delete-disks all --zone "${{ env.ZONE }}"
+          gcloud compute instances delete "zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}" --delete-disks all --zone "${{ env.ZONE }}"
+
+  test-stateful-sync:
+    name: Test blocks sync
+    runs-on: ubuntu-latest
+    needs: [ build, regenerate-stateful-disks]
+    if: ${{ github.event.inputs.regenerate-disks != 'true' }}
+    steps:
+      - uses: actions/checkout@v2.4.0
+        with:
+          persist-credentials: false
+
+      - name: Inject slug/short variables
+        uses: rlespinasse/github-slug-action@v4
+
+      - name: Downcase network name for disks
+        run: |
+          echo LOWER_NET_NAME="${{ github.event.inputs.network || env.NETWORK }}" | awk '{print tolower($0)}' >> $GITHUB_ENV
+
+      # Get the latest uploaded txt with the disk SHORT_SHA from this workflow
+      - name: Download latest disk state SHORT_SHA
+        uses: dawidd6/action-download-artifact@v2.17.0
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          workflow: test.yml
+          name: latest-disk-state-sha
+          check_artifacts:  true
+
+      - name: Get disk state SHA from txt
+        id: get-disk-sha
+        run: |
+          output=$(cat latest-disk-state-sha.txt)
+          echo "::set-output name=sha::$output"
+
+      # Setup gcloud CLI
+      - name: Authenticate to Google Cloud
+        id: auth
+        uses: google-github-actions/auth@v0.5.0
+        with:
+          credentials_json: ${{ secrets.GOOGLE_CREDENTIALS }}
+
+      # Creates Compute Engine virtual machine instance w/ disks
+      - name: Create GCP compute instance
+        id: create-instance
+        run: |
+          gcloud compute instances create-with-container "zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}" \
+          --boot-disk-size 100GB \
+          --boot-disk-type pd-ssd \
+          --disk=boot=no,mode=rw,name=zebrad-cache-${{ env.DISK_SHORT_SHA }}-${{ env.lower_net_name }}-canopy \
+          --container-mount-disk=mount-path='/zebrad-cache',name=zebrad-cache-${{ env.DISK_SHORT_SHA }}-${{ env.lower_net_name }}-canopy \
+          --container-image ${{ env.GAR_BASE }}/${{ env.IMAGE_NAME }}:${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }} \
+          --container-restart-policy=never \
+          --container-stdin \
+          --container-tty \
+          --container-command="cargo" \
+          --container-arg="test" \
+          --container-arg="--locked" \
+          --container-arg="--release" \
+          --container-arg="--features" \
+          --container-arg="enable-sentry,test_sync_past_mandatory_checkpoint_${{ env.lower_net_name }}" \
+          --container-arg="--manifest-path" \
+          --container-arg="zebrad/Cargo.toml" \
+          --container-arg="sync_past_mandatory_checkpoint_${{ env.lower_net_name }}" \
+          --container-env=ZEBRA_SKIP_IPV6_TESTS=1 \
+          --machine-type ${{ env.MACHINE_TYPE }} \
+          --scopes cloud-platform \
+          --metadata=google-monitoring-enabled=true,google-logging-enabled=true \
+          --tags zebrad \
+          --zone "${{ env.ZONE }}"
+        env:
+          DISK_SHORT_SHA: ${{ steps.get-disk-sha.outputs.sha }}
+
+      # TODO: this approach is very mesy, but getting the just created container name is very error prone and GCP doesn't have a workaround for this without requiring a TTY
+      # This TODO relates to the following issues:
+      # https://github.com/actions/runner/issues/241
+      # https://www.googlecloudcommunity.com/gc/Infrastructure-Compute-Storage/SSH-into-Compute-Container-not-easily-possible/td-p/170915
+      - name: Get container name from logs
+        id: get-container-name
+        if: steps.create-instance.outcome == 'success'
+        run: |
+          INSTANCE_ID=$(gcloud compute instances describe zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }} --zone ${{ env.ZONE }} --format='value(id)')
+          echo "Using instance: $INSTANCE_ID"
+          while [[ ${CONTAINER_NAME} != *"zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}"* ]]; do
+              CONTAINER_NAME=$(gcloud logging read 'log_name=projects/${{ env.PROJECT_ID }}/logs/cos_system AND jsonPayload.MESSAGE:zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}' --format='value(jsonPayload.MESSAGE)' --limit=1 | grep -o '...-zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}-....' | tr -d "'.")
+              echo "Using container: ${CONTAINER_NAME} from instance: ${INSTANCE_ID}"
+              sleep 10
+          done
+          CONTAINER_NAME=$(gcloud logging read 'log_name=projects/${{ env.PROJECT_ID }}/logs/cos_system AND jsonPayload.MESSAGE:zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}' --format='value(jsonPayload.MESSAGE)' --limit=1 | grep -o '...-zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}-....' | tr -d "'.")
+          echo "::set-output name=zebra_container::$CONTAINER_NAME"
+
+      - name: Sync past mandatory checkpoint logs
+        id: sync-past-checkpoint
+        run: |
+          gcloud compute ssh \
+          zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }} \
+          --zone ${{ env.ZONE }} \
+          --quiet \
+          --ssh-flag="-o ServerAliveInterval=5" \
+          --command="docker logs --follow ${{ env.ZEBRA_CONTAINER }}"
+        env:
+          ZEBRA_CONTAINER: ${{ steps.get-container-name.outputs.zebra_container }}
+
+      - name: Delete test instance
+        # Do not delete the instance if the sync timeouts in GitHub
+        if: ${{ steps.sync-past-checkpoint.outcome == 'success' }} || ${{ steps.sync-past-checkpoint.outcome == 'failure' }}
+        continue-on-error: true
+        run: |
+          gcloud compute instances delete "zebrad-tests-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_EVENT_PULL_REQUEST_HEAD_SHA_SHORT || env.GITHUB_SHA_SHORT }}" --delete-disks all --zone "${{ env.ZONE }}"
--- a/.github/workflows/zcashd-manual-deploy.yml
+++ b/.github/workflows/zcashd-manual-deploy.yml
@ -4,22 +4,22 @@ on:
  workflow_dispatch:
    inputs:
      network:
-        default: 'testnet'
+        default: 'Testnet'
      size:
        default: 10

 env:
  PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
  REGION: us-central1
-  MACHINE_TYPE: n2-standard-4
-  DEPLOY_SA: cos-vm@${{ secrets.GCP_PROJECT_ID }}.iam.gserviceaccount.com
+  ZONE: us-central1-a
+  MACHINE_TYPE: c2-standard-4

 jobs:
-
  deploy:
    name: Deploy zcashd nodes
    runs-on: ubuntu-latest
    timeout-minutes: 30
+
    steps:
      - uses: actions/checkout@v2.4.0
        with:
@ -38,14 +38,17 @@ jobs:
      # Create instance template from container image
      - name: Create instance template
        run: |
-          gcloud compute instance-templates create-with-container "zcashd-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_SHA_SHORT }}" \
-          --boot-disk-size 100GB \
-          --container-image "electriccoinco/zcashd" \
+          gcloud compute instance-templates create-with-container zcashd-${{ env.GITHUB_HEAD_REF_SLUG || env.GITHUB_REF_SLUG }}-${{ env.GITHUB_SHA_SHORT }} \
+          --boot-disk-size 10GB \
+          --boot-disk-type=pd-ssd \
+          --container-stdin \
+          --container-tty \
+          --container-image electriccoinco/zcashd \
          --container-env ZCASHD_NETWORK="${{ github.event.inputs.network }}" \
          --machine-type ${{ env.MACHINE_TYPE }} \
          --service-account ${{ env.DEPLOY_SA }} \
          --scopes cloud-platform \
-          --tags zcashd \
+          --tags zcashd

      # Check if our destination instance group exists already
      - name: Check if instance group exists