Commit Graph

291 Commits

Author SHA1 Message Date
teor 1896943f62
fix(ci): also check for duplicate dependencies with optional features off (#3592)
* fix(dependencies): update an unused duplicate dependency exception

This duplicate was removed by PR #3572, but other duplicates still exist.

* feat(ci): check for duplicate dependencies with optional features off
2022-02-21 17:41:31 +00:00
Gustavo Valverde 6fafd1af57
refactor(mergify): better test requirements and merge conditions (#3580)
* fix(mergify, actions): use better names and require tests

* feat(queue): do not update the actual PR, create a draft

Do not allow to update/rebase the original pull request to check its mergeability. Create a draft pull request instead.

This doesn't add Mergify as a co-author

* feat(queue): do not interrupt already running queues

Our queues might take more than 5 hours even if the priority is low.

Do not allow interrupting the ongoing speculative checks when a pull request with higher priority enters in the queue.

* fix(mergify): move 'allow' attributes to queue_rules

* fix(mergify): attributes are not conditions
2022-02-18 17:53:27 -05:00
teor e182c33273
fix(ci): make the purpose of each sync test clearer (#3574) 2022-02-17 21:42:26 +00:00
teor 92b561dc8a
refactor(state): split the database module (#3568)
* refactor(state): split the disk_format module

* refactor(ci): add the new disk_db file to the state CI list

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-17 19:08:49 +00:00
dependabot[bot] 61d3243da5
build(deps): bump google-github-actions/setup-gcloud from 0.5.0 to 0.5.1 (#3560)
Bumps [google-github-actions/setup-gcloud](https://github.com/google-github-actions/setup-gcloud) from 0.5.0 to 0.5.1.
- [Release notes](https://github.com/google-github-actions/setup-gcloud/releases)
- [Changelog](https://github.com/google-github-actions/setup-gcloud/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google-github-actions/setup-gcloud/compare/v0.5.0...v0.5.1)

---
updated-dependencies:
- dependency-name: google-github-actions/setup-gcloud
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-16 22:21:15 +00:00
teor d593347a3e
fix(ci): delete a redundant "test all" job (#3552)
* fix(ci): clarify ignored test name

`--include-ignored` runs all tests, including tests
that would normally be ignored.

`-Zunstable-options` enables all unstable options,
but it doesn't do anything by itself.

There is a lot of overlap with "test-all" in this job,
which we might want to fix in a future PR.

* fix(ci): remove unused -Zunstable-options

`--include-ignored` is now stable, so `unstable-options` is not needed.

* fix(test): delete a redundant test

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-16 20:53:37 +00:00
Gustavo Valverde 52ba1f0edd fix(test): create a new disk with the commit being tested 2022-02-16 15:52:12 -04:00
Gustavo Valverde 2e61998182
fix(test): evaluate "if" conditions correctly and use last disk SHA (#3556)
* fix(test): use the short SHA from actual run if valid

* fix(test): if condition must evaluate to a single false

* fix(test): do not run logs and upload if not needed

* imp(test): allow test stateful sync after disk regeneration

This takes is fast enough, so it shouldn't do any harm if run just after a ~3 hours test

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-16 15:56:09 +00:00
Gustavo Valverde 8c07d3906d
fix(actions): allow branches with dots in the name (#3557)
Dependabot creates branches with versions using a dot notation, and some tests fails because of this

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-16 15:47:02 +00:00
Gustavo Valverde 943a38a0d6
fix(clippy): allow unprivileged runs of clippy (#3558) 2022-02-16 15:15:19 +00:00
Gustavo Valverde 8023ab15cf
fix(test): create disk from image before mounting (#3550) 2022-02-16 02:43:29 +00:00
Gustavo Valverde fe2edca1ba
refactor(test): dockerize tests and run sync in detached mode (#3459)
* refactor (cd): overall pipeline improvement

- Use a more ENV configurable Dockerfile
- Remove cloudbuild dependency
- Use compute optimized machine types
- Use SSD instead of normal hard drives
- Move Sentry endpoint to secrets
- Use a single yml for auto & manual deploy
- Migrate to Google Artifact Registry

* refactor (cd): overall pipeline improvement

- Use a more ENV configurable Dockerfile
- Remove cloudbuild dependency
- Use compute optimized machine types
- Use SSD instead of normal hard drives
- Move Sentry endpoint to secrets
- Use a single yml for auto & manual deploy
- Migrate to Google Artifact Registry

* refactor (cd): use newer google auth action

* fix (cd): use newer secret as gcp credential

* fix (docker): do not create extra directories

* fix (docker): ignore .github for caching purposes

* fix (docker): use latest rust

* fix (cd): bump build timeout

* fix: use a better name for manual deployment

* refactor (docker): use standard directories for executable

* fix (cd): most systems expect a "latest" tag

Caching from the latest image is one of the main reasons to add this extra tag. Before this commit, the inline cache was not being used.

* fix (cd): push the build image and the cache separately

The inline cache exporter only supports `min` cache mode. To enable `max` cache mode, push the image and the cache separately by using the registry cache exporter.

This also allows for smaller release images.

* fix (cd): remove unused GHA cache

We're leveraging the registry to cache the actions, instead of using the 10GB limits from Github Actions cache storage

* refactor (cd): use cargo-chef for caching rust deps

* fix: move build system deps before cargo cheg cook

* fix (release): use newer debian to reduce vulnerabilities

* fix (cd): use same zone, region and service accounts

* fix (cd): use same disk size and type for all deployments

* refactor (cd): activate interactive shells

Use interactive shells for manual and test deployments. This allow greater flexibility if troubleshooting is needed inside the machines

* refactor (test): use docker artifact from registry

Instead of using a VM to SSH into in to build and test. Build in GHA (to have the logs available), run the workspace tests in GHA, and just run the sync tests in GCP

Use a cintainer VM with zebra's image directly on it, and pass the needed parameters to run the Sync past mandatory checkpoint.

* tmp (cd): bump timeout for building from scratch

* tmp (test): bump build time

* fix (cd, test): bump build time-out to 210 minutes

* fix (docker): do not build with different settings

Compiling might be slow because different steps are compiling the same code 2-4 times because of the variations

* revert (docker): do not fix the rust version

* fix (docker): build on the root directory

* refactor(docker): Use base image commands and tools

* fix (cd): use correct variables & values, add build concurrency

* fix(cd): use Mainnet instead of mainnet

* imp: remove checkout as Buildkit uses the git context

* fix (docker): just Buildkit uses a .dockerignore in a path

* imp (cd): just use needed variables in the right place

* imp (cd): do not checkout if not needed

* test: run on push

* refactor(docker): reduce build changes

* fix(cd): not checking out was limiting some variables

* refactor(test): add an multistage exclusive for testing

* fix(cd): remove tests as a runtime dependency

* fix(cd): use default service account with cloud-platform scope

* fix(cd): revert checkout actions

* fix: use GA c2 instead of Preview c2d machine types

* fix(actions): remove workflow_dispatch from patched actions

This causes GitHub confusion as it can't determined which of the actions using workflow_dispatch is the right one

* fix(actions): remove patches from push actions

* test: validate changes on each push

* fix(test): wrong file syntax on test job

* fix(test): add missing env parameters

* fix(docker): Do not rebuild to download params and run tests

* fix(test): setup gcloud and loginto artifact just when needed

Try not to rebuild the tests

* fix(test): use GCP container to sync past mandatory checkpoint

* fix(test): missing separators

* test

* fix(test): mount the available disk

* push

* refactor(test): merge disk regeneration into test.yml

* fix(cd): minor typo fixes

* fix(docker): rebuild on .github changes

* fix(cd): keep compatibility with gcr.io

To prevent conflicts between registries, and migrate when the time is right, we'll keep pushing to both registries and use github actions cache to prevent conflicts between artifacts.

* fix(cd): typo and scope

* fix(cd): typos everywhere

* refactor(test): use smarter docker wait and keep old registry

* fix(cd): do not constraint the CPUs for bigger machines

* revert(cd): reduce PR diff as there's a separate one for tests

* fix(docker): add .github as it has no impact on caching

* fix(test): run command correctly

* fix(test): wiat and create image if previous step succeded

* force rebuild

* fix(test): do not restrict interdependant steps based on event

* force push

* feat(docker): add google OS Config agent

Use a separate step to have better flexibility in case a better approach is available

* fix(test): remove all hardoced values and increase disks

* fix(test): use correct commands on deploy

* fix(test): use args as required by google

* fix(docker): try not to invalidate zebrad download cache

* fix(test): minor typo

* refactor(test): decouple jobs for better modularity

This also allows faster tests as testing Zunstable won't be a dependency and it can't stop already started jobs if it fails.

* fix(test): Do not try to execute ss and commands in one line

* fix(test): do not show undeeded information in the terminal

* fix(test): sleep befor/after machine creation/deletion

* fix(docker): do not download zcash params twice

* feat(docker): add google OS Config agent

Use a separate step to have better flexibility in case a better approach is available

* merge: docker-actions-refactor into docker-test-refactor

* test docker wait scenarios

* fix(docker): $HOME variables is not being expanded

* fix(test): allow docker wait to work correctly

* fix(docker): do not use variables while using COPY

* fix(docker): allow to use zebrad as a command

* fix(cd): use test .yml from main

* fix(cd): Do not duplicate network values

The Dockerfile has an ARG with a default value of 'Mainnet', if this value is changed it will be done manually on a workflow_dispatch, making the ENV option a uneeded duplicate in this workflow

* fix(test): use bigger machine type for compute intensive tasks

* refactor(test): add tests in CI file

* fix(test): remove duplicated tests

* fix(test): typo

* test: build on .github changes temporarily

* fix(test): bigger machines have no effect on sync times

* feat: add an image to inherit from with zcash params

* fix(cd): use the right image name and allow push to test

* fix(cd): use the right docker target and remove extra builds

* refactor(docker): use cached zcash params from previous build

* fix(cd): finalize for merging

* imp(cd): add double safety measure for production

* fix(cd): use specific SHA for containers

* fix(cd): use latest gcloud action version

* fix(test): use the network as Mainnet and remove the uppercase from tests

* fix(test): run disk regeneration on specific file change

Just run this regeneration when changing the following files:

https://github.com/ZcashFoundation/zebra/blob/main/zebra-state/src/service/finalized_state/disk_format.rs
https://github.com/ZcashFoundation/zebra/blob/main/zebra-state/src/service/finalized_state.rs
https://github.com/ZcashFoundation/zebra/blob/main/zebra-state/src/constants.rs

* refactor(test): seggregate disks regeneration from tests

Allow to regenerate disks without running tests, and to run tests from previous disk regeneration.

Disk will be regenerated just if specific files were changed, or triggered manually.

Tests will run just if a disk regeneration was not manually triggered.

* fix(test): gcp disks require lower case conventions

* fix(test): validate logs being emmited by docker

GHA is transforming is somehow transforwing the variable to lowercase also, so we're changint it to adapt to it

* test

* fix(test): force tty terminal

* fix(test): use a one line command to test terminal output

* fix(test): always delete test instance

* fix(test): use short SHA from the PR head

Using the SHA from the base, creates confusion and it's not accurate with the SHA being shown and used on GitHub.

We have to keep both as manual runs with `workflow_dispatch` does not have a PR SHA

* fix(ci): do not trigger CI on docker changes

There's no impact in this workflow when a change is done in the dockerfile

* Instead of runing cargo test when the instance gets created, run this commands afterwards in a different step.

As GHA TTY is not working as expected, and workarounds does not play nicely with `gcloud compute ssh` actions/runner#241 (comment) we decided to get the container name from the logs, log directly to the container and run the cargo command from there.

* doc(test): document reasoning for new steps

* fix(test): increase machine type and ssh timeout

* fix(test): run tests on creation and follow container logs

This allows to follow logs in Github Actions terminal, while the GCP container is still running.

Just delete the instance when following the logs ends successfully or fails

* finalize(test): do not rebuild image when changing actions

* fix(test): run tests on creation and follow container logs

This allows to follow logs in Github Actions terminal, while the GCP container is still running.

Just delete the instance when following the logs ends successfully or fails

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-16 00:54:16 +00:00
dependabot[bot] 2cae880e3e
Bump google-github-actions/auth from 0.5.0 to 0.6.0 (#3496)
Bumps [google-github-actions/auth](https://github.com/google-github-actions/auth) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/google-github-actions/auth/releases)
- [Changelog](https://github.com/google-github-actions/auth/blob/main/CHANGELOG.md)
- [Commits](https://github.com/google-github-actions/auth/compare/v0.5.0...v0.6.0)

---
updated-dependencies:
- dependency-name: google-github-actions/auth
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-13 12:39:18 +00:00
dependabot[bot] a471e9e3ee
Bump docker/build-push-action from 2.8.0 to 2.9.0 (#3497)
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 2.8.0 to 2.9.0.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v2.8.0...v2.9.0)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-13 12:37:06 +00:00
Gustavo Valverde fd8c727c31
fix(actions): typo on paths filtering (#3516)
The keyword is `paths` and the actions were using `path`

That's the reason why most actions have been running, and there's been no impact in time savings
2022-02-11 14:05:19 +00:00
Gustavo Valverde d2e58dfa37 fix(mergify): do not wait for depecrated Cloud Build
Cloud Build was recently removed, and Mergify is expecting a result from a non-existing action.
2022-02-10 18:44:10 -05:00
Gustavo Valverde 284aeb7106
fix: temporarily allow forked repos to run PR workflows (#3503)
* fix(zcash-params): Do not update parameters image on PR

We should not update a direct dependency of our Docker image to be writeable by a PR from anywhere, a local branch or a fork branch, before that change has been approved by a human and merged to #main

Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
2022-02-10 07:31:33 -04:00
Gustavo Valverde a872f1727a
refactor(cd): improve Docker and gcloud usage without Cloud Build (#3431)
* refactor (cd): overall pipeline improvement

- Use a more ENV configurable Dockerfile
- Remove cloudbuild dependency
- Use compute optimized machine types
- Use SSD instead of normal hard drives
- Move Sentry endpoint to secrets
- Use a single yml for auto & manual deploy
- Migrate to Google Artifact Registry

* refactor (cd): overall pipeline improvement

- Use a more ENV configurable Dockerfile
- Remove cloudbuild dependency
- Use compute optimized machine types
- Use SSD instead of normal hard drives
- Move Sentry endpoint to secrets
- Use a single yml for auto & manual deploy
- Migrate to Google Artifact Registry

* refactor (cd): use newer google auth action

* fix (cd): use newer secret as gcp credential

* fix (docker): do not create extra directories

* fix (docker): ignore .github for caching purposes

* fix (docker): use latest rust

* fix: use a better name for manual deployment

* refactor (docker): use standard directories for executable

* fix (cd): most systems expect a "latest" tag

Caching from the latest image is one of the main reasons to add this extra tag. Before this commit, the inline cache was not being used.

* fix (cd): push the build image and the cache separately

The inline cache exporter only supports `min` cache mode. To enable `max` cache mode, push the image and the cache separately by using the registry cache exporter.

This also allows for smaller release images.

* fix (cd): remove unused GHA cache

We're leveraging the registry to cache the actions, instead of using the 10GB limits from Github Actions cache storage

* refactor (cd): use cargo-chef for caching rust deps

* fix (release): use newer debian to reduce vulnerabilities

* fix (cd): use same zone, region and service accounts

* fix (cd): use same disk size and type for all deployments

* refactor (cd): activate interactive shells

Use interactive shells for manual and test deployments. This allow greater flexibility if troubleshooting is needed inside the machines

* fix (docker): do not build with different settings

Compiling might be slow because different steps are compiling the same code 2-4 times because of the variations

* fix(cd): use Mainnet instead of mainnet

* fix(docker): remove tests as a runtime dependency

* fix(cd): use default service account with cloud-platform scope

* fix(cd): keep compatibility with gcr.io

To prevent conflicts between registries, and migrate when the time is right, we'll keep pushing to both registries and use github actions cache to prevent conflicts between artifacts.

* fix(docker): do not download zcash params twice

* feat(docker): add google OS Config agent

Use a separate step to have better flexibility in case a better approach is available

* fix(docker): allow to use zebrad as a command

* feat: add an image to inherit from with zcash params

* refactor(docker): use cached zcash params from previous build

* imp(cd): add double safety measure for production
2022-02-08 20:50:13 -04:00
dependabot[bot] 35222fb30f
Bump google-github-actions/setup-gcloud from 0.4.0 to 0.5.0 (#3471)
Bumps [google-github-actions/setup-gcloud](https://github.com/google-github-actions/setup-gcloud) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/google-github-actions/setup-gcloud/releases)
- [Changelog](https://github.com/google-github-actions/setup-gcloud/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google-github-actions/setup-gcloud/compare/v0.4.0...v0.5.0)

---
updated-dependencies:
- dependency-name: google-github-actions/setup-gcloud
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-08 10:37:26 +00:00
teor 6ddbe7972d
fix(mergify): put PRs with no priority label in the low priority queue (#3454)
This also lets us handle dependabot PRs using the low queue rule.
2022-02-02 02:07:33 +00:00
Gustavo Valverde 4d32f9c765
fix(mergify): restrict merges if there are unresolved reviews (#3453) 2022-02-02 00:28:59 +00:00
Deirdre Connolly 07f120a21a
Restore coverage to use latest nightly (#3450)
Fixes #3367

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-01 01:10:28 +00:00
Gustavo Valverde 5fa40216df
refactor (actions): make better use of variables, secrets and versions (#3393)
* style: use global variables and don't double print

Remove repeated instances of global environment variables. Do not print ENV variables on the terminal as GitHub Actions already shows it.

* fix (actions): Use fixed major versions for actions

As actions get recurrent fixes, using a specific version causes more maintance on the pipelines.

On the other hand, using @master versions could make some action unreliable, as breaking changes might be included without further notice, and even change behavior on a daily basis.

* refactor: make better use of ENV variables

A whole step with refex was being used to extract different variables from GitHub's environment. This gets depecrated in favor of using `rlespinasse/github-slug-action@v4` which has slug URL variables.

A SLUG on a variable will:
- put the variable content in lower case
- replace any character by - except 0-9, a-z, ., and _
- remove leading and trailing - character
- limit the string size to 63 characters

This changes also takes care of using the Head or Base branch for deployments. This will allow us tomerge of workflows, as most steps on this deployment actions are very similar, with little variations between workflows.

* fix (actions): use secrets for sensitive information

* revert: use specific versions for dependabot

Reverting commit 8c93409902
2022-01-26 21:46:18 -04:00
Deirdre Connolly 39fc0e27d8
Bump cd workflow timeout to 60 mins (#3412) 2022-01-26 18:45:08 +00:00
teor aef75ee9ff
fix(mergify): use low-priority queue for the correct jobs (#3406)
This fixes an inverted match condition.
2022-01-26 07:45:14 -04:00
Gustavo Valverde 15eca05370
fix (actions): add duplicated workflows with path-ignore (#3405)
GitHub does not support skipped but required checks by default, this an official workaround to allow this feature; which requires to duplicate the same workflow using path-ignore and just executing a step that will always pass, for each job.

Reference: https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/troubleshooting-required-status-checks#handling-skipped-but-required-checks
2022-01-26 03:19:43 +00:00
Gustavo Valverde 6373a95405
Improve GitHub Actions checks based on files & folders (#3377)
* Segregate linting jobs from CI workflow

Lint on push to all branches, except for main, as this action will be required to merge.

Just run the lint action when a Rust file is changed, as it won't make sense to run it on other scenarios.

DRY with uneeded jobs

* Make actions dependable on changed files or folders

* Fix & add missing paths

* Revert changes removing cargo.lock and deny.toml checks

Also refactor this to use a more redable and change prone cargo-deny-action. And move this actions out of the clippy-deps job, as this are more related to CI than linting.

* Fix wrong indentation

* Add new configuration file from #3386

* Do not fail on licenses as this configuration is missing

* Do not add advisories features

Add advisories checks in a different PR

* Allow tests and coverage on PR series

If we only run CI on branches that are going to merge to main, then PR series become a lot harder to test. (Because each PR is based on the previous PR, not main.)
2022-01-25 07:58:11 -04:00
Conrado Gouvea 4aeabd0b52
Fix interstitial sprout anchors check (#3283)
* Fix interstitial Sprout anchors check

* Update state docs; add sprout_trees_by_anchor to comparisons

* Update book/src/dev/rfcs/0005-state-updates.md

Co-authored-by: Marek <mail@marek.onl>

* Rename `interstitial_roots` to `interstitial_trees`

* Document consensus rules

* Refactor the docs

* Improve the docs for consensus rules

* Update reference to cached state

* Update zebra-state/src/service/check/anchors.rs

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Fix formatting

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-01-18 20:18:49 +00:00
Conrado Gouvea 9b9e49953e
Use 2022-01-14 nightly to workaround coverage issue https://github.com/taiki-e/cargo-llvm-cov/issues/128 (#3366) 2022-01-18 19:31:09 +00:00
Gustavo Valverde ee9f081dd5
Do not update PRs to the latest tip automatically (#3351)
This causes the Mergify bot to commit to each PR, being also included in the squashed merge as an author.

As the queue merges the head branch (main) to latest tip before testing with the CI, having all those feature branches constantly updating with Mergify is not needed
2022-01-14 11:26:55 -05:00
Gustavo Valverde dc4533d368
Use the correct conditions and merge method in Mergify (#3341)
* fix: Use the correct conditions and merge method

Mergifys Status Checks conditions are based on the job name, not the worflow name. As our worflows have dynamic names, each variant must be considered.

Squash merges are the default being used in the Zebra repo, so mergify must comply with this configuration.

Use condition operators for labels in each pull request rule; previously it was expecting both labels to be set. And update names accordingly.

* fix: Allow mergify to merge dependabot PRs

Also adapt dependabot's configuration to use the recently adapted labels

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-01-13 23:34:09 +00:00
Gustavo Valverde fc1a1cdac1
Add mergify merging queues (#3324)
* Add mergify merging queues

* Fix Mergify invalid configuration

* Improve adaptability to the actual workflow

Do not merge if the pull-request test is not green. Do not move draft PRs to the queue. And update keep all open PRs updated.

* Fix a typo on check-success condition
2022-01-07 19:11:01 -05:00
Conrado Gouvea 1cbc06aa64
Add install.md to checklist, update to use a single PR (#3282)
* Add updating install.md to checklist

* Change to use a single PR for version increment and changelog
2021-12-23 10:16:15 -03:00
Janito Vaqueiro Ferreira Filho b71833292d
Use `MockedClientHandle` in other tests (#3241)
* Move `MockedClientHandle` to `peer` module

It's more closely related to a `Client` than the `PeerSet`, and this
prepares it to be used by other tests.

* Rename `MockedClientHandle` to `ClientTestHarness`

Reduce confusion, and clarify that the client is not mocked.

Co-authored-by: teor <teor@riseup.net>

* Add clarification to `mock_peers` documentation

Explicitly say how the generated data is returned.

* Rename method to `wants_connection_heartbeats`

The `Client` service only represents one direction of a connection, so
`is_connected` is not the exact term.

Co-authored-by: teor <teor@riseup.net>

* Mock `Client` instead of `LoadTrackedClient`

Move where the conversion from mocked `Client` to mocked
`LoadTrackedClient` in order to make the test helper more easily used by
other tests.

* Use `ClientTestHarness` in `initialize` tests

Replace the boilerplate code to create a fake `Client` instance with
usages of the `ClientTestHarness` constructor.

* Allow receiving requests from `Client` instance

Create a helper type to wrap the result, to make it easier to assert on
specific events after trying to receive a request.

* Allow inspecting the current error in the slot

Share the `ErrorSlot` between the `Client` and the handle, so that the
handle can be used to inspect the contents of the `ErrorSlot`.

* Allow placing an error into the `ErrorSlot`

Assuming it is initially empty. If it already has an error, the code
will panic.

* Allow gracefully closing the request receiver

Close the endpoint with the appropriate call to the `close()` method.

* Allow dropping the request receiver endpoint

Forcefully closes the endpoint.

* Rename field to `client_request_receiver`

Also rename the related methods to include
`outbound_client_request_receiver` to make it more precise.

Co-authored-by: teor <teor@riseup.net>

* Allow dropping the heartbeat shutdown receiver

Allows the `Client` to detect that the channel has been closed.

* Rename fn. to `drop_heartbeat_shutdown_receiver`

Make it clear that it affects the heartbeat task.

Co-authored-by: teor <teor@riseup.net>

* Move `NowOrLater` into a new `now-or-later` crate

Make it easily accessible to other crates.

* Add `IsReady` extension trait for `Service`

Simplifies checking if a service is immediately ready to be called.

* Add extension method to check for readiness error

Checks if the `Service` isn't immediately ready because a call to
`ready` immediately returns an error.

* Rename method to `is_failed`

Avoid negated method names.

Co-authored-by: teor <teor@riseup.net>

* Add a `IsReady::is_pending` extension method

Checks if a `Service` is not ready to be called.

* Use `ClientTestHarness` in `Client` test vectors

Reduce repeated code and try to improve readability.

* Create a new `ClientTestHarnessBuilder` type

A builder to create test `Client` instances using mock data which can be
tracked and manipulated through a `ClientTestHarness`.

* Allow configuring the `Client`'s mocked version

Add a `with_version` builder method.

* Use `ClientTestHarnessBuilder` in `PeerVersions`

Use the builder to set the peer version, so that the `version` parameter
can be removed from the constructor later.

* Use a default mock version where possible

Reduce noise when setting up the harness for tests that don't really
care about the remote peer version.

* Remove `Version` parameter from the `build` method

The `with_version` builder method should be used instead.

* Fix some typos and outdated info in the release checklist

* Add extra client tests for zero and multiple readiness checks (#3273)

And document existing tests.

* Replace `NowOrLater` with `futures::poll!` (#3272)

* Replace NowOrLater with the futures::poll! macro in zebrad

* Replace NowOrLater with the futures::poll! macro in zebra-test

* Remove the now-or-later crate

* remove unused imports

* rustfmt

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-12-22 06:13:26 +10:00
dependabot[bot] 5543b9b6bc
Bump EmbarkStudios/cargo-deny-action from 1.2.6 to 1.2.9 (#3163)
Bumps [EmbarkStudios/cargo-deny-action](https://github.com/EmbarkStudios/cargo-deny-action) from 1.2.6 to 1.2.9.
- [Release notes](https://github.com/EmbarkStudios/cargo-deny-action/releases)
- [Commits](https://github.com/EmbarkStudios/cargo-deny-action/compare/v1.2.6...v1.2.9)

---
updated-dependencies:
- dependency-name: EmbarkStudios/cargo-deny-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-12-07 11:08:51 +10:00
teor 022808d028
Release Zebra v1.0.0-beta.2 (#3132)
Zebra's latest beta continues implementing zero-knowledge proof and note commitment tree validation. In this release, we have finished implementing transaction header, transaction amount, and Zebra-specific NU5 validation. (NU5 mainnet validation is waiting on an `orchard` crate update, and some consensus parameter updates.)

We also fix a number of security issues that could pose a local denial of service risk, or make it easier for an attacker to make a node follow a false chain.

As of this release, Zebra will automatically download and cache the Sprout and Sapling Groth16 circuit parameters. The cache uses around 1 GB of disk space. These cached parameters are shared across all Zebra and `zcashd` instances run by the same user.

See CHANGELOG.md for the full list of changes in this release.
2021-12-03 06:54:14 +10:00
Marek 3c9ad89018
Add Sprout anchors to `zebra-state` (#3100)
* Add Sprout anchors to the state

* Update zebra-state/src/service/non_finalized_state/chain.rs

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Return new types of note commitments from Sprout transactions

* Refactor the tests

* Refactor some comments

Co-authored-by: teor <teor@riseup.net>

* Increment `DATABASE_FORMAT_VERSION`

* Update `test.yml` with the new image name

* Refactor the `version = 5` transaction description

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Update comment

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
2021-11-30 11:05:58 +01:00
teor 0ef4629232
Automatically download and load sprout parameters (#3085)
* Download and load Sprout parameters using zcash_proofs

Also update some librustzcash dependencies, to avoid duplicate dependencies.

* Update upstream orchard to avoid a compilation error

* Skip librustzcash batch refactor for now, to avoid compilation errors

* Change the cache ID, so we actually cache Sprout

* Move existing file checks into zcash_proofs

* Add a 1 hour timeout to parameter file downloads

* Give other tasks priority, before spawning the download task

* Update to the latest version of our modified librustzcash fork

* Change the cache key for Sprout

* Add 40 minutes to CI timeouts for occasional sprout downloads

* Update to zcash_proofs with split downloads

* Check file sizes to help debug parameter load failures in zcash_proofs

* Start the second download once the first has finished in zcash_proofs

* Document the parameter download task

* Stop hashing existing files twice
2021-11-25 13:26:32 -03:00
teor e054d57622
Speed up CI, and split unrelated and conflicting CI jobs (#3077)
* Move dependency checks to the clippy job

* Split the fake activation heights into their own job

* Fix expected types

* Minimise proptest cases on Windows, macOS, and coverage

We don't expect proptests to fail on different platforms.
2021-11-22 14:26:07 -03:00
teor f7202bfbc0
Download Zcash Sapling parameters and load them from cached files (#3057)
* Replace Zcash parameters crates with pre-downloaded local parameter files

* Download Zcash parameters using the `zcashd` script in CI and Docker

* Add a zcash_proofs dependency to zebra-consensus

* Download Sapling parameters using zcash_proofs, rather than fetch-params.sh

* Add a new `zebrad download` subcommand

This command isn't required for nomrmal usage.
But it's useful when testing, or launching multiple Zebra instances.

* Use `zebrad download` in CI to pre-download parameters

* Log a helpful hint if downloading fails

* Allow some duplicate dependencies currently hidden by orchard

* Spawn a separate task to download Groth16 parameters

* Run the parameter download with code coverage

This avoids re-compining Zebra with and without coverage.

* Update Cargo.lock after rebase

* Try to pass `download` as an argument to `zebrad` in coverage CI

* Fix copy and paste comment typos

* Add path and download examples, like zcash_proofs

* Download params in CI just like zcash_proofs does

* Delete a redundant build step

* Implement graceful shutdown for zebrad start

* Send coverage summary to /dev/null when getting the params path

* Use the correct parameters path and download commands in CI

* Explain pre-downloads

* Avoid calling params_folder twice

* Rename parameter types and methods for consistency

```sh
fastmod SaplingParams SaplingParameters zebra*
fastmod Groth16Params Groth16Parameters zebra*
fastmod PARAMS GROTH16_PARAMETERS zebra*
fastmod params_folder directory zebra*
```

And a manual variable name tweak.

* rustfmt

* Remove a redundant coverage step

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 23:02:56 +00:00
dependabot[bot] bdf5f70557
Bump actions/checkout from 2.3.5 to 2.4.0 (#3009)
Bumps [actions/checkout](https://github.com/actions/checkout) from 2.3.5 to 2.4.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2.3.5...v2.4.0)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
2021-11-05 15:08:39 +10:00
Janito Vaqueiro Ferreira Filho 0960e4fb0b
Update to Tokio 1.13.0 (#2994)
* Update `tower` to version `0.4.9`

Update to latest version to add support for Tokio version 1.

* Replace usage of `ServiceExt::ready_and`

It was deprecated in favor of `ServiceExt::ready`.

* Update Tokio dependency to version `1.13.0`

This will break the build because the code isn't ready for the update,
but future commits will fix the issues.

* Replace import of `tokio::stream::StreamExt`

Use `futures::stream::StreamExt` instead, because newer versions of
Tokio don't have the `stream` feature.

* Use `IntervalStream` in `zebra-network`

In newer versions of Tokio `Interval` doesn't implement `Stream`, so the
wrapper types from `tokio-stream` have to be used instead.

* Use `IntervalStream` in `inventory_registry`

In newer versions of Tokio the `Interval` type doesn't implement
`Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used
instead.

* Use `BroadcastStream` in `inventory_registry`

In newer versions of Tokio `broadcast::Receiver` doesn't implement
`Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This
also requires changing the error type that is used.

* Handle `Semaphore::acquire` error in `tower-batch`

Newer versions of Tokio can return an error if the semaphore is closed.
This shouldn't happen in `tower-batch` because the semaphore is never
closed.

* Handle `Semaphore::acquire` error in `zebrad` test

On newer versions of Tokio `Semaphore::acquire` can return an error if
the semaphore is closed. This shouldn't happen in the test because the
semaphore is never closed.

* Update some `zebra-network` dependencies

Use versions compatible with Tokio version 1.

* Upgrade Hyper to version 0.14

Use a version that supports Tokio version 1.

* Update `metrics` dependency to version 0.17

And also update the `metrics-exporter-prometheus` to version 0.6.1.
These updates are to make sure Tokio 1 is supported.

* Use `f64` as the histogram data type

`u64` isn't supported as the histogram data type in newer versions of
`metrics`.

* Update the initialization of the metrics component

Make it compatible with the new version of `metrics`.

* Simplify build version counter

Remove all constants and use the new `metrics::incement_counter!` macro.

* Change metrics output line to match on

The snapshot string isn't included in the newer version of
`metrics-exporter-prometheus`.

* Update `sentry` to version 0.23.0

Use a version compatible with Tokio version 1.

* Remove usage of `TracingIntegration`

This seems to not be available from `sentry-tracing` anymore, so it
needs to be replaced.

* Add sentry layer to tracing initialization

This seems like the replacement for `TracingIntegration`.

* Remove unnecessary conversion

Suggested by a Clippy lint.

* Update Cargo lock file

Apply all of the updates to dependencies.

* Ban duplicate tokio dependencies

Also ban git sources for tokio dependencies.

* Stop allowing sentry-tracing git repository in `deny.toml`

* Allow remaining duplicates after the tokio upgrade

* Use C: drive for CI build output on Windows

GitHub Actions uses a Windows image with two disk drives, and the
default D: drive is smaller than the C: drive. Zebra currently uses a
lot of space to build, so it has to use the C: drive to avoid CI build
failures because of insufficient space.

Co-authored-by: teor <teor@riseup.net>
2021-11-02 18:46:57 +00:00
Pili Guerra 989ba51cb9
Update release-checklist.md (#3001)
Include potential changes to the README in the release checklist

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-11-02 15:36:47 +00:00
teor 9963471b7c
Check for duplicate crate dependencies in CI (#2986)
* Add default deny.toml for "cargo deny check bans"

`cargo deny init`

* Delete unused "cargo deny" config entries

Also cleanup trailing whitespace.

* Deny duplicate crates and unexpected crate sources

Allow the current set of duplicates and sources,
with references to the tickets that will fix them.

* Check for duplicate dependencies in CI

Also check for:
- unexpected crate sources
- outdated Cargo.lock
  (required for accurate duplicate and source checks)

* Revert CI name changes so required statuses pass

* Fix ticket for sentry-tracing
2021-11-01 21:19:22 +00:00
Deirdre Connolly 0381c2347b
ZIP-401: weighted random mempool eviction (#2889)
* ZIP-401 weighted random mempool eviction

* rename zcash.mempool.total_cost.bytes to zcash.mempool.cost.bytes

Co-authored-by: teor <teor@riseup.net>

* Remove duplicated lines

* Add cost() method to UnminedTx

Update serialization failure messages

* More docs quoting ZIP-401 rules

* Change mempool::Storage::new() to handle Copy-less HashMap, HashSet

* mempool: tidy cost types and evict_one()

* More consensus rule docs

* Refactor calculating mempool costs for Unmined transactions

* Add a note on asympotic performance of calculating weights of txs in mempool

* Bump test mempool / storage config to avoid weighted random cost limits

* Use mempool tx_cost_limit = u64::MAX for some tests

* Remove failing tests for now

* Allow(clippy::field-reassign-with-default) because of a move on a type that doesn't impl Copy

* Fix mistaken doctest formatting

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Increase test timeout for Windows builds

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-27 00:21:19 +00:00
Deirdre Connolly 52357f7522
Remove dconnolly as default assignee and reviewer on dependabot PRs, and make them low priority (#2951)
* Remove dconnolly as default assignee and reviewer on dependabot PRs

* Assign dependabot PRs low priority by default

Co-authored-by: teor <teor@riseup.net>
2021-10-26 00:22:51 +00:00
teor 35290db052
Only run fake activation heights tests on zebra-state (#2929)
Also only run the zebrad acceptance tests on macOS.

Re-running the compiler and test binaries for unused crates is slow in CI.

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-10-21 23:51:20 +00:00
Marek ffa6fed7b3
Improve the template for release versioning (#2906)
* Specify the location of the user agent string

* Specify the `cargo install` command

* Fix a typo

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-10-20 18:02:07 +00:00
dependabot[bot] 0b25257bef
Bump actions/checkout from 2.3.4 to 2.3.5 (#2898)
Bumps [actions/checkout](https://github.com/actions/checkout) from 2.3.4 to 2.3.5.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2.3.4...v2.3.5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
2021-10-19 04:03:10 +00:00
dependabot[bot] d307d43e19 Bump codecov/codecov-action from 2.0.3 to 2.1.0
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 2.0.3 to 2.1.0.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v2.0.3...v2.1.0)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-13 15:14:19 -04:00