* Initialize the rayon threadpool with a new config for CPU-bound threads
* Verify proofs and signatures on the rayon thread pool
* Only spawn one concurrent batch per verifier, for now
* Allow tower-batch to queue multiple batches
* Fix up a potentially incorrect comment
* Rename some variables for concurrent batches
* Spawn multiple batches concurrently, without any limits
* Simplify batch worker loop using OptionFuture
* Clear pending batches once they finish
* Stop accepting new items when we're at the concurrent batch limit
* Fail queued requests on drop
* Move pending_items and the batch timer into the worker struct
* Add worker fields to batch trace logs
* Run docker tests on PR series
* During full verification, process 20 blocks concurrently
* Remove an outdated comment about yielding to other tasks
* Use a new channel for each batch
* Prefer the batch timer if there are also new batch requests
* Allow other tasks to run after each batch
* Label each batch worker with the verifier's type
* Rename Handle to ErrorHandle, and fix up some docs
* Check batch worker tasks for panics and task termination
* Use tokio's PollSemaphore instead of an outdated Semaphore impl
* Run all verifier cryptography on a blocking thread
Also use a new verifier channel for each batch.
* Make flush and drop behaviour consistent for all verifiers
* Partly fix an incorrect NU5 test
* Switch batch tests to the multi-threaded runtime
* Export all verifier primitive modules from zebra-consensus
* Remove outdated test code in tower-batch
* Use a watch channel to send batch verifier results
* Use spawn_blocking for batch fallback verifiers
* Spawn cryptography batches onto blocking tokio threads
* Use smaller batches for halo2
* Minor tower-batch cleanups
* Fix doc link in zebra-test
* Drop previous permit before acquiring another to avoid a deadlock edge case
* Wrap Sprout note commitment trees into `Arc`
* Remove a redundant comment
* Rephrase a comment about chain forking
* Remove a redundant comment
The comment is not valid because Zebra uses `bridgetree::Frontier`s from
the `incrementalmerkletree` crate to represent its note commitment
trees. This `struct` does not support popping elements from the tree.
* Wrap Sapling commitment trees into `Arc`
* Remove unnecessary `as_ref`s
* Wrap Orchard commitment trees into `Arc`
* Make the release checklist shorter and hide some details
* Ignore any `fastmod` updates to previous release notes in `CHANGELOG.md`
* Use recent versions in examples
* Fix markdown that doesn't render correctly
* Fix some weird line breaks
* Use capital letters to start list items
* Clarify `fastmod` and `CHANGELOG.md`
* Clarify version format by changing highlighting
* Return the maximum checkpoint height from the chain verifier
* Return the verified block height from the sync downloader
* Track the verified height in the syncer
* Use a lower concurrency limit during full verification
* Get the tip from the state before the first verified block
* Limit the number of submitted download and verify blocks in a batch
* Adjust lookahead limits when transitioning to full verification
* Keep unused extra hashes and submit them to the downloader later
* Remove redundant verified_height and state_tip()
* Split the checkpoint and full verify concurrency configs
* Decrease full verification concurrency to 5 blocks
10 concurrent blocks causes 3 minute stalls on some blocks on my machine.
(And it has about 4x as many cores as a standard machine.)
* cargo +stable fmt --all
* Remove a log that's verbose with smaller lookahead limits
* Apply the full verify concurrency limit to the inbound service
* Add a summary of the config changes to the CHANGELOG
* Increase the default full verify concurrency limit to 30
* Only verify halo2 proofs once per transaction (rather than once per action)
* Update comments on how there is one aggregate Halo2 proof instead of one per Action
Co-authored-by: Marek <mail@marek.onl>
* cargo +stable fmt --all
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Marek <mail@marek.onl>
* Rename `block_by_height` to `block_header_by_height` in fin state
* Rename `tx_by_hash` to `tx_loc_by_hash` in both (non & fin) states
* Rename `utxo_by_outpoint` to `utxo_by_out_loc` in finalized state
* Reorder the column families so that they match the docs
* Update `struct Chain` in the RFCs
* Increment `DATABASE_FORMAT_VERSION` to 25
* Remove obsolete docs from `0004-asynchronous-script-verification.md`
* Remove an obsolete `TODO` from `disk_db.rs`
* Delete unused snapshots
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Checkout zebra in each job to avoid warnings
But put TODOs where we might be able to skip checkouts
* Split log following into sprout checkpoints, sapling/orchard checkpoints, and full validation
* Make job IDs shorter
* Use /dev/stderr because docker doesn't have a tty
* remove pipefail
* Revert "remove pipefail"
This reverts commit a7ee37bebdc107a4215e7dd307b189d925969234.
* Make tee ignore errors writing to a grep pipe
* Avoid launching multiple docker instances for duplicate jobs
* Ignore broken pipe error messages and statuses
* fix(ci): docker wait not finding container
We had this issue before, I can't recall if this was a parsing error between GitHub Actions and gcloud `--command` parsing, but we had to change this into two pieces.
This implementation keeps it how we did it before 9b9578c999/.github/workflows/test.yml (L235-L243)
* docs: remove pending TODO
We can't remove `actions/checkout` nor set `create_credentials_file` to `false` as next steps won't be able to authenticate to GCP.
We can surely remove `actions/checkout` and leave `create_credentials_file` as `true`, but this will raise a warning on each step, and there's no benefit of doing so.
* Show `docker wait` and `gcloud ssh` output
* If `docker wait` fails, get the exit code using `docker inspect`
* Make full sync tests go all the way to the tip
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Checkout zebra in each job to avoid warnings
But put TODOs where we might be able to skip checkouts
* Split log following into sprout checkpoints, sapling/orchard checkpoints, and full validation
* Make job IDs shorter
* Use /dev/stderr because docker doesn't have a tty
* remove pipefail
* Revert "remove pipefail"
This reverts commit a7ee37bebdc107a4215e7dd307b189d925969234.
* Make tee ignore errors writing to a grep pipe
* Avoid launching multiple docker instances for duplicate jobs
* Ignore broken pipe error messages and statuses
* fix(ci): docker wait not finding container
We had this issue before, I can't recall if this was a parsing error between GitHub Actions and gcloud `--command` parsing, but we had to change this into two pieces.
This implementation keeps it how we did it before 9b9578c999/.github/workflows/test.yml (L235-L243)
* docs: remove pending TODO
We can't remove `actions/checkout` nor set `create_credentials_file` to `false` as next steps won't be able to authenticate to GCP.
We can surely remove `actions/checkout` and leave `create_credentials_file` as `true`, but this will raise a warning on each step, and there's no benefit of doing so.
* Show `docker wait` and `gcloud ssh` output
* If `docker wait` fails, get the exit code using `docker inspect`
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* add limits to rejection message and reason
* truncate bytes instead of characters
* change `MAX_REJECT_REASON_LENGTH` to match zcashd
* clippy
* avoid panix in the slice
* Escape and truncate error messages
Co-authored-by: teor <teor@riseup.net>
* Update Zebra to 1.0.0-beta.12
* Simplify tower-* versioning by updating both to 0.2.27
* Simplify version search and replace by removing an old version from the docs
* Update Cargo.lock
* Add draft changelog as of PR #4693
* Update CHANGELOG to PR #4680
* Document configs that need compile-time features
* Document features in zebrad/src/lib.rs
* Link to the feature list from the README
* Remove some outdated README info
* Move some detailed README info to the `zebrad` crate docs
* Improve time logging using humantime
* Only log full seconds, ignore the fractional part
* Move humantime_seconds to tracing::fmt
* Move the progress task to its own module
* Add missing humantime dependency
* Log the network upgrade in progress logs
* Log when Zebra verifies the final checkpoint
* Warn the user when Zebra cannot parse `zebrad.toml`
* Test that Zebra warns the user when it cannot parse `zebrad.toml`
* Fix up a mistaken merge change
* Suggest how to fix `zebrad.toml` when Zebra cannot parse it
Co-authored-by: teor <teor@riseup.net>
* Put arguments to "docker run" on different lines
And update some comments.
* Split docker run into launch, logs, and wait
* Remove mistaken "needs state" condition on log and results job
* Exit the ssh and the job with the container test's exit status
* change `initial_mainnet_peers` and `initial_testnet_peers` type to `IndexSet`
* add tests for zebra config files
* add serde feature to indexmap
* remove async
* update config
* fix `stored_config_path()`
* skip tests if config is not found
* improve error
* use CARGO_MANIFEST_DIR
* remove `stored_config_is_newest` test
* move `stored_config_works` test to the end of `valid_generated_config_test`
* space
* Split full sync into checkpoint and full validation
* Sort workflow variables into categories and add descriptions
* Split Create instance/volume and Run test into separate jobs
* Copy initial conditions to all jobs in the series
* Actually create a cached state image
* fix(state): use same disk naming convention for all test instances
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Decrease the default lookahead limit to 400
* Increase the block verification timeout to 10 minutes
* Halve the default concurrent downloads config
* Try to run the spawned download task before queueing the next download
* Allow verification to be cancelled if the verifier is busy
* Add an identifiable suffix to zcash-rpc-diff temp directories
* Explain why we need a fallback temp command
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feat(ci): build each crate individually
* fix(ci): use valid names for each job
* feat(ci): builds and checks with and without all features
* refactor(ci): build job matrix dinamically
* fix: use a JSON_CRATES variable with resulting values
* test: check-matrix
* fix(ci): use "crate" in singular for reference
* imp(ci): use a matrix for feature build arguments
* fix(ci): use correct naming and includes
* fix(ci): implement most recommendations given in review
* fix(ci): use simpler shell script
* fix: typo
* fix: add string to file, not cmd
* fix: some shellchecks
* fix(ci): remove warnings and errors from shellcheck
* imp(ci): add patch file for `Build crates individually` workflow
* Remove unused configs in patch job
Co-authored-by: teor <teor@riseup.net>
* delete old database directories
* check if state directory exists
* skip deleting when ephemeral
* split `check_and_delete_old_databases`
* move `check_and_delete_old_databases` to state
* spawn `check_and_delete_old_databases`
* simplity a bit
* fix(state): only delete old database directories inside the cache directory (#4631)
* Add function comments, tweak log
* Simplify version parsing
* Use spawn_blocking to launch the task on a separate thread, do the cleanup last
* Abort the cleanup task when Zebra exits
* Split directory deletion into its own function, handle ownership
* Rename cache_dir to state_dir
* If an outdated state directory is outside the cache directory, don't delete it
* Minimise diffs
* add test
* fix typos
Co-authored-by: teor <teor@riseup.net>
* add `canonicalize` to test regex
* add another match to test
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
teor
Marek
dependabot[bot]
Andrew Arnott
Alfredo Garcia
Conrado Gouvea
Gustavo Valverde
Deirdre Connolly