* Return the maximum checkpoint height from the chain verifier
* Return the verified block height from the sync downloader
* Track the verified height in the syncer
* Use a lower concurrency limit during full verification
* Get the tip from the state before the first verified block
* Limit the number of submitted download and verify blocks in a batch
* Adjust lookahead limits when transitioning to full verification
* Keep unused extra hashes and submit them to the downloader later
* Remove redundant verified_height and state_tip()
* Split the checkpoint and full verify concurrency configs
* Decrease full verification concurrency to 5 blocks
10 concurrent blocks causes 3 minute stalls on some blocks on my machine.
(And it has about 4x as many cores as a standard machine.)
* cargo +stable fmt --all
* Remove a log that's verbose with smaller lookahead limits
* Apply the full verify concurrency limit to the inbound service
* Add a summary of the config changes to the CHANGELOG
* Increase the default full verify concurrency limit to 30
* Only verify halo2 proofs once per transaction (rather than once per action)
* Update comments on how there is one aggregate Halo2 proof instead of one per Action
Co-authored-by: Marek <mail@marek.onl>
* cargo +stable fmt --all
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Marek <mail@marek.onl>
* Update Zebra to 1.0.0-beta.12
* Simplify tower-* versioning by updating both to 0.2.27
* Simplify version search and replace by removing an old version from the docs
* Update Cargo.lock
* Add draft changelog as of PR #4693
* Update CHANGELOG to PR #4680
* Improve time logging using humantime
* Only log full seconds, ignore the fractional part
* Move humantime_seconds to tracing::fmt
* Move the progress task to its own module
* Add missing humantime dependency
* Log the network upgrade in progress logs
* Log when Zebra verifies the final checkpoint
* Always activate tokio/tracing feature
And always build tests with all tokio features.
* Refactor tracing-subscriber init to simplify it
* Add the tokio-console feature and dependencies
* Add optional tokio-console support, and log the installed tracing layers at info level
Uses a tracing Registry for tokio-console, and a fmt::Subscriber otherwise.
* Add some TODOs based on tracing-subscriber features
* Fix up some spans
* Add a TODO for fixing a log filter bug in tokio-console mode
* Fix the syntax of links in comments
* Fix a mistake in the docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove unnecessary angle brackets from a link
* Revert the changes for links that serve as references
* Revert "Revert the changes for links that serve as references"
This reverts commit 8b091aa9fa.
* Remove `<` `>` from links that serve as references
This reverts commit 046ef25620.
* Don't use `<` `>` in normal comments
* Don't use `<` `>` for normal comments
* Revert changes for comments starting with `//`
* Fix some warnings produced by `cargo doc`
* Fix some rustdoc warnings
* Fix some warnings
* Refactor some changes
* Fix some rustdoc warnings
* Fix some rustdoc warnings
* Resolve various TODOs
Co-authored-by: teor <teor@riseup.net>
* Fix some unresolved links
* Allow links to private items
* Fix some unresolved links
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Fix the syntax of links in comments
* Fix a mistake in the docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove unnecessary angle brackets from a link
* Revert the changes for links that serve as references
* Revert "Revert the changes for links that serve as references"
This reverts commit 8b091aa9fa.
* Remove `<` `>` from links that serve as references
This reverts commit 046ef25620.
* Don't use `<` `>` in normal comments
* Don't use `<` `>` for normal comments
* Revert changes for comments starting with `//`
* Fix some warnings produced by `cargo doc`
* Fix some rustdoc warnings
* Fix some warnings
* Refactor some changes
* Fix some rustdoc warnings
* Fix some rustdoc warnings
* Resolve various TODOs
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Bump crate versions
* Increment the protocol user agent
* Increment the version in `README.md`
* Increment the version in `install.md`
* Update `README.md`
* Update the changelog
* Increment the versions of crates that depend on other crates
* Update `Cargo.lock`
* Upgrade tracing and related dependencies
```sh
cargo upgrade --workspace
tracing-error
tracing-subscrber
color-eyre
tracing-flame
tracing-journald
sentry
sentry-tracing
metrics
metrics-exporter-prometheus
reqwest
```
* Update duplicate dependency checks
* Enable the tracing/env-filter feature
* Fix type inference for metrics
Manual changes, plus:
```sh
fastmod "as _" "as f64"
```
* Tidy up some unrelated test code
* Update metrics-exporter-prometheus API
And make unused dependencies optional.
* Adjust test regexes to new tracing format
Also fix some regex bugs, and refactor to simplify.
* Disable color-eyre span traces and track caller in release builds
* Add a feature that enables extra debugging in release builds
* Clean up some redundant features
* Increase a test timeout
* Fix the syntax of links in comments
* Fix a mistake in the docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove unnecessary angle brackets from a link
* Revert the changes for links that serve as references
* Revert "Revert the changes for links that serve as references"
This reverts commit 8b091aa9fa.
* Remove `<` `>` from links that serve as references
This reverts commit 046ef25620.
* Don't use `<` `>` in normal comments
* Don't use `<` `>` for normal comments
* Revert changes for comments starting with `//`
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* bump versions
* add the changelog for beta 10
* Update CHANGELOG.md
Co-authored-by: Marek <mail@marek.onl>
* Distinguish CI and Rust dependency updates
* Include all the dep bumps in PR #4405
* join 2 dependencies
* Include incrementalmerkletree dep bump in changelog
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Marek <mail@marek.onl>
* change: set NU5 mainnet activation height and minimum protocol version
* fix tests
* test if V5 coinbase expiry height is not bound by non-coinbase limit
* update get_blockchain_info snapshot for NU5
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* upgrade dependencies to NU5 mainnet versions
* migrate to bitvec 1.0
* update deny.toml
* point to published redjubjub 0.5
* Point at zcash-script@0.1.6 on crates.io
* update Cargo.lock
* specify patch versions of dependencies
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Increase UTXO lookup timeout
Avoid block validation failures because UTXOs aren't available on time.
* Increase the block verification timeout
Attempt to reduce the synchronization restarts and consequently improve
performance.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Update changelog for 1.0.0-beta.8
* Bump Zebra crate versions
The tower crates and their dependencies haven't changed,
so their versions don't need to be bumped.
Some Zebra crates haven't changed,
but I bumped all the versions to keep them consistent.
* Explicitly say that we support NU5 testnet 2
* Expand PR #3799 into changelog entries
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* update librustzcash; adapt to new API
* add ticket reference for removing zcash_proofs duplicated dependencies
* update to new zcash_script V5 API
* use zp_tx shorthand
* update to Zcash 4.7.0 dependencies
* update protocol versions
* feat(rpc): Implement `getblockchaininfo` RPC method (#3891)
* Implement `getblockchaininfo` RPC method
* add a test for `get_blockchain_info`
* fix tohex/fromhex
* move comment
* Update lightwalletd acceptance test for getblockchaininfo RPC (#3914)
* change(rpc): Return getblockchaininfo network upgrades in height order (#3915)
* Update lightwalletd acceptance test for getblockchaininfo RPC
* Update some doc comments for network upgrades
* List network upgrades in order in the getblockchaininfo RPC
Also:
- Use a constant for the "missing consensus branch ID" RPC value
- Simplify fetching consensus branch IDs
- Make RPC type derives consistent
- Update RPC type documentation
* Make RPC type derives consistent
* Fix a confusing test comment
* get hashand height at the same time
* fix estimated_height
* fix lint
* add extra check
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix typo
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* split test
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix(rpc): ignore an expected error in the RPC acceptance tests (#3961)
* Add ignored regexes to test command failure regex methods
* Ignore empty chain error in getblockchaininfo
We expect this error when zebrad starts up with an empty state.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make sync error logs more user-friendly (#3944)
- use info level, there is nothing the user needs to do,
particularly for a single error
- explain that the errors are temporary
- hide backtraces, because they look like crashes
* Update test.patch.yml with lightwalletd job (#3970)
* Update test.patch.yml with lightwalletd job
* Remove a workflow condition that will always be false
In general, patch workflows need the
opposite conditions to the original workflow.
But in this case, we know the result of the
condition will always be true, so we can just delete it.
Co-authored-by: teor <teor@riseup.net>
* fix(doc): Fix bugs in the lightwalletd database design (#3964)
* Re-order column families in design in dependency order
* Minor RFC design tweaks and fixes
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Repoint zebra image links to our new zfnd.org site for now (#3949)
* Repoint zebra image links to our new zfnd.org site for now
* Remove images/
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Fix typos (#3956)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* bump database version to trigger testnet rollback
* reduce minimum protocol version for now (will be changed later)
* update dependencies
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* update versions to match zcash 4.7.0
* deny.toml: update 'darling'
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Dimitris Apostolou <dimitris.apostolou@icloud.com>
* Add Utxo constructors from output locations
* Store transparent outputs rather than Utxo structs
* Update raw data snapshots
* Increment the state version
* Update changelog for v1.0.0-beta.7
* Increment all crate versions
* Remove redundant release test that is now covered by CI
* Remove completed NU5 README check task from the release template
* Add Merge Freeze tool to the release checklist
* Simplify release checklist by removing unused steps
* Make proptest-impl features consistently depend on each other
* Use a non-test method for consensus checks
* Make proptest dependencies the same between crates
* Repoint zebra image links to our new zfnd.org site for now
* Remove images/
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* refactor(test/block): rename large single transaction function
```sh
fastmod single_transaction_block single_transaction_block_many_inputs
```
* rustfmt
* test(block): add a test block with many transparent outputs
* doc(db): explain why we can't just get the UTXOs right before they are deleted
* refactor(db): split out a block data write method
* refactor(block): add a height argument to new_outputs
* test(db): add block and transaction round-trip tests
Including large blocks and transactions.
* test(db): fix large block serialization instability in the tests
* doc(block): add TODOs for generating correct blocks
* Make transparent output functions which take a height test-only
* make sure generated blocks are actually over/under-sized
* replace println!() with an error!() log
* Enable `checkpoint_sync` by default
Provide fast synchronization by default.
* Add newline to separate fields
Make it slighly easier to read.
* Update `checkpoint_sync` documentation
Change the documentation to match the new default value, and explain
that changing the value can be used for debugging.
* Improve documentation
Remove `post-Canopy`, because Zebra will likely change checkpoint
heights in the future. Also explain better what the checkpoints are and
why it helps with debugging.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* Upgrade some dependencies
* Upgrade some dependencies
* Upgrade dependencies for zebrad
* Upgrade tracing dependencies
* Revert `tor` & `arti`
* Upgrade `criterion` & `pin-project` in `deny.toml`
* Remove some dependencies from `skip-tree` in `deny.toml`
* Revert some the versions of dependencies because of duplicates
* Revert proptest regressions
* Upgrade dependencies, then ignore some more duplicates (#3716)
* feat(actions)!: add full sync test (#3582)
* add(tests): full sync test
* fix(test): add build
* fix(deploy): escape double dashes '--' correctly
* fix(test): remove unexpected --no-capture arg
error: Found argument '--nocapture' which wasn't expected, or isn't valid in this context
* refactor(docker): use default executable as entrypoint
* refactor(startup): add a custom entrypoint
* fix(test): add missing TEST_FULL_SYNC variable
* test(timeout): use the biggest machine
* fix
* fix(deploy): use latest successful image
* typo
* refactor(docker): generate config file at startup
* revert(build): changes were made to docker
* fix(docker): send variables correctly to the entrypoint
* test different conf file approach
* fix(env): add RUN_TEST env variable
* ref: use previous approach
* fix(color): use environment variable
* fix(resources): use our normal machine size
* fix(ci): double CPU and RAM for full sync test
* fix(test): check for zebrad test output in the correct order
The mempool is only activated once, so we must check for that log first.
After mempool activation, the stop regex is logged at least once.
(It might be logged before as well, but we can't rely on that.)
When checking that the mempool didn't activate,
wait for the `zebrad` command to exit,
then check the entire log.
* fix(ci): run full sync test with full compiler optimisations
* fix(tests): reintroduce tests and run full sync on approval
* fix(tests): reduce the changelog
Co-authored-by: teor <teor@riseup.net>
* fix(ci): update CI job path triggers (#3692)
* ci(test): re-run tests when snapshot data changes
* fix(ci): rebuild state when disk format changes
* fix(ci): rebuild rust docs when code or dependencies change
* doc(ci): explain why we run jobs when files change
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(build): use the right multistage target (#3700)
* fix(review): only assign one reviewer to general Rust reviews (#3708)
If we assign two teams, GitHub assigns two reviewers.
* fix(ci): change the color-eyre ignore to a tracing-subscriber ignore
* fix(ci): ignore duplicate darling dependencies
* doc(ci): remove an alternative resolution doc
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* doc(utils): remove trailing spaces
* fix(consensus): update Zebra's hard-coded checkpoint lists
This updates the checkpoint lists as at 2022-02-22:
- mainnet: to the latest finalized tip
- testnet: to the latest Canopy checkpoint,
because we expect a NU5 testnet rollback
* doc(consensus): Add example commands to update the checkpoints
* docs: document consensus rules from 4.6 Action Descriptions
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Marek <mail@marek.onl>
* feat(log): log current height when logging sync progress
* fix(test): log the specific error when full sync tests fail
* doc(start): remove an obsolete TODO
We can't decrease this log level, because the tests rely on it.
* fix(test): wait until mempool activates in full sync tests
Changes the log message and log test so that the test only finishes
when the mempool has activated.
There is still a race condition here, between the log timer and
mempool activation. But it should be very rare, because the mempool
is activated immediately when `is_close_to_tip()` becomes true.
* fix(test): warn when Zebra stalls below the maximum checkpoint height
This also improves the full sync tests,
because the warning is checked before logging a successful sync.
* feat(log): warn when sync stalls downloading the genesis block
* fix(test): warn when the state hasn't committed a block for a long time
This also improves the full sync tests,
because the warning is checked before logging a successful sync.
* doc(test): update some sync acceptance test comments
* fix(log): use Display formatting to log chrono::Duration
Debug formatting is complicated and hard to read.
* fix(log): stop saying that we've activated the mempool without checking it
We're not checking if the mempool is active, so we can't say that.
* fix(log): minor tidying and TODOs
* fix(doc): fix a typo in the tests
* fix(log): explain the post-checkpoint blocks in progress warning calculations
* fix(doc): explain what could happen if we don't wait for extra blocks
* fix(log): add a percent symbol to a percent log
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(clippy): for loop with only one item
* fix(clippy): manual Range::contains
Also clarified the surrounding code because it was unclear.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
teor
Alfredo Garcia
Conrado Gouvea
dependabot[bot]
Marek
Janito Vaqueiro Ferreira Filho
Deirdre Connolly