* bump versions
* add the changelog for beta 10
* Update CHANGELOG.md
Co-authored-by: Marek <mail@marek.onl>
* Distinguish CI and Rust dependency updates
* Include all the dep bumps in PR #4405
* join 2 dependencies
* Include incrementalmerkletree dep bump in changelog
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Marek <mail@marek.onl>
* upgrade dependencies to NU5 mainnet versions
* migrate to bitvec 1.0
* update deny.toml
* point to published redjubjub 0.5
* Point at zcash-script@0.1.6 on crates.io
* update Cargo.lock
* specify patch versions of dependencies
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Impl the elementary structure of the `z_gettreestate` RPC
* Fix merging bugs
* Fix a merge bug
* Fix a merge bug
* Move a derive attribute
Co-authored-by: teor <teor@riseup.net>
* Clarify the support of negative heights
* Add Orchard note commitment trees to the response
* Add the time to the response
* Finalize the `z_gettreestate` RPC
* Add a note that verified blocks have coinbase height
* Refactor `from_str` for `HashOrHeight`
* Fix a mistake in the docs
Co-authored-by: teor <teor@riseup.net>
* Clarify request types
Co-authored-by: teor <teor@riseup.net>
* Simplify `hash_or_height` conversion to height
Co-authored-by: teor <teor@riseup.net>
* Add a TODO about optimization
Co-authored-by: teor <teor@riseup.net>
* Add a doc comment
* Make sure Sapling & Orchard trees don't get mixed up
* Serialize Sapling commitment trees
* Refactor some comments
* Serialize Orchard commitment trees
* Serialize block heights
* Simplify the serialization of commitment trees
* Remove the block time from the RPC response
* Simplify the serialization of block heights
* Put Sapling & Orchard requests together
* Remove a redundant TODO
* Add block times to the RPC response
* Derive `Clone, Debug, Eq, PartialEq` for `GetTreestate`
Co-authored-by: teor <teor@riseup.net>
* Derive `Clone`, `Debug`, `Eq` and `PartialEq` for `SerializedTree`
* Document the fields of `GetTreestate`
* Skip the serialization of empty trees
This ensures compatibility with `zcashd` in the `z_gettreestate` RPC.
* Document the `impl` of `merkle_tree::Hashable` for nodes
* Make the structure of the JSON response consistent with `zcashd`
* Derive `Eq` for nodes
Co-authored-by: teor <teor@riseup.net>
* Convert Sapling commitment trees to a format compatible with zcashd
* Refactor the conversion of Sapling commitment trees
* Refactor some comments
* Refactor comments
* Add a description of the conversion
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Fix comment indenting
* Document the conversion between the dense and sparse formats
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Move `init_tracing` to `lib.rs`
Allow it to be reused by other binaries.
* Fix `hex` dependency inclusion in `zebra-chain`
It would previously fail because `hex` was being included without the
`serde` feature, even though we required that for `transparent::Script`.
* Implement `FromStr` for `Network`
Make it easy to receive it as a command-line parameter.
* Add `zebra-tip-height` utility
Obtains the chain tip height of a directory containing Zebra state.
* Remove Tokio dependency from `zebra-utils`
It wasn't actually used by the `zebra-tip-height` utility.
* Remove `BoxStateService` type alias
It's not needed if we don't return the unused state service. This also
allows removing the `tower` dependency.
* Remove unnecessary attribute
Leftover from copied code.
* Make `cache_dir` argument optional
Fallback to the default Zebra state cache directory.
* Remove added newline
Minor formatting fix, to avoid adding an unnecessary newline.
* Move `tip-height` command into `zebrad`
Make it available as a sub-command in `zebrad`.
* Make some zebrad sub-commands only log warnings
Co-authored-by: teor <teor@riseup.net>
* Update changelog for 1.0.0-beta.8
* Bump Zebra crate versions
The tower crates and their dependencies haven't changed,
so their versions don't need to be bumped.
Some Zebra crates haven't changed,
but I bumped all the versions to keep them consistent.
* Explicitly say that we support NU5 testnet 2
* Expand PR #3799 into changelog entries
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* update librustzcash; adapt to new API
* add ticket reference for removing zcash_proofs duplicated dependencies
* update to new zcash_script V5 API
* use zp_tx shorthand
* update to Zcash 4.7.0 dependencies
* update protocol versions
* feat(rpc): Implement `getblockchaininfo` RPC method (#3891)
* Implement `getblockchaininfo` RPC method
* add a test for `get_blockchain_info`
* fix tohex/fromhex
* move comment
* Update lightwalletd acceptance test for getblockchaininfo RPC (#3914)
* change(rpc): Return getblockchaininfo network upgrades in height order (#3915)
* Update lightwalletd acceptance test for getblockchaininfo RPC
* Update some doc comments for network upgrades
* List network upgrades in order in the getblockchaininfo RPC
Also:
- Use a constant for the "missing consensus branch ID" RPC value
- Simplify fetching consensus branch IDs
- Make RPC type derives consistent
- Update RPC type documentation
* Make RPC type derives consistent
* Fix a confusing test comment
* get hashand height at the same time
* fix estimated_height
* fix lint
* add extra check
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix typo
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* split test
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix(rpc): ignore an expected error in the RPC acceptance tests (#3961)
* Add ignored regexes to test command failure regex methods
* Ignore empty chain error in getblockchaininfo
We expect this error when zebrad starts up with an empty state.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make sync error logs more user-friendly (#3944)
- use info level, there is nothing the user needs to do,
particularly for a single error
- explain that the errors are temporary
- hide backtraces, because they look like crashes
* Update test.patch.yml with lightwalletd job (#3970)
* Update test.patch.yml with lightwalletd job
* Remove a workflow condition that will always be false
In general, patch workflows need the
opposite conditions to the original workflow.
But in this case, we know the result of the
condition will always be true, so we can just delete it.
Co-authored-by: teor <teor@riseup.net>
* fix(doc): Fix bugs in the lightwalletd database design (#3964)
* Re-order column families in design in dependency order
* Minor RFC design tweaks and fixes
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Repoint zebra image links to our new zfnd.org site for now (#3949)
* Repoint zebra image links to our new zfnd.org site for now
* Remove images/
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Fix typos (#3956)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* bump database version to trigger testnet rollback
* reduce minimum protocol version for now (will be changed later)
* update dependencies
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* update versions to match zcash 4.7.0
* deny.toml: update 'darling'
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Dimitris Apostolou <dimitris.apostolou@icloud.com>
* Add test-only serialization, and make existing serialization test-only
* Make AddressLocations clearer in the API
* Add UnspentOutputAddressLocation
* Add the AddressLocation to the UTXO database value
* Update the snapshot test code for UnspentOutputAddressLocation
* Update the raw data snapshots
* Update the high-level data snapshots
* Increment the database version
* Make serialization clearer
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Fix code formatting
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Add an empty utxo_by_transparent_addr_loc column family
* Update snapshot data for the new column family
* Add an AddressUnspentOutputs type
* Add round-trip tests for AddressUnspentOutputs
* Move address balances into their own method
* Simplify updating address balances
* Fix utxo_by_out_loc column family name
* Implement reads and writes of address UTXOs
* Update raw data snapshots
* Update the snapshot tests for high-level address UTXOs
* Assert rather than taking empty address snapshots for genesis
* Update high-level address UTXO snapshot data, and delete empty snapshots
* Increment the database version
* Use typed values for all ReadDisk methods
* Implement test-only serialization for transparent::Address
* Implement FromDisk for ()
* Store AddressUnspentOutput as the column family key
* Update round-trip serialization tests for AddressUnspentOutput
* Update snapshot test code, and add a UTXO data snapshot
* Update existing snapshot data
* Add new UTXO snapshot data
* Update column family name
```sh
fastmod utxo_by_transparent_addr_loc utxo_loc_by_transparent_addr_loc zebra*
```
* cargo fmt --all
* cargo insta test --review --delete-unreferenced-snapshots
* Explain why it is ok to use invalid database iterator indexes
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add explanations of UTXO database updates
* Simplify an assertion
* Remove UnspentOutputAddressLocation and just store transparent::Output
* Update snapshot test data
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Update changelog for v1.0.0-beta.7
* Increment all crate versions
* Remove redundant release test that is now covered by CI
* Remove completed NU5 README check task from the release template
* Add Merge Freeze tool to the release checklist
* Simplify release checklist by removing unused steps
* Add round-trip test for OutputLocation disk serialization
* Make the transaction snapshot tests more accurate
Previously, we were showing the genesis transaction hash at height 0, index 0.
But its hash is actually not stored by location in the database,
because the genesis transaction is skipped due to a consensus rule.
* Update the transaction snapshot data
* Add history tree snapshot tests
At the current test heights, the history trees are all empty.
* Add the history tree snapshot data
* Update comments
* Simplify snapshot test code
* Make some serde::Serialize impls test-only, so we can hex-encode them
This should also speed up release compilation a bit.
* Add snapshot test code for UTXOs
* Add snapshot data for UTXOs
* Upgrade some dependencies
* Upgrade some dependencies
* Upgrade dependencies for zebrad
* Upgrade tracing dependencies
* Revert `tor` & `arti`
* Upgrade `criterion` & `pin-project` in `deny.toml`
* Remove some dependencies from `skip-tree` in `deny.toml`
* Revert some the versions of dependencies because of duplicates
* Revert proptest regressions
* Upgrade dependencies, then ignore some more duplicates (#3716)
* feat(actions)!: add full sync test (#3582)
* add(tests): full sync test
* fix(test): add build
* fix(deploy): escape double dashes '--' correctly
* fix(test): remove unexpected --no-capture arg
error: Found argument '--nocapture' which wasn't expected, or isn't valid in this context
* refactor(docker): use default executable as entrypoint
* refactor(startup): add a custom entrypoint
* fix(test): add missing TEST_FULL_SYNC variable
* test(timeout): use the biggest machine
* fix
* fix(deploy): use latest successful image
* typo
* refactor(docker): generate config file at startup
* revert(build): changes were made to docker
* fix(docker): send variables correctly to the entrypoint
* test different conf file approach
* fix(env): add RUN_TEST env variable
* ref: use previous approach
* fix(color): use environment variable
* fix(resources): use our normal machine size
* fix(ci): double CPU and RAM for full sync test
* fix(test): check for zebrad test output in the correct order
The mempool is only activated once, so we must check for that log first.
After mempool activation, the stop regex is logged at least once.
(It might be logged before as well, but we can't rely on that.)
When checking that the mempool didn't activate,
wait for the `zebrad` command to exit,
then check the entire log.
* fix(ci): run full sync test with full compiler optimisations
* fix(tests): reintroduce tests and run full sync on approval
* fix(tests): reduce the changelog
Co-authored-by: teor <teor@riseup.net>
* fix(ci): update CI job path triggers (#3692)
* ci(test): re-run tests when snapshot data changes
* fix(ci): rebuild state when disk format changes
* fix(ci): rebuild rust docs when code or dependencies change
* doc(ci): explain why we run jobs when files change
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(build): use the right multistage target (#3700)
* fix(review): only assign one reviewer to general Rust reviews (#3708)
If we assign two teams, GitHub assigns two reviewers.
* fix(ci): change the color-eyre ignore to a tracing-subscriber ignore
* fix(ci): ignore duplicate darling dependencies
* doc(ci): remove an alternative resolution doc
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(network): split synthetic NotFoundRegistry from message NotFoundResponse
* docs(network): Improve `notfound` message documentation
* refactor(network): Rename MustUseOneshotSender to MustUseClientResponseSender
```
fastmod MustUseOneshotSender MustUseClientResponseSender zebra*
```
* docs(network): fix a comment typo
* refactor(network): remove generics from MustUseClientResponseSender
* refactor(network): add an inventory collector to Client, but don't use it yet
* feat(network): register missing peer responses as missing inventory
We register this missing inventory based on peer responses,
or connection errors or timeouts.
Inbound message inventory tracking requires peers to send `notfound` messages.
But `zcashd` skips `notfound` for blocks, so we can't rely on peer messages.
This missing inventory tracking works regardless of peer `notfound` messages.
* refactor(network): rename ResponseStatus to InventoryResponse
```sh
fastmod ResponseStatus InventoryResponse zebra*
```
* refactor(network): rename InventoryStatus::inner() to to_inner()
* fix(network): remove a redundant runtime.enter() in a test
* doc(network): the exact time used to filter outbound peers doesn't matter
* fix(network): handle block requests slightly more efficiently
* doc(network): fix a typo
* fmt(network): `cargo fmt` after rename ResponseStatus to InventoryResponse
* doc(test): clarify some test comments
* test(network): test synthetic notfound from connection errors and peer inventory routing
* test(network): improve inbound test diagnostics
* feat(network): add a proptest-impl feature to zebra-network
* feat(network): add a test-only connect_isolated_with_inbound function
* test(network): allow a response on the isolated peer test connection
* test(network): fix failures in test synthetic notfound
* test(network): Simplify SharedPeerError test assertions
* test(network): test synthetic notfound from partially successful requests
* test(network): MissingInventoryCollector ignores local NotFoundRegistry errors
* fix(network): decrease the inventory rotation interval
This stops us waiting 3-4 sync resets (4 minutes) before we retry a missing block.
Now we wait 1-2 sync resets (2 minutes), which is still a reasonable rate limit.
This should speed up syncing near the tip, and on testnet.
* fmt(network): cargo fmt --all
* cleanup(network): remove unnecessary allow(dead_code)
* cleanup(network): stop importing the whole sync module into tests
* doc(network): clarify syncer inventory retry constraint
* doc(network): add a TODO for a fix to ensure API behaviour remains consistent
* doc(network): fix a function doc typo
* doc(network): clarify how we handle peers that don't send `notfound`
* docs(network): clarify a test comment
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Updating zebra-test to v1.0.0-beta.4
* Updating tower-fallback to v0.2.16
* Updating tower-batch to v0.2.20
* Updating zebra-chain to v1.0.0-beta.4
* Updating zebra-script to v1.0.0-beta.4
* Updating zebra-network to v1.0.0-beta.4
* Updating zebra-state to v1.0.0-beta.4
* Updating zebra-consensus to v1.0.0-beta.4
* Updating zebra-utils to v1.0.0-beta.4
* Updating zebrad to v1.0.0-beta.4
* Update Cargo.lock with updated crate versions
* Add section for 1.0.0-beta.4 in the CHANGELOG
* Update CHANGELOG.md
Co-authored-by: teor <teor@riseup.net>
* Update README.md
Co-authored-by: teor <teor@riseup.net>
* Update book/src/user/install.md
Co-authored-by: teor <teor@riseup.net>
* Update release description to mention Rust 2021
Update the changelog to mention that all crates now use Rust 2021
Edition.
* Elaborate on Section 3.6 documentation entry
Add the title of the section and mention that it's a section of the
Zcash protocol specification.
* Make changelog entries consistent
Use the same format for entries related to consensus rule documentation.
Co-authored-by: Marek <mail@marek.onl>
* Update README about validated consensus rules
Zebra now validates all documented consensus rules.
* Add changelog entry for newly merged PR
Describe the security fix in the changelog.
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
The 2021 edition activates the v2 cargo resolver.
This broke implicit feature resolution,
but only when `zebra-chain`'s dev or bench targets were compiled by themselves.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Refactor so that RetryLimit::Future is std::marker::Sync
* Make the syncer future std::marker::Send by spawning tips futures
* Download synced blocks in chain order, not HashSet order
* Improve MockService failure messages
* Add closure-based responses to the MockService API
* Move MockChainTip to zebra-chain
* Add a MockChainTipSender type alias
* Support MockChainTip in ChainSync and its downloader
* Add syncer tests for obtain tips, extend tips, and wrong block hashes
* Add block too high tests for obtain tips and extend tips
* Add syncer tests for duplicate FindBlocks response hashes
* Allow longer request delays for mocked services in syncer tests
Zebra's latest beta continues implementing zero-knowledge proof and note commitment tree validation. In this release, we have finished implementing transaction header, transaction amount, and Zebra-specific NU5 validation. (NU5 mainnet validation is waiting on an `orchard` crate update, and some consensus parameter updates.)
We also fix a number of security issues that could pose a local denial of service risk, or make it easier for an attacker to make a node follow a false chain.
As of this release, Zebra will automatically download and cache the Sprout and Sapling Groth16 circuit parameters. The cache uses around 1 GB of disk space. These cached parameters are shared across all Zebra and `zcashd` instances run by the same user.
See CHANGELOG.md for the full list of changes in this release.
* Add Transaction::sprout_joinsplits()
* Add Anchor variants to ValidateContextError
* Make Chain anchor collections pub(crate)
* tracing::instrument several methods in state
* Add contains_*_anchors methods to FinalizedState
* Add check::anchors module and function
* Verify that anchors_refer_to_earlier_treestates in when updating chains in non-finalized state
* Update zebra-state/src/service/check/anchors.rs
Co-authored-by: teor <teor@riseup.net>
* Add anchors() to sapling::ShieldedData
* Add sapling_anchors() to Transaction
* Use Transaction::sapling_anchors() in the anchors_refer_to_earlier_treestates() check
* Whoops, itertools
* Add a comment for improvement
Co-authored-by: teor <teor@riseup.net>
* Add & use a cfg(test) method on FinalizedState to prep test state with anchors to allow other tests to pass contextual checks
* Allow test nullifier checks to pass by populating anchor sets, allowing test anchor checks to pass
* Add mainnet block 419202 and its sapling note commitment tree root to test vectors
* Test sapling anchor verification using the first few Sapling blocks data
* Correct comment
* assert_eq instead of assert(bool)
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Update zebra-state/src/service/non_finalized_state.rs
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Download and load Sprout parameters using zcash_proofs
Also update some librustzcash dependencies, to avoid duplicate dependencies.
* Update upstream orchard to avoid a compilation error
* Skip librustzcash batch refactor for now, to avoid compilation errors
* Change the cache ID, so we actually cache Sprout
* Move existing file checks into zcash_proofs
* Add a 1 hour timeout to parameter file downloads
* Give other tasks priority, before spawning the download task
* Update to the latest version of our modified librustzcash fork
* Change the cache key for Sprout
* Add 40 minutes to CI timeouts for occasional sprout downloads
* Update to zcash_proofs with split downloads
* Check file sizes to help debug parameter load failures in zcash_proofs
* Start the second download once the first has finished in zcash_proofs
* Document the parameter download task
* Stop hashing existing files twice
* First pass at async Halo2 verification service
Stubs out a batch verifier for the future.
The dependencies for orchard, halo2, librustzcash, zcash_primitives, have
not been resolved.
* Halo2 verifier service and test
* Remove redundant conversion
* Test async halo2 verifier service with pre-computed Orchard shielded data test vectors
* Fix typo
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Assert future result is_ok() in Halo2 verifier test
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Shorten tower::Service trait constraints for Halo2 verifier tests
* Remove commented out trait constraints
* .expect() vs .unwrap() to parse orchard::redpallas::VerificationKey
* Use .to_vec() for some test vectors
* Fix self-referential Display impl
* Fix deps
* Distinguish orchard vs zebra_chain::orchard imports
* Add test that halo2 verifier fails with malformed proof inputs
* Use thiserror for Halo2Error
* Use ZcashFoundation/orchard instead of dconnolly/orchard
* Add a link to the issue to remove the zfnd fork of orchard crate
* Update zebra-consensus/Cargo.toml
Co-authored-by: teor <teor@riseup.net>
* Add note
* Move artificial Orchard shielded data test vectors to zebra-test
* Align brackets
* Tidy some trait constraints and debug statements
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* Upgrade aes and fpe
* Upgrade bellman, bls12_381, jubjub to latest
* Upgrade x25519-dalek to 1.2.0 and curve25519-dalek to 3.2.0 in the Cargo.lock
* Skip outdated hdrhistogram rather than its dependencies
Co-authored-by: teor <teor@riseup.net>
* Increment the crates that have new commits since the last version
* Increment the crates that depend on crates that have changed
* Increment the version of `zebra-script`
* Use the `zebrad` version in the `zebra-network` user agent string
* Use the `v1.0.0-alpha.19` git tag in `README.md`
* Copy the draft changelog into `CHANGELOG.md`
* Delete bumps
* Update CHANGELOG.md
Co-authored-by: teor <teor@riseup.net>
* Add newly merged PRs
Co-authored-by: teor <teor@riseup.net>