Commit Graph

43 Commits

Author SHA1 Message Date
Alfredo Garcia f2be73eebb
feat(rpc): OpenAPI spec (#8342)
* add an openapi generator

* fix run command

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2024-03-14 15:04:19 +00:00
Marek 2fad3573fd
change(state): Make the types for finalized blocks consistent (#7923)
* Stop using `SemanticallyVerifiedBlockWithTrees`

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Doc that default treestate corresponds to genesis

* Doc the diff between default values of tree roots

---------

Co-authored-by: teor <teor@riseup.net>
2023-11-08 22:06:16 +00:00
Marek 1dfebd791f
fix(state): Limit the retrieval of note commitment subtrees (#7734)
* Fix the combining of Sapling subtrees

This commit fixes a bug due to which the function `sapling_subtrees`
used to always include all subtrees from the non-finalized state without
respecting the given limit.

* Fix the combining of Orchard subtrees

This commit fixes a bug due to which the function `orchard_subtrees`
used to always include all subtrees from the non-finalized state without
respecting the given limit.

* Add additional checks when retrieving subtrees

* Allow raw subtree insertions into `Chain` in tests

* Derive `Default` for `Chain`

* Derive `Default` for note commitment tree nodes

* Use `pub(super)` for `DiskWriteBatch`

Set the visibility of `DiskWriteBatch` to `pub(super)`.

* Add tests for retrieving subtrees

* Use `default()` for subtree roots in tests

This change is unrelated to the PR.

* Refactor docs for inserting subtrees

This change is unrelated to the PR.
2023-10-12 20:00:43 +00:00
teor 92d6da3531
Fix a subtree comparison edge case (#7587) 2023-09-20 18:58:32 +00:00
teor 7a7d79dfaf
fix(state): Use correct end heights for end of block subtrees during the full sync (#7566)
* Avoid manual handling of previous sapling trees by using iterator windows instead

* Avoid manual sapling subtree index handling by comparing prev and current subtree indexes instead

* Simplify adding notes by using the exact number of remaining notes

* Simplify by skipping the first block, because it can't complete a subtree

* Re-use existing tree update code

* Apply the sapling changes to orchard subtree updates

* add a reverse database column family iterator function

* Make skipping the lowest tree independent of iteration order

* Move new subtree checks into the iterator, rename to end_height

* Split subtree calculation into a new method

* Split the calculate and write methods

* Quickly check the first subtree before running the full upgrade

* Do the quick checks every time Zebra runs, and refactor slow check error handling

* Do quick checks for orchard as well

* Make orchard tree upgrade match sapling upgrade code

* Upgrade subtrees in reverse height order

* Bump the database patch version so the upgrade runs again

* Reset previous subtree upgrade data before doing this one

* Add extra checks to subtree calculation to diagnose errors

* Use correct heights for subtrees completed at the end of a block

* Add even more checks to diagnose issues

* Instrument upgrade methods to improve diagnostics

* Prevent modification of re-used trees

* Debug with subtree positions as well

* Fix an off-by-one error with completed subtrees

* Fix typos and confusing comments

Co-authored-by: Marek <mail@marek.onl>

* Fix mistaken previous tree handling and end tree comments

* Remove unnecessary subtraction in remaining leaves calc

* Log heights when assertions fail

* Fix new subtree detection filter

* Move new subtree check into a method, cleanup unused code

* Remove redundant assertions

* Wait for subtree upgrade before testing RPCs

* Fix subtree search in quick check

* Temporarily upgrade subtrees in forward height order

* Clarify some comments

* Fix missing test imports

* Fix subtree logging

* Add a comment about a potential hang with future upgrades

* Fix zebrad var ownership

* Log more info when add_subtrees.rs fails

* cargo fmt --all

* Fix unrelated clippy::unnecessary_unwrap

* cargo clippy --fix --all-features --all-targets; cargo fmt --all

* Stop the quick check depending on tree de-duplication

* Refactor waiting for the upgrade into functions

* Wait for state upgrades whenever the cached state is updated

* Wait for the testnet upgrade in the right place

* Fix unused variable

* Fix a subtree detection bug and comments

* Remove an early reference to reverse direction

* Stop skipping subtrees completed at the end of blocks

* Actually fix new subtree code

---------

Co-authored-by: Marek <mail@marek.onl>
2023-09-19 14:49:36 +00:00
teor cc61bd50b9
change(state): Add note subtree indexes for new and existing blocks (#7437)
* Copy the add_subtrees upgrade from the original branch

* Copy the database write changes in shielded.rs from the original branch

* Copy the tree API changes from the original branch

* Simplify subtree APIs to avoid exposing frontiers

* Fix a dead code warning by re-using existing methods

* Use mpsc::Receiver<CancelFormatChange> in the subtree upgrade

* Run the subtree upgrade on startup

* Bump the database format version to 25.2.0

* Fix a confusing 'upgrade complete' log

* Clarify some comments and error messages

* Simplify prev_tree unwrap to avoid an (impossible?) concurrency bug

* Use separate subtree writing functions

* Use common note commitment list code

* Fix subtree completion condition and add asserts

* Simplify subtree API and avoid exposing Address

* Fix API compatibility when Arcs are removed

* Log when each subtree is added

* If a format change is cancelled, don't mark the database as upgraded or do format checks

* Log subtree progress about once every two minutes

* Adds a state validity check for subtrees upgrade

* Orchard is faster, decrease log interval

* Clarify subtree index docs

* Move a log to the correct location

* Refactor subtree upgrade to remove duplicate inverted loop conditions

* updates subtree state validity check

* Add a subtree format check when there is no upgrade

* Fix an off-by-one error with the final subtree check

* Use error-level logs for database format checks

* Skip format checks in tests that create invalid formats

* fix state validity test

* Add a concurrency comment to subtree by height methods

* Add individual subtree state methods: reverts removing these methods in an earlier PR

* fastmod "subtrees_by_index" "subtree_list_by_index_for_rpc"

---------

Co-authored-by: arya2 <aryasolhi@gmail.com>
2023-09-05 16:52:06 +00:00
teor 97b43fb4a6
Format subtree roots in little-endian order (#7466) 2023-09-05 12:27:26 +00:00
teor 188d06e7a1
change(state): Add state requests and support code for the `z_getsubtreesbyindex` RPC (#7408)
* Make NoteCommitmentSubtreeIndex compatible with serde-based RPCs

* Add a stub for z_getsubtreesbyindex

* Define a GetSubtrees RPC response type

* Reject invalid shielded pool names

* Make limit optional

* Define state request and response types for subtrees

* Implement FromDisk for NoteCommitmentSubtreeIndex and add a round-trip test

* Make subtrees compatible with round-trip proptests

* Add finalized state subtree list methods and delete unused methods

* Remove Arc from subtrees in zebra-chain

* Remove Arc from subtrees in zebra-state and use BTreeMap

* Implement subtree list lookups in the non-finalized state and delete unused methods

* Implement consistent concurrent subtree read requests

* Implement ToHex for sapling::Node

* Implement ToHex for orchard::Node

* Implement z_get_subtrees_by_index RPC

* Check for the start_index from the non-finalized state

* Remove an unused mut

* Fix missing doc links

* Fix RPC comments

* Temporarily remove the z_get_subtrees_by_index RPC method
2023-09-03 22:18:41 +00:00
Arya 94d9155adb
change(state): Add note subtree index handling to zebra-state, but don't write them to the finalized state yet (#7334)
* zebra-chain changes from the subtree-boundaries branch

```sh
git checkout -b subtree-boundaries-zebra-chain main
git checkout origin/subtree-boundaries zebra-chain
git commit
```

* Temporarily populate new subtree fields with None - for revert

This temporary commit needs to be reverted in the next PR.

* Applies suggestions from code review

* removes from_repr_unchecked methods

* simplifies loop

* adds subtrees to zebra-state

* uses split_at, from_repr, & updates state-db-upgrades.md

* Update book/src/dev/state-db-upgrades.md

Co-authored-by: teor <teor@riseup.net>

* renames partial_subtree to subtree_data

* tests that subtree serialization format

* adds raw data format serialization round-trip test

* decrements minor version and skips inserting subtrees in db

---------

Co-authored-by: teor <teor@riseup.net>
2023-08-28 08:50:31 +00:00
teor 62258d51da
0. Add note commitment subtree types to zebra-chain (#7371)
* zebra-chain changes from the subtree-boundaries branch

```sh
git checkout -b subtree-boundaries-zebra-chain main
git checkout origin/subtree-boundaries zebra-chain
git commit
```

* Temporarily populate new subtree fields with None - for revert

This temporary commit needs to be reverted in the next PR.

* Applies suggestions from code review

* removes from_repr_unchecked methods

* simplifies loop

---------

Co-authored-by: arya2 <aryasolhi@gmail.com>
2023-08-28 00:48:16 +00:00
teor 67e3c26190
Avoid expensive cryptographic tree root recalculations in eq() (#7386)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-08-26 01:45:46 +00:00
Alfredo Garcia e2f010eb4f
build(deps): update ecc dependencies for `zcashd` 5.6.0, and create legacy state format compatibility layer (#7053)
* update ecc dependencies no serialization

* use zcash_primitives merke tree functions for serialization/deserialization in orchard and sapling trees

* some more work in sapling/orchard serialization/deserialization

* clippy

* fix doc links

* fix missing doc

* make orchard trees serializa/deserialize as they were before upgrade

* make sapling trees serialize/deserialize as they were before upgrade

* use legacy for sprout

* remove unused code

* readd snapshot tests

* repalce some code

* upgrade zcash_proofs

* remove legacy code for sprout

* fix the count method

* add root to serialize

* fixes in as_bytes and from_bytes

* use legacy code

* add todo about pow2 tests

* remove unused sprout code

* fix doc typos

* Add a recalculate_root() method to trees for tests

* Rename test tree types to make them easier to change

* Add TODOs for tests for old and new serialization formats

* fix doc typos

* add more test to note commitment trees

* fix comment

* fix leaf serializatiuon, reverse to old hashes in pow2 tests

* fix serialization

* put sapling SerializedTree code back

* put orchard SerializedTree code back

* clippy

* add duplicated dependencies until zebra_script updates

* fix a doc link

* minor cleanup

* remove todo comment from tests

* add one more check to tests

* update zebra_script

* update deny.toml

* replace custom function with library

* fix some tests

* update docs

* Remove duplicate dependencies from deny.toml

Based on the list at:
https://github.com/ZcashFoundation/zebra/actions/runs/5557139662/jobs/10150543673?pr=7053#step:5:14

---------

Co-authored-by: teor <teor@riseup.net>
2023-07-17 22:06:27 +00:00
teor f9a48266ad
change(test): Add a recalculate_root() method to trees for tests (#7204)
* Add a recalculate_root() method to trees for tests

* Rename test tree types to make them easier to change

* Add TODOs for tests for old and new serialization formats

* fix doc typos

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2023-07-12 17:07:10 +00:00
teor 9df78ffdba
change(tests): Do round-trip tests for note commitment tree data structure and RPC serialisation (#7147)
* Add an assert_frontier_eq() method to note commitment trees for tests

* Check round-trip serialization for note commitment trees

* fix typos

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2023-07-06 01:04:28 +00:00
dependabot[bot] 45a96b5adc
build(deps): bump serde from 1.0.156 to 1.0.158 (#6358)
* build(deps): bump serde from 1.0.156 to 1.0.158

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.156 to 1.0.158.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.156...v1.0.158)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Make MERKLE_DEPTH into a u8 to avoid a spurious serde_derive warning

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
2023-03-22 17:12:34 +00:00
teor 256b1c0008
fix(clippy): Fix new lints in nightly clippy (#5959)
* Derive default using #[default]

* Implement PartialEq manually to satisfy clippy

* Allow a manual derive in test-only code

* Fix some missing docs warnings in the Docker build
2023-01-18 00:27:42 +00:00
teor 66ea40f215
Resolve unnecessary borrow lint and remove redundant imports (#5045)
```sh
cargo clippy --fix --all-features --all-targets
```
2022-09-02 18:20:32 +00:00
teor 6ad445eb97
1. fix(perf): Run CPU-intensive state updates in parallel rayon threads (#4802)
* Split disk reads from CPU-heavy Sprout interstitial tree cryptography

* Improve anchor validation debugging and error messages

* Work around a test data bug, and save some CPU

* Remove redundant checks for empty shielded data

* Skip generating unused interstitial treestates

* Do disk fetches and quick checks, then CPU-heavy cryptography

* Wrap HistoryTree in an Arc in the state

* Run CPU-intensive chain validation and updates in parallel rayon threads

* Refactor to prepare for parallel tree root calculations

* Run finalized state note commitment tree root updates in parallel rayon threads

* Update finalized state note commitment trees using parallel rayon threads

* Fix a comment typo and add a TODO

* Split sprout treestate fetch into its own function

* Move parallel note commitment trees to zebra-chain

* Re-calculate the tree roots in the same parallel batches

* Do non-finalized note commitment tree updates in parallel threads

* Update comments about note commitment tree rebuilds

* Do post-fork tree updates in parallel threads

* Add a TODO for parallel tree updates in tests

* Fix broken intra-doc links

* Clarify documentation for sprout treestates

* Sort Cargo.toml dependencies
2022-07-22 12:19:11 -04:00
Alfredo Garcia 97fb85dca9
lint(clippy): add `unwrap_in_result` lint (#4667)
* `unwrap_in_result` in zebra-chain crate

* `unwrap_in_result` in zebra-script crate

* `unwrap_in_result` in zebra-state crate

* `unwrap_in_result` in zebra-consensus crate

* `unwrap_in_result` in zebra-test crate

* `unwrap_in_result` in zebra-network crate

* `unwrap_in_result` in zebra-rpc crate

* `unwrap_in_result` in zebrad crate

* rustfmt

* revert `?` and add exceptions

* explain some panics better

* move some lint positions

* replace a panic with error

* Fix rustfmt?

Co-authored-by: teor <teor@riseup.net>
2022-06-28 06:22:07 +00:00
teor 3825caae03
fix(clippy): Resolve some lifetime and reference lints (#4578)
* Fix significant drop in match scrutinee

https://github.com/rust-lang/rust/issues/93883

* Fix deref immutable value

* Fix explicit 0 index when first() would do

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-14 06:21:07 +00:00
Marek b8b35f8da9
fix(doc): Fix various doc warnings, part 1 (#4514)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fa.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-06-02 15:07:35 +00:00
Marek 6f896ef5a5
fix(doc): Fix the syntax of links in comments (#4494)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fa.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-05-30 20:12:11 +00:00
teor be91ab29ee
fix(clippy): resolve various clippy warnings (#4473)
* clippy: unused import on non-linux platforms

* Fix some instances of clippy::derive_partial_eq_without_eq

* Move a deref to fix clippy::significant_drop_in_scrutinee

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-05-25 20:56:32 +00:00
Conrado Gouvea 55822dda41
change: NU5 mainnet dependency upgrades (#4405)
* upgrade dependencies to NU5 mainnet versions

* migrate to bitvec 1.0

* update deny.toml

* point to published redjubjub 0.5

* Point at zcash-script@0.1.6 on crates.io

* update Cargo.lock

* specify patch versions of dependencies

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
2022-05-19 01:03:55 +00:00
Marek 7c726b246d
feat(rpc): Implement `z_gettreestate` RPC (#3990)
* Impl the elementary structure of the `z_gettreestate` RPC

* Fix merging bugs

* Fix a merge bug

* Fix a merge bug

* Move a derive attribute

Co-authored-by: teor <teor@riseup.net>

* Clarify the support of negative heights

* Add Orchard note commitment trees to the response

* Add the time to the response

* Finalize the `z_gettreestate` RPC

* Add a note that verified blocks have coinbase height

* Refactor `from_str` for `HashOrHeight`

* Fix a mistake in the docs

Co-authored-by: teor <teor@riseup.net>

* Clarify request types

Co-authored-by: teor <teor@riseup.net>

* Simplify `hash_or_height` conversion to height

Co-authored-by: teor <teor@riseup.net>

* Add a TODO about optimization

Co-authored-by: teor <teor@riseup.net>

* Add a doc comment

* Make sure Sapling & Orchard trees don't get mixed up

* Serialize Sapling commitment trees

* Refactor some comments

* Serialize Orchard commitment trees

* Serialize block heights

* Simplify the serialization of commitment trees

* Remove the block time from the RPC response

* Simplify the serialization of block heights

* Put Sapling & Orchard requests together

* Remove a redundant TODO

* Add block times to the RPC response

* Derive `Clone, Debug, Eq, PartialEq` for `GetTreestate`

Co-authored-by: teor <teor@riseup.net>

* Derive `Clone`, `Debug`, `Eq` and `PartialEq` for `SerializedTree`

* Document the fields of `GetTreestate`

* Skip the serialization of empty trees

This ensures compatibility with `zcashd` in the `z_gettreestate` RPC.

* Document the `impl` of `merkle_tree::Hashable` for nodes

* Make the structure of the JSON response consistent with `zcashd`

* Derive `Eq` for nodes

Co-authored-by: teor <teor@riseup.net>

* Convert Sapling commitment trees to a format compatible with zcashd

* Refactor the conversion of Sapling commitment trees

* Refactor some comments

* Refactor comments

* Add a description of the conversion

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Fix comment indenting

* Document the conversion between the dense and sparse formats

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2022-05-12 07:00:12 +00:00
Conrado Gouvea d79e71e969
change(nu5): use new V5 transaction script verification API (#3799)
* update librustzcash; adapt to new API

* add ticket reference for removing zcash_proofs duplicated dependencies

* update to new zcash_script V5 API

* use zp_tx shorthand

* update to Zcash 4.7.0 dependencies

* update protocol versions

* feat(rpc): Implement `getblockchaininfo` RPC method (#3891)

* Implement `getblockchaininfo` RPC method

* add a test for `get_blockchain_info`

* fix tohex/fromhex

* move comment

* Update lightwalletd acceptance test for getblockchaininfo RPC (#3914)

* change(rpc): Return getblockchaininfo network upgrades in height order (#3915)

* Update lightwalletd acceptance test for getblockchaininfo RPC

* Update some doc comments for network upgrades

* List network upgrades in order in the getblockchaininfo RPC

Also:
- Use a constant for the "missing consensus branch ID" RPC value
- Simplify fetching consensus branch IDs
- Make RPC type derives consistent
- Update RPC type documentation

* Make RPC type derives consistent

* Fix a confusing test comment

* get hashand height at the same time

* fix estimated_height

* fix lint

* add extra check

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* fix typo

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* split test

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* fix(rpc): ignore an expected error in the RPC acceptance tests (#3961)

* Add ignored regexes to test command failure regex methods

* Ignore empty chain error in getblockchaininfo

We expect this error when zebrad starts up with an empty state.

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Make sync error logs more user-friendly (#3944)

- use info level, there is nothing the user needs to do,
  particularly for a single error
- explain that the errors are temporary
- hide backtraces, because they look like crashes

* Update test.patch.yml with lightwalletd job (#3970)

* Update test.patch.yml with lightwalletd job

* Remove a workflow condition that will always be false

In general, patch workflows need the
opposite conditions to the original workflow.

But in this case, we know the result of the
condition will always be true, so we can just delete it.

Co-authored-by: teor <teor@riseup.net>

* fix(doc): Fix bugs in the lightwalletd database design (#3964)

* Re-order column families in design in dependency order

* Minor RFC design tweaks and fixes

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Repoint zebra image links to our new zfnd.org site for now (#3949)

* Repoint zebra image links to our new zfnd.org site for now

* Remove images/

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Fix typos (#3956)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* bump database version to trigger testnet rollback

* reduce minimum protocol version for now (will be changed later)

* update dependencies

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* update versions to match zcash 4.7.0

* deny.toml: update 'darling'

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Dimitris Apostolou <dimitris.apostolou@icloud.com>
2022-04-19 10:14:16 +10:00
teor 3291db35c0
fix(shielded): use RwLock for note commitment tree root caches (#3809)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-09 23:26:49 +00:00
Marek ece2d0f09b
Document the consensus rules for note commitment trees (#3319)
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-01-14 01:28:35 +00:00
Janito Vaqueiro Ferreira Filho 339fefb6e6
Update shared NU5 dependencies, set the NU5 testnet activation network upgrade parameters (#2825)
* Check return value of zcash_script_new_precomputed_tx

* Set the NU5 testnet activation height to 1_590_000

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Update Nu5 constants to new values

* Update ZIP-244 test vectors for new branch ID

* Squashed commit of the following:

commit bdb120a249
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date:   Tue Oct 5 11:54:01 2021 -0400

    Use pallas::Base::from_str_vartime() in sinsemilla tests

commit e99fa49258
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date:   Tue Oct 5 11:45:24 2021 -0400

    Compiles

commit a520018114
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date:   Tue Oct 5 10:15:17 2021 -0400

    Incomplete upgrade of deps

* Squashed commit of the following:

commit 8d1b76ec5626517817c3a4d9f3950acc90a359df
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date:   Tue Oct 5 04:02:26 2021 +0000

    Update `zcash_script` to support V5 transactions

    Use a newer version of `zcash_script` that has been updated to support
    V5 transactions.

commit 371233628ae61e0c25d6ba8f31d9dba42823becb
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date:   Tue Oct 5 03:06:20 2021 +0000

    Update Zcash dependencies

    Update some Zcash crates:

    - `halo2`
    - `incrementalmerkletree' (patch version)
    - `orchard` (patch version)
    - `zcash_history` (patch version)
    - `zcash_note_encryption` (patch version)
    - `zcash_primitives` (patch version)

    And also update the `group` dependency so that the code remains
    compatible.

commit de5cf1ec40c3fc08670fc971cdf3e65e13d9f4c7
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date:   Tue Oct 5 03:04:13 2021 +0000

    Update error message assertion

    Use the updated message for the expected error variant.

* Update `zcash_script` to support V5 transactions

Use a newer version of `zcash_script` that has been updated to support
V5 transactions.

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
2021-10-06 11:08:41 +10:00
Conrado Gouvea eac95bdf10
Cache note commitment tree roots (#2584)
* Cache note commitment tree roots

* Add comments to cached root fields

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2021-08-10 10:33:34 -03:00
teor f09f2a9022
Check remaining transaction value & make value balance signs match the spec (#2566)
* Make Amount arithmetic more generic

To modify generated amounts, we need some extra operations on `Amount`.

We also need to extend existing operations to both `NonNegative` and
`NegativeAllowed` amounts.

* Add a constrain method for ValueBalance

* Derive Eq for ValueBalance

* impl Neg for ValueBalance

* Make some Amount arithmetic expectations explicit

* Explain why we use i128 for multiplication

And expand the overflow error details.

* Expand Amount::sum error details

* Make amount::Error field order consistent

* Rename an amount::Error variant to Constraint, so it's clearer

* Add specific pool variants to ValueBalanceError

* Update coinbase remaining value consensus rule comment

This consensus rule was updated recently to include coinbase transactions,
but Zebra doesn't check block subsidy or miner fees yet.

* Add test methods for modifying transparent values and shielded value balances

* Temporarily set values and value balances to zero in proptests

In both generated chains and proptests that construct their own transactions.

Using zero values reduces value calculation and value check test coverage.
A future change will use non-zero values, and fix them so the check passes.

* Add extra fields to remaining transaction value errors

* Swap the transparent value balance sign to match shielded value balances

This makes the signs of all the chain value pools consistent.

* Use a NonNegative constraint for transparent values

This fix:
* makes the type signature match the consensus rules
* avoids having to write code to handle negative values

* Allocate total generated transaction input value to outputs

If there isn't enough input value for an output, set it to zero.

Temporarily reduce all generated values to avoid overflow.
(We'll remove this workaround when we calculate chain value balances.)

* Consistently use ValueBalanceError for ValueBalances

* Make the value balance signs match the spec

And rename and document methods so their signs are clearer.

* Convert amount::Errors to specific pool ValueBalanceErrors

* Move some error changes to the next PR

* Add extra info to remaining transaction value errors (#2585)

* Distinguish between overflow and negative remaining transaction value errors

And make some error types cloneable.

* Add methods for updating chain value pools (#2586)

* Move amount::test to amount::tests:vectors

* Make ValueBalance traits more consistent with Amount

- implement Add and Sub variants with Result and Assign
- derive Hash

* Clarify some comments and expects

* Create ValueBalance update methods for blocks and transactions

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-09 14:22:26 -03:00
Deirdre Connolly e719c46b1b
Track anchors and note commitment trees in zebra-state (#2458)
* Tidy chain Cargo.toml

* Organize imports

* Add method to get note commitments from all Actions in Orchard shielded data

* Add method to get note commitments from all JoinSplits in Sprout JoinSplitData

* Add Request and Response variants for awaiting anchors

* Add anchors and note commitment trees to finalized state db

* Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees

* Track anchors and note commitment trees in Chain

Append note commitments to their trees when doing update_chain_state_with,
then use the resulting Sapling and Orchard roots to pass to history_tree, and add
new roots to the anchor sets.

* Handle errors when appending to note commitment trees

* Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with

* Implementing note commitments in finalized state

* Finish serialization of Orchard tree; remove old tree when updating finalize state

* Add serialization and finalized state updates for Sprout and Sapling trees

* Partially handle trees in non-finalized state. Use Option for trees in Chain

* Rebuild trees when forking; change finalized state tree getters to not require height

* Pass empty trees to tests; use empty trees by default in Chain

* Also rebuild anchor sets when forking

* Use empty tree as default in finalized state tree getters (for now)

* Use HashMultiSet for anchors in order to make pop_root() work correctly

* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS

* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more

* Apply suggestions from code review

* Add comments about order of note commitments and related methods/fields

* Don't use Option for trees

* Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS

* Remove unneeded anchor set rebuilding in fork()

* Improve proptest formatting

* Add missing comparisons to eq_internal_state

* Renamed sprout::tree::NoteCommitmentTree::hash() to root()

* Improve comments

* Add asserts, add issues to TODOs

* Remove impl Default for Chain since it was only used by tests

* Improve documentation and assertions; add tree serialization tests

* Remove Sprout code, which will be moved to another branch

* Add todo! in Sprout tree append()

* Remove stub request, response *Anchor* handling for now

* Add test for validating Sapling note commitment tree using test blocks

* Increase database version (new columns added for note commitment trees and anchors)

* Update test to make sure the order of sapling_note_commitments() is being tested

* Improve comments and structure of the test

* Improve variable names again

* Rustfmt

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
2021-07-29 09:37:18 -04:00
Deirdre Connolly 48cf52735c
Calculate incremental note commitment trees (#2407)
* Make sapling::tree::NoteCommitmentTree incrementally updateable

* Make orchard::tree::NoteCommitmentTree incrementally updateable

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Changed to incrementalmerkletree implementation; update MerkleCRH^Orchard

* Improvements from review; organize file names (vectors.rs -> tree.rs)

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com>
2021-07-15 13:58:36 +00:00
Deirdre Connolly bb974fd282
Orchard note commitment tree test vectors (#2384)
* Orchard note commitment tree and hash test vectors?

* Add failing sinsemilla test vector test

* Support incomplete Pallas addition, all the way down

* Fix sinsemilla sub function S(j), add note commitment tree empty root tests

* Clippy nightly lints

* allow(clippy::derive_hash_xor_eq) for orchard::tree

* Update zebra-chain/src/orchard/sinsemilla.rs

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-06-25 16:27:47 +00:00
Conrado Gouvea 9688811e35
Update multiple crates to ensure bitvec 0.22.3 is being used (#2351)
* Update multiple crates to ensure bitvec 0.22.3 is being used and avoid package conflicts

* Add documentation to zebra-chain::sapling to indicate that ZIP-216 rules are enforced by jubjub

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
2021-06-23 13:16:22 +10:00
Alfredo Garcia a57c09a3b8
Orchard data in V5 parsing (#2116)
* initialize the work on parsing orchard data in V5
* add the rest of orchard serialization
* fix serialization according to spec
* fix arbitrary for Signature<SpendAuth>
* move deserialization of AuthorizedAction to shielded_data module
* use `from_bits_truncate` to generate valid arbitrary flags
* change panic message
* fix serialization/deserialization when nActionsOrchard is empty
* fix Halo2Proof deserialization
* implement ZcashSerialize and ZcashDeserialize for flags
* implement ZcashSerialize and ZcashDeserialize for orchard::tree::Root
* use ZcashSerialize and ZcashDeserialize for binding_sig
* implement from_parts()
* implement Arbitrary for  Signature<Binding>
* add trusted preallocate with tests
* fix Arbitrary for orchard Nullifier
* Use zcash_serialize_bytes instead of write_compactsize

Co-authored-by: teor <teor@riseup.net>
2021-05-21 00:42:06 +00:00
Deirdre Connolly f5206af470 Rename all Orchard spec references to nu5.pdf 2021-04-30 00:48:23 -04:00
Deirdre Connolly a1a59a7f76 Clean up orchard::tree tests for now 2021-04-30 00:48:23 -04:00
Deirdre Connolly 87f65b8b01 Orchard: update merkle_crh_orchard to use the appropriate l=10 bits, not 6 2021-04-30 00:48:23 -04:00
Deirdre Connolly c892b93f61 Orchard: update merkle_crh_orchard correctly 2021-04-30 00:48:23 -04:00
Deirdre Connolly cb9d6956d7 Orchard: Fix trait imports and Arbitrary for Action 2021-04-30 00:48:23 -04:00
Deirdre Connolly 23e391894b Tidy 2021-04-30 00:48:23 -04:00
Deirdre Connolly 40383b2741 Partway done with typing out Orchard chain types 2021-04-30 00:48:23 -04:00