Commit Graph

25 Commits

Author SHA1 Message Date
Deirdre Connolly e719c46b1b
Track anchors and note commitment trees in zebra-state (#2458)
* Tidy chain Cargo.toml

* Organize imports

* Add method to get note commitments from all Actions in Orchard shielded data

* Add method to get note commitments from all JoinSplits in Sprout JoinSplitData

* Add Request and Response variants for awaiting anchors

* Add anchors and note commitment trees to finalized state db

* Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees

* Track anchors and note commitment trees in Chain

Append note commitments to their trees when doing update_chain_state_with,
then use the resulting Sapling and Orchard roots to pass to history_tree, and add
new roots to the anchor sets.

* Handle errors when appending to note commitment trees

* Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with

* Implementing note commitments in finalized state

* Finish serialization of Orchard tree; remove old tree when updating finalize state

* Add serialization and finalized state updates for Sprout and Sapling trees

* Partially handle trees in non-finalized state. Use Option for trees in Chain

* Rebuild trees when forking; change finalized state tree getters to not require height

* Pass empty trees to tests; use empty trees by default in Chain

* Also rebuild anchor sets when forking

* Use empty tree as default in finalized state tree getters (for now)

* Use HashMultiSet for anchors in order to make pop_root() work correctly

* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS

* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more

* Apply suggestions from code review

* Add comments about order of note commitments and related methods/fields

* Don't use Option for trees

* Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS

* Remove unneeded anchor set rebuilding in fork()

* Improve proptest formatting

* Add missing comparisons to eq_internal_state

* Renamed sprout::tree::NoteCommitmentTree::hash() to root()

* Improve comments

* Add asserts, add issues to TODOs

* Remove impl Default for Chain since it was only used by tests

* Improve documentation and assertions; add tree serialization tests

* Remove Sprout code, which will be moved to another branch

* Add todo! in Sprout tree append()

* Remove stub request, response *Anchor* handling for now

* Add test for validating Sapling note commitment tree using test blocks

* Increase database version (new columns added for note commitment trees and anchors)

* Update test to make sure the order of sapling_note_commitments() is being tested

* Improve comments and structure of the test

* Improve variable names again

* Rustfmt

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
2021-07-29 09:37:18 -04:00
teor 544be14c70
Reject duplicate Sapling and Orchard nullifiers (#2497)
* Add sapling and orchard duplicate nullifier errors

* Reject duplicate finalized sapling and orchard nullifiers

Reject duplicate sapling and orchard nullifiers in a new block,
when the block is added to a non-finalized chain,
and the duplicate nullifier is already in the finalized state.

* Reject duplicate non-finalized sapling and orchard nullifiers

Reject duplicate sapling and orchard nullifiers in a new block,
when the block is added to a non-finalized chain,
and the duplicate nullifier is in:
* the same shielded data,
* the same transaction,
* the same block, or
* an earlier block in the non-finalized chain.

* Refactor sprout nullifier tests to remove common code

* Add sapling nullifier tests

Test that the state rejects duplicate sapling nullifiers in a new block,
when the block is added to a non-finalized chain,
and the duplicate nullifier is in:
* the same shielded data,
* the same transaction,
* the same block,
* an earlier block in the non-finalized chain, or
* the finalized state.

* Add orchard nullifier tests

Test that the state rejects duplicate orchard nullifiers in a new block,
when the block is added to a non-finalized chain,
and the duplicate nullifier is in:
* the same shielded data,
* the same transaction,
* the same block,
* an earlier block in the non-finalized chain, or
* the finalized state.

* Check for specific nullifiers in the state in tests

* Replace slices with vectors in arguments

* Remove redundant code and variables

* Simplify sapling TransferData tests

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Remove an extra :

* Remove redundant vec!

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-19 21:39:05 -03:00
teor 0f5eced5c7
Reject duplicate sprout nullifiers in the state (#2477)
* Reject duplicate sprout nullifiers in the state

* Improve docs and error messages

* Clarify "must be present" assert logs

* Move nullifier checks to their own module

Also:
* make non-finalized nullifier checks and errors generic over
  sprout, sapling, and orchard
* create and update module and function documentation

* Fix a block type name in docs

* Move state assertions or skip them during tests

These changes enable state testing, while still asserting in production.

* Add sprout duplicate nullifier tests

* Improve comments

* Set value balance to 0 to pass future chain value pool checks

* Test finalized state in sprout nullifier accept test

* Replace assert with expect

* Improve assertion messages
2021-07-14 09:06:43 -03:00
teor 82696b150b
Document some consensus-critical finalized state behaviour (#2476)
* Document the new genesis transaction consensus rule

Zebra previously implemented this rule, but we documented it as a bug in
`zcashd`.

* Document the actual behaviour of zs_insert
2021-07-12 17:11:33 -03:00
Alfredo Garcia f7026d728f
move `Utxo` type to zebra-chain (#2481) 2021-07-12 12:49:33 +10:00
teor f745333ff2
Use the block verifier and non-finalized state in the cached state tests (#2463)
* Use the block verifier and non-finalized state in the cached state tests

This substantially increases test coverage.

Previously, the cached state tests were configured with
`checkpoint_sync = true`, which only uses the checkpoint
verifier and the finalized state.

* Log the source of blocks in commit_finalized_direct

This lets us check that we're actually testing the non-finalized state
and block verifier in the cached state tests.

It also improves diagnostics for state errors.

* Fail cached state tests if they're using incorrect heights or configs

This makes sure that the cached state tests actually test the transition
from checkpoint to block verification, and the non-finalized state.
2021-07-09 09:47:09 -03:00
teor 2aaf1d245f
Revert "ZIP-221: Validate chain history commitments in the non-finalized state (#2301)" (#2464)
This reverts commit 91b1fcb37b.
2021-07-08 09:25:49 -03:00
Conrado Gouvea 91b1fcb37b
ZIP-221: Validate chain history commitments in the non-finalized state (#2301)
* sketch of implementation

* refined implementation; still incomplete

* update librustzcash, change zcash_history to work with it

* simplified code per review; renamed MMR to HistoryTree

* expand HistoryTree implementation

* handle and propagate errors

* simplify check.rs tracing

* add suggested TODO

* add HistoryTree::prune

* fix bug in pruning

* fix compilation of tests; still need to make them pass

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* improvements from code review

* improve check.rs comments and variable names

* fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests

* fix finalized_state proptest

* fix non_finalized_state tests by setting the correct commitments

* renamed mmr.rs to history_tree.rs

* Add HistoryTree struct

* expand non_finalized_state protest

* fix typo

* Add HistoryTree struct

* Update zebra-chain/src/primitives/zcash_history.rs

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* fix formatting

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* history_tree.rs: fixes from code review

* fixes to work with updated HistoryTree

* Improvements from code review

* Add Debug implementations to allow comparing Chains with proptest_assert_eq

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-07-07 10:29:08 +10:00
Alfredo Garcia 76ad543ec5 remove assert 2021-06-21 21:46:09 -04:00
Alfredo Garcia 22cf3eed9e add path() to FinalizedState 2021-06-21 21:46:09 -04:00
teor 8ebb415e7c Clippy: remove needless borrows 2021-06-07 18:33:58 -04:00
Alfredo Garcia 1685611592
Store orchard nullifiers into the state (#2185)
* add nullifier methods to orchard
* store orchard nullifiers
* bump database version
* update `IntoDisk`
* support V5 in `UpdateWith`
* add a test for finalized state
* Use the latest network upgrade in state proptests
2021-06-01 17:53:13 +10:00
teor f94033df08
Make arbitrary block chains pass some genesis checks (#2208)
* Clarify the finalized state assertion that checks the genesis block

* Make arbitrary block chains pass some genesis checks

Use the genesis previous block hash for
- the first arbitrary block in each chain, and
- individual arbitrary blocks.

This setting can be adjusted by individual proptests as needed.
2021-05-27 12:41:20 -03:00
Alfredo Garcia 4b34482264
Add hints to port conflict and lock file panics (#1535)
* add hint for port error
* add issue filter for port panic
* add lock file hint
* add metrics endpoint port conflict hint
* add hint for tracing endpoint port conflict
* add acceptance test for resource conflics
* Split out common conflict test code into a function
* Add state, metrics, and tracing conflict tests

* Add a full set of stderr acceptance test functions

This change makes the stdout and stderr acceptance test interfaces
identical.

* move Zcash listener opening
* add todo about hint for disk full
* add constant for lock file
* match path in state cache
* don't match windows cache path

* Use Display for state path logs

Avoids weird escaping on Windows when using Debug

* Add Windows conflict error messages

* Turn PORT_IN_USE_ERROR into a regex

And add another alternative Windows-specific port error

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Jane Lusby <jane@zfnd.org>
2021-01-29 22:36:33 +10:00
teor 270070f2d9
Change a metrics sentinel to f64::NAN (#1642)
* Change a metrics sentinel to f64::NAN

And document a tricky part of the code

* rustfmt

* rustfmt again
2021-01-28 17:29:57 +10:00
Jane Lusby 15698245e1
Deduplicate metrics dependencies (#1561)
## Motivation

This PR is motivated by the regression identified in https://github.com/ZcashFoundation/zebra/issues/1349. That PR notes that the metrics stopped working for most of the crates other than `zebrad`.

## Solution

This PR resolves the regression by deduplicating the `metrics` crate dependency. During a recent change we upgraded the metrics version in `zebrad` and a couple other of our crates, but we never updated the dependencies in `zebra-state`, `zebra-consensus`, or `zebra-network`. This caused the metrics macros to attempt to retrieve the current metrics exporter through the wrong function. We would install the metrics exporter in `0.13`, but then attempt to look it up through the `0.12` crate, which contains a different instance of the metrics exporter static variable which is unset. Doing this causes the metrics macros to return `None` for the current exporter after which they just silently give up.

## Related Issues

closes https://github.com/ZcashFoundation/zebra/issues/1349

## Follow Up Work

I noticed we have quite a few duplicate dependencies in our tree. We might be able to save some compilation time by auditing those and deduplicating them as much as possible.

- https://github.com/ZcashFoundation/zebra/issues/1582
Co-authored-by: teor <teor@riseup.net>
2021-01-12 12:28:56 +10:00
teor 44f2326672 Move the RocksDB column family list into finalized_state
The list was previously split between config and finalized_state.
2020-12-02 15:32:36 +10:00
Henry de Valence 7c08c0c315 consensus: check Merkle roots
As a side effect of computing Merkle roots, we build a list of
transaction hashes.  Instead of discarding these, add them to
PreparedBlock and FinalizedBlock so that they can be reused rather than
recomputed.

This commit adds Merkle root validation to:

1. the block verifier;
2. the checkpoint verifier.

In the first case, Bitcoin Merkle tree malleability has no effect,
because only a single Merkle tree in each malleablity set is valid (the
others have duplicate transactions).

In the second case, we need to check that the Merkle tree does not contain any
duplicate transactions.

Closes #1385
Closes #906
2020-12-01 10:14:44 +10:00
teor b1bbb13978
Make debug_stop_at_height and ephemeral work together (#1339)
* Make debug_stop_at_height and ephemeral work together

* if `debug_stop_at_height` and `ephemeral` are set, delete the database
  files after reaching the stop height
* drop or flush the database before `debug_stop_at_height` exits Zebra
2020-11-25 15:04:18 +10:00
Henry de Valence 342eb166ff state: track UTXO provenance
This commit changes the state system and database format to track the
provenance of UTXOs, in addition to the outputs themselves.
Specifically, it tracks the following additional metadata:

- the height at which the UTXO was created;
- whether or not the UTXO was created from a coinbase transaction or
  not.

This metadata will allow us to:

- check the coinbase maturity consensus rule;
- check the coinbase inputs => no transparent outputs rule;
- implement lookup of transactions by utxo (using the height to find the
  block and then scanning the block) for a future RPC mechanism.

Closes #1342
2020-11-23 22:18:43 -08:00
Henry de Valence e0817d1747 state: introduce PreparedBlock, FinalizedBlock
This change introduces two new types:

- `PreparedBlock`, representing a block which has undergone semantic
  validation and has been prepared for contextual validation;
- `FinalizedBlock`, representing a block which is ready to be finalized
  immediately;

and changes the `Request::CommitBlock`,`Request::CommitFinalizedBlock`
variants to use these types instead of their previous fields.

This change solves the problem of passing data between semantic
validation and contextual validation, and cleans up the state code by
allowing it to pass around a bundle of data.  Previously, the state code
just passed around an `Arc<Block>`, which forced it to needlessly
recompute block hashes and other data, and was incompatible with the
already-known but not-yet-implemented data transfer requirements, namely
passing in the Sprout and Sapling anchors computed during contextual
validation.

This commit propagates the `PreparedBlock` and `FinalizedBlock` types
through the state code but only uses their data opportunistically, e.g.,
changing .hash() computations to use the precomputed hash.  In the
future, these structures can be extended to pass data through the
verification pipeline for reuse as appropriate.  For instance, these
changes allow the sprout and sapling anchors to be propagated through
the state.
2020-11-23 14:16:39 +10:00
Henry de Valence f0810b028d state,consensus,sync: shorten span lengths
These changes help reduce the size of the resulting spans, making the
output more compact.  Together they save about 30-40 characters.
2020-11-23 14:16:39 +10:00
Jane Lusby 4c9bb87df2
zebra-state: replace sled with rocksdb (#1325)
## Motivation

Prior to this PR we've been using `sled` as our database for storing persistent chain data on the disk between boots. We picked sled over rocksdb to minimize our c++ dependencies despite it being a less mature codebase. The theory was if it worked well enough we'd prefer to have a pure rust codebase, but if we ever ran into problems we knew we could easily swap it out with rocksdb.

Well, we ran into problems. Sled's memory usage was particularly high, and it seemed to be leaking memory. On top of all that, the performance for writes was pretty poor, causing us to become bottle-necked on sled instead of the network.

## Solution

This PR replaces `sled` with `rocksdb`. We've seen a 10x improvement in memory usage out of the box, no more leaking, and much better write performance. With this change writing chain data to disk is no longer a limiting factor in how quickly we can sync the chain.

The code in this pull request has:
  - [x] Documentation Comments
  - [x] Unit Tests and Property Tests

## Review

@hdevalence
2020-11-18 18:05:06 -08:00
Jane Lusby 5a6a9fd51e remove some references to sled in serialization definition module 2020-11-18 15:09:43 -05:00
Jane Lusby a122a547be reorganize modules for consistency 2020-11-18 15:09:43 -05:00