Commit Graph

302 Commits

Author SHA1 Message Date
Deirdre Connolly 89b0403582
Enforce Rust edition 2021 (#3332)
* Rust edition 2021: zebra-network, cargo fix --edition and clippy --fix

* Rust edition 2021: zebra-chain, cargo fix --edition

* Rust edition 2021: tower-batch, cargo fix --edition

* Rust edition 2021: tower-fallback, cargo fix --edition

* Rust edition 2021: zebra-client, cargo fix --edition

* Rust edition 2021: zebra-consensus, cargo fix --edition

* Rust edition 2021: zebra-rpc, cargo fix --edition

* Rust edition 2021: zebra-state, cargo fix --edition

* Rust edition 2021: zebra-state, cargo fix --edition

* Rust edition 2021: zebra-test, cargo fix --edition

* Rust edition 2021: zebra-utils, cargo fix --edition

* Rust edition 2021: zebrad, cargo fix --edition

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-01-14 12:10:18 +00:00
Dimitris Apostolou 1a1ce3dbff
Fix typos (#3314) 2022-01-04 11:25:00 +01:00
Alfredo Garcia 918a337d8b
Document part of the block header consensus rules (#3296)
* document header version consensus rule

* document nbits threshold consensus rule

* document difficulty filter consensus rule

* document header solution consensus rule

* document header time consensus rule

* document upper time limit consensus rule

* document max block size consensus rule

* skip genesis in conesnsus rule check

* remove fixed comment

Co-authored-by: teor <teor@riseup.net>
2021-12-29 23:07:27 +00:00
teor ea01e305ed
Silence expected anchor errors in tests (#3237) 2021-12-15 22:20:53 +00:00
Conrado Gouvea bd9c954667
Disable Sprout anchor check (#3206)
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-12-15 11:16:59 -03:00
Conrado Gouvea ee073c0876
Validate JoinSplit proofs (#3128)
* Validate JoinSplit proofs

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Move primary input encoding to zebra_consensus

* Improve typing of h_sig; add RandomSeed

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-12-10 16:33:15 +00:00
Deirdre Connolly b973b7a622
Checking of Sprout anchors in non-finalized state (#3123)
* Do prelim checking of Sprout anchors in non-finalized state

Does not check intra-transaction interstitial states yet

* Populate sprout anchors to allow other state tests to pass

* Preliminary interstitial sprout note commitment tree anchor checks implementation

* Make sure only prior anchors are checked in the same transaction

* Add tests

* Refactor a comment

* Refactor rustdoc

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Use the first `JoinSplit`s from mainnet

* Print debug messages

* Use correct blocks for the tests

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-12-09 16:50:26 +00:00
teor 4ce6fbccc4
Fix new clippy lints in clippy nightly (#3176) 2021-12-09 14:19:14 +00:00
Deirdre Connolly e6ffe374d4
Validate sapling, orchard anchors (#3084)
* Add Transaction::sprout_joinsplits()

* Add Anchor variants to ValidateContextError

* Make Chain anchor collections pub(crate)

* tracing::instrument several methods in state

* Add contains_*_anchors methods to FinalizedState

* Add check::anchors module and function

* Verify that anchors_refer_to_earlier_treestates in when updating chains in non-finalized state

* Update zebra-state/src/service/check/anchors.rs

Co-authored-by: teor <teor@riseup.net>

* Add anchors() to sapling::ShieldedData

* Add sapling_anchors() to Transaction

* Use Transaction::sapling_anchors() in the anchors_refer_to_earlier_treestates() check

* Whoops, itertools

* Add a comment for improvement

Co-authored-by: teor <teor@riseup.net>

* Add & use a cfg(test) method on FinalizedState to prep test state with anchors to allow other tests to pass contextual checks

* Allow test nullifier checks to pass by populating anchor sets, allowing test anchor checks to pass

* Add mainnet block 419202 and its sapling note commitment tree root to test vectors

* Test sapling anchor verification using the first few Sapling blocks data

* Correct comment

* assert_eq instead of assert(bool)

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Update zebra-state/src/service/non_finalized_state.rs

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-11-30 16:05:35 +00:00
Marek 3c9ad89018
Add Sprout anchors to `zebra-state` (#3100)
* Add Sprout anchors to the state

* Update zebra-state/src/service/non_finalized_state/chain.rs

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Return new types of note commitments from Sprout transactions

* Refactor the tests

* Refactor some comments

Co-authored-by: teor <teor@riseup.net>

* Increment `DATABASE_FORMAT_VERSION`

* Update `test.yml` with the new image name

* Refactor the `version = 5` transaction description

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Update comment

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
2021-11-30 11:05:58 +01:00
Janito Vaqueiro Ferreira Filho ec2c980bb1
Validate transaction lock times (#3060)
* Create a `LockTime::unlocked` helper constructor

Returns a `LockTime` that is unlocked at the genesis block.

* Return `Option<LockTime>` from `lock_time` method

Prepare to return `None` for when a transaction has its lock time
disabled.

* Return `None` instead of zero `LockTime`

Because a zero lock time means that the transaction was unlocked at the
genesis block, so it was never actually locked.

* Rephrase zero lock time check comment

Clarify that the check is not redundant, and is necessary for the
genesis transaction.

Co-authored-by: teor <teor@riseup.net>

* Add a `transparent::Input::sequence` getter method

Retrieve a transparent input's sequence number.

* Check if lock time is enabled by a sequence number

Validate the consensus rule that the lock time is only enabled if at
least one transparent input has a value different from `u32::MAX` as its
sequence number.

* Add more Zcash specific details to comment

Explain the Zcash specific lock time behaviors.

Co-authored-by: teor <teor@riseup.net>

* Add `time` field to `Request::Block` variant

The block time to use to check if the transaction was unlocked and
allowed to be included in the block.

* Add `Request::block_time` getter

Returns the block time for the block that owns the transaction being
validated or the current time plus a tolerance for mempool transactions.

* Validate transaction lock times

If they are enabled by a transaction's transparent input sequence
numbers, make sure that they are in the past.

* Add comments with consensus rule parts

Make it easier to map what part of the consensus rule each match arm is
responsible for.

Co-authored-by: teor <teor@riseup.net>
2021-11-23 05:53:53 +00:00
teor 1df3bdb089
Check MAX_BLOCK_SIGOPS in the block verifier (#3049)
* Cleanup a function that calls zcash_script

* Remove zebra_test::prelude macros that conflict with the Rust prelude

* Add sigops count support to zebra-script

* Check MAX_BLOCK_SIGOPS in the block verifier

* Test MAX_BLOCK_SIGOPS on generated and historic blocks

* Add SAFETY comments for all unsafe zebra-script code

* Explain where the consensus rule comes from

* Remove unused pretty_assertions dependency

* Allow large test block generation functions with the proptest-impl feature

* Replace `as` with `try_into` for integer conversions in unsafe code

* Expand SAFETY comments
2021-11-15 20:55:32 +00:00
Dimitris Apostolou afb8b3d477
Fix typos (#3055)
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-12 19:30:22 +00:00
teor 7d8240fac3
Fix verbose add_initial_peers logs (#3019)
And update some function docs.
2021-11-07 22:21:51 +00:00
teor b1303ab8d7
Replace read_compactsize and write_compactsize with CompactSizeMessage (#3014)
* Replace read_compactsize and write_compactsize with CompactSizeMessage

* Add tests for CompactSize64

* Add compact size range and conversion tests
2021-11-05 15:24:24 -03:00
Janito Vaqueiro Ferreira Filho 0960e4fb0b
Update to Tokio 1.13.0 (#2994)
* Update `tower` to version `0.4.9`

Update to latest version to add support for Tokio version 1.

* Replace usage of `ServiceExt::ready_and`

It was deprecated in favor of `ServiceExt::ready`.

* Update Tokio dependency to version `1.13.0`

This will break the build because the code isn't ready for the update,
but future commits will fix the issues.

* Replace import of `tokio::stream::StreamExt`

Use `futures::stream::StreamExt` instead, because newer versions of
Tokio don't have the `stream` feature.

* Use `IntervalStream` in `zebra-network`

In newer versions of Tokio `Interval` doesn't implement `Stream`, so the
wrapper types from `tokio-stream` have to be used instead.

* Use `IntervalStream` in `inventory_registry`

In newer versions of Tokio the `Interval` type doesn't implement
`Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used
instead.

* Use `BroadcastStream` in `inventory_registry`

In newer versions of Tokio `broadcast::Receiver` doesn't implement
`Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This
also requires changing the error type that is used.

* Handle `Semaphore::acquire` error in `tower-batch`

Newer versions of Tokio can return an error if the semaphore is closed.
This shouldn't happen in `tower-batch` because the semaphore is never
closed.

* Handle `Semaphore::acquire` error in `zebrad` test

On newer versions of Tokio `Semaphore::acquire` can return an error if
the semaphore is closed. This shouldn't happen in the test because the
semaphore is never closed.

* Update some `zebra-network` dependencies

Use versions compatible with Tokio version 1.

* Upgrade Hyper to version 0.14

Use a version that supports Tokio version 1.

* Update `metrics` dependency to version 0.17

And also update the `metrics-exporter-prometheus` to version 0.6.1.
These updates are to make sure Tokio 1 is supported.

* Use `f64` as the histogram data type

`u64` isn't supported as the histogram data type in newer versions of
`metrics`.

* Update the initialization of the metrics component

Make it compatible with the new version of `metrics`.

* Simplify build version counter

Remove all constants and use the new `metrics::incement_counter!` macro.

* Change metrics output line to match on

The snapshot string isn't included in the newer version of
`metrics-exporter-prometheus`.

* Update `sentry` to version 0.23.0

Use a version compatible with Tokio version 1.

* Remove usage of `TracingIntegration`

This seems to not be available from `sentry-tracing` anymore, so it
needs to be replaced.

* Add sentry layer to tracing initialization

This seems like the replacement for `TracingIntegration`.

* Remove unnecessary conversion

Suggested by a Clippy lint.

* Update Cargo lock file

Apply all of the updates to dependencies.

* Ban duplicate tokio dependencies

Also ban git sources for tokio dependencies.

* Stop allowing sentry-tracing git repository in `deny.toml`

* Allow remaining duplicates after the tokio upgrade

* Use C: drive for CI build output on Windows

GitHub Actions uses a Windows image with two disk drives, and the
default D: drive is smaller than the C: drive. Zebra currently uses a
lot of space to build, so it has to use the C: drive to avoid CI build
failures because of insufficient space.

Co-authored-by: teor <teor@riseup.net>
2021-11-02 18:46:57 +00:00
Marek 4f7a977565
Test multiple chain resets (#2897)
* Try simulating a chain growth

* Adjust the transaction expiry height

The mempool evicts expired transactions. When working with mocked data,
appending a new block typically clears the mempool because transactions become
expired. For this reason, the expiry height of each transactions is adjusted so
that it is greater than the new chain tip's height.

* Refactor the code so that it works with `VerifiedUnminedTx`

* Fix a typo

* Fix clippy warnings

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-10-22 02:54:08 +00:00
teor 67327ac462
Downgrade some less interesting info-level logs to debug (#2938)
There are a lot of these messages when Zebra starts up.
They might be slowing down CI and causing timeouts.

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-10-22 02:11:09 +00:00
teor e277975d85
Try flushing streams before exiting Zebra (#2911) 2021-10-20 13:57:09 +00:00
teor 92634f788b
Increase Zebra's restart acceptance test timeout (#2910)
* Increase the restart test timeout to 10 seconds

It shouldn't take this long.
But maybe the CI VMs are under a lot of load?

* Add extensive logging to diagnose CI state reload failures
2021-10-20 01:59:09 +00:00
teor f4118dadda
Make block metrics more accurate (#2835)
* Check for state errors before updating metrics

Previously, the metrics would be updated for some rejected blocks.

* Clarify and expand block verification metrics

Rename checkpoint-specific metrics to clarify their purpose.

Add metrics for:
- finalized blocks on disk
- blocks verified using the full block verifier
  (this metric was previously incorrectly called `zcash_chain_verified_block_height`)

* Update dashboard metric names

Also:
- add some extra block height metrics
- fix a dashboard name

* Add exact block heights to Grafana dashboards

* Add a missing comment

* grafana: use 0 decimals for metrics

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* grafana: show the entire height instead of abbreviated

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* grafana: show the entire height instead of abbreviated

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Fix typo in metric name

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Move height gauges to the state, so they are correct

If we update height gauges in futures, they can execute out of order,
so the metrics can be incorrect.

Instead:
- move the height gauges to the state, and update them based on the best tip
- move the verified block counts to the state
- continue to include all verified blocks on all non-finalized chains
  (not just the best chain)

* Show exact checkpoint heights in the dashboard

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-07 13:42:38 +00:00
teor 04d2cfb3d0
Gossip recently verified block hashes to peers (#2729)
* Implement a task that gossips verified block hashes

* Log an info message for block broadcasts

* Simplify the gossip task

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Re-use the old tip change if there is no new tip change

Also improve the comments.

* Add an assertion message

* Rename task join handles and futures in start method

* Add a dedicated BlockGossipError type

This type helps distinguish between syncer and state errors.

* Test that committed blocks are gossiped to peers

Also do a minor type cleanup on the existing test code,
replacing `Option<Vec<_>>` with `Vec<_>`.

* Formatting

* Remove excess newlines

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Clear the initial gossiped blocks during test setup

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-10-07 07:46:37 -03:00
teor 0b82298645 Stop dropping queued block senders without sending a response
This causes a rare panic, because Zebra expects every queued sender
to send a response.
2021-10-07 00:12:45 -04:00
teor c7c1a8a469 Remove known UTXOs when pruning blocks 2021-10-07 00:12:45 -04:00
teor e5f5ac9ce8
Fix or disable recent nightly clippy lints (#2817)
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-01 15:26:06 +00:00
Janito Vaqueiro Ferreira Filho 50a5728d0b
Test if the mempool storage is cleared (#2815)
* Move mempool tests into `tests::vector` sub-module

Make it consistent with other test modules and prepare for adding
property tests.

* Reorder imports

Make it consistent with the general guidelines followed on other
modules.

* Export `ChainTipBlock` and `ChainTipSender`

Allow these types to be used in other crates for testing purposes.

* Derive `Arbitrary` for `ChainTipBlock`

Make it easy to generate random `ChainTipBlock`s for usage in property
tests.

* Refactor to move test methods into `tests` module

Reduce the repeated test configuration attributes and make it easier to
see what is test specific and what is part of the general
implementation.

* Add a `Mempool::dummy_call` test helper method

Performs a dummy call just so that `poll_ready` gets called.

* Use `dummy_call` in existing tests

Replace the custom dummy requests with the helper method.

* Test if the mempool is cleared on chain reset

A chain reset should force the mempool storage to be cleared so that
transaction verification can restart using the new chain tip.

* Test if mempool is cleared on syncer restart

If the block synchronizer falls behind and then starts catching up
again, the mempool should be disabled and therefore the storage should
be cleared.
2021-10-01 14:44:25 +00:00
teor 1601c9fbb3
Stop panicking when using sync and async methods on the same ChainTipChange (#2800)
* Instrument chain tip methods

* Expand tests to cover last_tip_change and multiple change checks

* Expand tests to cover Grow as well as Reset

* Support sync and async methods on the same ChainTipChange

* Add a Tokio 1.0 TODO

* Clarify a comment

* Manual rustfmt inside a proptest

* Remove tracing clones, and instrument ChainTipSender::new

* Add the tokio issue number to a TODO comment
2021-09-28 07:48:19 -03:00
teor 4567701933
Fix an incorrect assertion when the block locator is at the tip (#2789)
* Fix an incorrect assertion when the block locator is at the tip

This might have been triggered by receiving block hash gossips
from the new Zebra code.

* Add missing tests for zebra-state requests and responses

Specifically:
* `BlockLocator` (populated state only)
* `FindBlockHashes`
* `FindBlockHeaders`

* Test `FindBlock*` before and after the current block

* Add a specific test for bug #2789

* Refactor collect_best_chain_hashes to avoid manual index calculations

* Reword a comment

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-09-27 22:43:05 +00:00
Marek 30c9618207
Clear mempool at a network upgrade (#2773)
* Update the expiry TODO

* Clear the mempool at a chain tip reset

* Clear the mempool by using a sync method (#2777)

* Clear the mempool by using a sync method

* Update docs

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Refactor last_tip_change()

* Apply suggestions from code review

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Fix brackets

* Use best_tip_block instead of manual borrowing

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-09-23 19:09:44 +00:00
Marek 061ad55144
Sneak chain_tip_change into mempool (#2785)
* Pass ChainTipChange to the mempool

* Fix nits
2021-09-21 17:06:52 +00:00
teor 44ac06775b
Reset `ChainTipChange`s on chain fork and network upgrade activation (#2721)
* Use `TipAction::Reset` for initialization and out-of-order blocks

Needs tests for:
- cloning a `ChainTipChange` resets the cloned instance
- skipped updates reset the cloned instance
- changing forks resets the cloned instance

* Use `TipAction::Reset` for network upgrade activation blocks

* Use an `if` expression

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Another if expression

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-09-02 12:25:42 +10:00
Conrado Gouvea 1ccb2de7c7
Add transaction downloader and verifier (#2679)
* Add transaction downloader

* Changed mempool downloader to be like inbound

* Verifier working (logs result)

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Fix coinbase check for mempool, improve is_coinbase() docs

* Change other downloads.rs docs to reflect the mempool downloads.rs changes

* Change TIMEOUTs to downloads.rs; add docs

* Renamed is_coinbase() to has_valid_coinbase_transaction_inputs() and contains_coinbase_input() to has_any_coinbase_inputs(); reorder checks

* Validate network upgrade for V4 transactions; check before computing sighash (for V5 too)

* Add block_ prefix to downloads and verifier

* Update zebra-consensus/src/transaction.rs

Co-authored-by: teor <teor@riseup.net>

* Add consensus doc; add more Block prefixes

Co-authored-by: teor <teor@riseup.net>
2021-09-02 00:06:20 +00:00
teor b6fe816473
Add a `ChainTipChange` type to `await` chain tip changes (#2715)
* Rename ChainTipReceiver to CurrentChainTip

`fastmod ChainTipReceiver CurrentChainTip zebra*`

* Update chain tip documentation and variable names

* Basic chain tip change implementation, without resets

Also includes the following name changes:
```
fastmod CurrentChainTip LatestChainTip zebra*
fastmod chain_tip_receiver latest_chain_tip zebra*
```

* Clarify the difference between `LatestChainTip` and `ChainTipChange`
2021-09-01 22:31:16 +00:00
teor 2dac0dda47
Stop calculating transaction hashes twice in the checkpoint verifier (#2696)
* Stop precalculating transaction hashes twice during checkpointing

* Refactor a complex type using a new `RequestBlock` type

* Comment formatting

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-08-31 10:55:39 +10:00
teor a66ecbc16d
Make chain tip updates and access more efficient (#2695)
* Store precalculated transactions in an `Arc`

Transaction `Hash`es are 32 bytes,
and the minimun transparent transaction size is 54 bytes.
So a full 2MB block can create 1.1MB of transaction hashes.

We use an `Arc` to avoid repeatedly cloning that much data.

* Remove the unused `Block` from `ChainTipBlock`

This drops the block as soon as it isn't needed any more.

Previously, it would stick around until every `ChainTipReceiver`
dropped their `ChainTipBlock`, even if they didn't use the `Block`
at all.
2021-08-30 15:42:07 -03:00
teor 2e1d857b27
Add transaction IDs to the chain tip channel (#2686)
* Re-use finalized blocks for chain tip updates

This avoids serializing and deserializing blocks from the finalized state.

* Optimise tip sender equality checks

* Re-use precalculated block hashes and heights for chain tip updates

* Add chain tip mined transaction IDs

* Doc comment typo

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-08-30 12:38:41 +10:00
teor f9c90b3d86
Implement best tip block hashes for ChainTip receivers (#2677)
* Always prefer the non-finalized tip in ChainTipSender

This significantly simplifies the internal implementation of ChainTipSender.

Also make the methods and types a bit more generic.

* Update ChainTipSender with blocks, not heights

Also fix a bug where queued non-finalized blocks would clear the chain tip.

* Provide a best tip hash in ChainTip receivers

* Skip finalized blocks once the non-finalized state is active

* Add tip hash and NoChainTip tests

* Remove a redundant finalized tip update

* Skip `None` updates to the finalized tip

The finalized and non-finalized tips never update to `None`
once they have added at least one block.

* Stop committing finalized queued blocks if there is an error

Also return the highest committed queued block.

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-08-27 19:18:47 +00:00
teor d2e14b22f9
Refactor BestTipHeight into a generic ChainTip sender and receiver (#2676)
* Rename BestTipHeight so it can be generalised to ChainTipSender

`fastmod BestTipHeight ChainTipSender zebra*`

For senders:
`fastmod best_tip_height chain_tip_sender zebra*`

For receivers:
`fastmod best_tip_height chain_tip_receiver zebra*`

* Rename best_tip_height module to chain_tip

* Wrap the chain tip watch channel in a ChainTipReceiver type

* Create a ChainTip trait to avoid tricky crate dependencies

And add convenience impls for optional and empty chain tips.

* Use the ChainTip trait in zebra-network

* Replace `Option<ChainTip>` with `NoChainTip`

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-08-27 11:34:33 +10:00
teor d7eb01d7f0
Add value pool/balances to non-finalized state (#2656)
* add value balances to non finalized state

* fix 2 tests

* fix remaining constrain issues in tests

* extend value pool test to non finalized

* WIP: fix tests after adding value pools to non-finalized state (#2647)

* Update Chain::eq_internal_state with Chain.value_balance

Also increase the number of cases in its tests,
because they didn't detect this bug.

* Calculate the chain value pool change before `Chain::push`

Code
- store the chain value pool change in `ContextuallyValidBlock`
- convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos`
  (rather than `from` or `into`)
- replace `block_utxos` with `new_outputs` in `PreparedBlock`
- replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock`

Tests
- create test methods for `PreparedBlock` and `ContextuallyValidBlock`
- use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos`
  to make tests pass

* fix conflicts

* build `set_current_value_pool()` only for tests

* remove redundant cfgs

* change cfg of set_current_value_pool()

* Clarify some chain field documentation

* Fix bugs in the non-finalized chain value pool calculations

1. Only revert the chain value pool balances when the tip is popped.
   Don't modify them when the root is finalized.

2. Only update or revert the chain value pool balances once per block.
   (Previously, the block changes were multiplied by the number of *transactions*.)

And make corresponding changes to method names and documentation.

* Add extra proptests to try to identify value balance failures

* Simplify some transaction generation code

* Add extra debugging info to value balance errors

* Actually update non-finalized chain value pools in `UpdateWith`

Previously, we were dropping the updated value pools in the `Ok` result.
So the initial (finalized) chain value pool balances were never modified.

* Rename and document value balance add methods

The names and documentation of these methods were confusing.

* Create genesis-based proptests that check chain value pools

* Increase coverage for some test vectors

* Test each chain value balance calculation for blocks 0-10

* Make continuous blockchain test errors easier to debug

* Test the exact transparent pool values for the first few blocks

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 10:57:07 -03:00
Conrado Gouvea bc4194fcb9
ZIP-221/244 auth data commitment validation in checkpoint verifier (#2633)
* Add validation of ZIP-221 and ZIP-244 commitments

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Add auth commitment check in the finalized state

* Reset the verifier when comitting to state fails

* Add explanation comment

* Add test with fake activation heights

* Add generate_valid_commitments flag

* Enable fake activation heights using env var instead of feature

* Also update initial_tip_hash; refactor into progress_from_tip()

* Improve comments

* Add fake activation heights test to CI

* Fix bug that caused commitment trees to not match when generating partial arbitrary chains

* Add ChainHistoryBlockTxAuthCommitmentHash::from_commitments to organize and deduplicate code

* Remove stale comment, improve readability

* Allow overriding with PROPTEST_CASES

* partial_chain_strategy(): don't update note commitment trees when not needed; add comment

Co-authored-by: teor <teor@riseup.net>
2021-08-23 14:17:33 +00:00
teor e6dc03329e
Remove a finalized state test dependency on ContextuallyValidBlock (#2648) 2021-08-20 09:59:12 -03:00
teor 2ec6ae364f
Improve some test errors (#2643)
* Improve error logging in rejection_restores_internal_state

* Summarise history tree peaks in debug output

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-19 19:44:02 +00:00
Alfredo Garcia d2e417cf48
Add value pools to FinalizedState (#2599)
* add value pools to the database

* remove redundant genesis block check

* use update_with_chain_value_pool_change()

* remove constrains

* remove height from the database

* remove calls to chain_value_pool_change

* clippy

* use the "correct" value balances

* bump the database format

* remove everything that is not finalized state

* clippy

* rustfmt

* use all spent utxos

* add new_outputs utxos to all_utxos_spent_by_block

* remove panic

* add finalized state value pool test

* clippy

* clippy 2

* move import

* fix import

* rustfmt

Co-authored-by: teor <teor@riseup.net>
2021-08-19 13:55:36 -03:00
Conrado Gouvea 5c5abf6171
ZIP-221 and ZIP-244 commitment validation in non-finalized state (#2609)
* Add validation of ZIP-221 and ZIP-244 commitments

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2021-08-17 11:49:27 -03:00
teor 76591ceeed
Generate test chains with valid chain value pools (#2597)
* Generate chains with valid chain value pool balances

* Move MAX_PARTIAL_CHAIN_BLOCKS to zebra-chain

* Fix generated value overflow based on the maximum number of values

And split it into its own method.

* Split fix_remaining_value into smaller methods

* Remove unused methods

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-12 12:38:16 +00:00
Conrado Gouvea 94175c6955
Add ZIP-221 history tree to non-finalized state (#2583)
* Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree

* HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq

* ZIP-221: Validate chain history commitments in the non-finalized state (#2301)

* sketch of implementation

* refined implementation; still incomplete

* update librustzcash, change zcash_history to work with it

* simplified code per review; renamed MMR to HistoryTree

* expand HistoryTree implementation

* handle and propagate errors

* simplify check.rs tracing

* add suggested TODO

* add HistoryTree::prune

* fix bug in pruning

* fix compilation of tests; still need to make them pass

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* improvements from code review

* improve check.rs comments and variable names

* fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests

* fix finalized_state proptest

* fix non_finalized_state tests by setting the correct commitments

* renamed mmr.rs to history_tree.rs

* Add HistoryTree struct

* expand non_finalized_state protest

* fix typo

* Add HistoryTree struct

* Update zebra-chain/src/primitives/zcash_history.rs

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* fix formatting

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* history_tree.rs: fixes from code review

* fixes to work with updated HistoryTree

* Improvements from code review

* Add Debug implementations to allow comparing Chains with proptest_assert_eq

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Improvements from code review

* Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 10:42:40 -03:00
Conrado Gouvea eac95bdf10
Cache note commitment tree roots (#2584)
* Cache note commitment tree roots

* Add comments to cached root fields

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2021-08-10 10:33:34 -03:00
Conrado Gouvea 9fc49827d6
Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree (#2582)
* Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree

* HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq
2021-08-10 09:51:50 -03:00
teor f09f2a9022
Check remaining transaction value & make value balance signs match the spec (#2566)
* Make Amount arithmetic more generic

To modify generated amounts, we need some extra operations on `Amount`.

We also need to extend existing operations to both `NonNegative` and
`NegativeAllowed` amounts.

* Add a constrain method for ValueBalance

* Derive Eq for ValueBalance

* impl Neg for ValueBalance

* Make some Amount arithmetic expectations explicit

* Explain why we use i128 for multiplication

And expand the overflow error details.

* Expand Amount::sum error details

* Make amount::Error field order consistent

* Rename an amount::Error variant to Constraint, so it's clearer

* Add specific pool variants to ValueBalanceError

* Update coinbase remaining value consensus rule comment

This consensus rule was updated recently to include coinbase transactions,
but Zebra doesn't check block subsidy or miner fees yet.

* Add test methods for modifying transparent values and shielded value balances

* Temporarily set values and value balances to zero in proptests

In both generated chains and proptests that construct their own transactions.

Using zero values reduces value calculation and value check test coverage.
A future change will use non-zero values, and fix them so the check passes.

* Add extra fields to remaining transaction value errors

* Swap the transparent value balance sign to match shielded value balances

This makes the signs of all the chain value pools consistent.

* Use a NonNegative constraint for transparent values

This fix:
* makes the type signature match the consensus rules
* avoids having to write code to handle negative values

* Allocate total generated transaction input value to outputs

If there isn't enough input value for an output, set it to zero.

Temporarily reduce all generated values to avoid overflow.
(We'll remove this workaround when we calculate chain value balances.)

* Consistently use ValueBalanceError for ValueBalances

* Make the value balance signs match the spec

And rename and document methods so their signs are clearer.

* Convert amount::Errors to specific pool ValueBalanceErrors

* Move some error changes to the next PR

* Add extra info to remaining transaction value errors (#2585)

* Distinguish between overflow and negative remaining transaction value errors

And make some error types cloneable.

* Add methods for updating chain value pools (#2586)

* Move amount::test to amount::tests:vectors

* Make ValueBalance traits more consistent with Amount

- implement Add and Sub variants with Result and Assign
- derive Hash

* Clarify some comments and expects

* Create ValueBalance update methods for blocks and transactions

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-09 14:22:26 -03:00
Alfredo Garcia 910f0ff5dc
Add Chain ValueBalance serialization (#2554)
* add serialization for value balances

* change test names

* change panic messages

* add a deserialization test

* return the errors from `from_bytes()` methods

* add prop test for serialize/deserialize Amount

* Move amount proptests to amount::tests::prop

Co-authored-by: teor <teor@riseup.net>
2021-08-09 11:22:27 +10:00