Commit Graph

376 Commits

Author SHA1 Message Date
teor 7d8240fac3
Fix verbose add_initial_peers logs (#3019)
And update some function docs.
2021-11-07 22:21:51 +00:00
teor b1303ab8d7
Replace read_compactsize and write_compactsize with CompactSizeMessage (#3014)
* Replace read_compactsize and write_compactsize with CompactSizeMessage

* Add tests for CompactSize64

* Add compact size range and conversion tests
2021-11-05 15:24:24 -03:00
Janito Vaqueiro Ferreira Filho 0960e4fb0b
Update to Tokio 1.13.0 (#2994)
* Update `tower` to version `0.4.9`

Update to latest version to add support for Tokio version 1.

* Replace usage of `ServiceExt::ready_and`

It was deprecated in favor of `ServiceExt::ready`.

* Update Tokio dependency to version `1.13.0`

This will break the build because the code isn't ready for the update,
but future commits will fix the issues.

* Replace import of `tokio::stream::StreamExt`

Use `futures::stream::StreamExt` instead, because newer versions of
Tokio don't have the `stream` feature.

* Use `IntervalStream` in `zebra-network`

In newer versions of Tokio `Interval` doesn't implement `Stream`, so the
wrapper types from `tokio-stream` have to be used instead.

* Use `IntervalStream` in `inventory_registry`

In newer versions of Tokio the `Interval` type doesn't implement
`Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used
instead.

* Use `BroadcastStream` in `inventory_registry`

In newer versions of Tokio `broadcast::Receiver` doesn't implement
`Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This
also requires changing the error type that is used.

* Handle `Semaphore::acquire` error in `tower-batch`

Newer versions of Tokio can return an error if the semaphore is closed.
This shouldn't happen in `tower-batch` because the semaphore is never
closed.

* Handle `Semaphore::acquire` error in `zebrad` test

On newer versions of Tokio `Semaphore::acquire` can return an error if
the semaphore is closed. This shouldn't happen in the test because the
semaphore is never closed.

* Update some `zebra-network` dependencies

Use versions compatible with Tokio version 1.

* Upgrade Hyper to version 0.14

Use a version that supports Tokio version 1.

* Update `metrics` dependency to version 0.17

And also update the `metrics-exporter-prometheus` to version 0.6.1.
These updates are to make sure Tokio 1 is supported.

* Use `f64` as the histogram data type

`u64` isn't supported as the histogram data type in newer versions of
`metrics`.

* Update the initialization of the metrics component

Make it compatible with the new version of `metrics`.

* Simplify build version counter

Remove all constants and use the new `metrics::incement_counter!` macro.

* Change metrics output line to match on

The snapshot string isn't included in the newer version of
`metrics-exporter-prometheus`.

* Update `sentry` to version 0.23.0

Use a version compatible with Tokio version 1.

* Remove usage of `TracingIntegration`

This seems to not be available from `sentry-tracing` anymore, so it
needs to be replaced.

* Add sentry layer to tracing initialization

This seems like the replacement for `TracingIntegration`.

* Remove unnecessary conversion

Suggested by a Clippy lint.

* Update Cargo lock file

Apply all of the updates to dependencies.

* Ban duplicate tokio dependencies

Also ban git sources for tokio dependencies.

* Stop allowing sentry-tracing git repository in `deny.toml`

* Allow remaining duplicates after the tokio upgrade

* Use C: drive for CI build output on Windows

GitHub Actions uses a Windows image with two disk drives, and the
default D: drive is smaller than the C: drive. Zebra currently uses a
lot of space to build, so it has to use the C: drive to avoid CI build
failures because of insufficient space.

Co-authored-by: teor <teor@riseup.net>
2021-11-02 18:46:57 +00:00
Conrado Gouvea e54917ae7c
V1.0.0-beta.0 (#2973)
* V1.0.0-beta.0

* Bump version in install.md
2021-10-29 20:21:26 +00:00
Marek 4f7a977565
Test multiple chain resets (#2897)
* Try simulating a chain growth

* Adjust the transaction expiry height

The mempool evicts expired transactions. When working with mocked data,
appending a new block typically clears the mempool because transactions become
expired. For this reason, the expiry height of each transactions is adjusted so
that it is greater than the new chain tip's height.

* Refactor the code so that it works with `VerifiedUnminedTx`

* Fix a typo

* Fix clippy warnings

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-10-22 02:54:08 +00:00
teor 67327ac462
Downgrade some less interesting info-level logs to debug (#2938)
There are a lot of these messages when Zebra starts up.
They might be slowing down CI and causing timeouts.

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-10-22 02:11:09 +00:00
Marek d2a5af0ea5
V1.0.0 alpha.19 (#2907)
* Increment the crates that have new commits since the last version

* Increment the crates that depend on crates that have changed

* Increment the version of `zebra-script`

* Use the `zebrad` version in the `zebra-network` user agent string

* Use the `v1.0.0-alpha.19` git tag in `README.md`

* Copy the draft changelog into `CHANGELOG.md`

* Delete bumps

* Update CHANGELOG.md

Co-authored-by: teor <teor@riseup.net>

* Add newly merged PRs

Co-authored-by: teor <teor@riseup.net>
2021-10-21 12:33:35 +02:00
teor e277975d85
Try flushing streams before exiting Zebra (#2911) 2021-10-20 13:57:09 +00:00
teor 92634f788b
Increase Zebra's restart acceptance test timeout (#2910)
* Increase the restart test timeout to 10 seconds

It shouldn't take this long.
But maybe the CI VMs are under a lot of load?

* Add extensive logging to diagnose CI state reload failures
2021-10-20 01:59:09 +00:00
dependabot[bot] 7a5f419cb6
Bump thiserror from 1.0.29 to 1.0.30 (#2856)
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.29 to 1.0.30.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.29...1.0.30)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-10-11 20:36:44 +00:00
teor f4118dadda
Make block metrics more accurate (#2835)
* Check for state errors before updating metrics

Previously, the metrics would be updated for some rejected blocks.

* Clarify and expand block verification metrics

Rename checkpoint-specific metrics to clarify their purpose.

Add metrics for:
- finalized blocks on disk
- blocks verified using the full block verifier
  (this metric was previously incorrectly called `zcash_chain_verified_block_height`)

* Update dashboard metric names

Also:
- add some extra block height metrics
- fix a dashboard name

* Add exact block heights to Grafana dashboards

* Add a missing comment

* grafana: use 0 decimals for metrics

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* grafana: show the entire height instead of abbreviated

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* grafana: show the entire height instead of abbreviated

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Fix typo in metric name

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Move height gauges to the state, so they are correct

If we update height gauges in futures, they can execute out of order,
so the metrics can be incorrect.

Instead:
- move the height gauges to the state, and update them based on the best tip
- move the verified block counts to the state
- continue to include all verified blocks on all non-finalized chains
  (not just the best chain)

* Show exact checkpoint heights in the dashboard

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-07 13:42:38 +00:00
teor 04d2cfb3d0
Gossip recently verified block hashes to peers (#2729)
* Implement a task that gossips verified block hashes

* Log an info message for block broadcasts

* Simplify the gossip task

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Re-use the old tip change if there is no new tip change

Also improve the comments.

* Add an assertion message

* Rename task join handles and futures in start method

* Add a dedicated BlockGossipError type

This type helps distinguish between syncer and state errors.

* Test that committed blocks are gossiped to peers

Also do a minor type cleanup on the existing test code,
replacing `Option<Vec<_>>` with `Vec<_>`.

* Formatting

* Remove excess newlines

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Clear the initial gossiped blocks during test setup

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-10-07 07:46:37 -03:00
teor 0b82298645 Stop dropping queued block senders without sending a response
This causes a rare panic, because Zebra expects every queued sender
to send a response.
2021-10-07 00:12:45 -04:00
teor c7c1a8a469 Remove known UTXOs when pruning blocks 2021-10-07 00:12:45 -04:00
Pili Guerra a85e80a0ec
Update versions for zebra v1.0.0-alpha.18 release (#2828)
* Update versions for zebra v1.0.0-alpha.18 release

* WIP: Initial PR list

* Remove uninteresting version bumps from CHANGELOG

* Categorise and group PRs in CHANGELOG, removing uninteresting PRs

* Further refine and categorise changelog entries

* Fix tag url

* Final changes to CHANGELOG

* Add a changelog description

* Spacing

* Clarify and fix changelog PR descriptions

* Add PRs that are about to be merged

* More slight clarifications

* Spacing

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-10-05 23:22:31 -03:00
Janito Vaqueiro Ferreira Filho 339fefb6e6
Update shared NU5 dependencies, set the NU5 testnet activation network upgrade parameters (#2825)
* Check return value of zcash_script_new_precomputed_tx

* Set the NU5 testnet activation height to 1_590_000

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Update Nu5 constants to new values

* Update ZIP-244 test vectors for new branch ID

* Squashed commit of the following:

commit bdb120a249
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date:   Tue Oct 5 11:54:01 2021 -0400

    Use pallas::Base::from_str_vartime() in sinsemilla tests

commit e99fa49258
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date:   Tue Oct 5 11:45:24 2021 -0400

    Compiles

commit a520018114
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date:   Tue Oct 5 10:15:17 2021 -0400

    Incomplete upgrade of deps

* Squashed commit of the following:

commit 8d1b76ec5626517817c3a4d9f3950acc90a359df
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date:   Tue Oct 5 04:02:26 2021 +0000

    Update `zcash_script` to support V5 transactions

    Use a newer version of `zcash_script` that has been updated to support
    V5 transactions.

commit 371233628ae61e0c25d6ba8f31d9dba42823becb
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date:   Tue Oct 5 03:06:20 2021 +0000

    Update Zcash dependencies

    Update some Zcash crates:

    - `halo2`
    - `incrementalmerkletree' (patch version)
    - `orchard` (patch version)
    - `zcash_history` (patch version)
    - `zcash_note_encryption` (patch version)
    - `zcash_primitives` (patch version)

    And also update the `group` dependency so that the code remains
    compatible.

commit de5cf1ec40c3fc08670fc971cdf3e65e13d9f4c7
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date:   Tue Oct 5 03:04:13 2021 +0000

    Update error message assertion

    Use the updated message for the expected error variant.

* Update `zcash_script` to support V5 transactions

Use a newer version of `zcash_script` that has been updated to support
V5 transactions.

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
2021-10-06 11:08:41 +10:00
teor e5f5ac9ce8
Fix or disable recent nightly clippy lints (#2817)
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-01 15:26:06 +00:00
Janito Vaqueiro Ferreira Filho 50a5728d0b
Test if the mempool storage is cleared (#2815)
* Move mempool tests into `tests::vector` sub-module

Make it consistent with other test modules and prepare for adding
property tests.

* Reorder imports

Make it consistent with the general guidelines followed on other
modules.

* Export `ChainTipBlock` and `ChainTipSender`

Allow these types to be used in other crates for testing purposes.

* Derive `Arbitrary` for `ChainTipBlock`

Make it easy to generate random `ChainTipBlock`s for usage in property
tests.

* Refactor to move test methods into `tests` module

Reduce the repeated test configuration attributes and make it easier to
see what is test specific and what is part of the general
implementation.

* Add a `Mempool::dummy_call` test helper method

Performs a dummy call just so that `poll_ready` gets called.

* Use `dummy_call` in existing tests

Replace the custom dummy requests with the helper method.

* Test if the mempool is cleared on chain reset

A chain reset should force the mempool storage to be cleared so that
transaction verification can restart using the new chain tip.

* Test if mempool is cleared on syncer restart

If the block synchronizer falls behind and then starts catching up
again, the mempool should be disabled and therefore the storage should
be cleared.
2021-10-01 14:44:25 +00:00
teor 1601c9fbb3
Stop panicking when using sync and async methods on the same ChainTipChange (#2800)
* Instrument chain tip methods

* Expand tests to cover last_tip_change and multiple change checks

* Expand tests to cover Grow as well as Reset

* Support sync and async methods on the same ChainTipChange

* Add a Tokio 1.0 TODO

* Clarify a comment

* Manual rustfmt inside a proptest

* Remove tracing clones, and instrument ChainTipSender::new

* Add the tokio issue number to a TODO comment
2021-09-28 07:48:19 -03:00
teor 4567701933
Fix an incorrect assertion when the block locator is at the tip (#2789)
* Fix an incorrect assertion when the block locator is at the tip

This might have been triggered by receiving block hash gossips
from the new Zebra code.

* Add missing tests for zebra-state requests and responses

Specifically:
* `BlockLocator` (populated state only)
* `FindBlockHashes`
* `FindBlockHeaders`

* Test `FindBlock*` before and after the current block

* Add a specific test for bug #2789

* Refactor collect_best_chain_hashes to avoid manual index calculations

* Reword a comment

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-09-27 22:43:05 +00:00
Marek 30c9618207
Clear mempool at a network upgrade (#2773)
* Update the expiry TODO

* Clear the mempool at a chain tip reset

* Clear the mempool by using a sync method (#2777)

* Clear the mempool by using a sync method

* Update docs

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Refactor last_tip_change()

* Apply suggestions from code review

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Fix brackets

* Use best_tip_block instead of manual borrowing

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-09-23 19:09:44 +00:00
dependabot[bot] e2cc6e12f8 Bump dirs from 3.0.2 to 4.0.0
Bumps [dirs](https://github.com/soc/dirs-rs) from 3.0.2 to 4.0.0.
- [Release notes](https://github.com/soc/dirs-rs/releases)
- [Commits](https://github.com/soc/dirs-rs/commits)

---
updated-dependencies:
- dependency-name: dirs
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-22 20:15:32 -04:00
Marek 061ad55144
Sneak chain_tip_change into mempool (#2785)
* Pass ChainTipChange to the mempool

* Fix nits
2021-09-21 17:06:52 +00:00
Conrado Gouvea 8971d62727
Update version strings for v1.0.0 alpha.17 release (#2750)
* Change versions for v1.0.0-alpha.17 release
2021-09-14 17:41:50 +00:00
dependabot[bot] af8dd3e4b1 Bump thiserror from 1.0.28 to 1.0.29
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.28 to 1.0.29.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.28...1.0.29)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 15:41:24 -04:00
teor 44ac06775b
Reset `ChainTipChange`s on chain fork and network upgrade activation (#2721)
* Use `TipAction::Reset` for initialization and out-of-order blocks

Needs tests for:
- cloning a `ChainTipChange` resets the cloned instance
- skipped updates reset the cloned instance
- changing forks resets the cloned instance

* Use `TipAction::Reset` for network upgrade activation blocks

* Use an `if` expression

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Another if expression

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-09-02 12:25:42 +10:00
Conrado Gouvea 1ccb2de7c7
Add transaction downloader and verifier (#2679)
* Add transaction downloader

* Changed mempool downloader to be like inbound

* Verifier working (logs result)

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Fix coinbase check for mempool, improve is_coinbase() docs

* Change other downloads.rs docs to reflect the mempool downloads.rs changes

* Change TIMEOUTs to downloads.rs; add docs

* Renamed is_coinbase() to has_valid_coinbase_transaction_inputs() and contains_coinbase_input() to has_any_coinbase_inputs(); reorder checks

* Validate network upgrade for V4 transactions; check before computing sighash (for V5 too)

* Add block_ prefix to downloads and verifier

* Update zebra-consensus/src/transaction.rs

Co-authored-by: teor <teor@riseup.net>

* Add consensus doc; add more Block prefixes

Co-authored-by: teor <teor@riseup.net>
2021-09-02 00:06:20 +00:00
teor b6fe816473
Add a `ChainTipChange` type to `await` chain tip changes (#2715)
* Rename ChainTipReceiver to CurrentChainTip

`fastmod ChainTipReceiver CurrentChainTip zebra*`

* Update chain tip documentation and variable names

* Basic chain tip change implementation, without resets

Also includes the following name changes:
```
fastmod CurrentChainTip LatestChainTip zebra*
fastmod chain_tip_receiver latest_chain_tip zebra*
```

* Clarify the difference between `LatestChainTip` and `ChainTipChange`
2021-09-01 22:31:16 +00:00
dependabot[bot] cdfc380d2b Bump thiserror from 1.0.25 to 1.0.28
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.25 to 1.0.28.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.25...1.0.28)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-08-31 22:19:04 -04:00
dependabot[bot] c9fbd0939f Bump futures from 0.3.15 to 0.3.17
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.15 to 0.3.17.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.15...0.3.17)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-08-31 18:08:51 -04:00
teor 2dac0dda47
Stop calculating transaction hashes twice in the checkpoint verifier (#2696)
* Stop precalculating transaction hashes twice during checkpointing

* Refactor a complex type using a new `RequestBlock` type

* Comment formatting

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-08-31 10:55:39 +10:00
teor a66ecbc16d
Make chain tip updates and access more efficient (#2695)
* Store precalculated transactions in an `Arc`

Transaction `Hash`es are 32 bytes,
and the minimun transparent transaction size is 54 bytes.
So a full 2MB block can create 1.1MB of transaction hashes.

We use an `Arc` to avoid repeatedly cloning that much data.

* Remove the unused `Block` from `ChainTipBlock`

This drops the block as soon as it isn't needed any more.

Previously, it would stick around until every `ChainTipReceiver`
dropped their `ChainTipBlock`, even if they didn't use the `Block`
at all.
2021-08-30 15:42:07 -03:00
teor 2e1d857b27
Add transaction IDs to the chain tip channel (#2686)
* Re-use finalized blocks for chain tip updates

This avoids serializing and deserializing blocks from the finalized state.

* Optimise tip sender equality checks

* Re-use precalculated block hashes and heights for chain tip updates

* Add chain tip mined transaction IDs

* Doc comment typo

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-08-30 12:38:41 +10:00
Alfredo Garcia 968f20d423
Update versions for zebra v1.0.0-alpha.16 release (#2670)
* bump crate versions

* update zebra-script

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-27 22:13:54 +00:00
teor f9c90b3d86
Implement best tip block hashes for ChainTip receivers (#2677)
* Always prefer the non-finalized tip in ChainTipSender

This significantly simplifies the internal implementation of ChainTipSender.

Also make the methods and types a bit more generic.

* Update ChainTipSender with blocks, not heights

Also fix a bug where queued non-finalized blocks would clear the chain tip.

* Provide a best tip hash in ChainTip receivers

* Skip finalized blocks once the non-finalized state is active

* Add tip hash and NoChainTip tests

* Remove a redundant finalized tip update

* Skip `None` updates to the finalized tip

The finalized and non-finalized tips never update to `None`
once they have added at least one block.

* Stop committing finalized queued blocks if there is an error

Also return the highest committed queued block.

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-08-27 19:18:47 +00:00
teor d2e14b22f9
Refactor BestTipHeight into a generic ChainTip sender and receiver (#2676)
* Rename BestTipHeight so it can be generalised to ChainTipSender

`fastmod BestTipHeight ChainTipSender zebra*`

For senders:
`fastmod best_tip_height chain_tip_sender zebra*`

For receivers:
`fastmod best_tip_height chain_tip_receiver zebra*`

* Rename best_tip_height module to chain_tip

* Wrap the chain tip watch channel in a ChainTipReceiver type

* Create a ChainTip trait to avoid tricky crate dependencies

And add convenience impls for optional and empty chain tips.

* Use the ChainTip trait in zebra-network

* Replace `Option<ChainTip>` with `NoChainTip`

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-08-27 11:34:33 +10:00
teor d7eb01d7f0
Add value pool/balances to non-finalized state (#2656)
* add value balances to non finalized state

* fix 2 tests

* fix remaining constrain issues in tests

* extend value pool test to non finalized

* WIP: fix tests after adding value pools to non-finalized state (#2647)

* Update Chain::eq_internal_state with Chain.value_balance

Also increase the number of cases in its tests,
because they didn't detect this bug.

* Calculate the chain value pool change before `Chain::push`

Code
- store the chain value pool change in `ContextuallyValidBlock`
- convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos`
  (rather than `from` or `into`)
- replace `block_utxos` with `new_outputs` in `PreparedBlock`
- replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock`

Tests
- create test methods for `PreparedBlock` and `ContextuallyValidBlock`
- use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos`
  to make tests pass

* fix conflicts

* build `set_current_value_pool()` only for tests

* remove redundant cfgs

* change cfg of set_current_value_pool()

* Clarify some chain field documentation

* Fix bugs in the non-finalized chain value pool calculations

1. Only revert the chain value pool balances when the tip is popped.
   Don't modify them when the root is finalized.

2. Only update or revert the chain value pool balances once per block.
   (Previously, the block changes were multiplied by the number of *transactions*.)

And make corresponding changes to method names and documentation.

* Add extra proptests to try to identify value balance failures

* Simplify some transaction generation code

* Add extra debugging info to value balance errors

* Actually update non-finalized chain value pools in `UpdateWith`

Previously, we were dropping the updated value pools in the `Ok` result.
So the initial (finalized) chain value pool balances were never modified.

* Rename and document value balance add methods

The names and documentation of these methods were confusing.

* Create genesis-based proptests that check chain value pools

* Increase coverage for some test vectors

* Test each chain value balance calculation for blocks 0-10

* Make continuous blockchain test errors easier to debug

* Test the exact transparent pool values for the first few blocks

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-25 10:57:07 -03:00
Conrado Gouvea bc4194fcb9
ZIP-221/244 auth data commitment validation in checkpoint verifier (#2633)
* Add validation of ZIP-221 and ZIP-244 commitments

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Add auth commitment check in the finalized state

* Reset the verifier when comitting to state fails

* Add explanation comment

* Add test with fake activation heights

* Add generate_valid_commitments flag

* Enable fake activation heights using env var instead of feature

* Also update initial_tip_hash; refactor into progress_from_tip()

* Improve comments

* Add fake activation heights test to CI

* Fix bug that caused commitment trees to not match when generating partial arbitrary chains

* Add ChainHistoryBlockTxAuthCommitmentHash::from_commitments to organize and deduplicate code

* Remove stale comment, improve readability

* Allow overriding with PROPTEST_CASES

* partial_chain_strategy(): don't update note commitment trees when not needed; add comment

Co-authored-by: teor <teor@riseup.net>
2021-08-23 14:17:33 +00:00
teor e6dc03329e
Remove a finalized state test dependency on ContextuallyValidBlock (#2648) 2021-08-20 09:59:12 -03:00
teor 2ec6ae364f
Improve some test errors (#2643)
* Improve error logging in rejection_restores_internal_state

* Summarise history tree peaks in debug output

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-19 19:44:02 +00:00
Alfredo Garcia d2e417cf48
Add value pools to FinalizedState (#2599)
* add value pools to the database

* remove redundant genesis block check

* use update_with_chain_value_pool_change()

* remove constrains

* remove height from the database

* remove calls to chain_value_pool_change

* clippy

* use the "correct" value balances

* bump the database format

* remove everything that is not finalized state

* clippy

* rustfmt

* use all spent utxos

* add new_outputs utxos to all_utxos_spent_by_block

* remove panic

* add finalized state value pool test

* clippy

* clippy 2

* move import

* fix import

* rustfmt

Co-authored-by: teor <teor@riseup.net>
2021-08-19 13:55:36 -03:00
Conrado Gouvea 5c5abf6171
ZIP-221 and ZIP-244 commitment validation in non-finalized state (#2609)
* Add validation of ZIP-221 and ZIP-244 commitments

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2021-08-17 11:49:27 -03:00
Pili Guerra 234953e620
Update versions for zebra v1.0.0-alpha.15 release (#2612) 2021-08-16 10:06:26 +00:00
teor 76591ceeed
Generate test chains with valid chain value pools (#2597)
* Generate chains with valid chain value pool balances

* Move MAX_PARTIAL_CHAIN_BLOCKS to zebra-chain

* Fix generated value overflow based on the maximum number of values

And split it into its own method.

* Split fix_remaining_value into smaller methods

* Remove unused methods

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-12 12:38:16 +00:00
teor a176c499ab Make zebra-state compile successfully by itself
The zebra-state integration test compiled successfully during workspace
builds.

But it couldn't find the `init_test` function when the crate was built
by itself. This happens because the function is only active for `test`
and `proptest-impl` builds.

This commit removes the dependency on that function.
2021-08-11 21:09:32 -04:00
Conrado Gouvea 94175c6955
Add ZIP-221 history tree to non-finalized state (#2583)
* Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree

* HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq

* ZIP-221: Validate chain history commitments in the non-finalized state (#2301)

* sketch of implementation

* refined implementation; still incomplete

* update librustzcash, change zcash_history to work with it

* simplified code per review; renamed MMR to HistoryTree

* expand HistoryTree implementation

* handle and propagate errors

* simplify check.rs tracing

* add suggested TODO

* add HistoryTree::prune

* fix bug in pruning

* fix compilation of tests; still need to make them pass

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* improvements from code review

* improve check.rs comments and variable names

* fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests

* fix finalized_state proptest

* fix non_finalized_state tests by setting the correct commitments

* renamed mmr.rs to history_tree.rs

* Add HistoryTree struct

* expand non_finalized_state protest

* fix typo

* Add HistoryTree struct

* Update zebra-chain/src/primitives/zcash_history.rs

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* fix formatting

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* history_tree.rs: fixes from code review

* fixes to work with updated HistoryTree

* Improvements from code review

* Add Debug implementations to allow comparing Chains with proptest_assert_eq

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Improvements from code review

* Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-11 10:42:40 -03:00
Conrado Gouvea eac95bdf10
Cache note commitment tree roots (#2584)
* Cache note commitment tree roots

* Add comments to cached root fields

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2021-08-10 10:33:34 -03:00
Conrado Gouvea 9fc49827d6
Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree (#2582)
* Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree

* HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq
2021-08-10 09:51:50 -03:00
teor f09f2a9022
Check remaining transaction value & make value balance signs match the spec (#2566)
* Make Amount arithmetic more generic

To modify generated amounts, we need some extra operations on `Amount`.

We also need to extend existing operations to both `NonNegative` and
`NegativeAllowed` amounts.

* Add a constrain method for ValueBalance

* Derive Eq for ValueBalance

* impl Neg for ValueBalance

* Make some Amount arithmetic expectations explicit

* Explain why we use i128 for multiplication

And expand the overflow error details.

* Expand Amount::sum error details

* Make amount::Error field order consistent

* Rename an amount::Error variant to Constraint, so it's clearer

* Add specific pool variants to ValueBalanceError

* Update coinbase remaining value consensus rule comment

This consensus rule was updated recently to include coinbase transactions,
but Zebra doesn't check block subsidy or miner fees yet.

* Add test methods for modifying transparent values and shielded value balances

* Temporarily set values and value balances to zero in proptests

In both generated chains and proptests that construct their own transactions.

Using zero values reduces value calculation and value check test coverage.
A future change will use non-zero values, and fix them so the check passes.

* Add extra fields to remaining transaction value errors

* Swap the transparent value balance sign to match shielded value balances

This makes the signs of all the chain value pools consistent.

* Use a NonNegative constraint for transparent values

This fix:
* makes the type signature match the consensus rules
* avoids having to write code to handle negative values

* Allocate total generated transaction input value to outputs

If there isn't enough input value for an output, set it to zero.

Temporarily reduce all generated values to avoid overflow.
(We'll remove this workaround when we calculate chain value balances.)

* Consistently use ValueBalanceError for ValueBalances

* Make the value balance signs match the spec

And rename and document methods so their signs are clearer.

* Convert amount::Errors to specific pool ValueBalanceErrors

* Move some error changes to the next PR

* Add extra info to remaining transaction value errors (#2585)

* Distinguish between overflow and negative remaining transaction value errors

And make some error types cloneable.

* Add methods for updating chain value pools (#2586)

* Move amount::test to amount::tests:vectors

* Make ValueBalance traits more consistent with Amount

- implement Add and Sub variants with Result and Assign
- derive Hash

* Clarify some comments and expects

* Create ValueBalance update methods for blocks and transactions

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-09 14:22:26 -03:00
Alfredo Garcia 910f0ff5dc
Add Chain ValueBalance serialization (#2554)
* add serialization for value balances

* change test names

* change panic messages

* add a deserialization test

* return the errors from `from_bytes()` methods

* add prop test for serialize/deserialize Amount

* Move amount proptests to amount::tests::prop

Co-authored-by: teor <teor@riseup.net>
2021-08-09 11:22:27 +10:00