Commit Graph

86 Commits

Author SHA1 Message Date
Arya 99b017e2a4
add(consensus): Add a `target_difficulty_limit` field on `testnet::Parameters` (#8518)
* Adds a `target_difficulty_limit` field on `testnet::Parameters`

* updates test to increment block nonce until finding a block that's below the difficulty threshold

* increment the nonce while the difficulty is invalid instead of while the difficulty threshold is invalid

* Adds comments
2024-05-17 23:06:08 +00:00
Arya d84d7356dd
add(consensus): Adds `disable_pow` and `genesis_hash` fields, converts `equihash::Solution` to an enum (#8421)
* minor cleanup and rename

* Adds an empty NetworkParameters struct to Network::Testnet variant, updates production code.

* Updates tests

* Adds `NetworkKind` and uses it instead of `Network` in `HistoryTreeParts` and `transparent::Address`

* Adds a [network.testnet_parameters] section to the config, uses `NetworkKind` as zebra_network::Config::network field type, and converts 'Network' to `NetworkKind` before serializing

* Applies some suggestions from code review

* Applies suggestions from code review

* returns b58 prefix constants directly to remove From<NetworkKind> impl for zcash_primitives::consensus::Network

* Applies more suggestions from code review.

* moves conversions to zcash_primitives::consensus::Network to where they're used.

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

* rename `network` variables and method names typed as NetworkKind to `network_kind`

* use only test block heights for the network associated with them

* Applies more suggestions from code review.

* Rename `NetworkParameters` to `Parameters` and move it a new `testnet` module

* adds activation heights field

* updates stored test config, adds a quicker test for checking that stored configs can be parsed, adds an intermediate representation of activation heights

* implement Parameters for Network

* Passes &Network directly instead of converting to zp_consensus::Network where there were conversions

* fixes a bad merge (removes a network conversion in zcash_note_encryption)

* Adds a test for the Parameters impl for zebra_chain::Network

* fixes doc links

* - Makes the `activation_heights` config field optional by adding a #[serde(default)]
- Panics if a non-zero activation height is provided for the `Genesis` network upgrade
- Always sets the `Genesis` and `BeforeOverwinter` network upgrade activation heights to 0 and 1, `BeforeOverwinter` could be overwritten by a later network upgrade
- Makes the `activation_heights` field on `Parameters` private, adds/uses an accessor method instead, and adds a builder struct and `build()` method

* small refactor of activation_heights() method

* check that activation heights are in the right order

* Updates `NetworkUpgrade::activation_height()` to return the height of the next NetworkUpgrade if it doesn't find the activation height of `&self`

* checks that the miner address is of TestnetKind on Regtest and update assertion message

* Adds a DNetworkUpgradeActivationHeights struct for better control over how activation heights can be configured

* moves all ordered network upgrades to a constant, adds a no_duplicates test, moves struct with activation heights outside deserialization impl and accepts it in `ParametersBuilder::activation_heights()` instead of a Vec

* panics if any network upgrades are configured to activate at Height(0) because it's reserved for Genesis

* Simplifies the `ParametersBuilder::activation_heights()` method and removes an unnecessary test.

* Adds Sapling HRPs as fields on testnet params. (#8398)

* Applies suggestions from code review.

* Update zebra-chain/src/parameters/network_upgrade.rs

Co-authored-by: Marek <mail@marek.onl>

* Adds `network_name` field and accessor method on `Parameters`, uses it in the `Display` impl for `Network`

* Removes unnecessary `.filter()`

* adds constraints on valid network names and a test

* adds config field for setting network name, adds "with_" prefix to ParameterBuilder setter methods

* Adds `MainnetKind`, `TestnetKind`, and `RegtestKind` to reserved network names

* updates stored test configs and fixes `network_name()` docs

* Adds a `new_regtest()` method on `Network` and `testnet::Parameters`, updates config deserialization to return an error if the initial_testnet_peers include any default initial peers AND contain configured activation heights

* Updates `activates_network_upgrades_correctly` test to check Regtest activation heights (and default Mainnet/Testnet)

* Refactors if-let & match statement into general match statement, removes constraint against including `testnet_parameters` field/section in the config when using `Mainnet` or `Regtest`

* Removes outdated TODO

* Restores `testnet_parameters` section of the latest stored config.

* Adds `with_sapling_hrps()` method and uses it to set the Regtest HRPs in `new_regtest()`.

Adds a test for Sapling HRP validation

* Checks that default Mainnet/Testnet/Regtest Sapling HRPs pass validation in `with_sapling_hrps()`

* Uses the correct constant in test

* Adds `is_regtest()` methods

* Adds genesis hash methods and fields to `testnet::Parameters` and its builder

* Copies Regtest genesis block from zcashd

* moves genesis hash check to Network.checkpoint_list()

* Checks the first line in the checkpoint list instead of the first item in the BTreeMap

* Checks that there is _some_ genesis block in the checkpoints list in `from_list()`, adds Regtest checkpoints (which only includes the genesis block)

* Adds a doc comment to `ParametersBuilder::default()`

* Adds a `disable_pow` field and skips checking proof of work if it's true

* Makes `equihash::Solution` an enum to support Regtest solutions, adds a test for validating and committing the Regtest genesis block

* use genesis_hash as first checkpoint when checkpoints are missing for configured testnets too

* Applies suggestions from code review.

* Reverts changes to `activation_height()` method

* Avoids panic in test

* Updates test docs

* drop custom panic hooks after expected panics

---------

Co-authored-by: Marek <mail@marek.onl>
2024-04-26 05:30:36 +00:00
idky137 4579722833
change(chain): Remove `Copy` trait impl from `Network` (#8354)
* removed derive copy from network, address and ledgerstate

* changed is_max_block_time_enforced to accept ref

* changed NetworkUpgrade::Current to accept ref

* changed NetworkUpgrade::Next to accept ref

* changed NetworkUpgrade::IsActivationHeight to accept ref

* changed NetworkUpgrade::TargetSpacingForHeight to accept ref

* changed NetworkUpgrade::TargetSpacings to accept ref

* changed NetworkUpgrade::MinimumDifficultySpacing_forHeight to accept ref

* changed NetworkUpgrade::IsTestnetMinDifficultyBlock to accept ref

* changed NetworkUpgrade::AveragingWindowTimespanForHeight to accept ref

* changed NetworkUpgrade::ActivationHeight to accept ref

* changed sapling_activation_height to accept ref

* fixed lifetime for target_spacings

* fixed sapling_activation_height

* changed transaction_to_fake_v5 and fake_v5_transactions_for_network to accept ref to network

* changed Input::vec_strategy to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_history.rs to accept ref to network

* changed functions in zebra-chain/src/history_tree.rs to accept ref to network

* changed functions in zebra-chain/src/history_tree.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/address.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/viewing_key* to accept ref to network

* changed functions in zebra-chain/src/transparent/address.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_primitives.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_note_encryption.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/history_tree* to accept ref to network

* changed functions in zebra-chain/src/block* to accept ref to network

* fixed errors in zebra-chain::parameters::network

* fixed errors in zebra-chain::parameters::network

* fixed errors in zebra-chain

* changed NonEmptyHistoryTree and InnerHistoryTree to hold value instead of ref

* changed NonEmptyHistoryTree and InnerHistoryTree to hold value instead of ref

* fixed errors in zebra-chain/src/block/arbitrary.rs

* finished fixing errors in zebra-chain - all crate tests pass

* changed functions in zebra-state::service::finalized_state to accept &Network

* changed functions in zebra-state::service::non_finalized_state to accept &Network

* zebra-state tests run but fail with overflow error

* zebra-state tests all pass

* converted zebra-network -- all crate tests pass

* applied all requested changes from review

* converted zebra-consensus -- all crate tests pass

* converted zebra-scan -- all crate tests pass

* converted zebra-rpc -- all crate tests pass

* converted zebra-grpc -- all crate tests pass

* converted zebrad -- all crate tests pass

* applied all requested changes from review

* fixed all clippy errors

* fixed build error in zebrad/src/components/mempool/crawler.rs
2024-03-19 20:45:27 +00:00
Za Wilcox 3bef54b764
change(consensus): Refactor production code for network consensus rules to `Network` methods (#8340)
* begin refactor suggested as "step 2": https://github.com/ZcashFoundation/zebra/issues/7968#issue-2003245309
Squashed from multiple commits to enable partial rebase

* break out more little traits

* add activation implementation leveraging From<Network> for lrz::cons::

* for transfer of ownership I cannot return a type that's owned by the method

* hrp_sapling_extended_full_viewing_key

* complete implementation of interface of Parameters on Network reuse Parameters on zcash Network where possible

* move doc-comments to trait declarations (from impls)

* Simplify/complete Parameters impl for Network

* Add checkpoint_list method, move documentation, etc

* move last match network to inside network method

* add back comment

* use zcash_address for parameter types in zebra-chain

* use inherent methods instead of big parameters passthrough

* revert to implementation of From on zcash_primitives::..::Network vs &zcash_prim...

* move match

* add test to block maximum time rule

* update changelog

* finish porting target_difficutly_limit

* remove obscelete code comment

* revert non-functional change

* finish migrating target_difficulty_limit, checkpoint_list

* update changelog

---------

Co-authored-by: Hazel OHearn <gygaxis@zingolabs.org>
2024-03-12 21:41:44 +00:00
dependabot[bot] f21d7c6934
build(deps): bump the prod group with 6 updates (#8125)
* build(deps): bump the prod group with 6 updates

Bumps the prod group with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [futures](https://github.com/rust-lang/futures-rs) | `0.3.29` | `0.3.30` |
| [tokio](https://github.com/tokio-rs/tokio) | `1.35.0` | `1.35.1` |
| [metrics](https://github.com/metrics-rs/metrics) | `0.21.1` | `0.22.0` |
| [metrics-exporter-prometheus](https://github.com/metrics-rs/metrics) | `0.12.2` | `0.13.0` |
| [reqwest](https://github.com/seanmonstar/reqwest) | `0.11.22` | `0.11.23` |
| [owo-colors](https://github.com/jam1garner/owo-colors) | `3.5.0` | `4.0.0` |


Updates `futures` from 0.3.29 to 0.3.30
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.29...0.3.30)

Updates `tokio` from 1.35.0 to 1.35.1
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.35.0...tokio-1.35.1)

Updates `metrics` from 0.21.1 to 0.22.0
- [Changelog](https://github.com/metrics-rs/metrics/blob/main/release.toml)
- [Commits](https://github.com/metrics-rs/metrics/compare/metrics-v0.21.1...metrics-v0.22.0)

Updates `metrics-exporter-prometheus` from 0.12.2 to 0.13.0
- [Changelog](https://github.com/metrics-rs/metrics/blob/main/release.toml)
- [Commits](https://github.com/metrics-rs/metrics/compare/metrics-exporter-prometheus-v0.12.2...metrics-exporter-prometheus-v0.13.0)

Updates `reqwest` from 0.11.22 to 0.11.23
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.22...v0.11.23)

Updates `owo-colors` from 3.5.0 to 4.0.0
- [Commits](https://github.com/jam1garner/owo-colors/compare/v3.5.0...v4.0.0)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: metrics
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: metrics-exporter-prometheus
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: owo-colors
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: prod
...

Signed-off-by: dependabot[bot] <support@github.com>

* update all metric macros

* fix deprecated function

* fix duplicated deps

* Fix an incorrect gauge method call

* Expand documentation and error messages for best chain length

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
2024-01-02 01:26:54 +00:00
teor f2a2a403a8
fix(log): Remove redundant startup logs, fix progress bar number, order, and wording (#7087)
* Remove duplicate "running" message, send log file opening logs to stderr

* Add missing block progress bar desc, make them consistent

* Put progress bars in a specific order in each section

* Actually make block progress bar desc consistent

* Make progress bar order even more consistent

* Fix fork blocks plural for 1 block

* Use the correct number of chain fork bars

* Disable confusing partial work display

* Add struct field category comments

* Silence a verbose inventory log
2023-07-05 07:08:59 +00:00
Alfredo Garcia 5859fac5b1
docs(state): Use different terms for block verification and state queues (#7061)
* claridy some checkpoint verifier docs

* update documentation of `Request::CommitSemanticallyVerifiedBlock` and `Request::CommitCheckpointVerifiedBlock`

* replace `prepared` with `semantically_verified` in state service checks code

* replace `non-finalized` where needed in docs of the state service

* fix double space in doc

* replace `finalized` where needed in docs of the state service

* change some docs in state queued_blocks.rs

* Rewrite pending UTXO checkpoint block comment

* Fix trailing space in docs

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

---------

Co-authored-by: teor <teor@riseup.net>
2023-07-04 21:29:41 +00:00
Alfredo Garcia 8a7c871480
rename(state): do additional renaming for clarification purposes (#6967)
* do renames by script

```
git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/CommitFinalized/CommitCheckpointVerified/g'

git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/commit_finalized_block/commit_checkpoint_verified/g'

git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/drain_queue_and_commit_finalized/drain_finalized_queue_and_commit/g'

git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/queue_and_commit_finalized/queue_and_commit_to_finalized_state/g'

git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/queued_finalized_blocks/finalized_state_queued_blocks/g'

git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/max_queued_finalized_height/max_finalized_queue_height/g'

git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/send_finalized_block_error/send_checkpoint_verified_block_error/g'

git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/QueuedFinalized/QueuedCheckpointVerified/g'

git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/queue_and_commit_non_finalized/queue_and_commit_to_non_finalized_state/g'

git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/queued_non_finalized_blocks/non_finalized_state_queued_blocks/g'

git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/send_non_finalized_block_error/send_semantically_verified_block_error/g'

git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/QueuedNonFinalized/QueuedSemanticallyVerified/g'

git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/last_sent_finalized_block_hash/finalized_block_write_last_sent_hash/g'

git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/sent_non_finalized_block_hashes/non_finalized_block_write_sent_hashes/g'

git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/invalid_block_reset_receiver/invalid_block_write_reset_receiver/g'

cargo fmt --all
```

* add missing log renames by script

```
git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/queued finalized block/finalized state queue block/g'

git ls-tree --full-tree -r --name-only HEAD | \
grep -v 'book' | grep -v 'CHANGELOG.md' | \
xargs sed -i 's/queued non-finalized block/non-finalized state queue block/g'

cargo fmt --all
```
2023-06-15 05:12:45 +00:00
Alfredo Garcia eb07bb31d6
rename(state): Rename state verifiers and related code (#6762)
* rename verifiers

* rename `PreparedBlock` to `SemanticallyVerifiedBlock`

* rename `CommitBlock` to `SemanticallyVerifiedBlock`

* rename `FinalizedBlock` to `CheckpointVerifiedBlock`

* rename `CommitFinalizedBlock` to `CommitCheckpointVerifiedBlock`

* rename `FinalizedWithTrees` to `ContextuallyVerifiedBlockWithTrees`

* rename `ContextuallyValidBlock` to `ContextuallyVerifiedBlock`

* change some `finalized` variables or function arguments to `checkpoint_verified`

* fix docs

* document the difference between `CheckpointVerifiedBlock` and `ContextuallyVerifiedBlock`

* fix doc links

* apply suggestions to request

Co-authored-by: Marek <mail@marek.onl>

* apply suggestions to service

Co-authored-by: Marek <mail@marek.onl>

* apply suggestions to finalized_state.rs and write.rs

Co-authored-by: Marek <mail@marek.onl>

* fmt

* change some more variable names

* change a few missing generics

* fix checkpoint log issue

* rename more `prepared` vars `semantically_verified`

* fix test regex

* fix test regex 2

---------

Co-authored-by: Marek <mail@marek.onl>
2023-06-01 12:29:03 +00:00
teor 166526a088
feat(ui): Add a terminal-based progress bar to Zebra (#6235)
* Implement Display and to_string() for NetworkUpgrade

* Add a progress-bar feature to zebrad

* Add the progress bar writer to the tracing component

* Add a block progress bar transmitter

* Correctly shut down the progress bar, and shut it down on an interrupt

* Make it clearer that the progress task never exits

* Add a config for writing logs to a file

* Add a progress-bar feature to zebra-network

* Add a progress bar for the address book size

* Add progress bars for never attempted and failed peers

* Add an optional limit and label to connection counters

* Add open connection progress bars

* Improve CheckpointList API and CheckpointVerifier debugging

* Add checkpoint index and checkpoint queue progress bars

* Security: Limit the number of non-finalized chains tracked by Zebra

* Make some NonFinalizedState methods available with proptest-impl

* Add a non-finalized chain count progress bar

* Track the last fork height for newly forked chains

* Add a should_count_metrics to Chain

* Add a display method for PartialCumulativeWork

* Add a progress bar for each chain fork

* Add a NonFinalizedState::disable_metrics() method and switch to using it

* Move metrics out of Chain because we can't update Arc<Chain>

* Fix: consistently use best chain order when searching chains

* Track Chain progress bars in NonFinalizedState

* Display work as bits, not a multiple of the target difficulty

* Handle negative fork lengths by reporting "No fork"

* Correctly disable unused fork bars

* clippy: rewrite using `match _.cmp(_) { ... }`

* Initial mempool progress bar implementation

* Update Cargo.lock

* Add the actual transaction size as a description to the cost bar

* Only show mempool progress bars after first activation

* Add queued and rejected mempool progress bars

* Clarify cost note is actual size

* Add tracing.log_file config and progress-bar feature to zebrad docs

* Derive Clone for Chain

* Upgrade to howudoin 0.1.2 and remove some bug workarounds

* Directly call the debug formatter to Display a Network

Co-authored-by: Arya <aryasolhi@gmail.com>

* Rename the address count metric to num_addresses

Co-authored-by: Arya <aryasolhi@gmail.com>

* Simplify reverse checkpoint lookup

Co-authored-by: Arya <aryasolhi@gmail.com>

* Simplify progress bar shutdown code

Co-authored-by: Arya <aryasolhi@gmail.com>

* Remove unused MIN_TRANSPARENT_TX_MEMPOOL_SIZE

* Document that the progress task runs forever

* Fix progress log formatting

* If progress-bar is on, log to a file by default

* Create missing directories for log files

* Add file security docs for running Zebra with elevated permissions

* Document automatic log file, spell progress-bar correctly

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-04-13 08:42:17 +00:00
teor c2896cce4b
change(deps): Remove unused zebra-utils dependencies (#5961)
* Remove unused zebra-utils dependencies

* Cleanup zebra-checkpoints
2023-01-16 21:39:47 +00:00
teor 4078e244d3
fix(lint): Box large error types to resolve the clippy large result err variant lint (#5759)
* Box errors to deal with large error warnings, add accessor methods for error properties

* Remove or explain some large enum variant lints

* Turn some tickets into TODOs

* Allow missing docs on single-field error enum variants

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-12-08 06:11:33 +00:00
Alfredo Garcia 97fb85dca9
lint(clippy): add `unwrap_in_result` lint (#4667)
* `unwrap_in_result` in zebra-chain crate

* `unwrap_in_result` in zebra-script crate

* `unwrap_in_result` in zebra-state crate

* `unwrap_in_result` in zebra-consensus crate

* `unwrap_in_result` in zebra-test crate

* `unwrap_in_result` in zebra-network crate

* `unwrap_in_result` in zebra-rpc crate

* `unwrap_in_result` in zebrad crate

* rustfmt

* revert `?` and add exceptions

* explain some panics better

* move some lint positions

* replace a panic with error

* Fix rustfmt?

Co-authored-by: teor <teor@riseup.net>
2022-06-28 06:22:07 +00:00
teor d4b9353d67
feat(log): Show the current network upgrade in progress logs (#4694)
* Improve time logging using humantime

* Only log full seconds, ignore the fractional part

* Move humantime_seconds to tracing::fmt

* Move the progress task to its own module

* Add missing humantime dependency

* Log the network upgrade in progress logs

* Log when Zebra verifies the final checkpoint
2022-06-28 02:51:41 +00:00
teor ca0520b2e8
change(deps): Upgrade tracing-subscriber and related dependencies (#4517)
* Upgrade tracing and related dependencies

```sh
cargo upgrade --workspace
tracing-error
tracing-subscrber

color-eyre

tracing-flame
tracing-journald

sentry
sentry-tracing

metrics
metrics-exporter-prometheus
reqwest
```

* Update duplicate dependency checks

* Enable the tracing/env-filter feature

* Fix type inference for metrics

Manual changes, plus:
```sh
fastmod "as _" "as f64"
```

* Tidy up some unrelated test code

* Update metrics-exporter-prometheus API

And make unused dependencies optional.

* Adjust test regexes to new tracing format

Also fix some regex bugs, and refactor to simplify.

* Disable color-eyre span traces and track caller in release builds

* Add a feature that enables extra debugging in release builds

* Clean up some redundant features

* Increase a test timeout
2022-06-01 13:53:51 +10:00
teor d767f43fd9
fix(build): Stop using a test-only method in production code (#4000)
* Make proptest-impl features consistently depend on each other

* Use a non-test method for consensus checks

* Make proptest dependencies the same between crates
2022-03-30 18:23:55 +00:00
teor 5c60528b84
fix(test): make full sync test more accurate (#3555)
* feat(log): log current height when logging sync progress

* fix(test): log the specific error when full sync tests fail

* doc(start): remove an obsolete TODO

We can't decrease this log level, because the tests rely on it.

* fix(test): wait until mempool activates in full sync tests

Changes the log message and log test so that the test only finishes
when the mempool has activated.

There is still a race condition here, between the log timer and
mempool activation. But it should be very rare, because the mempool
is activated immediately when `is_close_to_tip()` becomes true.

* fix(test): warn when Zebra stalls below the maximum checkpoint height

This also improves the full sync tests,
because the warning is checked before logging a successful sync.

* feat(log): warn when sync stalls downloading the genesis block

* fix(test): warn when the state hasn't committed a block for a long time

This also improves the full sync tests,
because the warning is checked before logging a successful sync.

* doc(test): update some sync acceptance test comments

* fix(log): use Display formatting to log chrono::Duration

Debug formatting is complicated and hard to read.

* fix(log): stop saying that we've activated the mempool without checking it

We're not checking if the mempool is active, so we can't say that.

* fix(log): minor tidying and TODOs

* fix(doc): fix a typo in the tests

* fix(log): explain the post-checkpoint blocks in progress warning calculations

* fix(doc): explain what could happen if we don't wait for extra blocks

* fix(log): add a percent symbol to a percent log

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-18 02:00:24 +00:00
Dimitris Apostolou afb8b3d477
Fix typos (#3055)
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-12 19:30:22 +00:00
Conrado Gouvea 84f2c07fbc
Ignore AlreadyInChain error in the syncer (#2890)
* Ignore AlreadyInChain error in the syncer

* Split Cancelled errors; add them to should_restart_sync exceptions

* Also filter 'block is already comitted'; try to detect a wrong downcast
2021-10-20 11:07:19 +10:00
teor 34c7a27c2a
Security: Replace queued checkpoint blocks with duplicate hashes (#2697)
We don't check the authorizing data hash until checkpoint blocks reach the state.

So signatures, proofs, or scripts could be different,
even if the block hash is the same.

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-08-31 13:50:47 -03:00
teor 2dac0dda47
Stop calculating transaction hashes twice in the checkpoint verifier (#2696)
* Stop precalculating transaction hashes twice during checkpointing

* Refactor a complex type using a new `RequestBlock` type

* Comment formatting

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-08-31 10:55:39 +10:00
Conrado Gouvea bc4194fcb9
ZIP-221/244 auth data commitment validation in checkpoint verifier (#2633)
* Add validation of ZIP-221 and ZIP-244 commitments

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Add auth commitment check in the finalized state

* Reset the verifier when comitting to state fails

* Add explanation comment

* Add test with fake activation heights

* Add generate_valid_commitments flag

* Enable fake activation heights using env var instead of feature

* Also update initial_tip_hash; refactor into progress_from_tip()

* Improve comments

* Add fake activation heights test to CI

* Fix bug that caused commitment trees to not match when generating partial arbitrary chains

* Add ChainHistoryBlockTxAuthCommitmentHash::from_commitments to organize and deduplicate code

* Remove stale comment, improve readability

* Allow overriding with PROPTEST_CASES

* partial_chain_strategy(): don't update note commitment trees when not needed; add comment

Co-authored-by: teor <teor@riseup.net>
2021-08-23 14:17:33 +00:00
teor 8ebb415e7c Clippy: remove needless borrows 2021-06-07 18:33:58 -04:00
Alfredo Garcia 29893f2b9b
Validate nConsensusBranchId (#2100)
* validate nConsensusBranchId
* add tests

* fix bug in transaction_to_fake_v5

Co-authored-by: teor <teor@riseup.net>
2021-05-10 01:31:45 +00:00
teor 8103e89739 Check proof-of-work in the CheckpointVerifier
These checks make some resource exhaustion attacks harder to carry out,
because the malicious blocks need a valid proof-of-work.
2021-03-12 08:55:29 +10:00
teor c1a910942d Re-use some block checks in the CheckpointVerifier 2021-03-12 08:55:29 +10:00
teor 252affdf84 Add network to CheckpointVerifier
This refactor allows us to re-use some block checks in the checkpoint
verifier.
2021-03-12 08:55:29 +10:00
teor a91006afa7
Actually use `VerifyCheckpointError::CommitFinalized` (#1706)
Also:
* use `ServiceExt::oneshot` in the checkpoint verifier
* make some error messages more specific
* clean up the `std::future`/`futures` imports
2021-02-09 07:16:36 +10:00
Alfredo Garcia d7c40af2a8
Fix shutdown panics (#1637)
* add a shutdown flag in zebra_chain::shutdown
* fix network panic on shutdown
* fix checkpoint panic on shutdown
2021-02-03 19:03:28 +10:00
teor 3699bbdae6 Add some additional sync correctness constraints
And adjust the sync restart delay as a consequence.
2021-01-13 20:46:25 -05:00
Jane Lusby 15698245e1
Deduplicate metrics dependencies (#1561)
## Motivation

This PR is motivated by the regression identified in https://github.com/ZcashFoundation/zebra/issues/1349. That PR notes that the metrics stopped working for most of the crates other than `zebrad`.

## Solution

This PR resolves the regression by deduplicating the `metrics` crate dependency. During a recent change we upgraded the metrics version in `zebrad` and a couple other of our crates, but we never updated the dependencies in `zebra-state`, `zebra-consensus`, or `zebra-network`. This caused the metrics macros to attempt to retrieve the current metrics exporter through the wrong function. We would install the metrics exporter in `0.13`, but then attempt to look it up through the `0.12` crate, which contains a different instance of the metrics exporter static variable which is unset. Doing this causes the metrics macros to return `None` for the current exporter after which they just silently give up.

## Related Issues

closes https://github.com/ZcashFoundation/zebra/issues/1349

## Follow Up Work

I noticed we have quite a few duplicate dependencies in our tree. We might be able to save some compilation time by auditing those and deduplicating them as much as possible.

- https://github.com/ZcashFoundation/zebra/issues/1582
Co-authored-by: teor <teor@riseup.net>
2021-01-12 12:28:56 +10:00
Henry de Valence 706f1fff81 consensus: tidy merkle validation in checkpointer
This moves it in with the existing `check_block` method and expands that
method's contract to cover general block validation checks.
2020-12-01 10:14:44 +10:00
Henry de Valence 7c08c0c315 consensus: check Merkle roots
As a side effect of computing Merkle roots, we build a list of
transaction hashes.  Instead of discarding these, add them to
PreparedBlock and FinalizedBlock so that they can be reused rather than
recomputed.

This commit adds Merkle root validation to:

1. the block verifier;
2. the checkpoint verifier.

In the first case, Bitcoin Merkle tree malleability has no effect,
because only a single Merkle tree in each malleablity set is valid (the
others have duplicate transactions).

In the second case, we need to check that the Merkle tree does not contain any
duplicate transactions.

Closes #1385
Closes #906
2020-12-01 10:14:44 +10:00
Henry de Valence e0817d1747 state: introduce PreparedBlock, FinalizedBlock
This change introduces two new types:

- `PreparedBlock`, representing a block which has undergone semantic
  validation and has been prepared for contextual validation;
- `FinalizedBlock`, representing a block which is ready to be finalized
  immediately;

and changes the `Request::CommitBlock`,`Request::CommitFinalizedBlock`
variants to use these types instead of their previous fields.

This change solves the problem of passing data between semantic
validation and contextual validation, and cleans up the state code by
allowing it to pass around a bundle of data.  Previously, the state code
just passed around an `Arc<Block>`, which forced it to needlessly
recompute block hashes and other data, and was incompatible with the
already-known but not-yet-implemented data transfer requirements, namely
passing in the Sprout and Sapling anchors computed during contextual
validation.

This commit propagates the `PreparedBlock` and `FinalizedBlock` types
through the state code but only uses their data opportunistically, e.g.,
changing .hash() computations to use the precomputed hash.  In the
future, these structures can be extended to pass data through the
verification pipeline for reuse as appropriate.  For instance, these
changes allow the sprout and sapling anchors to be propagated through
the state.
2020-11-23 14:16:39 +10:00
Henry de Valence aa8d95bd23 consensus: improve checkpoint request replacement diagnostics 2020-11-17 14:56:27 -08:00
Henry de Valence a3ab589d89 consensus,state: document cancellation contracts for services
This change explicitly documents cancellation contracts for our Tower services,
and tries to correct a bug in the implementation of the CheckpointVerifier,
which duplicates information from the state service but did not ensure that it
would be kept in sync.
2020-11-17 14:56:27 -08:00
teor af4797130b
Replace Target<block::Height> with TargetHeight (#1289)
We don't use this generic, so let's just remove it.
2020-11-12 16:11:25 -08:00
teor 96b7572bb5 Make the checkpoint metrics more accurate 2020-11-13 09:41:14 +10:00
teor f90a749910
Clarify CheckpointVerifier errors (#1260)
And make an unreachable error into a panic.
2020-11-06 11:07:30 -08:00
Henry de Valence a1a3e4db5a consensus: simplify block verify tracing output
The previous debug output printed a message that the chain verifier had
recieved a block.  But this provides no additional information compared
to printing no message in chain::Verifier and a message in whichever
verifier the block was sent to, since the resulting spans indicate where
the block was dispatched.

This commit also removes the "unexpected high block" detection; this was
an artefact of the original sync algorithm failing to handle block
advertisements, but we don't have that problem any more, so we can
simplify the code by eliminating that logic.
2020-10-26 12:05:35 -07:00
teor 20dfd04463 Reduce maximum checkpoint size in the Zebra code
The new limits are 400 blocks and 32 MB.
2020-10-26 20:59:40 +10:00
teor 6dc95b1d6d
Revise the checkpoint verifier metrics (#1195)
* update continuous and processing.next metrics correctly
* remove duplicate metrics
* rename ambiguous metrics
2020-10-21 20:06:26 -07:00
Deirdre Connolly a7ef6f6a40 Allow dead_code for checkpoint::Verifier for now 2020-10-20 11:16:22 -04:00
Henry de Valence eb43893de0 consensus: minimize API, clean docs
This reduces the API surface to the minimum required for functionality,
and cleans up module documentation.  The stub mempool module is deleted
entirely, since it will need to be redone later anyways.
2020-10-20 11:16:22 -04:00
Alfredo Garcia c0a14ecc8c
move genesis parameters to zebra-chain (#1151) 2020-10-12 14:08:23 -07:00
Jane Lusby eaf5473efc
Add verification and state tracing events and spans (#1131) 2020-10-09 19:05:09 +10:00
Alfredo Garcia c616c7ffd2
Move consensus verification errors (#1070)
* refactor block and tx validation errors

* rename errors module to error

* move NoTransactions to BlockError

* clarify some errors, use dbg format for hash in error

* mnake is_coinbase_first return BlockError

* add new error types for each consensus Service

Co-authored-by: Jane Lusby <jane@zfnd.org>
2020-09-21 11:54:06 -07:00
Henry de Valence aa70db1867 consensus: remove dead code in checkpointer
This was never used anywhere, not even by tests, but nothing noticed
because it had an #[allow(dead_code)].
2020-09-11 13:37:49 -07:00
Henry de Valence 93586867bc consensus: change verifier poll_ready handling.
This makes the component verifiers both always return `poll_ready`,
because they do not exert backpressure and cannot fail.

The checkpoint verifier now immediately rejects any blocks that arrive
after it finishes checkpointing, instead of marking the service itself
as failed.

The chain verifier is agnostic to the readiness behavior of its
components, and reports readiness when they are both ready.
2020-09-11 13:37:49 -07:00
Henry de Valence 0d6303a56f consensus: avoid tower::ServiceExt::oneshot()
This is a really nice function but there might be a bug in its future
implementation: https://github.com/tower-rs/tower/issues/469

This bug may have already been fixed for the 0.4.0 release, so we could change
back then.
2020-09-11 13:37:49 -07:00