RFC: Contextual Difficulty (#1246)
* Difficulty Contextual RFC: Introduction Add a header, summary, and motivation * Difficulty RFC: Add draft definitions And update the state RFC definitions to match. * Difficulty RFC: Add relevant chain * Difficulty RFC: draft guide-level explanation Outline the core calculations and checks. * Difficulty RFC: Revised based on spec fixes Update the design based on the spec bugs in #1276, #1277, and zcash/zips#416. These changes make the difficulty filter into a context-free check, so we remove it from this contextual validation RFC. * Difficulty RFC: Explain how Zebra's calculations can match the spec * Difficulty RFC: write most of the reference section Includes most of the implementation, modules for each function, and draft notes for some of the remaining parts of the RFC. * Difficulty RFC: Add an AdjustedDifficulty struct * Difficulty RFC: Summarise module structure in the one place * Difficulty RFC: Create implementation notes subsections * Difficulty RFC: add consensus critical order of operations * Difficulty RFC: Use the ValidateContextError type * Difficulty RFC: make the median_time arg mut owned We have to clone the data to pass a fixed-length array to a function, so we might as well sort that array to find the median, and avoid a copy.
This commit is contained in:
parent
e11e8e1373
commit
fc7d37c984
|
@ -34,15 +34,20 @@ state service.
|
|||
|
||||
* **block chain**: A sequence of valid blocks linked by inclusion of the
|
||||
previous block hash in the subsequent block. Chains are rooted at the
|
||||
*genesis* block and extend to a *tip*.
|
||||
genesis block and extend to a **tip**.
|
||||
|
||||
* **chain state**: The state of the ledger after application of a particular
|
||||
sequence of blocks (state transitions).
|
||||
|
||||
* **cumulative difficulty**: The cumulative proof-of-work from genesis to the
|
||||
chain tip.
|
||||
* **block work**: The approximate amount of work required for a miner to generate
|
||||
a block hash that passes the difficulty filter. The number of block header
|
||||
attempts and the mining time are proportional to the work value. Numerically
|
||||
higher work values represent longer processing times.
|
||||
|
||||
* **best chain**: The chain with the greatest cumulative difficulty. This chain
|
||||
* **cumulative work**: The sum of the **block work** of all blocks in a chain, from
|
||||
genesis to the chain tip.
|
||||
|
||||
* **best chain**: The chain with the greatest **cumulative work**. This chain
|
||||
represents the consensus state of the Zcash network and transactions.
|
||||
|
||||
* **side chain**: A chain which is not contained in the best chain.
|
||||
|
@ -70,6 +75,11 @@ state service.
|
|||
be lower during the initial sync, and after a chain reorganization, if the new
|
||||
best chain is at a lower height.
|
||||
|
||||
* **relevant chain**: The relevant chain for a block starts at the previous
|
||||
block, and extends back to genesis.
|
||||
|
||||
* **relevant tip**: The tip of the relevant chain.
|
||||
|
||||
# Guide-level explanation
|
||||
[guide-level-explanation]: #guide-level-explanation
|
||||
|
||||
|
@ -230,7 +240,6 @@ chains, so that the map ordering is the ordering of worst to best chains.
|
|||
### The `Chain` type
|
||||
[chain-type]: #chain-type
|
||||
|
||||
|
||||
The `Chain` type represents a chain of blocks. Each block represents an
|
||||
incremental state update, and the `Chain` type caches the cumulative state
|
||||
update from its root to its tip.
|
||||
|
@ -706,6 +715,59 @@ These updates can be performed in a batch or without necessarily iterating
|
|||
over all transactions, if the data is available by other means; they're
|
||||
specified this way for clarity.
|
||||
|
||||
## Accessing previous blocks for contextual validation
|
||||
[previous-block-context]: #previous-block-context
|
||||
|
||||
The state service performs contextual validation of blocks received via the
|
||||
`CommitBlock` request. Since `CommitBlock` is synchronous, contextual validation
|
||||
must also be performed synchronously.
|
||||
|
||||
The relevant chain for a block starts at its previous block, and follows the
|
||||
chain of previous blocks back to the genesis block.
|
||||
|
||||
### Relevant chain iterator
|
||||
[relevant-chain-iterator]: #relevant-chain-iterator
|
||||
|
||||
The relevant chain can be retrieved from the state service as follows:
|
||||
* if the previous block is the finalized tip:
|
||||
* get recent blocks from the finalized state
|
||||
* if the previous block is in the non-finalized state:
|
||||
* get recent blocks from the relevant chain, then
|
||||
* get recent blocks from the finalized state, if required
|
||||
|
||||
The relevant chain can start at any non-finalized block, or at the finalized tip.
|
||||
|
||||
### Relevant chain implementation
|
||||
[relevant-chain-implementation]: #relevant-chain-implementation
|
||||
|
||||
The relevant chain is implemented as a `StateService` iterator, which returns
|
||||
`Arc<Block>`s.
|
||||
|
||||
The chain iterator implements `ExactSizeIterator`, so Zebra can efficiently
|
||||
assert that the relevant chain contains enough blocks to perform each contextual
|
||||
validation check.
|
||||
|
||||
```rust
|
||||
impl StateService {
|
||||
/// Return an iterator over the relevant chain of the block identified by
|
||||
/// `hash`.
|
||||
///
|
||||
/// The block identified by `hash` is included in the chain of blocks yielded
|
||||
/// by the iterator.
|
||||
pub fn chain(&self, hash: block::Hash) -> Iter<'_> { ... }
|
||||
}
|
||||
|
||||
impl Iterator for Iter<'_> {
|
||||
type Item = Arc<Block>;
|
||||
...
|
||||
}
|
||||
impl ExactSizeIterator for Iter<'_> { ... }
|
||||
impl FusedIterator for Iter<'_> {}
|
||||
```
|
||||
|
||||
For further details, see [PR 1271].
|
||||
|
||||
[PR 1271]: https://github.com/ZcashFoundation/zebra/pull/1271
|
||||
|
||||
## Request / Response API
|
||||
[request-response]: #request-response
|
||||
|
|
|
@ -0,0 +1,784 @@
|
|||
- Feature Name: contextual_difficulty_validation
|
||||
- Start Date: 2020-11-02
|
||||
- Design PR: [ZcashFoundation/zebra#0000](https://github.com/ZcashFoundation/zebra/pull/0000)
|
||||
- Zebra Issue: [ZcashFoundation/zebra#1036](https://github.com/ZcashFoundation/zebra/issues/1036)
|
||||
|
||||
# Summary
|
||||
[summary]: #summary
|
||||
|
||||
Zcash nodes use a Proof of Work algorithm to reach consensus on the best chain.
|
||||
Valid blocks must reach a difficulty threshold, which is adjusted after every
|
||||
block. The difficulty adjustment calculations depend on the difficulties and
|
||||
times of recent blocks. So Zebra performs contextual validation [RFC2] of
|
||||
difficulty adjustments as part of committing blocks to the state.
|
||||
|
||||
[RFC2]: ./0002-parallel-verification.md
|
||||
|
||||
# Motivation
|
||||
[motivation]: #motivation
|
||||
|
||||
The Zcash block difficulty adjustment is one of the core Zcash consensus rules.
|
||||
Zebra must implement this consensus rule to make sure that its cached chain
|
||||
state is consistent with the consensus of Zcash nodes.
|
||||
|
||||
Difficulty adjustment is also a significant part of Zcash's security guarantees.
|
||||
It ensures that the network continues to resist takeover attacks, even as the
|
||||
number of Zcash miners grows.
|
||||
|
||||
Difficulty adjustment also ensures that blocks are regularly spaced, which
|
||||
allows users to create and finalise transactions with short, consistent delays.
|
||||
These predictable delays contribute to Zcash's usability.
|
||||
|
||||
# Definitions
|
||||
[definitions]: #definitions
|
||||
|
||||
Difficulty:
|
||||
* **hash difficulty**: An arbitrary ranking of blocks, based on their hashes.
|
||||
Defined as the hash of the block, interpreted as a big-endian 256-bit number.
|
||||
Numerically smaller difficulties are harder to generate.
|
||||
|
||||
* **difficulty threshold**: The easiest valid hash difficulty for a block.
|
||||
Numerically lower thresholds are harder to satisfy.
|
||||
|
||||
* **difficulty filter**: A block passes the difficulty filter if the hash
|
||||
difficulty is less than or equal to the difficulty threshold (based on the
|
||||
block's difficulty field).
|
||||
|
||||
* **block work**: The approximate amount of work required for a miner to generate
|
||||
a block hash that passes the difficulty filter. The number of block header
|
||||
attempts and the mining time are proportional to the work value. Numerically
|
||||
higher work values represent longer processing times.
|
||||
|
||||
* **averaging window**: The 17 most recent blocks in the relevant chain.
|
||||
|
||||
* **median block span**: The 11 most recent blocks from a chosen tip, typically
|
||||
the relevant tip.
|
||||
|
||||
* **target spacing**: 150 seconds per block before Blossom activation, 75 seconds
|
||||
per block from Blossom activation onwards.
|
||||
|
||||
* **adjusted difficulty**: After each block is mined, the difficulty threshold of
|
||||
the next block is adjusted, to keep the block gap close to the target spacing.
|
||||
|
||||
* **mean target difficulty**: The arithmetic mean of the difficulty thresholds
|
||||
of the blocks in the averaging window.
|
||||
|
||||
* **median timespan**: The average number of seconds taken to generate the blocks
|
||||
in the averaging window. Calculated using the difference of median block spans
|
||||
in and after the averaging window, then damped and bounded.
|
||||
|
||||
* **target timespan**: The target spacing for an averaging window's worth of
|
||||
blocks.
|
||||
|
||||
Consensus:
|
||||
* **consensus rule:** A protocol rule which all nodes must apply consistently,
|
||||
so they can converge on the same chain fork.
|
||||
|
||||
* **structural/semantic/contextual verification**: as defined in [RFC2].
|
||||
|
||||
State:
|
||||
* **block chain**: A sequence of valid blocks linked by inclusion of the
|
||||
previous block hash in the subsequent block. Chains are rooted at the
|
||||
genesis block and extend to a tip.
|
||||
|
||||
* **relevant chain**: The relevant chain for a block starts at the previous
|
||||
block, and extends back to genesis.
|
||||
|
||||
* **relevant tip**: The tip of the relevant chain.
|
||||
|
||||
* **non-finalized state**: State data corresponding to blocks above the reorg
|
||||
limit. This data can change in the event of a chain reorg.
|
||||
|
||||
* **finalized state**: State data corresponding to blocks below the reorg
|
||||
limit. This data cannot change in the event of a chain reorg.
|
||||
|
||||
* **non-finalized tips**: The highest blocks in each non-finalized chain. These
|
||||
tips might be at different heights.
|
||||
|
||||
* **finalized tip**: The highest block in the finalized state. The tip of the best
|
||||
chain is usually 100 blocks (the reorg limit) above the finalized tip. But it can
|
||||
be lower during the initial sync, and after a chain reorganization, if the new
|
||||
best chain is at a lower height.
|
||||
|
||||
# Guide-level explanation
|
||||
[guide-level-explanation]: #guide-level-explanation
|
||||
|
||||
Zcash's difficulty consensus rules are similar to Bitcoin.
|
||||
|
||||
Each block contains a **difficulty threshold** in its header. The hash of the
|
||||
block header must be less than this **difficulty threshold**. (When interpreted
|
||||
as a 256-bit integer in big-endian byte order.) This context-free semantic
|
||||
verification check is performed by the `BlockVerifier`.
|
||||
|
||||
After each block, the difficulty threshold is adjusted so that the block gap is
|
||||
close to the target spacing. On average, harder blocks take longer to mine, and
|
||||
easier blocks take less time.
|
||||
|
||||
The **adjusted difficulty** for the next block is calculated using the difficulty
|
||||
thresholds and times of recent blocks. Zcash uses the most recent 28 blocks in
|
||||
the **relevant chain** in its difficulty adjustment calculations.
|
||||
|
||||
The difficulty adjustment calculations adjust the **mean target difficulty**,
|
||||
based on the difference between the **median timespan** and the
|
||||
**target timespan**. If the median timespan is less than the target timespan, the
|
||||
next block is harder to mine.
|
||||
|
||||
The `StateService` calculates the adjusted difficulty using the context from the
|
||||
**relevant chain**. The difficulty contextual verification check ensures that the
|
||||
**difficulty threshold** of the next block is equal to the **adjusted difficulty**
|
||||
for its relevant chain.
|
||||
|
||||
## State service interface changes
|
||||
[state-service-interface]: #state-service-interface
|
||||
|
||||
Contextual validation accesses recent blocks. So we modify the internal state
|
||||
service interface to provide an abstraction for accessing recent blocks.
|
||||
|
||||
### The relevant chain
|
||||
[relevant-chain]: #relevant-chain
|
||||
|
||||
The relevant chain consists of the ancestors of a block, starting with its
|
||||
parent block, and extending back to the genesis block.
|
||||
|
||||
In Zebra, recent blocks are part of the non-finalized state, which can contain
|
||||
multiple chains. Past the reorganization limit, Zebra commits a single chain to
|
||||
the finalized state.
|
||||
|
||||
The relevant chain can start at any block in the non-finalized state, or at the
|
||||
finalized tip. See [RFC5] for details.
|
||||
|
||||
[RFC5]: ./0005-state-updates.md
|
||||
|
||||
## Contextual validation design
|
||||
[contextual-validation-design]: #contextual-validation-design
|
||||
|
||||
Contextual validation is performed synchronously by the state service, as soon
|
||||
as the state has:
|
||||
* received the semantically valid next block (via `CommitBlock`), and
|
||||
* committed the previous block.
|
||||
|
||||
The difficulty adjustment check calculates the correct adjusted difficulty
|
||||
threshold value for a candidate block, and ensures that the block's
|
||||
`difficulty_threshold` field is equal to that value.
|
||||
|
||||
This check is implemented as follows:
|
||||
|
||||
### Difficulty adjustment
|
||||
[difficulty-adjustment]: #difficulty-adjustment
|
||||
|
||||
The block difficulty threshold is adjusted by scaling the mean target difficulty
|
||||
by the median timespan.
|
||||
|
||||
On Testnet, if a long time has elapsed since the previous block, the difficulty
|
||||
adjustment is modified to allow minimum-difficulty blocks.
|
||||
|
||||
#### Mean target difficulty
|
||||
[mean-target-difficulty]: #mean-target-difficulty
|
||||
|
||||
The mean target difficulty is the arithmetic mean of the difficulty
|
||||
thresholds of the `PoWAveragingWindow` (17) most recent blocks in the relevant
|
||||
chain.
|
||||
|
||||
Zcash uses block difficulty thresholds in its difficulty adjustment calculations.
|
||||
(Block hashes are not used for difficulty adjustment.)
|
||||
|
||||
#### Median timespan
|
||||
[median-timespan]: #median-timespan
|
||||
|
||||
The average number of seconds taken to generate the 17 blocks in the averaging
|
||||
window.
|
||||
|
||||
The median timespan is calculated by taking the difference of the median times
|
||||
for:
|
||||
* the relevant tip: the `PoWMedianBlockSpan` (11) most recent blocks, and
|
||||
* the 11 blocks after the 17-block `PoWAveragingWindow`: that is, blocks 18-28
|
||||
behind the relevant tip.
|
||||
|
||||
The median timespan is damped by the `PoWDampingFactor`, and bounded by
|
||||
`PoWMaxAdjustDown` and `PoWMaxAdjustUp`.
|
||||
|
||||
#### Test network minimum difficulty blocks
|
||||
[test-net-min-difficulty]: #test-net-min-difficulty
|
||||
|
||||
If there is a large gap after a Testnet block, the next block becomes a minimum
|
||||
difficulty block. Testnet minimum difficulty blocks have their
|
||||
`difficulty_threshold` set to the minimum difficulty for Testnet.
|
||||
|
||||
#### Block difficulty threshold
|
||||
[block-difficulty-threshold]: #block-difficulty-threshold
|
||||
|
||||
The block difficulty threshold for the next block is calculated by scaling the
|
||||
mean target difficulty by the ratio between the median timespan and the averaging
|
||||
window timespan.
|
||||
|
||||
The result of this calculation is limited by `ToCompact(PoWLimit(network))`, a
|
||||
per-network minimum block difficulty. This minimum difficulty is also used when
|
||||
a Testnet block's time gap exceeds the minimum difficulty gap.
|
||||
|
||||
# Reference-level explanation
|
||||
[reference-level-explanation]: #reference-level-explanation
|
||||
|
||||
## Contextual validation
|
||||
[contextual-validation]: #contextual-validation
|
||||
|
||||
Contextual validation is implemented in
|
||||
`StateService::check_contextual_validity`, which calls a separate function for
|
||||
each contextual validity check.
|
||||
|
||||
In Zebra, contextual validation starts after Sapling activation, so we can assume
|
||||
that the relevant chain contains at least 28 blocks on Mainnet and Testnet. (And
|
||||
panic if this assumption does not hold at runtime.)
|
||||
|
||||
## Fundamental data types
|
||||
[fundamental-data-types]: #fundamental-data-types
|
||||
|
||||
Zebra is free to implement its difficulty calculations in any way that produces
|
||||
equivalent results to `zcashd` and the Zcash specification.
|
||||
|
||||
### Difficulty
|
||||
|
||||
In Zcash block headers, difficulty thresholds are stored as a "compact" `nBits`
|
||||
value, which uses a custom 32-bit floating-point encoding. Zebra calls this type
|
||||
`CompactDifficulty`.
|
||||
|
||||
In Zcash, difficulty threshold calculations are performed using unsigned 256-bit
|
||||
integers. Rust has no standard `u256` type, but there are a number of crates
|
||||
available which implement the required operations on 256-bit integers. Zebra
|
||||
abstracts over the chosen `u256` implementation using its `ExpandedDifficulty`
|
||||
type.
|
||||
|
||||
### Time
|
||||
|
||||
In Zcash, time values are unsigned 32-bit integers. But the difficulty adjustment
|
||||
calculations include time subtractions which could overflow an unsigned type, so
|
||||
they are performed using signed 64-bit integers in `zcashd`.
|
||||
|
||||
Zebra parses the `header.time` field into a `DateTime<Utc>`. Conveniently, the
|
||||
`chrono::DateTime<_>::timestamp()` function returns `i64` values. So Zebra can do
|
||||
its signed time calculations using `i64` values internally.
|
||||
|
||||
Note: `i32` is an unsuitable type for signed time calculations. It is
|
||||
theoretically possible for the time gap between blocks to be larger than
|
||||
`i32::MAX`, because those times are provided by miners. Even if the median time
|
||||
gap is that large, the bounds and minimum difficulty in Zcash's difficulty
|
||||
adjustment algorithm will preserve a reasonable difficulty threshold. So Zebra
|
||||
must support this edge case.
|
||||
|
||||
### Consensus-Critical Operations
|
||||
|
||||
The order of operations and overflow semantics for 256-bit integers can be
|
||||
consensus-critical.
|
||||
|
||||
For example:
|
||||
- dividing before multiplying discards lower-order bits, but
|
||||
- multiplying before dividing can cause overflow.
|
||||
|
||||
Zebra's implementation should try to match zcashd's order of operations and
|
||||
overflow handling as closely as possible.
|
||||
|
||||
## Difficulty adjustment check
|
||||
[difficulty-adjustment-check]: #difficulty-adjustment-check
|
||||
|
||||
The difficulty adjustment check calculates the correct difficulty threshold
|
||||
value for a candidate block, and ensures that the block's
|
||||
`difficulty_threshold` field is equal to that value.
|
||||
|
||||
### Context data type
|
||||
[context-data-type]: #context-data-type
|
||||
|
||||
The difficulty adjustment functions use a context consisting of the difficulties
|
||||
and times from the previous 28 blocks in the relevant chain.
|
||||
|
||||
These functions also use the candidate block's `height` and `network`.
|
||||
|
||||
To make these functions more ergonomic, we create a `AdjustedDifficulty`
|
||||
type, and implement the difficulty adjustment calculations as methods on that
|
||||
type.
|
||||
|
||||
```rust
|
||||
/// The averaging window for difficulty threshold arithmetic mean calculations.
|
||||
///
|
||||
/// `PoWAveragingWindow` in the Zcash specification.
|
||||
pub const POW_AVERAGING_WINDOW: usize = 17;
|
||||
|
||||
/// The median block span for time median calculations.
|
||||
///
|
||||
/// `PoWMedianBlockSpan` in the Zcash specification.
|
||||
pub const POW_MEDIAN_BLOCK_SPAN: usize = 11;
|
||||
|
||||
/// Contains the context needed to calculate the adjusted difficulty for a block.
|
||||
struct AdjustedDifficulty {
|
||||
candidate_time: DateTime<Utc>,
|
||||
candidate_height: block::Height,
|
||||
network: Network,
|
||||
relevant_difficulty_thresholds: [CompactDifficulty; POW_AVERAGING_WINDOW + POW_MEDIAN_BLOCK_SPAN],
|
||||
relevant_times: [DateTime<Utc>; POW_AVERAGING_WINDOW + POW_MEDIAN_BLOCK_SPAN],
|
||||
}
|
||||
```
|
||||
|
||||
We implement some initialiser methods on `AdjustedDifficulty` for convenience.
|
||||
We might want to validate downloaded headers in future, so we include a
|
||||
`new_from_header` initialiser.
|
||||
|
||||
```rust
|
||||
/// Initialise and return a new `AdjustedDifficulty` using a `candidate_block`,
|
||||
/// `network`, and a `context`.
|
||||
///
|
||||
/// The `context` contains the previous
|
||||
/// `PoWAveragingWindow + PoWMedianBlockSpan` (28) `difficulty_threshold`s and
|
||||
/// `time`s from the relevant chain for `candidate_block`, in reverse height
|
||||
/// order, starting with the previous block.
|
||||
///
|
||||
/// Note that the `time`s might not be in reverse chronological order, because
|
||||
/// block times are supplied by miners.
|
||||
///
|
||||
/// Panics:
|
||||
/// If the `context` contains fewer than 28 items.
|
||||
pub fn new_from_block<C>(candidate_block: &Block,
|
||||
network: Network,
|
||||
context: C)
|
||||
-> AdjustedDifficulty
|
||||
where
|
||||
C: IntoIterator<Item = (CompactDifficulty, DateTime<Utc>)>,
|
||||
{ ... }
|
||||
|
||||
/// Initialise and return a new `AdjustedDifficulty` using a
|
||||
/// `candidate_header`, `previous_block_height`, `network`, and a `context`.
|
||||
///
|
||||
/// Designed for use when validating block headers, where the full block has not
|
||||
/// been downloaded yet.
|
||||
///
|
||||
/// See `new_from_block` for detailed information about the `context`.
|
||||
///
|
||||
/// Panics:
|
||||
/// If the context contains fewer than 28 items.
|
||||
pub fn new_from_header<C>(candidate_header: &block::Header,
|
||||
previous_block_height: block::Height,
|
||||
network: Network,
|
||||
context: C)
|
||||
-> AdjustedDifficulty
|
||||
where
|
||||
C: IntoIterator<Item = (CompactDifficulty, DateTime<Utc>)>,
|
||||
{ ... }
|
||||
```
|
||||
|
||||
#### Memory usage note
|
||||
|
||||
Copying `CompactDifficulty` values into the `AdjustedDifficulty` struct uses
|
||||
less memory than borrowing those values. `CompactDifficulty` values are 32 bits,
|
||||
but pointers are 64-bit on most modern machines. (And since they all come from
|
||||
different blocks, we need a pointer to each individual value.)
|
||||
|
||||
Borrowing `DateTime<Utc>` values might use slightly less memory than copying
|
||||
them - but that depends on the exact way that Rust stores associated types
|
||||
derived from a generic argument.
|
||||
|
||||
In any case, the overall size of each `AdjustedDifficulty` is only a few
|
||||
hundred bytes. If it turns up in profiles, we can look at borrowing the block
|
||||
header data.
|
||||
|
||||
### Difficulty adjustment check implementation
|
||||
[difficulty-adjustment-check-implementation]: #difficulty-adjustment-check-implementation
|
||||
|
||||
The difficulty adjustment check ensures that the
|
||||
`candidate_difficulty_threshold` is equal to the `difficulty_threshold` value
|
||||
calculated using `AdjustedDifficulty::adjusted_difficulty_threshold`.
|
||||
|
||||
We implement this function:
|
||||
```rust
|
||||
/// Validate the `difficulty_threshold` from a candidate block's header, based
|
||||
/// on an `expected_difficulty` for that block.
|
||||
///
|
||||
/// Uses `expected_difficulty` to calculate the expected `ToCompact(Threshold())`
|
||||
/// value, then compares that value to the `difficulty_threshold`. Returns
|
||||
/// `Ok(())` if the values are equal.
|
||||
pub fn difficulty_threshold_is_valid(difficulty_threshold: CompactDifficulty,
|
||||
expected_difficulty: AdjustedDifficulty)
|
||||
-> Result<(), ValidateContextError> { ... }
|
||||
```
|
||||
|
||||
[Issue 1166]: https://github.com/ZcashFoundation/zebra/issues/1166
|
||||
|
||||
### Mean target difficulty calculation
|
||||
[mean-target-difficulty-calculation]: #mean-target-difficulty-calculation
|
||||
|
||||
The mean target difficulty is the arithmetic mean of the difficulty
|
||||
thresholds of the `PoWAveragingWindow` (17) most recent blocks in the relevant
|
||||
chain.
|
||||
|
||||
We implement this method on `AdjustedDifficulty`:
|
||||
```rust
|
||||
/// Calculate the arithmetic mean of the averaging window thresholds: the
|
||||
/// expanded `difficulty_threshold`s from the previous `PoWAveragingWindow` (17)
|
||||
/// blocks in the relevant chain.
|
||||
///
|
||||
/// Implements `MeanTarget` from the Zcash specification.
|
||||
fn mean_target_difficulty(&self) -> ExpandedDifficulty { ... }
|
||||
```
|
||||
|
||||
#### Implementation notes
|
||||
|
||||
Since the `PoWLimit`s are `2^251 − 1` for Testnet, and `2^243 − 1` for Mainnet,
|
||||
the sum of these difficulty thresholds will be less than or equal to
|
||||
`(2^251 − 1)*17 = 2^255 + 2^251 - 17`. Therefore, this calculation can not
|
||||
overflow a `u256` value. So the function is infalliable.
|
||||
|
||||
In Zebra, contextual validation starts after Sapling activation, so we can assume
|
||||
that the relevant chain contains at least 17 blocks. Therefore, the `PoWLimit`
|
||||
case of `MeanTarget()` in the Zcash specification is unreachable.
|
||||
|
||||
### Median timespan calculation
|
||||
[median-timespan-calculation]: #median-timespan-calculation
|
||||
|
||||
The median timespan is the difference of the median times for:
|
||||
* the relevant tip: the `PoWMedianBlockSpan` (11) most recent blocks, and
|
||||
* the 11 blocks after the 17-block `PoWAveragingWindow`: that is, blocks 18-28
|
||||
behind the relevant tip.
|
||||
|
||||
(The median timespan is known as the `ActualTimespan` in the Zcash specification,
|
||||
but this terminology is confusing, because it is a difference of medians, rather
|
||||
than any "actual" elapsed time.)
|
||||
|
||||
Zebra implements the median timespan using the following methods on
|
||||
`AdjustedDifficulty`:
|
||||
```rust
|
||||
/// Calculate the bounded median timespan. The median timespan is the
|
||||
/// difference of medians of the timespan times, which are the `time`s from
|
||||
/// the previous `PoWAveragingWindow + PoWMedianBlockSpan` (28) blocks in the
|
||||
/// relevant chain.
|
||||
///
|
||||
/// Uses the candidate block's `height' and `network` to calculate the
|
||||
/// `AveragingWindowTimespan` for that block.
|
||||
///
|
||||
/// The median timespan is damped by the `PoWDampingFactor`, and bounded by
|
||||
/// `PoWMaxAdjustDown` and `PoWMaxAdjustUp`.
|
||||
///
|
||||
/// Implements `ActualTimespanBounded` from the Zcash specification.
|
||||
///
|
||||
/// Note: This calculation only uses `PoWMedianBlockSpan` (11) times at the
|
||||
/// start and end of the timespan times. timespan times `[11..=16]` are ignored.
|
||||
fn median_timespan_bounded(&self) -> Duration { ... }
|
||||
|
||||
/// Calculate the median timespan. The median timespan is the difference of
|
||||
/// medians of the timespan times, which are the `time`s from the previous
|
||||
/// `PoWAveragingWindow + PoWMedianBlockSpan` (28) blocks in the relevant chain.
|
||||
///
|
||||
/// Implements `ActualTimespan` from the Zcash specification.
|
||||
///
|
||||
/// See `median_timespan_bounded` for details.
|
||||
fn median_timespan(&self) -> Duration { ... }
|
||||
|
||||
/// Calculate the median of the `median_block_span_times`: the `time`s from a
|
||||
/// slice of `PoWMedianBlockSpan` (11) blocks in the relevant chain.
|
||||
///
|
||||
/// Implements `MedianTime` from the Zcash specification.
|
||||
fn median_time(mut median_block_span_times: [DateTime<Utc>; POW_MEDIAN_BLOCK_SPAN])
|
||||
-> DateTime<Utc> { ... }
|
||||
```
|
||||
|
||||
Zebra implements the `AveragingWindowTimespan` using the following methods on
|
||||
`NetworkUpgrade`:
|
||||
```rust
|
||||
impl NetworkUpgrade {
|
||||
/// Returns the `AveragingWindowTimespan` for the network upgrade.
|
||||
pub fn averaging_window_timespan(&self) -> Duration { ... }
|
||||
|
||||
/// Returns the `AveragingWindowTimespan` for `network` and `height`.
|
||||
pub fn averaging_window_timespan_for_height(network: Network,
|
||||
height: block::Height)
|
||||
-> Duration { ... }
|
||||
}
|
||||
```
|
||||
|
||||
#### Implementation notes
|
||||
|
||||
In Zebra, contextual validation starts after Sapling activation, so we can assume
|
||||
that the relevant chain contains at least 28 blocks. Therefore:
|
||||
* `max(0, height − PoWMedianBlockSpan)` in the `MedianTime()` calculation
|
||||
simplifies to `height − PoWMedianBlockSpan`, and
|
||||
* there is always an odd number of blocks in `MedianTime()`, so the median is
|
||||
always the exact middle of the sequence.
|
||||
|
||||
Therefore, the function is infalliable.
|
||||
|
||||
### Test network minimum difficulty calculation
|
||||
[test-net-min-difficulty-calculation]: #test-net-min-difficulty-calculation
|
||||
|
||||
A block is a Testnet minimum difficulty block if:
|
||||
* the block is a Testnet block,
|
||||
* the block's height is 299188 or greater, and
|
||||
* the time gap from the previous block is greater than the Testnet minimum
|
||||
difficulty gap, which is 6 times the target spacing for the block's height.
|
||||
(The target spacing was halved from the Blossom network upgrade onwards.)
|
||||
|
||||
The difficulty adjustment is modified for Testnet minimum difficulty blocks as
|
||||
follows:
|
||||
* the difficulty threshold in the block header is set to the Testnet minimum
|
||||
difficulty threshold, `ToCompact(PoWLimit(network))`.
|
||||
|
||||
Since the new difficulty changes the block header, Testnet blocks can only
|
||||
satisfy one of the alternate difficulty adjustment rules:
|
||||
* if the time gap is less than or equal to the Testnet minimum difficulty gap:
|
||||
the difficulty threshold is calculated using the default difficulty adjustment
|
||||
rule,
|
||||
* if the time gap is greater than the Testnet minimum difficulty gap:
|
||||
the difficulty threshold is the Testnet minimum difficulty threshold.
|
||||
|
||||
See [ZIP-208] for details.
|
||||
|
||||
Note: some older versions of ZIPs 205 and 208 incorrectly said that:
|
||||
* the time gap threshold uses an "at least" check (it is strictly greater than),
|
||||
* the minimum difficulty threshold value was `PoWLimit`
|
||||
(it is `ToCompact(PoWLimit)`),
|
||||
* the `difficulty_threshold` (`nBits`) field is not modified in Testnet minimum
|
||||
difficulty blocks (the field is modified), and
|
||||
* the Testnet minimum difficulty value is not used to calculate future difficulty
|
||||
adjustments (the modified value is used in future adjustments).
|
||||
|
||||
ZIP 205 and 208 were fixed on 14 November 2020, see [ZIP PR 417] and
|
||||
[ZIP commit 806076c] for details.
|
||||
|
||||
[ZIP-208]: https://zips.z.cash/zip-0208#minimum-difficulty-blocks-on-the-test-network
|
||||
[ZIP PR 417]: https://github.com/zcash/zips/pull/417
|
||||
[ZIP commit 806076c]: https://github.com/zcash/zips/commit/806076c48c9834fd9941b940a32310d737975a3a
|
||||
|
||||
#### Test network minimum difficulty implementation
|
||||
[test-net-min-difficulty-implementation]: #test-net-min-difficulty-implementation
|
||||
|
||||
The Testnet minimum difficulty calculation uses the existing
|
||||
`NetworkUpgrade::minimum_difficulty_spacing_for_height` function to calculate the
|
||||
minimum difficulty gap.
|
||||
|
||||
We implement this method on `NetworkUpgrade`:
|
||||
```rust
|
||||
/// Returns true if the gap between `block_time` and `previous_block_time` is
|
||||
/// greater than the Testnet minimum difficulty time gap. This time gap
|
||||
/// depends on the `network` and `block_height`.
|
||||
///
|
||||
/// Returns false on Mainnet, when `block_height` is less than the minimum
|
||||
/// difficulty start height, and when the time gap is too small.
|
||||
///
|
||||
/// `block_time` can be less than, equal to, or greater than
|
||||
/// `previous_block_time`, because block times are provided by miners.
|
||||
///
|
||||
/// Implements the Testnet minimum difficulty adjustment from ZIPs 205 and 208.
|
||||
///
|
||||
/// Spec Note: Some parts of ZIPs 205 and 208 previously specified an incorrect
|
||||
/// check for the time gap. This function implements the correct "greater than"
|
||||
/// check.
|
||||
pub fn is_testnet_min_difficulty_block(
|
||||
network: Network,
|
||||
block_height: block::Height,
|
||||
block_time: DateTime<Utc>,
|
||||
previous_block_time: DateTime<Utc>,
|
||||
) -> bool { ... }
|
||||
```
|
||||
|
||||
#### Implementation notes
|
||||
|
||||
In Zcash, the Testnet minimum difficulty rule starts at block 299188, and in
|
||||
Zebra, contextual validation starts after Sapling activation. So we can assume
|
||||
that there is always a previous block.
|
||||
|
||||
Therefore, this function is infalliable.
|
||||
|
||||
### Block difficulty threshold calculation
|
||||
[block-difficulty-threshold-calculation]: #block-difficulty-threshold-calculation
|
||||
|
||||
The block difficulty threshold for the next block is calculated by scaling the
|
||||
mean target difficulty by the ratio between the median timespan and the averaging
|
||||
window timespan.
|
||||
|
||||
The result of the scaled threshold calculation is limited by
|
||||
`ToCompact(PoWLimit(network))`, a per-network minimum block difficulty. This
|
||||
minimum difficulty is also used when a Testnet block's time gap exceeds the
|
||||
minimum difficulty gap. We use the existing
|
||||
`ExpandedDifficulty::target_difficulty_limit` function to calculate the value of
|
||||
`ToCompact(PoWLimit(network))`.
|
||||
|
||||
In Zebra, contextual validation starts after Sapling activation, so the genesis
|
||||
case of `Threshold()` in the Zcash specification is unreachable.
|
||||
|
||||
#### Block difficulty threshold implementation
|
||||
[block-difficulty-threshold-implementation]: #block-difficulty-threshold-implementation
|
||||
|
||||
We implement these methods on `AdjustedDifficulty`:
|
||||
```rust
|
||||
/// Calculate the expected `difficulty_threshold` for a candidate block, based
|
||||
/// on the `candidate_time`, `candidate_height`, `network`, and the
|
||||
/// `difficulty_threshold`s and `time`s from the previous
|
||||
/// `PoWAveragingWindow + PoWMedianBlockSpan` (28) blocks in the relevant chain.
|
||||
///
|
||||
/// Implements `ThresholdBits` from the Zcash specification, and the Testnet
|
||||
/// minimum difficulty adjustment from ZIPs 205 and 208.
|
||||
pub fn expected_difficulty_threshold(&self) -> CompactDifficulty { ... }
|
||||
|
||||
/// Calculate the `difficulty_threshold` for a candidate block, based on the
|
||||
/// `candidate_height`, `network`, and the relevant `difficulty_threshold`s and
|
||||
/// `time`s.
|
||||
///
|
||||
/// See `expected_difficulty_threshold` for details.
|
||||
///
|
||||
/// Implements `ThresholdBits` from the Zcash specification. (Which excludes the
|
||||
/// Testnet minimum difficulty adjustment.)
|
||||
fn threshold_bits(&self) -> CompactDifficulty { ... }
|
||||
```
|
||||
|
||||
#### Implementation notes
|
||||
|
||||
Since:
|
||||
* the `PoWLimit`s are `2^251 − 1` for Testnet, and `2^243 − 1` for Mainnet,
|
||||
* the `ActualTimespanBounded` can be at most `MaxActualTimespan`, which is
|
||||
`floor(PoWAveragingWindow * PoWTargetSpacing * (1 + PoWMaxAdjustDown))` or
|
||||
`floor(17 * 150 * (1 + 32/100)) = 3366`,
|
||||
* `AveragingWindowTimespan` is at most `17 * 150 = 2250`, and
|
||||
* `MeanTarget` is at most `PoWLimit`, ...
|
||||
|
||||
The maximum scaled value inside the `Threshold()` calculation is:
|
||||
* `floor(PoWLimit / 2250) * 3366`, which equals
|
||||
* `floor((2^251 − 1) / 2250) * 3366`, which equals
|
||||
* `(2^251 − 1) * 132/100`,
|
||||
* which is less than `2^252`.
|
||||
|
||||
Therefore, this calculation can not overflow a `u256` value. (And even if it did
|
||||
overflow, it would be constrained to a valid value by the `PoWLimit` minimum.)
|
||||
|
||||
Note that the multiplication by `ActualTimespanBounded` must happen after the
|
||||
division by `AveragingWindowTimespan`. Performing the multiplication first
|
||||
could overflow.
|
||||
|
||||
If implemented in this way, the function is infalliable.
|
||||
|
||||
`zcashd` truncates the `MeanTarget` after the mean calculation, and
|
||||
after dividing by `AveragingWindowTimespan`. But as long as there is no overflow,
|
||||
this is [equivalent to the single truncation of the final result] in the Zcash
|
||||
specification. However, Zebra should follow the order of operations in `zcashd`,
|
||||
and use repeated divisions, because that can't overflow. See the relevant
|
||||
[comment in the zcashd souce code].
|
||||
|
||||
[equivalent to the single truncation of the final result]: https://math.stackexchange.com/questions/147771/rewriting-repeated-integer-division-with-multiplication
|
||||
[comment in the zcashd souce code]: https://github.com/zcash/zcash/pull/4860/files
|
||||
|
||||
## Module Structure
|
||||
[module-structure]: #module-structure
|
||||
|
||||
The structs and functions in this RFC are implemented in a new
|
||||
`zebra_state::service::check::difficulty` module.
|
||||
|
||||
This module has two entry points:
|
||||
* `DifficultyAdjustment::new_from_block`
|
||||
* `difficulty_threshold_is_valid`
|
||||
|
||||
These entry points are both called from
|
||||
`StateService::check_contextual_validity`.
|
||||
|
||||
## Test Plan
|
||||
[test-plan]: #test-plan
|
||||
|
||||
Explain how the feature will be tested, including:
|
||||
- [ ] tests for consensus-critical functionality
|
||||
- [ ] existing test vectors, if available
|
||||
- [ ] Zcash blockchain block test vectors (specify the network upgrade, feature, or block height and network)
|
||||
- [ ] property testing or fuzzing
|
||||
|
||||
The tests should cover:
|
||||
- [ ] positive cases: make sure the feature accepts valid inputs
|
||||
- using block test vectors for each network upgrade provides some coverage of valid inputs
|
||||
- [ ] negative cases: make sure the feature rejects invalid inputs
|
||||
- make sure there is a test case for each error condition in the code
|
||||
- if there are lots of potential errors, prioritise:
|
||||
- consensus-critical errors
|
||||
- security-critical errors, and
|
||||
- likely errors
|
||||
- [ ] edge cases: make sure that boundary conditions are correctly handled
|
||||
|
||||
# Drawbacks
|
||||
[drawbacks]: #drawbacks
|
||||
|
||||
Why should we *not* do this?
|
||||
|
||||
## Alternate consensus parameters
|
||||
[alternate-consensus-parameters]: #alternate-consensus-parameters
|
||||
|
||||
Any alternate consensus parameters or `regtest` mode would have to respect the constraints set by this design.
|
||||
|
||||
In particular:
|
||||
* the `PoWLimit` must be less than or equal to
|
||||
`(2^256 - 1) / PoWAveragingWindow` (approximately `2^251`) to avoid overflow,
|
||||
* the `PoWAveragingWindow` and `PoWMedianBlockSpan` are fixed by function argument types
|
||||
(at least until Rust gets stable const generics), and
|
||||
* the design eliminates a significant number of edge cases by assuming that difficulty adjustments aren't
|
||||
validated for the first `PoWAveragingWindow + PoWMedianBlockSpan` (28) blocks in the chain.
|
||||
|
||||
# Rationale and alternatives
|
||||
[rationale-and-alternatives]: #rationale-and-alternatives
|
||||
|
||||
## Is this design a good basis for later designs or implementations?
|
||||
[good-basis]: #good-basis
|
||||
|
||||
The design includes specific methods for a future header-only validation design.
|
||||
|
||||
## What other designs have been considered and what is the rationale for not choosing them?
|
||||
[alternate-designs]: #alternate-designs
|
||||
|
||||
A previous version of the RFC did not have the `AdjustedDifficulty` struct and
|
||||
methods. That design was easy to misuse, because each function had a complicated
|
||||
argument list.
|
||||
|
||||
## What is the impact of not doing this?
|
||||
[no-action]: #no-action
|
||||
|
||||
Zebra could accept invalid, low-difficulty blocks from arbitrary miners. That
|
||||
would be a security issue.
|
||||
|
||||
# Prior art
|
||||
[prior-art]: #prior-art
|
||||
|
||||
* `zcashd`
|
||||
* the Zcash specification
|
||||
* Bitcoin
|
||||
|
||||
# Unresolved questions
|
||||
[unresolved-questions]: #unresolved-questions
|
||||
|
||||
- What parts of the design do you expect to resolve through the implementation of this feature before stabilization?
|
||||
- Guide-level examples
|
||||
- Reference-level examples
|
||||
- Corner case examples
|
||||
- Testing
|
||||
|
||||
- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC?
|
||||
- Monitoring and maintainence
|
||||
|
||||
# Future possibilities
|
||||
[future-possibilities]: #future-possibilities
|
||||
|
||||
## Re-using the relevant chain API in other contextual checks
|
||||
[relevant-chain-api-reuse]: #relevant-chain-api-reuse
|
||||
|
||||
The relevant chain iterator can be re-used to implement other contextual
|
||||
validation checks.
|
||||
|
||||
For example, responding to peer requests for block locators, which means
|
||||
implementing relevant chain hash queries as a `StateService` request
|
||||
|
||||
## Header-only difficulty adjustment validation
|
||||
[header-only-validation]: #header-only-validation
|
||||
|
||||
Implementing header-only difficulty adjustment validation as a `StateService` request.
|
||||
|
||||
## Caching difficulty calculations
|
||||
[caching-calculations]: #caching-calculations
|
||||
|
||||
Difficulty calculations use `u256` could be a bit expensive, particularly if we
|
||||
get a flood of low-difficulty blocks. To reduce the impact of this kind of DoS,
|
||||
we could cache the value returned by `threshold_bits` for each block in the
|
||||
non-finalized state, and for the finalized tip. This value could be used to
|
||||
quickly calculate the difficulties for any child blocks of these blocks.
|
||||
|
||||
There's no need to persist this cache, or pre-fill it. (Minimum-difficulty
|
||||
Testnet blocks don't call `threshold_bits`, and some side-chain blocks will
|
||||
never have a next block.)
|
||||
|
||||
This caching is only worth implementing if these calculations show up in `zebrad`
|
||||
profiles.
|
Loading…
Reference in New Issue