Commit Graph

1845 Commits

Author SHA1 Message Date
teor cee0e86190 Increase the open file limit on unix platforms
If the limit is less than the ideal, try to increase it to the ideal.
If that doesn't work, try to increase the limit as high as possible.
If the limit is still less than the minimum, panic.
2020-12-02 15:32:36 +10:00
teor 44f2326672 Move the RocksDB column family list into finalized_state
The list was previously split between config and finalized_state.
2020-12-02 15:32:36 +10:00
teor 4109534c9b
Run `cargo fmt --all` (#1427) 2020-12-02 14:43:33 +10:00
Henry de Valence 00c4f4f0e6 network: record cause of handshake failure 2020-12-01 19:16:41 -08:00
Henry de Valence 1606b28d14 chain: add missing sources to SerializationError 2020-12-01 19:16:41 -08:00
Henry de Valence 5ccd1905fc network: avoid putting null bytes in trace output 2020-12-01 19:16:41 -08:00
Henry de Valence f93deb1cac network: fix missing {0} in PeerError::Serialization 2020-12-01 19:16:41 -08:00
Henry de Valence 18cf5e0249 network: use short Display for Message in spans
This makes the span data more compact (e.g., `msg_as_req{msg=block}`) and
restores the Debug impl for Message to show all of the data contained in the
message.  The full message is added as a single event at trace level in the
span to preserve the previous full-inspectability.
2020-12-01 19:16:41 -08:00
Henry de Valence f0db75e712 cargo fmt 2020-12-01 19:16:41 -08:00
Jane Lusby a91d0f0bb6
Include short sha in log messages and error urls (#1410)
As we approach our alpha release we've decided we want to plan ahead for the user bug reports we will eventually receive. One of the bigger issues we foresee is determining exactly what version of the software users are running, and particularly how easy it may or may not be for users to accidentally discard this information when reporting bugs.

To defend against this, we've decided to include the exact git sha for any given build in the compiled artifact. This information will then be re-exported as a span early in the application startup process, so that all logs and error messages should include the sha as their very first span. We've also added this sha as issue metadata for `color-eyre`'s github issue url auto generation feature, which should make sure that the sha is easily available in bug reports we receive, even in the absence of logs.

Co-authored-by: teor <teor@riseup.net>
2020-12-01 12:13:20 -08:00
Jane Lusby de34c47cc2 enable tracing acceptance test 2020-12-01 11:03:13 -05:00
Jane Lusby fceef849cf remove unused mutability to defuse deadlock 2020-12-01 11:03:13 -05:00
teor 4d5ea4897c Log peer set ready and unready peers
* warn: if there are no peers at all
* info: if there are no ready peers
* trace: the number of ready and unready peers for every request

Log at most one warn or info log per minute, to avoid flooding the
terminal with log lines. Suppress warn and info logs for the first
minute, while the peer set is starting up.
2020-12-01 11:00:21 -05:00
dependabot[bot] 2dd0fa0e07 build(deps): bump tokio from 0.3.4 to 0.3.5
Bumps [tokio](https://github.com/tokio-rs/tokio) from 0.3.4 to 0.3.5.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-0.3.4...tokio-0.3.5)

Signed-off-by: dependabot[bot] <support@github.com>
2020-12-01 10:39:39 -05:00
dependabot[bot] 61d0f02c57 build(deps): bump inferno from 0.10.1 to 0.10.2
Bumps [inferno](https://github.com/jonhoo/inferno) from 0.10.1 to 0.10.2.
- [Release notes](https://github.com/jonhoo/inferno/releases)
- [Changelog](https://github.com/jonhoo/inferno/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jonhoo/inferno/compare/v0.10.1...v0.10.2)

Signed-off-by: dependabot[bot] <support@github.com>
2020-12-01 10:35:14 -05:00
dependabot[bot] 1a77122acb build(deps): bump structopt from 0.3.20 to 0.3.21
Bumps [structopt](https://github.com/TeXitoi/structopt) from 0.3.20 to 0.3.21.
- [Release notes](https://github.com/TeXitoi/structopt/releases)
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md)
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.20...v0.3.21)

Signed-off-by: dependabot[bot] <support@github.com>
2020-12-01 10:34:12 -05:00
teor 92eb92d1dd
Disable the nightly clippy unnecessary_wraps lint (#1403)
It seems to be a bit broken - some of our functions return `Result` for
consistency with similar functions. But the lint picks them up anyway.
2020-12-01 12:20:57 +10:00
Henry de Valence 4fd9203785 consensus: check Merkle root immediately after difficulty 2020-12-01 10:14:44 +10:00
Henry de Valence 4fa119dd1f chain: fix consensus-critical coinbase encoding bug
The `CoinbaseData` parses the block height separately from the rest of the
free-form coinbase data.  However, it had two bugs:

1. It did not require that the height was canonically encoded;
2. Its canonical encoding was incorrect relative to the BIP34-inherited encoding.

This meant that we computed some transaction hashes incorrectly, because when
we re-serialized the coinbase transaction, we would canonically serialize the
coinbase transaction (using the incorrect definition of canonical, bug 2).  And
we didn't notice that the wrong definition of canonical encoding was being used
because we accepted what we thought were non-canonically encoded heights.

The relevant rules are here: 877212414a/src/script/script.h (L307-L346)

This commit changes the encoding to reject non-canonically encoded heights, and
to match the correct encoding rules.  We check that at least one
non-canonically encoded height is correctly rejected using a new test vector.

The database format increments because we saved a bunch of wrongly encoded blocks.

This discrepancy was originally noticed by @teor2345, who pointed out that a
previous version of the block 202 test vector (now preserved as "bad block
202") did not match the block from zcashd.
2020-12-01 10:14:44 +10:00
teor 15be1b81cb Show transaction hashes on merkle failure
Also show the block height and block hash.
2020-12-01 10:14:44 +10:00
Henry de Valence f7baee77ff test: add mainnet block 202 test vector 2020-12-01 10:14:44 +10:00
Henry de Valence 4906a191f9 consensus: check for duplicate transactions in blocks
Change the Merkle root validation logic to also check that a block does not
contain duplicate transactions.  This check is redundant with later
double-spend checks, but is a useful defense-in-depth.
2020-12-01 10:14:44 +10:00
Henry de Valence 706f1fff81 consensus: tidy merkle validation in checkpointer
This moves it in with the existing `check_block` method and expands that
method's contract to cover general block validation checks.
2020-12-01 10:14:44 +10:00
Henry de Valence 7c08c0c315 consensus: check Merkle roots
As a side effect of computing Merkle roots, we build a list of
transaction hashes.  Instead of discarding these, add them to
PreparedBlock and FinalizedBlock so that they can be reused rather than
recomputed.

This commit adds Merkle root validation to:

1. the block verifier;
2. the checkpoint verifier.

In the first case, Bitcoin Merkle tree malleability has no effect,
because only a single Merkle tree in each malleablity set is valid (the
others have duplicate transactions).

In the second case, we need to check that the Merkle tree does not contain any
duplicate transactions.

Closes #1385
Closes #906
2020-12-01 10:14:44 +10:00
Henry de Valence 738b5b0f1b chain: implement Bitcoin Merkle root computation 2020-12-01 10:14:44 +10:00
teor 440e183d32
Decrease the UTXO request timeout (#1407)
UTXO requests during transaction input verification can time out because:

1. The block that creates the UTXO is queued for download or verify, but
   it hasn't been committed yet. The creating block might spend UTXOs
   that come from other recent blocks, so UTXO verification can depend on
   a (non-contiguous) sequence of block verifications.

   In this case, Zebra should wait for additional block download and
   verify tasks to complete.

2. The block that creates the UTXO isn't queued for download. This can
   happen because the block is gossiped block that's much higher than the
   current tip, or because a peer sent the syncer a bad list of block
   hashes.

   In this case, Zebra should discard the timed out block, and restart
   the sync.

We need to choose a timeout that balances these two cases, so we time
out after 180 seconds.

Assuming Zebra can download at least 1 MB per second, 180 seconds is
enough time to download a few hundred blocks. So Zebra should be able to
download and verify the next block before the UTXOs that it creates time
out. (Since Zebra has already verified all the blocks before the next
block, its UTXO requests should return immediately.)

Even if some peers time out downloads, a block can only be pending
download for 80 seconds (4 retries * 20 second timeout) before the
download fails. So the UTXO timeout doesn't need to be much larger than
this overall download timeout - because the download timeout will happen
first on slow networks.

Alternately, if the download for the creating block was never queued,
Zebra should timeout as soon as possible - so it can restart the sync
and download the creating block.

As a side-effect, a lower UTXO timeout also makes it slightly easier to
debug UTXO issues, because unsatisfiable queries fail faster.
2020-12-01 10:12:25 +10:00
Henry de Valence 1df9284444 zebrad: add a use_color option to the tracing config.
This is useful for creating searchable logs without having to filter color codes after the fact.
2020-11-30 15:25:50 -08:00
Henry de Valence e8c16b172f zebrad: pass TracingSection to Tracing component 2020-11-30 15:25:50 -08:00
Alfredo Garcia 4544463059
Inbound `FindBlocks` and `FindHeaders` (#1347)
* implement inbound `FindBlocks`
* Handle inbound peer FindHeaders requests
* handle request before having any chain tip
* Split `find_chain_hashes` into smaller functions

Add a `max_len` argument to support `FindHeaders` requests.

Rewrite the hash collection code to use heights, so we can handle the
`stop` hash and "no intersection" cases correctly.

* Split state height functions into "any chain" and "best chain"
* Rename the best chain block method to `best_block`
* Move fmt utilities to zebra_chain::fmt
* Summarise Debug for some Message variants

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
2020-12-01 07:30:37 +10:00
teor d007c76488 Turn a chain length check into an assert 2020-12-01 07:27:30 +10:00
teor fb2f2a97f2 Delete an outdated canopy activation comment 2020-12-01 07:27:30 +10:00
teor d1ba1146d4 Add intra-doc links 2020-12-01 07:27:30 +10:00
teor 1e4ce74c93 Turn the relevant chain into a Vec before using it
Some checks use the same blocks, so we take a copy of the block borrows
before using them. That way, we don't have to manage the position of the
iterator between checks.
2020-12-01 07:27:30 +10:00
teor 712dd9ddf3 Make a module `pub(crate)` rather than `pub` 2020-12-01 07:27:30 +10:00
teor b6ce509363 Make order of operations clearer 2020-12-01 07:27:30 +10:00
teor ec6ef93b7b Simplify an ExpandedDifficulty division 2020-12-01 07:27:30 +10:00
teor d64c2976e3 Rewrite iterator processing using unzip
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
2020-12-01 07:27:30 +10:00
teor 91476535d3 Doc comment formatting
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
2020-12-01 07:27:30 +10:00
teor 678e6ad090 Implement difficulty_threshold_is_valid 2020-12-01 07:27:30 +10:00
teor 750f096a99 Implement testnet minimum difficulty 2020-12-01 07:27:30 +10:00
teor bb9c4918bf Implement threshold_bits 2020-12-01 07:27:30 +10:00
teor f0a49d64bf Split out a median_timespan function 2020-12-01 07:27:30 +10:00
teor 054d6f0525 Implement median_timespan_bounded 2020-12-01 07:27:30 +10:00
teor 75519b0ae9 Implement averaging_window_timespan 2020-12-01 07:27:30 +10:00
teor bcabf75fe9 Replace integer lengths with named constants 2020-12-01 07:27:30 +10:00
teor e07b0bc8da Implement median_time
And enough stubs to run it on real data.
2020-12-01 07:27:30 +10:00
teor 741c44cd55 Implement mean_target_difficulty
And enough stub code to actually run it on the context.
2020-12-01 07:27:30 +10:00
teor 939c2b97a6 Implement AdjustedDifficulty creation
Also:
* call the difficulty check from `block_is_contextually_valid`
* add a stub `difficulty_threshold_is_valid` function
2020-12-01 07:27:30 +10:00
teor fa03b83351 Update some contextual validation comments and error messages 2020-12-01 07:27:30 +10:00
teor 1bf5ff07fb Fix a state config comment 2020-11-30 15:57:46 -05:00