Commit Graph

1056 Commits

Author SHA1 Message Date
teor 2550c44d48
Make sync ignore known hashes (#853)
* fix: Handle known ObtainTips correctly

enumerate never returns a value beyond the end of the vector.

* fix: Ignore known tips in ExtendTips

Some peers send us known tips when we try to extend.

* fix: Ignore known hashes when downloading

Despite all our other checks, we still end up downloading some hashes
multiple times.

* fix: Increase the number of retries

The old sync code relied on duplicate block fetches to make progress,
but the last few commits have removed some of those duplicates.

Instead, just retry the fetches that fail.

* fix: Tweak comments

Co-authored-by: Jane Lusby <jlusby42@gmail.com>

* fix: Cleanup the state_contains interface in Sync

* Fix brackets

Oops

Co-authored-by: Jane Lusby <jlusby42@gmail.com>
2020-08-10 16:17:50 -07:00
Alfredo Garcia c9093e4d59
Make more checks in non server acceptance tests (#860)
* make sure no info is printed in non server tests

* check exact full output for validity instead of log msgs

* add end of output character to version regex

* use coercions, use equality operator

Co-authored-by: Jane Lusby <jlusby42@gmail.com>

Co-authored-by: Jane Lusby <jlusby42@gmail.com>
2020-08-10 12:50:48 -07:00
Alfredo Garcia 9c387521bd
Print endpoint addresses at startup (#867)
* print tracing and metrics endpoints in startup

* print network address in startup
2020-08-10 12:47:26 -07:00
Henry de Valence f8a24f9c11
Rename script fields to lock/unlock scripts. (#866)
This is slightly clearer and more correct than the previous pk_script.
2020-08-10 11:49:38 -07:00
dependabot[bot] 4ec011f32d build(deps): bump indexmap from 1.5.0 to 1.5.1
Bumps [indexmap](https://github.com/bluss/indexmap) from 1.5.0 to 1.5.1.
- [Release notes](https://github.com/bluss/indexmap/releases)
- [Commits](https://github.com/bluss/indexmap/compare/1.5.0...1.5.1)

Signed-off-by: dependabot[bot] <support@github.com>
2020-08-10 06:06:00 -04:00
teor 98659ac565 fix: Stop duplicating genesis hashes in the block locator
And add some tests to avoid future bugs.
2020-08-10 18:58:21 +10:00
teor e95358dbe3 fix: Increase the number of retries
The old sync code relied on duplicate block fetches to make progress,
but the last few commits have removed some of those duplicates.

Instead, just retry the fetches that fail.
2020-08-10 18:58:21 +10:00
teor 23469b2347 fix: Only ask for blocks within the reorg limit
Instead of creating a block locator all the way back to the genesis
block, only ask for blocks within the reorg limit (99 blocks).

Use the reorg limit as the final locator. (Or if the chain is less
than 99 blocks, use the genesis block.)

Fixes some instances of #818 at very small block heights.
2020-08-10 18:58:21 +10:00
teor 1d3dd35175 fix: Include the current tip in the block locator
The state service was providing block locators starting at the parent of
the current tip. Instead, include the current tip in the block locator.

Also handle an edge case where we could include the genesis block twice,
if the current tip height was a power of two.

Fixes an instance of #818 where we re-download the current tip.
2020-08-10 18:58:21 +10:00
teor 7afd76f5fb feature: Add light_client_root_hash(network) to Block
* use the right variant in LightClientRootHash::from_bytes()
* make block.header.light_client_root_hash pub(super)
* add tests for LightClientRootHash and block.light_client_root_hash
2020-08-10 18:54:42 +10:00
teor ee6f0de14d refactor: Move NetworkUpgrade to zebra-chain 2020-08-10 18:54:42 +10:00
teor c4eb136426 feature: Add a LightClientRootHash type 2020-08-10 18:54:42 +10:00
teor 53ce01fd23 fix: Rename the light client hash field 2020-08-10 18:54:42 +10:00
teor faac50697c feature: Add a verified blocks metrics counter
We have a counter for pending "download and verify" futures. But these
futures are spawned, so they can complete in any order. They can also
complete before we receive their results.
2020-08-10 15:12:08 +10:00
teor 6aeefcee8b fix: Improve sync diagnostics 2020-08-10 15:12:08 +10:00
teor 65c3c503d5 fix: Improve verifier diagnostics
Log a message with the height when we get duplicate blocks.

Downgrade some verifier errors and warnings to info and debug, because
some peers on mainnet consistently provide bad blocks.
2020-08-10 15:12:08 +10:00
Jane Lusby a6b9de6938
Add a definitions section to our RFC template (#846)
* Add a definitions section to our RFC template

* Update 0000-template.md
2020-08-07 13:29:00 -04:00
teor 62225653f0 fix: Reduce the previous block waiting time
Reduce the amount of time that the block verifier waits for the previous
block, before polling again.

(Waiting for 2 seconds resulted in some apparent block verifier hangs.)

This is a temporary fix, until the state layer handles context checks.
2020-08-07 13:28:02 -04:00
dependabot[bot] 3de99bd360
build(deps): bump color-eyre from 0.5.0 to 0.5.1 (#855)
Bumps [color-eyre](https://github.com/yaahc/color-eyre) from 0.5.0 to 0.5.1.
- [Release notes](https://github.com/yaahc/color-eyre/releases)
- [Commits](https://github.com/yaahc/color-eyre/commits)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-08-07 10:12:50 -07:00
Henry de Valence 6d1a4b2218
Load config after initializing the Terminal (#848) 2020-08-06 17:22:40 -07:00
Alfredo Garcia c52481c041 fix logs 2020-08-07 09:21:57 +10:00
Henry de Valence bdf66387d6
Reorganize the book. (#843)
* Reorganize the book.

This PR has one unfortunate change, which is that the README.md and
CONTRIBUTING.md files in the book are symlinks to files in the parent
directory.  The motivation for this is to ensure that we don't maintain two
copies of the same data, and that the landing page of the website matches the
landing page of the Github repo, etc.   However, I'm not sure whether these
symlinks will work correctly on Windows.

The alternatives are:

- Duplicate the contents of the files and expect that people will know to keep
  them in sync;

- Use relative links `../../README.md` in the `SUMMARY.md`.  This seemed like
  it caused mdbook to dump the rendered files into the repository root rather
  than keeping them in the `book` directory.

- Use a symlink (chosen option).  This may not work on Windows but I think that
  the worst outcome would be that the book would be unbuildable unless someone
  used WSL or something.  This seems like the least bad option.

* Remove symlinks in favor of #include 

Turns out the symlinks aren't required!
2020-08-06 15:39:54 -07:00
Henry de Valence 3d46ab746a
Clean up options in network config section. (#839)
Closes #536.

This removes:

- the user-agent (we can add a mechanism to specify extra BIP14 components later, if any users ask us for that feature);
- the EWMA parameters (these were put in the config just to avoid making a choice);
- the peer connection timeout (we can change the default value if anyone ever has a problem with it);
- the peer set request buffer size (setting this too low can make the application deadlock);

The new peer interval is left in.
2020-08-06 11:29:00 -07:00
Jane Lusby 3e9c6f054b
fix log level default for server commands (#840)
* fix log level default for server commands

* remove dbg
2020-08-06 11:23:00 -07:00
Henry de Valence a77328ad7c
Refactor tracing components (#834)
* Split tracing component code into modules.

* Repatriate Tracing and simplify config handling.

We upstreamed our Tracing component, expecting not to have to exert fine
control over the tracing settings.  But this turned out not to be the case, and
now that we want to do other things (flamegraphs, journalctl, opentelemetry,
etc), we end up with really awkward code (as in the current flamegraph
handling).

This also makes use of the changes to `init()` to load the config early to pass
configuration data into the components, which avoids the need for the
refactoring in #775.

Finally, we restore support for the `-v` flag when the filter is unset.  Closes #831.

* Disable tracing and metrics endpoints by default.

Closes #660.

* Switch back to upstream Abscissa.

* Integrate flamegraph support into the new Tracing component.

* Pass -v in acceptance tests to get info-level output.

* Clean up acceptance test code.
2020-08-06 10:29:31 -07:00
dependabot[bot] c0caba401c build(deps): bump structopt from 0.3.15 to 0.3.16
Bumps [structopt](https://github.com/TeXitoi/structopt) from 0.3.15 to 0.3.16.
- [Release notes](https://github.com/TeXitoi/structopt/releases)
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md)
- [Commits](https://github.com/TeXitoi/structopt/commits)

Signed-off-by: dependabot[bot] <support@github.com>
2020-08-06 06:22:55 -04:00
Henry de Valence 6c379be7f2 consensus: temporarily allow dead code in zebra_consensus::primitives. 2020-08-05 21:28:55 -07:00
Henry de Valence 2efb4eb262 consensus: add a static redjubjub::VERIFIER with fallback 2020-08-05 21:28:55 -07:00
Henry de Valence 8d4f154c9f consensus: move redjubjub to new primitives module
This will let it sit alongside a groth16 verifier.
2020-08-05 21:28:55 -07:00
Henry de Valence a987394782
Add KaTeX to rendered docs. (#832)
This adds a single test formula to the `zebra-chain` docs.  We can remove it
after merging this and running it through CI to confirm that it works.
2020-08-05 17:34:30 -07:00
Jane Lusby 867dd0b475
Setup tracing-flame for use profiling zebrad (#436)
* Setup tracing-flame for use profiling zebrad

* start work on conditional flamegraph generation

* review time!

* update comments

* Update Cargo.toml

* disable default features for inferno

* reorganize

* missing one trait

* Apply suggestions from code review

* graceful shutdown!

* remove special case handling on ctrlc for cleanup

* rename signal fn to better represent its responsibility

* remove unused global hook for flushing flamegraph

* move tracing logic to the right file

* just copy linkerd's signal handling logic

* update book

* make zebrad app drop on shutdown normally

* Update zebrad/src/components/tokio.rs

Co-authored-by: teor <teor@riseup.net>

* Update zebrad/src/application.rs

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* cleanup a little

* ooh yea there's an API for that

* setup env-filter for backup subscriber

* document env filter

* document return codes

* forgot to save

* Update book/src/applications/zebrad.md

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2020-08-05 16:35:56 -07:00
Alfredo Garcia ded273413a
Use transcripts in block verify tests (#675)
* change consensus block tests to transcripts

* fix lints

Co-authored-by: Jane Lusby <jlusby42@gmail.com>

Co-authored-by: Jane Lusby <jlusby42@gmail.com>
2020-08-05 14:41:41 -07:00
Henry de Valence 4a03d76a41
Remove environment variables in favor of documented config options. (#827)
* Load tracing filter only from config and simplify logic.

* Configure the state storage in the config, not an environment variable.

This also changes the config so that the path is always set rather than being
optional, because Zebra always needs a place to store its config.
2020-08-05 11:48:08 -07:00
teor d49c11ee70 fix: Allow PROPTEST_CASES to override the lower block case default 2020-08-05 02:07:05 -07:00
Henry de Valence 82da4a5326 Remove connect command. 2020-08-04 23:34:45 -07:00
Deirdre Connolly 1f57b3a948 Update data-flow.md 2020-08-04 22:44:39 -07:00
Deirdre Connolly 6024ffdc97 Update data-flow.md 2020-08-04 22:44:39 -07:00
Deirdre Connolly f2f5641691 Move data-flow.md to design/ 2020-08-04 22:44:39 -07:00
Deirdre Connolly a99330f025 Create data-flow.md 2020-08-04 22:44:39 -07:00
teor ac7a4ae517 fix: Speed up the block round trip proptest
Reduce the number of cases run by the block round trip proptest, to
speed up the Zebra tests.
2020-08-04 22:35:01 -07:00
Alfredo Garcia 70597c1da3
Add testnet blocks to zebra_state (#770)
* add testnet blocks, use them in state tests

* remove mutex and just add more transcripts

Co-authored-by: Jane Lusby <jane@zfnd.org>
2020-08-04 13:39:32 -07:00
Alfredo Garcia e037466e26
Acceptance tests - check kill signal (#814)
* check kill signal exit code

* change names and add docs

* change exit_status() to was_killed()

* change assert calls
2020-08-04 13:38:39 -07:00
teor 78b5bf5e9a refactor: Move the difficulty tests into their own file 2020-08-04 15:24:52 -04:00
teor c95d980bc2 doc: Explain current and minimum network protocol versions 2020-08-04 15:11:16 -04:00
teor ffb6aacf33 feature: Check difficulty in BlockVerifier 2020-08-04 12:29:07 +10:00
teor efb9313460 fix: Resolve a type ambiguity in a proptest
For some reason, implementing
`PartialOrd<BlockHeaderHash> for ExpandedDifficulty` (or its converse)
makes this unrelated code ambiguous.
2020-08-04 12:29:07 +10:00
teor c8b502a5da feature: Compare ExpandedDifficulty and BlockHeaderHash 2020-08-04 12:29:07 +10:00
teor b7fac7b3bc feature: Make ExpandedDifficulty use U256 internally
Adds a dependency on the primitive-types crate.

Also adds custom hex debug formatting for compact and expanded
difficulties.
2020-08-04 12:29:07 +10:00
teor c04d1b7b8f doc: Consensus-critical difficulty encodings 2020-08-04 12:29:07 +10:00
teor 06157a7c8e
Update the BlockHeader fields for Heartwood (#767)
The Heartwood upgrade changes the meaning of the hashFinalSaplingRoot to
hashLightClientRoot. Since we don't know the network upgrade heights in
zebra-chain, we just use [u8; 32] for now.
2020-08-04 12:15:28 +10:00