Commit Graph

228 Commits

Author SHA1 Message Date
Pili Guerra 26b3a50e01
Updates for zebra v1.0.0-beta.1 release (#3073)
* Update versions for zebra v1.0.0-beta.1 release

* Adding original PR list for comparison and tracking as PRs merge

* First pass at categorising changes

* Merge and clarify description of related changes

* Remove or merge trivial changes

* Improve change descriptions

* Add new PRs merged

* CHANGELOG: Improve release summary

* CHANGELOG: categorise changes further

* README: Remove resolved issues and items

* Update CHANGELOG.md

Co-authored-by: teor <teor@riseup.net>

* CHANGELOG: Add new PRs merged

* CHANGELOG: Move change category

* CHANGELOG: Update release date ready for tagging

Co-authored-by: teor <teor@riseup.net>
2021-11-19 13:05:11 +01:00
Dimitris Apostolou afb8b3d477
Fix typos (#3055)
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-12 19:30:22 +00:00
teor 01e63da3fe
Create CompactSize64 for non-message-length fields (#3008)
* Add CompactSize64 and CompactSizeMessage types

But don't remove read_compactsize and write_compactsize yet.

* Fix CompactSize capitalisation

```sh
fastmod compactSize CompactSize zebra* book
fastmod compactsize CompactSize zebra* book
```

* Make CompactSize patterns consistent with integer lengths

* Replace unwrap_err with asserting is_err

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Replace a panic with an assertion

* Make generic serialization use CompactSizeMessage

* Fix type inference and borrow-checker errors

* Doctest fixes

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-04 15:57:30 +00:00
Conrado Gouvea e54917ae7c
V1.0.0-beta.0 (#2973)
* V1.0.0-beta.0

* Bump version in install.md
2021-10-29 20:21:26 +00:00
Conrado Gouvea 4367b232fc
Improve documentation for Beta release (#2894)
* Improve documentation

* Incorporate text about Zebra from the last newsletter

* Organize README and user docs

* Add table of contents, organize heading levels

* Fix link

* capitalize list items

* fix table of contents

* format spacing issue

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-10-29 19:25:30 +00:00
Marek 86d05c5e90
Update the documentation for value pools (#2919)
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-10-21 20:30:11 +00:00
Deirdre Connolly 5afefd55bf
Specify Zebra Client will only support Unifed Addresses (#2706) 2021-08-30 22:40:07 +00:00
Marek 985747861e
Refactor Zebra metrics docs (#2687)
* Refactor Zebra metrics docs

* Refactor Zebra metrics docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-27 20:11:36 +00:00
Conrado Gouvea a4b0b46d21
Describe how to import a Grafana dashboard (#2678) 2021-08-26 18:34:17 +02:00
Marek bacc0f3bbf
Fix a typo in metrics.md (#2658) 2021-08-23 13:56:58 +01:00
teor a1cec27871
Update the signs in the value pools design to match the spec (#2576)
* Update the signs in the value pools design to match the spec

Also split the definitions and guide sections into transaction value balances
and chain value pools. And explain the sign differences between them in terms of
inputs and outputs.

* fix typo

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-06 22:00:02 +00:00
teor 7218aec19f
Design: Fix Transparent Value Sign and Definition (#2555)
* Design: Fix Transparent Value Sign and Definition

* make the transparent value pool the sum of unspent output values, like other pools
* swap the sign of the transparent value balance
* make the remaining transaction value balance the sum of unspent input values
* update the consensus rules based on recent spec changes
* explain why the transparent chain value pool balance can't be negative
* clarify the guide and consensus rules sections

* Use consistent notation for sprout value balances

* Fix typos

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-02 23:52:21 +00:00
Alfredo Garcia fba8a63912
Add value pools design to book summary (#2520)
* add value pools design to book summary
2021-07-23 14:49:47 +01:00
teor ccf93cf5c6
Update state RFC for incremental trees, value pools, and RocksDB (#2456)
* Update state RFC for double-spends and other recent designs

* Update the value pool column family name

* Mark incremental note commitment trees as tentative

* Change history tree type

* Apply suggestions from code review
2021-07-12 10:33:02 -03:00
Alfredo Garcia d1ab8a8946
Value pools design (#2430)
* add a draft for value pools design

* add some `ValueBalance` operators

* add `value_balance` methods to modules

* fix some minors

* finalize the last part of the implementation design

* replace wrong AllowNegative with correct NegativeAllowed

* add design PR to header

* update scope

* remove details from transaction `value_balance()`

* update definitions

* change method name

* return Result in operators

* fix the TODOs

* implement `UpdateWith<PreparedBlock>` for `ValueBalance`

* add details to `ValueBalance` serialization

* add a panic to block value balance

* fix `remaining_transaction_value()`

* add the block value balance into `FinalizedState`

* populate the `Chain` with the finalized tip value balance

* remove redundant text from definition

* add docs to `Chain` field

* trigger the `remaining_transaction_value()` check

* fix mistake

* add some tests, remove some not needed sections

* add a summary of the implementation

* do some minor fixes to consensus rules text

* clarify some names

* fix `UpdateWith`

* move the remaining transaction value consensus rule

* fix serialization bug

* fix typo

* Add a missing test

* typo

Co-authored-by: Conrado Gouvea <conradoplg@gmail.com>

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conradoplg@gmail.com>
2021-07-09 09:53:14 +10:00
teor 0be7470601
Modify UTXO and state designs for transparent coinbase output checks (#2413)
* Modify UTXO and state designs for transparent coinbase output checks

* Add missing word

* Clarify unspent transparent transaction outputs

And add a definition for OutPoint

* Fix coinbase flag derivation

* Rename SpendPools to SpendRestriction

* Replace coinbase_height with Coinbase.height

And clarify a height to spend_height

* Make block height italic to match spec formatting

* Provide a value for MIN_TRANSPARENT_COINBASE_MATURITY

* Add new v1-4 transaction ID consensus rule

* Make the design match the existing Utxo code

This minimises the changes we'll need to make.

* Add a justification for parallel coinbase verification

* Change to Response::SpendableUtxo

To avoid confusion.

* Update the state RFC to match the current design

* Move a change to another PR to avoid conflicts

* Fix spacing

* Clarify the future returned by `PendingUtxos::queue(u)`

* Add missing PendingUtxos::queue arguments
2021-07-09 07:11:43 +10:00
Deirdre Connolly 06eddf7fb3
Add notes about note commitment trees storage (#2259)
* Add notes about note commitment trees storage

* Updates

* Update book/src/dev/rfcs/drafts/0005-treestate.md

Co-authored-by: teor <teor@riseup.net>

* Update book/src/dev/rfcs/drafts/0005-treestate.md

Co-authored-by: teor <teor@riseup.net>

* Update book/src/dev/rfcs/drafts/0005-treestate.md

Co-authored-by: teor <teor@riseup.net>

* Update book/src/dev/rfcs/drafts/0005-treestate.md

Co-authored-by: teor <teor@riseup.net>

* Update book/src/dev/rfcs/drafts/0005-treestate.md

Co-authored-by: teor <teor@riseup.net>

* Update book/src/dev/rfcs/drafts/0005-treestate.md

Co-authored-by: teor <teor@riseup.net>

* Update book/src/dev/rfcs/drafts/0005-treestate.md

Co-authored-by: teor <teor@riseup.net>

* Update book/src/dev/rfcs/drafts/0005-treestate.md

Co-authored-by: teor <teor@riseup.net>

* Update book/src/dev/rfcs/drafts/0005-treestate.md

Co-authored-by: teor <teor@riseup.net>

* Update book/src/dev/rfcs/drafts/0005-treestate.md

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2021-07-02 02:52:19 -04:00
Deirdre Connolly e7b4abcbad
Add a note about maybe using the tui crate for zebra-cli (#2367) 2021-06-23 08:12:07 +10:00
teor 30b3212261 Reference Zebra ticket 2021-06-18 14:45:18 -04:00
teor f15aa52751 Client RFC: add BlazeSync and cap-std 2021-06-18 14:45:18 -04:00
Alfredo Garcia 3dcd407d66
Security: Stop panicking on invalid reserved orchard::Flags bits (#2284)
* stop panicking in deserialize orchard flags

* make the test simpler

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-06-15 10:04:18 +10:00
teor 71c10af7d9
RFC: Add portability risk for atomics (#2269) 2021-06-11 12:22:59 +10:00
teor b07b825286
RFC: add more background to atomics (#2260)
* RFC: add more background to atomics

- Provide explicit tested alternatives to atomics
- Explain the differences between x86 and other processors

* Add nomicon link and further explanation
2021-06-08 18:08:41 -03:00
teor 3d2fb1c13e
Update GitHub and RFC templates based on retrospectives (#2242)
Updates:
- GitHub Issue templates
- GitHub PR templates
- RFC template

Focusing on:
- consensus rule / network reference sections
- design sections
- review/test checklist

Process changes:
- add new team members to RFC approval
- change RFC approval to "most of the team"

And general cleanup:
- delete docs from the checklist, because we now `warn(missing_docs)`
- shorter explanations
- consistent headings
- consistent order
- consistent formatting
2021-06-04 14:40:53 +10:00
teor 2f0f379a9e
Standardise clippy lints and require docs (#2238)
* Standardise lints across Zebra crates, and add missing docs

The only remaining module with missing docs is `zebra_test::command`

* Todo -> TODO

* Clarify what a transcript ErrorChecker does

Also change `Error` -> `BoxError`

* TransError -> ExpectedTranscriptError

* Output Descriptions -> Output descriptions
2021-06-04 08:48:40 +10:00
Alfredo Garcia a9fe0d9d3e
Make sure the mandatory checkpoint includes Canopy activation (#2235)
* Make sure the Canopy activation block is a finalized checkpoint block

This enables ZIP-221 chain history from Canopy activation onwards.

* Clarify that the mandatory checkpoint test includes Canopy activation

The test was correct, but the docs and assertion message did not include activation.

* Document that the mandatory checkpoint includes Canopy activation

Co-authored-by: teor <teor@riseup.net>
2021-06-03 10:24:08 +10:00
Alfredo Garcia 1685611592
Store orchard nullifiers into the state (#2185)
* add nullifier methods to orchard
* store orchard nullifiers
* bump database version
* update `IntoDisk`
* support V5 in `UpdateWith`
* add a test for finalized state
* Use the latest network upgrade in state proptests
2021-06-01 17:53:13 +10:00
Conrado Gouvea f222bf94ea
Update Zebra Book TOC (#2124) 2021-05-07 10:50:51 -03:00
teor 328f343515
RFC: add tokio-console/TurboWish prototype (#2111) 2021-05-06 09:59:29 +10:00
teor 74fd2a65fe
Rename feature to match RFC filename 2021-05-04 10:20:52 +10:00
teor 6f2dbd9de8
Async in Zebra RFC (#1965)
* Initial async RFC version

* Add a table of contents

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Add a toc anchor

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Add some words that need definitions

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Write guide intro based on feedback
* Add a code example for each reference section
* Link to code examples using commit hashes
* Link to PR and commit for each code example

* Fix typos

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Remove redundant version in docs.rs link

* Link the guide to the reference

And expand the guide descriptions

* Mention TurboWish as a future diagnostic tool
* Add an example of a compiler error that prevents deadlock

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-05-03 21:00:49 -03:00
teor 0793c903b4
Clarify that orchard::ShieldedData has an orchard::tree::Root 2021-05-04 05:51:26 +10:00
Alfredo Garcia a63c2e8c40 Update book/src/dev/proptests.md
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
2021-04-29 18:29:53 -04:00
teor 661a8d57dc Explain how to derive arbitrary impls 2021-04-29 18:29:53 -04:00
teor 4b948e67a7
Add extra crate dependencies to the proptest docs (#2071) 2021-04-27 09:42:22 -03:00
teor 3377a486c6 Document how Zebra does cross-crate proptests 2021-04-26 14:07:04 -04:00
Kirill Fomichev 5b2f1cdfd5
Add journald support through tracing-journald (#2034)
* Add journald support through tracing-journald

* change journald to use_journald

* more fixes
2021-04-22 09:31:06 +10:00
teor 53779d2a3c
Redesign Sapling data model for V5 shared anchor and spends (#2021)
* Redesign Sapling data model for V5 shared anchor and spends

The shared anchor is only present if there are any spends.

As part of this change, delete the manual PartialEq impl and its tests,
because we can derive PartialEq now.

* Stop creating a temporary Vec for the spend and output iterators

* Rename TransferData variants

Interactive rename using the following commands:
```sh
fastmod Spends SpendsAndMaybeOutputs
fastmod NoSpends JustOutputs
```

* Refactor out common sprout nullifier code

* Implement the AtLeastOne constrained vector type

This vector wrapper ensures that it always contains at least one element.

* Simplify Sapling TransferData using AtLeastOne

Also update the RFC to use AtLeastOne for Orchard.
2021-04-20 16:22:25 +10:00
Alfredo Garcia e42442d48b
Redesign Transaction V5 serialization, impl trusted vector security, nullifier utility functions (#1996)
* add sapling shielded data to transaction V5

* implement nullifiers

* test v5 in shielded_data_roundtrip

* Explicitly design serialization for Transaction V5

Implement serialization for V4 and V5 spends and outputs, to make sure
that the design works.

* Test serialization for v5 spends and outputs

Also add a few missing v4 tests.

* Delete a disabled proptest

* Make v5 transactions a top-level heading

And add a missing serialized type.

* Fix a comment typo

* v5 transaction RFC: split array serialization

Based on #2017

* RFC: explicitly describe serialized field order

And link to the spec

* RFC: add the shared anchor serialization rule test

Co-authored-by: teor <teor@riseup.net>
2021-04-16 08:19:28 +10:00
Deirdre Connolly 370d0480b4
Merge pull request #983 from ZcashFoundation/treestate-design
Draft RFC: Treestate management
2021-04-15 13:39:34 -04:00
teor 418575458e
Rename the () placeholder to FieldNotPresent (#1987)
* Rename the () placeholder to FieldNotPresent

* Use a unit struct

* Update the RFC
2021-04-07 09:34:58 -03:00
teor 109ca4db86 Zebra Client RFC: add mandatory sweep fast startup option
A mandatory sweep would help `zebra-client` match `zebrad`'s
blazingly fast speeds.
2021-04-01 07:41:10 -04:00
Alfredo Garcia 1302ffe0a1
Draft RFC: Block subsidy (#1129)
* create block subsidy rfc
* add reference in miner subsidy

* remove `or 0` from function descriptions.

It is clear from the signature that the function will return `Error` on any failure(including input errors, for example a wrong height). Describing this will force us to write it for all functions that returns an error so better remove it.

* add links to subsidy categories
* fix design in Funding streams parameter constants
* fix order in FUNDING_STREAM_RECEIVER_NUMERATORS
* return `PayToScriptHash` in `funding_stream_address`
* add shielded coinbase

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Daira Hopwood <daira@jacaranda.org>

* improve  transparent value pool section
* add a test plan section
* rename `subsidy_is_correct` to `subsidy_is_valid`
* add funding streams address constants
* fix funding stream addresses
* add errors
* remove ending dot for  error descriptions

Co-authored-by: Jane Lusby <jlusby42@gmail.com>

* modify founders reward
* change ECC to BP
* add note about state needed for transparent pool
* split constants
* Tweak block subsidy RFC wording

Co-authored-by: Jane Lusby <jlusby42@gmail.com>

* Clarify transaction fees definition
* Use consistent fee terminology
* Clarify transparent value pool
* Revise implementation plan based on latest priorities
* Change tests based on new priorities
* Fix markdown

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Fix value pool description
* change block and miner subsidy definitions
* fee pool definitions, fix all links to protocol
* add reference to zip209
* add note for MG stream number of addresses

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Daira Hopwood <daira@jacaranda.org>
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
2021-03-30 16:25:41 +10:00
Henry de Valence 0e5a6d7efd
RFC: Release process (#1063)
* rfc: initial draft of release process proposal

This is a version of the notes I posted in slack for preliminary
discussion, slightly reformatted to fit the Zebra RFC template.

@yaahc suggested I post them as an RFC, to have a concrete proposal for
discussion.

* Add draft note and link to ticket and PR

Co-authored-by: teor <teor@riseup.net>
2021-03-30 15:55:23 +10:00
teor 258bd881aa
Draft RFC: Initial draft for basic network integration testing (#1007)
* Initial draft for basic integration testing
* Update draft, comment out template
2021-03-30 15:44:57 +10:00
Henry de Valence 73708eebff
RFC: Transaction diffusion with isolated, minimally-distinguishable Tor connections (Stolon) (#1006)
* stolon: Initial RFC draft.
* rfc: add spec of minimally-distinguished handshake
* Add ticket references and a draft note

Co-authored-by: teor <teor@riseup.net>
2021-03-30 15:33:08 +10:00
teor 0562fa7e23
Clarify anchor names (#1955)
When we're naming the anchors without the corresponding type or struct,
it's not clear if they are shared or per spend.

Rename the fields as `shared_anchor` or `per_spend_anchor`.
2021-03-30 09:22:07 +10:00
Alfredo Garcia 32beef233e
V5 transaction rfc (#1886)
* propose a v5 transaction rfc
* define orchard flags
* Update test sections
* Add security section
* move some code into sapling and orchard crates, do renames
* Move sapling value balance into shielded data
* Add AuthorizedAction type
* Order fields based on last serialized data
* Add the proofsOrchard field
* Delete visibility modifiers for readability

All those `pub`s make the design harder to read.

* Model anchor variants as generic trait types
* Be specific about the network upgrade
* Specify a file for each new type
* Document how the Orchard flags are used

Co-authored-by: teor <teor@riseup.net>
2021-03-25 10:30:30 +10:00
teor 9da220517b Update docs for Sapling to Canopy checkpoint change 2021-03-18 10:13:47 +10:00
Alfredo Garcia a292cbe611 change the mandatory checkpoint to Canopy 2021-03-18 10:13:47 +10:00
Alfredo Garcia 65fa1c6bd9 replace canopy.pdf with protocol.pdf 2021-03-02 16:07:04 -05:00
teor 4a812af5d7 Move design/data-flow to rfcs/drafts 2021-03-01 17:58:50 -05:00
teor 9736abdf07 Update grafana instructions 2021-03-01 16:50:06 -05:00
teor 20486be042 Client design: add "one database per key" alternative 2021-02-16 23:25:45 -05:00
Deirdre Connolly 32ed09262d Stub out Sled trees for zebra-client / blockchain scanning state
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
2021-01-08 19:02:40 -05:00
Deirdre Connolly 31421b55d3 Preallocate 0009 for Zebra Client RFC 2021-01-08 19:02:40 -05:00
Deirdre Connolly 62362ef31e Add more detail about the differences between this design and light client protocol
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
2021-01-08 19:02:40 -05:00
Deirdre Connolly 5d45bd0116 Clarify that zebra-cli will wrap zebra-client, a library that implements wallet stuff
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
2021-01-08 19:02:40 -05:00
Deirdre Connolly 6a4152fe6c Update PR link 2021-01-08 19:02:40 -05:00
Henry de Valence a6b07bdc3d client: further work on client RFC
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-01-08 19:02:40 -05:00
Deirdre Connolly 69a8107c86 Add wip rfc for zebra-client/cli design 2021-01-08 19:02:40 -05:00
Alfredo Garcia d725eeb4d6
Add documentation to zebra-utils and checkpoint generation (#1491)
* create readme for utils and checkpoints
* add link to checkpoint usage to book

Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: teor <teor@riseup.net>
2020-12-14 11:34:22 +10:00
teor 621dc23082 Link to where the deployments docs will be 2020-12-07 21:11:24 -05:00
Pili Guerra 113c27196e Update alpha-release-criteria.md
Update after Team Sync on 2020.12.07
2020-12-07 21:11:24 -05:00
teor 81d19ca196
Move the system requirements to the README 2020-12-08 07:45:05 +10:00
teor d965a4cd24
Check crate versions before a release 2020-12-08 07:36:59 +10:00
teor c75a659d03
Update release criteria (#1469)
Add checkpoint list and known serious issues updates
Update statuses based on recent changes
2020-12-07 12:14:58 +01:00
teor e2792fa656
Add crate dependencies to the release checklist (#1459)
* Add crate dependencies to the release checklist

* Update "Last updated" date

Co-authored-by: Pili Guerra <pili@piliguerra.com>
2020-12-04 11:47:17 +01:00
Pili Guerra 19d25fb7be
WIP: First draft of alpha release criteria for review (#1374)
* WIP: First draft of release criteria for review

* Update release-criteria.md

Fix formatting

* Added more details to release criteria

Co-authored-by: teor <teor@riseup.net>

* Add "Future Releases" section

* Remove Alpha Release criteria items

These should be included and expanded upon in future releases

Co-authored-by: teor <teor@riseup.net>

* Formatting fixes

* Remove support and troubleshooting criteria from first alpha release

* Switching functionality criteria between future and alpha release

* Remove redundant statement from "Network Readiness" section for "Future Releases"

* Go/No-Go checklist should be a living document

Let's make this a living document by making it clear that this reflects the latest status as of the "Last updated" date

* Update release-criteria.md

Update status after Go/No-Go meeting

* Make RAG status symbols more accessible

* Update release-criteria.md

- Remove "Future Releases" section
- Clean up formatting

* Update book/src/dev/release-criteria.md

change "`zebrad` can validate proof of work" from green to amber

Co-authored-by: teor <teor@riseup.net>

* Update book/src/dev/release-criteria.md

Change "Build completes within 30 minutes in Zebra's CI" from green to amber

Co-authored-by: teor <teor@riseup.net>

* Update book/src/dev/release-criteria.md

Change "known panics, errors and warnings have open tickets" and "`zebrad` executes normally" form amber to green

Co-authored-by: teor <teor@riseup.net>

* Rename release-criteria.md to alpha-release-criteria.md

We will have new release criteria for future releases

Co-authored-by: teor <teor@riseup.net>
2020-12-03 15:44:33 +01:00
teor fc7d37c984
RFC: Contextual Difficulty (#1246)
* Difficulty Contextual RFC: Introduction
Add a header, summary, and motivation

* Difficulty RFC: Add draft definitions
And update the state RFC definitions to match.

* Difficulty RFC: Add relevant chain
* Difficulty RFC: draft guide-level explanation
Outline the core calculations and checks.

* Difficulty RFC: Revised based on spec fixes
Update the design based on the spec bugs in #1276, #1277, and
zcash/zips#416.

These changes make the difficulty filter into a context-free check,
so we remove it from this contextual validation RFC.

* Difficulty RFC: Explain how Zebra's calculations can match the spec
* Difficulty RFC: write most of the reference section
Includes most of the implementation, modules for each function, and
draft notes for some of the remaining parts of the RFC.

* Difficulty RFC: Add an AdjustedDifficulty struct
* Difficulty RFC: Summarise module structure in the one place
* Difficulty RFC: Create implementation notes subsections
* Difficulty RFC: add consensus critical order of operations
* Difficulty RFC: Use the ValidateContextError type
* Difficulty RFC: make the median_time arg mut owned

We have to clone the data to pass a fixed-length array to a function,
so we might as well sort that array to find the median, and avoid a
copy.
2020-11-26 11:45:01 +10:00
Deirdre Connolly 2a21c86b91 I before E except after C (or uh, not-english) 2020-11-24 22:23:57 -05:00
teor d2173cae7e
Basic documentation for zebra-checkpoints (#1361) 2020-11-24 10:49:41 +10:00
Jane Lusby 4c9bb87df2
zebra-state: replace sled with rocksdb (#1325)
## Motivation

Prior to this PR we've been using `sled` as our database for storing persistent chain data on the disk between boots. We picked sled over rocksdb to minimize our c++ dependencies despite it being a less mature codebase. The theory was if it worked well enough we'd prefer to have a pure rust codebase, but if we ever ran into problems we knew we could easily swap it out with rocksdb.

Well, we ran into problems. Sled's memory usage was particularly high, and it seemed to be leaking memory. On top of all that, the performance for writes was pretty poor, causing us to become bottle-necked on sled instead of the network.

## Solution

This PR replaces `sled` with `rocksdb`. We've seen a 10x improvement in memory usage out of the box, no more leaking, and much better write performance. With this change writing chain data to disk is no longer a limiting factor in how quickly we can sync the chain.

The code in this pull request has:
  - [x] Documentation Comments
  - [x] Unit Tests and Property Tests

## Review

@hdevalence
2020-11-18 18:05:06 -08:00
teor 0dc38608ef Add missing anchors to the RFC template 2020-11-12 21:59:42 -05:00
Jane Lusby 104b5406d5 Update book/src/dev/rfcs/0005-state-updates.md
Co-authored-by: teor <teor@riseup.net>
2020-11-12 09:14:52 -05:00
Jane Lusby 74af22e5ca Add unit tests for 2020-11-12 09:14:52 -05:00
Jane Lusby 34f50d7ebb
Fix inconsistencies related to best chain order in RFC and state impl (#1267)
Prior to this PR we realized that the RFC had been drafted with the assumption that chains would be ordered from best to worst in `NonFinalizedState`. This assumption was incorrect, since `BTreeSet` only ever orders values in ascending order. This discrepancy was noticed and fixed in the code, but there were still some inconsistencies that needed to be cleaned up.

This PR updates all the incorrect or confusing comments about chain ordering in the RFC and code.
2020-11-09 15:53:16 -08:00
teor 9f261e2213 Fix options order in metrics docker commands
And use long-form command options for readability
2020-11-05 08:39:14 -05:00
teor 776e49ff0c State RFC: clarify difficulty
The difficulty validation RFC will introduce a definition of
per-block difficulty. Make it clear that the state RFC
definition is cumulative difficulty.
2020-10-30 19:12:32 -04:00
teor f338048012 Clarify edge case 2020-10-29 15:41:45 -04:00
teor b742657e45 State RFC: Clarify genesis sprout anchors 2020-10-29 15:41:45 -04:00
Alfredo Garcia 97c93daca7
Remove template section from Pipelinable Block Syncing RFC (#1219)
* remove template section and add summary
2020-10-29 20:10:17 +10:00
teor b4ce442cea
State RFC: Use escaped concatenation operator
And use BE32 consistently
2020-10-27 20:50:29 +10:00
teor adbdb3c76e
Note that `blocks_by_height` is also a bijection 2020-10-27 20:13:49 +10:00
teor 7ea92283a7
Fix a State RFC rendering issue 2020-10-27 20:07:43 +10:00
teor 08910e0378
State RFC: fix block height contextual validation 2020-10-27 19:29:32 +10:00
Jane Lusby 971765ab30
Handle duplicate blocks in zebra-state (#1198)
## Motivation

The zebra-state service needs to be able to handle duplicate blocks.

## Solution

This implements changes already outlined by [The State
RFC](https://zebra.zfnd.org/dev/rfcs/0005-state-updates.html). We check for
successfully committed blocks first, since interacting with the queued blocks
struct at this point just complicates the implimentation. If the block has not
already been committed we then check if the block has already been queued, if
not we handle the block normally (normally here being the bit we already had
implemented).

## Documentation Changes

- [x] Update the state RFC to match the ways this fix departs from the design
	- the main thing is that I switched the order of checking for duplicates
- [x] ~~Add newly added functions to the state rfc~~ Decided not to do this because they're minor getters that don't influence the rest of the design and aren't exposed as part of the API
- [x] Document newly added functions inline

## Testing

## Related Issues

- fixes https://github.com/ZcashFoundation/zebra/issues/1182
- tracking issue https://github.com/ZcashFoundation/zebra/issues/1049

Co-authored-by: teor <teor@riseup.net>
2020-10-26 13:54:19 -07:00
teor fb079c2ca1
Replace BlockHeaderHash with block::Hash 2020-10-26 22:27:57 +10:00
teor a9102e8d6d
Fix State RFC rendering ambiguities 2020-10-26 22:02:45 +10:00
teor 0935b3305a
Fix more state RFC function heading sizes 2020-10-26 21:14:14 +10:00
teor 7bf2fdd6d7
Fix a header level in the state RFC 2020-10-26 21:11:26 +10:00
teor 90e755472c Add data source instructions to the metrics help 2020-10-23 15:06:37 -04:00
teor b492cabeee Bind grafana to localhost in metrics instructions
Binding grafana to localhost makes it inaccessible from the wider internet,
which is a secure default.

Since we run docker with host networking, docker containers have access to D-Bus and other
security-related services on localhost. So it's risky to also expose them to the wider internet.
2020-10-23 15:06:37 -04:00
Alfredo Garcia 21ad6ffc47
Reverse displayed endianness of transaction and block hashes (#1171)
* Reverse displayed endianness of transaction and block hashes
* fix zebra-checkpoints utility for new hash order
* Stop using "zebrad revhex" in zebrad-hash-lookup
* Rebuild checkpoint lists in new hash order
This change also adds additional checkpoints to the end of each list.

* Replace TransactionHash with transaction::Hash
This change should have been made in #905, but we missed Debug impls
and some docs.

Co-authored-by: Ramana Venkata <vramana@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
2020-10-22 07:54:02 +10:00
teor 351e5013ae
Expand the template testing section (#1157)
Based on team discussions
2020-10-14 12:08:52 -07:00
teor 691ad12cc9
Add modules and test plans to the RFC template (#1145) 2020-10-12 12:46:52 -07:00
teor fd0fac3a61
State RFC: Handle duplicate block edge cases (#1136)
Handle the following duplicate block edge cases:
* duplicate in finalized state
* duplicate in queue

* Handle the broadcast channel correctly
2020-10-10 11:50:12 +10:00
Jane Lusby 855f9b5bcb
Implement MVP of NonFinalizedState and integrate it with the state service (#1101)
* implement most of the chain functions
* implement fork
* fix outpoint handling in Chain struct
* update expect for work
* split utxo into two sets
* update the Chain definition
* remove allow attribute in zebra-state/lib.rs
* merge ChainSet type into MemoryState
* Add error messages to asserts
* export proptest impls for use in downstream crates
* add testjob for disabled feature in zebra-chain
* try to fix github actions syntax
* add module doc comment
* update RFC for utxos
* add missing header
* working proptest for Chain
* propagate back results over channel
* Start updating RFC to match changes
* implement queued block pruning
* and now it syncs wooo!
* remove empty modules
* setup config for proptests
* re-enable missing_docs lint
* update RFC to match changes in impl
* add documentation
* use more explicit variable names
2020-10-08 13:07:32 +10:00
Dimitris Apostolou 36279621f0 Fix typos 2020-10-06 12:16:41 +10:00
teor b05273ac16
State RFC: remove duplicate validation steps (#1123) 2020-10-05 19:03:17 -07:00
Jane Lusby 86ed13060f
Add tests for `Chain` implementation (#1093)
* Begin work on RFC5 implementation

* I think this is necessary

* holy shit supertrait implemented via subtrait

* implement most of the chain functions

* change to slightly better name

* implement fork

* fix outpoint handling in Chain struct

* update expect for work

* resolve review comment

* split utxo into two sets

* update the Chain definition

* just a little more

* update comment

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* apply changes from code review

* remove allow attribute in zebra-state/lib.rs

* Update zebra-state/src/memory_state.rs

Co-authored-by: teor <teor@riseup.net>

* merge ChainSet type into MemoryState

* rename state impl types

* Add error messages to asserts

* checkpoint so I can split off arbitrary changes into a PR

* export proptest impls for use in downstream crates

* add testjob for disabled feature in zebra-chain

* run rustfmt

* try to fix github actions syntax

* differentiate name

* prove that github action tests zebra-chain build without features

* revert change from last commit now that test is running

* remove accidentally introduced newline

* checkpoint

* add module doc comment

* update RFC for utxos

* add missing header

* working proptest for Chain

* apply change from chain impl PR

* setup config for proptests

* Update zebra-chain/src/block/arbitrary.rs

Co-authored-by: teor <teor@riseup.net>

* run rustfmt

Co-authored-by: teor <teor@riseup.net>
2020-10-02 15:51:51 -07:00
teor 5ed27dc8a1 State RFC - handle duplicates of finalized blocks 2020-09-29 12:19:56 +10:00
teor df9252c759 Refactor out push_block_on_chain
And tidy up some error handling and panics
2020-09-29 12:19:56 +10:00
teor b264e24cd0 Panic on all commit_block errors 2020-09-29 12:19:56 +10:00
teor 2d183cbff3 Tweak State RFC to handle edge cases
* Reject CommitBlock with pre-sapling blocks: they must use CommitFinalizedBlock
* Clarify adding a new Chain to an empty ChainSet
* Handle duplicate blocks correctly
2020-09-29 12:19:56 +10:00
Jane Lusby 352721bd88
Implement RFC5: State updates `Chain` type (#1069)
* Begin work on RFC5 implementation

* I think this is necessary

* holy shit supertrait implemented via subtrait

* implement most of the chain functions

* change to slightly better name

* implement fork

* fix outpoint handling in Chain struct

* update expect for work

* resolve review comment

* split utxo into two sets

* update the Chain definition

* just a little more

* update comment

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* apply changes from code review

* remove allow attribute in zebra-state/lib.rs

* Update zebra-state/src/memory_state.rs

Co-authored-by: teor <teor@riseup.net>

* merge ChainSet type into MemoryState

* rename state impl types

* Add error messages to asserts

* add module doc comment

* update RFC for utxos

* add missing header

Co-authored-by: teor <teor@riseup.net>
2020-09-24 15:46:04 -07:00
Henry de Valence 16cc095484
RFC: state updates (#902)
* wip

Co-authored-by: Jane Lusby <jlusby42@gmail.com>

* wip2: add nullifiers

Co-authored-by: Jane Lusby <jlusby42@gmail.com>

* Update book/src/dev/rfcs/0003-state-updates.md

Co-authored-by: teor <teor@riseup.net>

* Move to RFC number 5

* rfc: add PR link to state update RFC

* rfc: change state RFC to store blocks by height.

The rationale for this change is described in the document: it means
that we write blocks only to one end of the Sled tree, and hopefully
helps us with spatial access patterns.

This should help alleviate a major cause of memory use in Zebra's
current WIP Sled structure, which is that:

- blocks are stored in random, sparse order (by hash) in the B-tree;
- the `Request::GetDepth` method opens the entire block store and
  queries a random part of its block data to determine whether a hash is
  present;
- if present, it deserializes the complete block data of both the given
  block and the current tip block, to compute the difference in block
  heights.

This access pattern forces a large amount of B-tree data to remain
resident, and could probably be avoided if we didn't do that.

* rfc: add sprout and sapling anchors to sled trees.

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* rfc: fill in details of state service requests.

* rfc: extract commit process from API description

* rfc: add anchor parameters to CommitBlock.

These have to be computed by a verifier, so passing them as parameters
means we don't recompute them.

* WIP for in memory state structs

* tweeks from end of session with henry

* more updates from pairing

* rewrite non-finalized state sections

* update query instructions for each request

* more updates

* updates from pairing with henry

* updates from proofreading solo

* add guide level explanation to state rfc

* add drawbacks section

* Update book/src/dev/rfcs/0005-state-updates.md

Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>

* Apply suggestions from code review

Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>

* Update book/src/dev/rfcs/0005-state-updates.md

Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>

* apply changes from code review

* clarify iteration

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* apply changes from code review

* Update book/src/dev/rfcs/0005-state-updates.md

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* add info about default constructing chains when forking from finalized state

* Update book/src/dev/rfcs/0005-state-updates.md

Co-authored-by: teor <teor@riseup.net>

* move contextual verification out of Chain

Co-authored-by: Jane Lusby <jlusby42@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Jane Lusby <jane@zfnd.org>
2020-09-21 12:05:31 -07:00
Henry de Valence 1d0ebf89c6 zebrad: move seed command into inbound component
Remove the seed command entirely, and make the behavior it provided
(responding to `Request::Peers`) part of the ordinary functioning of the
start command.

The new `Inbound` service should be expanded to handle all request
types.
2020-09-18 18:34:25 -07:00
Jane Lusby 3c40eddc17 Update book/src/dev/rfcs.md 2020-09-17 13:23:58 -07:00
noone 92f3ae2c00 Linkify the reference to the rust RFC process so readers can read up on it easily. 2020-09-17 13:23:58 -07:00
Alfredo Garcia 454e75e7c0
Rename old references to BlockHeaderHash and BlockHeight (#1002)
* rename some references

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: teor <teor@riseup.net>

Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: teor <teor@riseup.net>
2020-09-04 15:40:48 -07:00
Henry de Valence e1e299906f rfc: add number to script verification rfc 2020-08-31 00:58:50 -04:00
Henry de Valence b212becb67 Add issue and PR templates, document RFC process 2020-08-31 00:58:50 -04:00
Deirdre Connolly 50af921d5d Use zebra favicon for zebra.zfnd.org book 2020-08-29 03:33:31 -04:00
Henry de Valence 4561f1d25b
rfc: initial inventory tracking (#952)
* rfc: initial inventory tracking

This just describes the design, not the design alternatives.

* rfc: finish inventory tracking rfc

Also assign it #3.  The async script verification RFC should have had a number
assigned before merging but it didn't.  I don't want to fix that in this PR
because I don't want those changes to block on each other.  The fix is to (1)
document the RFC flow better and (2) add issue templates for RFCs.

* rfc: touch up inventory tracking rfc

* rfc: prune inventory entries generationally.

Based on a suggestion by @yaahc.

* Update book/src/dev/rfcs/0003-inventory-tracking.md

Co-authored-by: Jane Lusby <jlusby42@gmail.com>
2020-08-28 14:19:18 -07:00
Henry de Valence dda0d2dd41 Remove check_block
Per discussion where @yaahc suggested that it would be simpler to delete this function entirely and treat it as an implementation detail.

Co-authored-by: Jane Lusby <jlusby42@gmail.com>
2020-08-21 16:38:33 -04:00
Jane Lusby 7dcac97d0b Update book/src/dev/rfcs/XXXX-asynchronous-script-verification.md
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2020-08-21 16:38:33 -04:00
Henry de Valence a3a4ba858a Add async script RFC 2020-08-21 16:38:33 -04:00
teor 120c7ef648
rfc: Parallel Verification (#763)
* rfc: Parallel Verification Draft

An initial draft RFC for parallel verification.

* rfc: Integrate the CheckpointVerifier design

Describe how the CheckpointVerifier interacts with chain state updates.

* rfc: Add a chain tips update service

* rfc: Add network upgrade context changes

* rfc: Add main chain tip section

* rfc: Clarify and expand genesis block rules

* rfc: More genesis special cases

* Add another chain tips edge case

* Remove the final tie-breaker for tip ties

Instead, change the design to make them impossible.

* rfc: add a definitions section to parallel verification

* rfc: Split parallel verification into two RFCs

This is the semantic verification RFC.

* rfc: Add guide and examples for parallel verification

* rfc: Fix GitHub markdown

* rfc: Fix parallel function design

We don't need separate functions, we can just do the awaits as late as
possible.

* rfc: Fix typo

* rfc: Stop assigning responsibilities to modules

* rfc: Add more parallel verification definitions

* rfc: Say "block height consensus rule"

* rfc: Tidy some of the TODOs

* rfc: Expand rationale and alternatives

* rfc: Delete "try to depend on older blocks"

* rfc: Delete coinbase checks which are unrelated to BlockHeight

And remove some duplicate references to BlockHeight checks.

* rfc: Focus on verification stages

And rewrite some stages for clarity.

* rfc: Remove reference to zebra-network
2020-08-14 11:04:17 -07:00
teor aa922564b4
rfc: Update the RFC template to talk about testing and maintenance (#875) 2020-08-11 13:26:35 -07:00
Jane Lusby a6b9de6938
Add a definitions section to our RFC template (#846)
* Add a definitions section to our RFC template

* Update 0000-template.md
2020-08-07 13:29:00 -04:00
Henry de Valence bdf66387d6
Reorganize the book. (#843)
* Reorganize the book.

This PR has one unfortunate change, which is that the README.md and
CONTRIBUTING.md files in the book are symlinks to files in the parent
directory.  The motivation for this is to ensure that we don't maintain two
copies of the same data, and that the landing page of the website matches the
landing page of the Github repo, etc.   However, I'm not sure whether these
symlinks will work correctly on Windows.

The alternatives are:

- Duplicate the contents of the files and expect that people will know to keep
  them in sync;

- Use relative links `../../README.md` in the `SUMMARY.md`.  This seemed like
  it caused mdbook to dump the rendered files into the repository root rather
  than keeping them in the `book` directory.

- Use a symlink (chosen option).  This may not work on Windows but I think that
  the worst outcome would be that the book would be unbuildable unless someone
  used WSL or something.  This seems like the least bad option.

* Remove symlinks in favor of #include 

Turns out the symlinks aren't required!
2020-08-06 15:39:54 -07:00
Jane Lusby 867dd0b475
Setup tracing-flame for use profiling zebrad (#436)
* Setup tracing-flame for use profiling zebrad

* start work on conditional flamegraph generation

* review time!

* update comments

* Update Cargo.toml

* disable default features for inferno

* reorganize

* missing one trait

* Apply suggestions from code review

* graceful shutdown!

* remove special case handling on ctrlc for cleanup

* rename signal fn to better represent its responsibility

* remove unused global hook for flushing flamegraph

* move tracing logic to the right file

* just copy linkerd's signal handling logic

* update book

* make zebrad app drop on shutdown normally

* Update zebrad/src/components/tokio.rs

Co-authored-by: teor <teor@riseup.net>

* Update zebrad/src/application.rs

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* cleanup a little

* ooh yea there's an API for that

* setup env-filter for backup subscriber

* document env filter

* document return codes

* forgot to save

* Update book/src/applications/zebrad.md

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2020-08-05 16:35:56 -07:00
Henry de Valence 4a03d76a41
Remove environment variables in favor of documented config options. (#827)
* Load tracing filter only from config and simplify logic.

* Configure the state storage in the config, not an environment variable.

This also changes the config so that the path is always set rather than being
optional, because Zebra always needs a place to store its config.
2020-08-05 11:48:08 -07:00
Alfredo Garcia 89ef08793b delete file 2020-07-29 19:44:21 -07:00
Alfredo Garcia 617f1d80ef move docs to zebra book 2020-07-29 19:44:21 -07:00
teor 0ef8e38266 fix: Tweak the RFC template
* replace references to "(Rust) language" with Zebra, Zcash or project as appropriate
* talk about impacts on the broader ecosystem
2020-07-29 11:32:24 +10:00
Jane Lusby e06357f795
Add skeleton of eventual zebra book (#653)
* Add skeleton of eventual zebra book

* reorg sections

* restore file and reorg book a little

* try setting up a firebase deployment

* allow firebase ci to work on test

* download mdbook

* fix book path

* use newer version of mdbook

* remove event hook for book branch pre merge

* Apply suggestions from code review

Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>

Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
2020-07-17 20:15:50 -07:00