* create readme for utils and checkpoints
* add link to checkpoint usage to book
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* WIP: First draft of release criteria for review
* Update release-criteria.md
Fix formatting
* Added more details to release criteria
Co-authored-by: teor <teor@riseup.net>
* Add "Future Releases" section
* Remove Alpha Release criteria items
These should be included and expanded upon in future releases
Co-authored-by: teor <teor@riseup.net>
* Formatting fixes
* Remove support and troubleshooting criteria from first alpha release
* Switching functionality criteria between future and alpha release
* Remove redundant statement from "Network Readiness" section for "Future Releases"
* Go/No-Go checklist should be a living document
Let's make this a living document by making it clear that this reflects the latest status as of the "Last updated" date
* Update release-criteria.md
Update status after Go/No-Go meeting
* Make RAG status symbols more accessible
* Update release-criteria.md
- Remove "Future Releases" section
- Clean up formatting
* Update book/src/dev/release-criteria.md
change "`zebrad` can validate proof of work" from green to amber
Co-authored-by: teor <teor@riseup.net>
* Update book/src/dev/release-criteria.md
Change "Build completes within 30 minutes in Zebra's CI" from green to amber
Co-authored-by: teor <teor@riseup.net>
* Update book/src/dev/release-criteria.md
Change "known panics, errors and warnings have open tickets" and "`zebrad` executes normally" form amber to green
Co-authored-by: teor <teor@riseup.net>
* Rename release-criteria.md to alpha-release-criteria.md
We will have new release criteria for future releases
Co-authored-by: teor <teor@riseup.net>
* Difficulty Contextual RFC: Introduction
Add a header, summary, and motivation
* Difficulty RFC: Add draft definitions
And update the state RFC definitions to match.
* Difficulty RFC: Add relevant chain
* Difficulty RFC: draft guide-level explanation
Outline the core calculations and checks.
* Difficulty RFC: Revised based on spec fixes
Update the design based on the spec bugs in #1276, #1277, and
zcash/zips#416.
These changes make the difficulty filter into a context-free check,
so we remove it from this contextual validation RFC.
* Difficulty RFC: Explain how Zebra's calculations can match the spec
* Difficulty RFC: write most of the reference section
Includes most of the implementation, modules for each function, and
draft notes for some of the remaining parts of the RFC.
* Difficulty RFC: Add an AdjustedDifficulty struct
* Difficulty RFC: Summarise module structure in the one place
* Difficulty RFC: Create implementation notes subsections
* Difficulty RFC: add consensus critical order of operations
* Difficulty RFC: Use the ValidateContextError type
* Difficulty RFC: make the median_time arg mut owned
We have to clone the data to pass a fixed-length array to a function,
so we might as well sort that array to find the median, and avoid a
copy.
## Motivation
Prior to this PR we've been using `sled` as our database for storing persistent chain data on the disk between boots. We picked sled over rocksdb to minimize our c++ dependencies despite it being a less mature codebase. The theory was if it worked well enough we'd prefer to have a pure rust codebase, but if we ever ran into problems we knew we could easily swap it out with rocksdb.
Well, we ran into problems. Sled's memory usage was particularly high, and it seemed to be leaking memory. On top of all that, the performance for writes was pretty poor, causing us to become bottle-necked on sled instead of the network.
## Solution
This PR replaces `sled` with `rocksdb`. We've seen a 10x improvement in memory usage out of the box, no more leaking, and much better write performance. With this change writing chain data to disk is no longer a limiting factor in how quickly we can sync the chain.
The code in this pull request has:
- [x] Documentation Comments
- [x] Unit Tests and Property Tests
## Review
@hdevalence
Prior to this PR we realized that the RFC had been drafted with the assumption that chains would be ordered from best to worst in `NonFinalizedState`. This assumption was incorrect, since `BTreeSet` only ever orders values in ascending order. This discrepancy was noticed and fixed in the code, but there were still some inconsistencies that needed to be cleaned up.
This PR updates all the incorrect or confusing comments about chain ordering in the RFC and code.
The difficulty validation RFC will introduce a definition of
per-block difficulty. Make it clear that the state RFC
definition is cumulative difficulty.
## Motivation
The zebra-state service needs to be able to handle duplicate blocks.
## Solution
This implements changes already outlined by [The State
RFC](https://zebra.zfnd.org/dev/rfcs/0005-state-updates.html). We check for
successfully committed blocks first, since interacting with the queued blocks
struct at this point just complicates the implimentation. If the block has not
already been committed we then check if the block has already been queued, if
not we handle the block normally (normally here being the bit we already had
implemented).
## Documentation Changes
- [x] Update the state RFC to match the ways this fix departs from the design
- the main thing is that I switched the order of checking for duplicates
- [x] ~~Add newly added functions to the state rfc~~ Decided not to do this because they're minor getters that don't influence the rest of the design and aren't exposed as part of the API
- [x] Document newly added functions inline
## Testing
## Related Issues
- fixes https://github.com/ZcashFoundation/zebra/issues/1182
- tracking issue https://github.com/ZcashFoundation/zebra/issues/1049
Co-authored-by: teor <teor@riseup.net>
Binding grafana to localhost makes it inaccessible from the wider internet,
which is a secure default.
Since we run docker with host networking, docker containers have access to D-Bus and other
security-related services on localhost. So it's risky to also expose them to the wider internet.
* Reverse displayed endianness of transaction and block hashes
* fix zebra-checkpoints utility for new hash order
* Stop using "zebrad revhex" in zebrad-hash-lookup
* Rebuild checkpoint lists in new hash order
This change also adds additional checkpoints to the end of each list.
* Replace TransactionHash with transaction::Hash
This change should have been made in #905, but we missed Debug impls
and some docs.
Co-authored-by: Ramana Venkata <vramana@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
* implement most of the chain functions
* implement fork
* fix outpoint handling in Chain struct
* update expect for work
* split utxo into two sets
* update the Chain definition
* remove allow attribute in zebra-state/lib.rs
* merge ChainSet type into MemoryState
* Add error messages to asserts
* export proptest impls for use in downstream crates
* add testjob for disabled feature in zebra-chain
* try to fix github actions syntax
* add module doc comment
* update RFC for utxos
* add missing header
* working proptest for Chain
* propagate back results over channel
* Start updating RFC to match changes
* implement queued block pruning
* and now it syncs wooo!
* remove empty modules
* setup config for proptests
* re-enable missing_docs lint
* update RFC to match changes in impl
* add documentation
* use more explicit variable names
* Begin work on RFC5 implementation
* I think this is necessary
* holy shit supertrait implemented via subtrait
* implement most of the chain functions
* change to slightly better name
* implement fork
* fix outpoint handling in Chain struct
* update expect for work
* resolve review comment
* split utxo into two sets
* update the Chain definition
* just a little more
* update comment
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* apply changes from code review
* remove allow attribute in zebra-state/lib.rs
* Update zebra-state/src/memory_state.rs
Co-authored-by: teor <teor@riseup.net>
* merge ChainSet type into MemoryState
* rename state impl types
* Add error messages to asserts
* checkpoint so I can split off arbitrary changes into a PR
* export proptest impls for use in downstream crates
* add testjob for disabled feature in zebra-chain
* run rustfmt
* try to fix github actions syntax
* differentiate name
* prove that github action tests zebra-chain build without features
* revert change from last commit now that test is running
* remove accidentally introduced newline
* checkpoint
* add module doc comment
* update RFC for utxos
* add missing header
* working proptest for Chain
* apply change from chain impl PR
* setup config for proptests
* Update zebra-chain/src/block/arbitrary.rs
Co-authored-by: teor <teor@riseup.net>
* run rustfmt
Co-authored-by: teor <teor@riseup.net>
* Reject CommitBlock with pre-sapling blocks: they must use CommitFinalizedBlock
* Clarify adding a new Chain to an empty ChainSet
* Handle duplicate blocks correctly
* Begin work on RFC5 implementation
* I think this is necessary
* holy shit supertrait implemented via subtrait
* implement most of the chain functions
* change to slightly better name
* implement fork
* fix outpoint handling in Chain struct
* update expect for work
* resolve review comment
* split utxo into two sets
* update the Chain definition
* just a little more
* update comment
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* apply changes from code review
* remove allow attribute in zebra-state/lib.rs
* Update zebra-state/src/memory_state.rs
Co-authored-by: teor <teor@riseup.net>
* merge ChainSet type into MemoryState
* rename state impl types
* Add error messages to asserts
* add module doc comment
* update RFC for utxos
* add missing header
Co-authored-by: teor <teor@riseup.net>
* wip
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
* wip2: add nullifiers
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
* Update book/src/dev/rfcs/0003-state-updates.md
Co-authored-by: teor <teor@riseup.net>
* Move to RFC number 5
* rfc: add PR link to state update RFC
* rfc: change state RFC to store blocks by height.
The rationale for this change is described in the document: it means
that we write blocks only to one end of the Sled tree, and hopefully
helps us with spatial access patterns.
This should help alleviate a major cause of memory use in Zebra's
current WIP Sled structure, which is that:
- blocks are stored in random, sparse order (by hash) in the B-tree;
- the `Request::GetDepth` method opens the entire block store and
queries a random part of its block data to determine whether a hash is
present;
- if present, it deserializes the complete block data of both the given
block and the current tip block, to compute the difference in block
heights.
This access pattern forces a large amount of B-tree data to remain
resident, and could probably be avoided if we didn't do that.
* rfc: add sprout and sapling anchors to sled trees.
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* rfc: fill in details of state service requests.
* rfc: extract commit process from API description
* rfc: add anchor parameters to CommitBlock.
These have to be computed by a verifier, so passing them as parameters
means we don't recompute them.
* WIP for in memory state structs
* tweeks from end of session with henry
* more updates from pairing
* rewrite non-finalized state sections
* update query instructions for each request
* more updates
* updates from pairing with henry
* updates from proofreading solo
* add guide level explanation to state rfc
* add drawbacks section
* Update book/src/dev/rfcs/0005-state-updates.md
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
* Apply suggestions from code review
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
* Update book/src/dev/rfcs/0005-state-updates.md
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
* apply changes from code review
* clarify iteration
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* apply changes from code review
* Update book/src/dev/rfcs/0005-state-updates.md
Co-authored-by: teor <teor@riseup.net>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* add info about default constructing chains when forking from finalized state
* Update book/src/dev/rfcs/0005-state-updates.md
Co-authored-by: teor <teor@riseup.net>
* move contextual verification out of Chain
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Jane Lusby <jane@zfnd.org>
Remove the seed command entirely, and make the behavior it provided
(responding to `Request::Peers`) part of the ordinary functioning of the
start command.
The new `Inbound` service should be expanded to handle all request
types.
* rfc: initial inventory tracking
This just describes the design, not the design alternatives.
* rfc: finish inventory tracking rfc
Also assign it #3. The async script verification RFC should have had a number
assigned before merging but it didn't. I don't want to fix that in this PR
because I don't want those changes to block on each other. The fix is to (1)
document the RFC flow better and (2) add issue templates for RFCs.
* rfc: touch up inventory tracking rfc
* rfc: prune inventory entries generationally.
Based on a suggestion by @yaahc.
* Update book/src/dev/rfcs/0003-inventory-tracking.md
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
Per discussion where @yaahc suggested that it would be simpler to delete this function entirely and treat it as an implementation detail.
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
* rfc: Parallel Verification Draft
An initial draft RFC for parallel verification.
* rfc: Integrate the CheckpointVerifier design
Describe how the CheckpointVerifier interacts with chain state updates.
* rfc: Add a chain tips update service
* rfc: Add network upgrade context changes
* rfc: Add main chain tip section
* rfc: Clarify and expand genesis block rules
* rfc: More genesis special cases
* Add another chain tips edge case
* Remove the final tie-breaker for tip ties
Instead, change the design to make them impossible.
* rfc: add a definitions section to parallel verification
* rfc: Split parallel verification into two RFCs
This is the semantic verification RFC.
* rfc: Add guide and examples for parallel verification
* rfc: Fix GitHub markdown
* rfc: Fix parallel function design
We don't need separate functions, we can just do the awaits as late as
possible.
* rfc: Fix typo
* rfc: Stop assigning responsibilities to modules
* rfc: Add more parallel verification definitions
* rfc: Say "block height consensus rule"
* rfc: Tidy some of the TODOs
* rfc: Expand rationale and alternatives
* rfc: Delete "try to depend on older blocks"
* rfc: Delete coinbase checks which are unrelated to BlockHeight
And remove some duplicate references to BlockHeight checks.
* rfc: Focus on verification stages
And rewrite some stages for clarity.
* rfc: Remove reference to zebra-network
* Reorganize the book.
This PR has one unfortunate change, which is that the README.md and
CONTRIBUTING.md files in the book are symlinks to files in the parent
directory. The motivation for this is to ensure that we don't maintain two
copies of the same data, and that the landing page of the website matches the
landing page of the Github repo, etc. However, I'm not sure whether these
symlinks will work correctly on Windows.
The alternatives are:
- Duplicate the contents of the files and expect that people will know to keep
them in sync;
- Use relative links `../../README.md` in the `SUMMARY.md`. This seemed like
it caused mdbook to dump the rendered files into the repository root rather
than keeping them in the `book` directory.
- Use a symlink (chosen option). This may not work on Windows but I think that
the worst outcome would be that the book would be unbuildable unless someone
used WSL or something. This seems like the least bad option.
* Remove symlinks in favor of #include
Turns out the symlinks aren't required!
* Setup tracing-flame for use profiling zebrad
* start work on conditional flamegraph generation
* review time!
* update comments
* Update Cargo.toml
* disable default features for inferno
* reorganize
* missing one trait
* Apply suggestions from code review
* graceful shutdown!
* remove special case handling on ctrlc for cleanup
* rename signal fn to better represent its responsibility
* remove unused global hook for flushing flamegraph
* move tracing logic to the right file
* just copy linkerd's signal handling logic
* update book
* make zebrad app drop on shutdown normally
* Update zebrad/src/components/tokio.rs
Co-authored-by: teor <teor@riseup.net>
* Update zebrad/src/application.rs
Co-authored-by: teor <teor@riseup.net>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* cleanup a little
* ooh yea there's an API for that
* setup env-filter for backup subscriber
* document env filter
* document return codes
* forgot to save
* Update book/src/applications/zebrad.md
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* Load tracing filter only from config and simplify logic.
* Configure the state storage in the config, not an environment variable.
This also changes the config so that the path is always set rather than being
optional, because Zebra always needs a place to store its config.
* Add skeleton of eventual zebra book
* reorg sections
* restore file and reorg book a little
* try setting up a firebase deployment
* allow firebase ci to work on test
* download mdbook
* fix book path
* use newer version of mdbook
* remove event hook for book branch pre merge
* Apply suggestions from code review
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
Alfredo Garcia
teor
Deirdre Connolly
Henry de Valence
Pili Guerra
Jane Lusby
Jane Lusby
Dimitris Apostolou
noone