## Motivation
The zebra-state service needs to be able to handle duplicate blocks.
## Solution
This implements changes already outlined by [The State
RFC](https://zebra.zfnd.org/dev/rfcs/0005-state-updates.html). We check for
successfully committed blocks first, since interacting with the queued blocks
struct at this point just complicates the implimentation. If the block has not
already been committed we then check if the block has already been queued, if
not we handle the block normally (normally here being the bit we already had
implemented).
## Documentation Changes
- [x] Update the state RFC to match the ways this fix departs from the design
- the main thing is that I switched the order of checking for duplicates
- [x] ~~Add newly added functions to the state rfc~~ Decided not to do this because they're minor getters that don't influence the rest of the design and aren't exposed as part of the API
- [x] Document newly added functions inline
## Testing
## Related Issues
- fixes https://github.com/ZcashFoundation/zebra/issues/1182
- tracking issue https://github.com/ZcashFoundation/zebra/issues/1049
Co-authored-by: teor <teor@riseup.net>
Binding grafana to localhost makes it inaccessible from the wider internet,
which is a secure default.
Since we run docker with host networking, docker containers have access to D-Bus and other
security-related services on localhost. So it's risky to also expose them to the wider internet.
* Reverse displayed endianness of transaction and block hashes
* fix zebra-checkpoints utility for new hash order
* Stop using "zebrad revhex" in zebrad-hash-lookup
* Rebuild checkpoint lists in new hash order
This change also adds additional checkpoints to the end of each list.
* Replace TransactionHash with transaction::Hash
This change should have been made in #905, but we missed Debug impls
and some docs.
Co-authored-by: Ramana Venkata <vramana@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
* implement most of the chain functions
* implement fork
* fix outpoint handling in Chain struct
* update expect for work
* split utxo into two sets
* update the Chain definition
* remove allow attribute in zebra-state/lib.rs
* merge ChainSet type into MemoryState
* Add error messages to asserts
* export proptest impls for use in downstream crates
* add testjob for disabled feature in zebra-chain
* try to fix github actions syntax
* add module doc comment
* update RFC for utxos
* add missing header
* working proptest for Chain
* propagate back results over channel
* Start updating RFC to match changes
* implement queued block pruning
* and now it syncs wooo!
* remove empty modules
* setup config for proptests
* re-enable missing_docs lint
* update RFC to match changes in impl
* add documentation
* use more explicit variable names
* Begin work on RFC5 implementation
* I think this is necessary
* holy shit supertrait implemented via subtrait
* implement most of the chain functions
* change to slightly better name
* implement fork
* fix outpoint handling in Chain struct
* update expect for work
* resolve review comment
* split utxo into two sets
* update the Chain definition
* just a little more
* update comment
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* apply changes from code review
* remove allow attribute in zebra-state/lib.rs
* Update zebra-state/src/memory_state.rs
Co-authored-by: teor <teor@riseup.net>
* merge ChainSet type into MemoryState
* rename state impl types
* Add error messages to asserts
* checkpoint so I can split off arbitrary changes into a PR
* export proptest impls for use in downstream crates
* add testjob for disabled feature in zebra-chain
* run rustfmt
* try to fix github actions syntax
* differentiate name
* prove that github action tests zebra-chain build without features
* revert change from last commit now that test is running
* remove accidentally introduced newline
* checkpoint
* add module doc comment
* update RFC for utxos
* add missing header
* working proptest for Chain
* apply change from chain impl PR
* setup config for proptests
* Update zebra-chain/src/block/arbitrary.rs
Co-authored-by: teor <teor@riseup.net>
* run rustfmt
Co-authored-by: teor <teor@riseup.net>
* Reject CommitBlock with pre-sapling blocks: they must use CommitFinalizedBlock
* Clarify adding a new Chain to an empty ChainSet
* Handle duplicate blocks correctly
* Begin work on RFC5 implementation
* I think this is necessary
* holy shit supertrait implemented via subtrait
* implement most of the chain functions
* change to slightly better name
* implement fork
* fix outpoint handling in Chain struct
* update expect for work
* resolve review comment
* split utxo into two sets
* update the Chain definition
* just a little more
* update comment
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* apply changes from code review
* remove allow attribute in zebra-state/lib.rs
* Update zebra-state/src/memory_state.rs
Co-authored-by: teor <teor@riseup.net>
* merge ChainSet type into MemoryState
* rename state impl types
* Add error messages to asserts
* add module doc comment
* update RFC for utxos
* add missing header
Co-authored-by: teor <teor@riseup.net>
* wip
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
* wip2: add nullifiers
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
* Update book/src/dev/rfcs/0003-state-updates.md
Co-authored-by: teor <teor@riseup.net>
* Move to RFC number 5
* rfc: add PR link to state update RFC
* rfc: change state RFC to store blocks by height.
The rationale for this change is described in the document: it means
that we write blocks only to one end of the Sled tree, and hopefully
helps us with spatial access patterns.
This should help alleviate a major cause of memory use in Zebra's
current WIP Sled structure, which is that:
- blocks are stored in random, sparse order (by hash) in the B-tree;
- the `Request::GetDepth` method opens the entire block store and
queries a random part of its block data to determine whether a hash is
present;
- if present, it deserializes the complete block data of both the given
block and the current tip block, to compute the difference in block
heights.
This access pattern forces a large amount of B-tree data to remain
resident, and could probably be avoided if we didn't do that.
* rfc: add sprout and sapling anchors to sled trees.
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* rfc: fill in details of state service requests.
* rfc: extract commit process from API description
* rfc: add anchor parameters to CommitBlock.
These have to be computed by a verifier, so passing them as parameters
means we don't recompute them.
* WIP for in memory state structs
* tweeks from end of session with henry
* more updates from pairing
* rewrite non-finalized state sections
* update query instructions for each request
* more updates
* updates from pairing with henry
* updates from proofreading solo
* add guide level explanation to state rfc
* add drawbacks section
* Update book/src/dev/rfcs/0005-state-updates.md
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
* Apply suggestions from code review
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
* Update book/src/dev/rfcs/0005-state-updates.md
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
* apply changes from code review
* clarify iteration
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* apply changes from code review
* Update book/src/dev/rfcs/0005-state-updates.md
Co-authored-by: teor <teor@riseup.net>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* add info about default constructing chains when forking from finalized state
* Update book/src/dev/rfcs/0005-state-updates.md
Co-authored-by: teor <teor@riseup.net>
* move contextual verification out of Chain
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Jane Lusby <jane@zfnd.org>
Remove the seed command entirely, and make the behavior it provided
(responding to `Request::Peers`) part of the ordinary functioning of the
start command.
The new `Inbound` service should be expanded to handle all request
types.
* rfc: initial inventory tracking
This just describes the design, not the design alternatives.
* rfc: finish inventory tracking rfc
Also assign it #3. The async script verification RFC should have had a number
assigned before merging but it didn't. I don't want to fix that in this PR
because I don't want those changes to block on each other. The fix is to (1)
document the RFC flow better and (2) add issue templates for RFCs.
* rfc: touch up inventory tracking rfc
* rfc: prune inventory entries generationally.
Based on a suggestion by @yaahc.
* Update book/src/dev/rfcs/0003-inventory-tracking.md
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
Per discussion where @yaahc suggested that it would be simpler to delete this function entirely and treat it as an implementation detail.
Co-authored-by: Jane Lusby <jlusby42@gmail.com>
* rfc: Parallel Verification Draft
An initial draft RFC for parallel verification.
* rfc: Integrate the CheckpointVerifier design
Describe how the CheckpointVerifier interacts with chain state updates.
* rfc: Add a chain tips update service
* rfc: Add network upgrade context changes
* rfc: Add main chain tip section
* rfc: Clarify and expand genesis block rules
* rfc: More genesis special cases
* Add another chain tips edge case
* Remove the final tie-breaker for tip ties
Instead, change the design to make them impossible.
* rfc: add a definitions section to parallel verification
* rfc: Split parallel verification into two RFCs
This is the semantic verification RFC.
* rfc: Add guide and examples for parallel verification
* rfc: Fix GitHub markdown
* rfc: Fix parallel function design
We don't need separate functions, we can just do the awaits as late as
possible.
* rfc: Fix typo
* rfc: Stop assigning responsibilities to modules
* rfc: Add more parallel verification definitions
* rfc: Say "block height consensus rule"
* rfc: Tidy some of the TODOs
* rfc: Expand rationale and alternatives
* rfc: Delete "try to depend on older blocks"
* rfc: Delete coinbase checks which are unrelated to BlockHeight
And remove some duplicate references to BlockHeight checks.
* rfc: Focus on verification stages
And rewrite some stages for clarity.
* rfc: Remove reference to zebra-network
* Reorganize the book.
This PR has one unfortunate change, which is that the README.md and
CONTRIBUTING.md files in the book are symlinks to files in the parent
directory. The motivation for this is to ensure that we don't maintain two
copies of the same data, and that the landing page of the website matches the
landing page of the Github repo, etc. However, I'm not sure whether these
symlinks will work correctly on Windows.
The alternatives are:
- Duplicate the contents of the files and expect that people will know to keep
them in sync;
- Use relative links `../../README.md` in the `SUMMARY.md`. This seemed like
it caused mdbook to dump the rendered files into the repository root rather
than keeping them in the `book` directory.
- Use a symlink (chosen option). This may not work on Windows but I think that
the worst outcome would be that the book would be unbuildable unless someone
used WSL or something. This seems like the least bad option.
* Remove symlinks in favor of #include
Turns out the symlinks aren't required!
* Setup tracing-flame for use profiling zebrad
* start work on conditional flamegraph generation
* review time!
* update comments
* Update Cargo.toml
* disable default features for inferno
* reorganize
* missing one trait
* Apply suggestions from code review
* graceful shutdown!
* remove special case handling on ctrlc for cleanup
* rename signal fn to better represent its responsibility
* remove unused global hook for flushing flamegraph
* move tracing logic to the right file
* just copy linkerd's signal handling logic
* update book
* make zebrad app drop on shutdown normally
* Update zebrad/src/components/tokio.rs
Co-authored-by: teor <teor@riseup.net>
* Update zebrad/src/application.rs
Co-authored-by: teor <teor@riseup.net>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* cleanup a little
* ooh yea there's an API for that
* setup env-filter for backup subscriber
* document env filter
* document return codes
* forgot to save
* Update book/src/applications/zebrad.md
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* Load tracing filter only from config and simplify logic.
* Configure the state storage in the config, not an environment variable.
This also changes the config so that the path is always set rather than being
optional, because Zebra always needs a place to store its config.
* Add skeleton of eventual zebra book
* reorg sections
* restore file and reorg book a little
* try setting up a firebase deployment
* allow firebase ci to work on test
* download mdbook
* fix book path
* use newer version of mdbook
* remove event hook for book branch pre merge
* Apply suggestions from code review
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
Jane Lusby
teor
Alfredo Garcia
Dimitris Apostolou
Henry de Valence
noone
Deirdre Connolly