9e78a8af40
* Add `HashSet`s to help spend conflict detection Keep track of the spent transparent outpoints and the revealed nullifiers. Clippy complained that the `ActiveState` had variants with large size differences, but that was expected, so I disabled that lint on that `enum`. * Clear the `HashSet`s when clearing the mempool Clear them so that they remain consistent with the set of verified transactions. * Use `HashSet`s to check for spend conflicts Store new outputs into its respective `HashSet`, and abort if a duplicate output is found. * Remove inserted outputs when aborting Restore the `HashSet` to its previous state. * Remove tracked outputs when removing a transaction Keep the mempool storage in a consistent state when a transaction is removed. * Remove tracked outputs when evicting from mempool Ensure eviction also keeps the tracked outputs consistent with the verified transactions. * Refactor to create a `VerifiedSet` helper type Move the code to handle the output caches into the new type. Also move the eviction code to make things a little simpler. * Refactor to have a single `remove` method Centralize the code that handles the removal of a transaction to avoid mistakes. * Move mempool size limiting back to `Storage` Because the evicted transactions must be added to the rejected list. * Remove leftover `dbg!` statement Leftover from some temporary testing code. Co-authored-by: teor <teor@riseup.net> * Remove unnecessary `TODO` It is more speculation than planning, so it doesn't add much value. Co-authored-by: teor <teor@riseup.net> * Fix typo in documentation The verb should match the subject "transactions" which is plural. Co-authored-by: teor <teor@riseup.net> * Add a comment to warn about correctness There's a subtle but important detail in the implementation that should be made more visible to avoid mistakes in the future. Co-authored-by: teor <teor@riseup.net> * Remove outdated comment Left-over from the attempt to move the eviction into the `VerifiedSet`. * Improve comment explaining lint removal Rewrite the comment explaining why the Clippy lint was ignored. * Check for spend conflicts in `VerifiedSet` Refactor to avoid API misuse. * Test rejected transaction rollback Using two transactions, perform the same test adding a conflict to both of them to check if the second inserted transaction is properly rejected. Then remove any conflicts from the second transaction and add it again. That should work, because if it doesn't it means that when the second transaction was rejected it left things it shouldn't in the cache. * Test removal of multiple transactions When removing multiple transactions from the mempool storage, all of the ones requested should be removed and any other transaction should be still be there afterwards. * Increase mempool size to 4, so that spend conflict tests work If the mempool size is smaller than 4, these tests don't fail on a trivial removal bug. Because we need a minimum number of transactions in the mempool to trigger the bug. Also commit a proptest seed that fails on a trivial removal bug. (This seed fails if we remove indexes in order, because every index past the first removes the wrong transaction.) * Summarise transaction data in proptest error output * Summarise spend conflict field data in proptest error output * Summarise multiple removal field data in proptest error output And replace the very large proptest debug output with the new summary. Co-authored-by: teor <teor@riseup.net> |
||
---|---|---|
.github | ||
book | ||
docker | ||
grafana | ||
tower-batch | ||
tower-fallback | ||
zebra-chain | ||
zebra-client | ||
zebra-consensus | ||
zebra-network | ||
zebra-rpc | ||
zebra-script | ||
zebra-state | ||
zebra-test | ||
zebra-utils | ||
zebrad | ||
.gitignore | ||
CHANGELOG.md | ||
CODE_OF_CONDUCT.md | ||
CONTRIBUTING.md | ||
Cargo.lock | ||
Cargo.toml | ||
LICENSE-APACHE | ||
LICENSE-MIT | ||
README.md | ||
SECURITY.md | ||
clippy.toml | ||
cloudbuild.yaml | ||
codecov.yml | ||
firebase.json | ||
katex-header.html | ||
prometheus.yaml |
README.md
About
Zebra is the Zcash Foundation's independent, consensus-compatible implementation of the Zcash protocol, currently under development. Please join us on Discord if you'd like to find out more or get involved!
Alpha Releases
Every few weeks, we release a new Zebra alpha release.
The goals of the alpha release series are to:
- participate in the Zcash network,
- replicate the Zcash chain state,
- implement the Zcash proof of work consensus rules, and
- sync on Mainnet under excellent network conditions.
Currently, Zebra does not validate all the Zcash consensus rules. It may be unreliable on Testnet, and under less-than-perfect network conditions. See our current features and roadmap for details.
Getting Started
Building zebrad
requires Rust,
libclang, and a C++ compiler.
Detailed Build and Run Instructions
- Install
cargo
andrustc
.- Using
rustup
installs the stable Rust toolchain, whichzebrad
targets.
- Using
- Install Zebra's build dependencies:
- libclang: the
libclang
,libclang-dev
,llvm
, orllvm-dev
packages, depending on your package manager - clang or another C++ compiler:
g++
,Xcode
, orMSVC
- libclang: the
- Run
cargo install --locked --git https://github.com/ZcashFoundation/zebra --tag v1.0.0-alpha.18 zebrad
- Run
zebrad start
If you're interested in testing out zebrad
please feel free, but keep in mind
that there is a lot of key functionality still missing.
Build Troubleshooting
If you're having trouble with:
- dependencies:
- install both
libclang
andclang
- they are usually different packages - use
cargo install
without--locked
to build with the latest versions of each dependency
- install both
- libclang: check out the clang-sys documentation
- g++ or MSVC++: try using clang or Xcode instead
- rustc: use rustc 1.48 or later
- Zebra does not have a minimum supported Rust version (MSRV) policy yet
System Requirements
We usually build zebrad
on systems with:
- 2+ CPU cores
- 7+ GB RAM
- 14+ GB of disk space
On many-core machines (like, 32-core) the build is very fast; on 2-core machines it's less fast.
We continuously test that our builds and tests pass on:
- Windows Server 2019
- macOS Big Sur 11.0
- Ubuntu 18.04 / the latest LTS
- Debian Buster
We usually run zebrad
on systems with:
- 4+ CPU cores
- 16+ GB RAM
- 50GB+ available disk space for finalized state
- 100+ Mbps network connections
zebrad
might build and run fine on smaller and slower systems - we haven't
tested its exact limits yet.
Network Ports and Data Usage
By default, Zebra uses the following inbound TCP listener ports:
- 8233 on Mainnet
- 18233 on Testnet
If Zebra is configured with a specific listen_addr
,
it will advertise this address to other nodes for inbound connections.
Zebra makes outbound connections to peers on any port.
But zcashd
prefers peers on the default ports,
so that it can't be used for DDoS attacks on other networks.
zebrad
's typical network usage is:
- initial sync: 30 GB download
- ongoing updates: 10-50 MB upload and download per day, depending on peer requests
The major constraint we've found on zebrad
performance is the network weather,
especially the ability to make good connections to other Zcash network peers.
Current Features
Network:
- synchronize the chain from peers
- download gossiped blocks from peers
- answer inbound peer requests for hashes, headers, and blocks
State:
- persist block, transaction, UTXO, and nullifier indexes
- handle chain reorganizations
Proof of Work:
- validate equihash, block difficulty threshold, and difficulty adjustment
- validate transaction merkle roots
Validating proof of work increases the cost of creating a consensus split
between zebrad
and zcashd
.
This release also implements some other Zcash consensus rules, to check that Zebra's validation architecture supports future work on a full validating node:
- block and transaction structure
- checkpoint-based verification up to and including Canopy activation
- transaction validation (incomplete)
- transaction cryptography (incomplete)
- transaction scripts (incomplete)
- batch verification (incomplete)
Dependencies
Zebra primarily depends on pure Rust crates, and some Rust/C++ crates:
Known Issues
There are a few bugs in Zebra that we're still working on fixing:
- In rare cases, Zebra panics on shutdown #1678
- Interrupt handler does not work when a blocking task is running #1351
- Zebra should eventually exit once the task finishes. Or you can forcibly terminate the process.
- Duplicate block errors #1372
- These errors can be ignored, unless they happen frequently
Zebra's state commits changes using database transactions. If you forcibly terminate it, or it panics, any incomplete changes will be rolled back the next time it starts.
Future Work
In 2021, we intend to finish validation, add RPC support, and add wallet integration. This phased approach allows us to test Zebra's independent implementation of the consensus rules, before asking users to entrust it with their funds.
Features:
- full consensus rule validation
- transaction mempool
- wallet functionality
- RPC functionality
Performance and Reliability:
- reliable syncing on Testnet
- reliable syncing under poor network conditions
- batch verification
- performance tuning
Documentation
The Zebra website contains user documentation, such as how to run or configure Zebra, set up metrics integrations, etc., as well as developer documentation, such as design documents. We also render API documentation for the external API of our crates, as well as internal documentation for private APIs.
Architecture
Unlike zcashd
, which originated as a Bitcoin Core fork and inherited its
monolithic architecture, Zebra has a modular, library-first design, with the
intent that each component can be independently reused outside of the zebrad
full node. For instance, the zebra-network
crate containing the network stack
can also be used to implement anonymous transaction relay, network crawlers, or
other functionality, without requiring a full node.
At a high level, the fullnode functionality required by zebrad
is factored
into several components:
-
zebra-chain
, providing definitions of core data structures for Zcash, such as blocks, transactions, addresses, etc., and related functionality. It also contains the implementation of the consensus-critical serialization formats used in Zcash. The data structures inzebra-chain
are defined to enforce structural validity by making invalid states unrepresentable. For instance, theTransaction
enum has variants for each transaction version, and it's impossible to construct a transaction with, e.g., spend or output descriptions but no binding signature, or, e.g., a version 2 (Sprout) transaction with Sapling proofs. Currently,zebra-chain
is oriented towards verifying transactions, but will be extended to support creating them in the future. -
zebra-network
, providing an asynchronous, multithreaded implementation of the Zcash network protocol inherited from Bitcoin. In contrast tozcashd
, each peer connection has a separate state machine, and the crate translates the external network protocol into a stateless, request/response-oriented protocol for internal use. The crate provides two interfaces:- an auto-managed connection pool that load-balances local node requests over available peers, and sends peer requests to a local inbound service, and
- a
connect_isolated
method that produces a peer connection completely isolated from all other node state. This can be used, for instance, to safely relay data over Tor, without revealing distinguishing information.
-
zebra-script
provides script validation. Currently, this is implemented by linking to the C++ script verification code fromzcashd
, but in the future we may implement a pure-Rust script implementation. -
zebra-consensus
performs semantic validation of blocks and transactions: all consensus rules that can be checked independently of the chain state, such as verification of signatures, proofs, and scripts. Internally, the library usestower-batch
to perform automatic, transparent batch processing of contemporaneous verification requests. -
zebra-state
is responsible for storing, updating, and querying the chain state. The state service is responsible for contextual verification: all consensus rules that check whether a new block is a valid extension of an existing chain, such as updating the nullifier set or checking that transaction inputs remain unspent. -
zebrad
contains the full node, which connects these components together and implements logic to handle inbound requests from peers and the chain sync process. -
zebra-rpc
andzebra-client
will eventually contain the RPC and wallet functionality, but as mentioned above, our goal is to implement replication of chain state first before asking users to entrust Zebra with their funds.
All of these components can be reused as independent libraries, and all communication between stateful components is handled internally by internal asynchronous RPC abstraction ("microservices in one process").
Security
Zebra has a responsible disclosure policy, which we encourage security researchers to follow.
License
Zebra is distributed under the terms of both the MIT license and the Apache License (Version 2.0).
See LICENSE-APACHE and LICENSE-MIT.