2021-10-29 12:25:30 -07:00
|
|
|
# Design Overview
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
This document sketches the future design for Zebra.
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
## Desiderata
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
The following are general desiderata for Zebra:
|
|
|
|
|
|
|
|
* [George's list..]
|
|
|
|
|
|
|
|
* As much as reasonably possible, it and its dependencies should be
|
|
|
|
implemented in Rust. While it may not make sense to require this in
|
|
|
|
every case (for instance, it probably doesn't make sense to rewrite
|
|
|
|
libsecp256k1 in Rust, instead of using the same upstream library as
|
|
|
|
Bitcoin), we should generally aim for it.
|
|
|
|
|
|
|
|
* As much as reasonably possible, Zebra should minimize trust in
|
|
|
|
required dependencies. Note that "minimize number of dependencies"
|
|
|
|
is usually a proxy for this desideratum, but is not exactly the same:
|
|
|
|
for instance, a collection of crates like the tokio crates are all
|
|
|
|
developed together and have one trust boundary.
|
2020-06-02 16:16:17 -07:00
|
|
|
|
2019-10-21 11:04:28 -07:00
|
|
|
* Zebra should be well-factored internally into a collection of
|
|
|
|
component libraries which can be used by other applications to
|
|
|
|
perform Zcash-related tasks. Implementation details of each
|
2019-10-22 19:03:04 -07:00
|
|
|
component should not leak into all other components.
|
2020-06-02 16:16:17 -07:00
|
|
|
|
2021-03-14 17:39:47 -07:00
|
|
|
* Zebra should checkpoint on Canopy activation and drop all
|
|
|
|
Sprout-related functionality not required post-Canopy.
|
2020-06-02 16:16:17 -07:00
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
## Non-Goals
|
2020-06-03 15:24:52 -07:00
|
|
|
|
|
|
|
* Zebra keeps a copy of the chain state, so it isn't intended for
|
|
|
|
lightweight applications like light wallets. Those applications
|
|
|
|
should use a light client protocol.
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
## Architecture
|
|
|
|
|
|
|
|
Unlike `zcashd`, which originated as a Bitcoin Core fork and inherited its
|
|
|
|
monolithic architecture, Zebra has a modular, library-first design, with the
|
|
|
|
intent that each component can be independently reused outside of the `zebrad`
|
|
|
|
full node. For instance, the `zebra-network` crate containing the network stack
|
|
|
|
can also be used to implement anonymous transaction relay, network crawlers, or
|
|
|
|
other functionality, without requiring a full node.
|
|
|
|
|
|
|
|
At a high level, the fullnode functionality required by `zebrad` is factored
|
|
|
|
into several components:
|
|
|
|
|
|
|
|
- [`zebra-chain`](https://doc.zebra.zfnd.org/zebra_chain/index.html), providing
|
|
|
|
definitions of core data structures for Zcash, such as blocks, transactions,
|
|
|
|
addresses, etc., and related functionality. It also contains the
|
|
|
|
implementation of the consensus-critical serialization formats used in Zcash.
|
|
|
|
The data structures in `zebra-chain` are defined to enforce
|
|
|
|
[*structural validity*](https://zebra.zfnd.org/dev/rfcs/0002-parallel-verification.html#verification-stages)
|
|
|
|
by making invalid states unrepresentable. For instance, the
|
|
|
|
`Transaction` enum has variants for each transaction version, and it's
|
|
|
|
impossible to construct a transaction with, e.g., spend or output
|
|
|
|
descriptions but no binding signature, or, e.g., a version 2 (Sprout)
|
|
|
|
transaction with Sapling proofs. Currently, `zebra-chain` is oriented
|
|
|
|
towards verifying transactions, but will be extended to support creating them
|
|
|
|
in the future.
|
|
|
|
|
|
|
|
- [`zebra-network`](https://doc.zebra.zfnd.org/zebra_network/index.html),
|
|
|
|
providing an asynchronous, multithreaded implementation of the Zcash network
|
|
|
|
protocol inherited from Bitcoin. In contrast to `zcashd`, each peer
|
|
|
|
connection has a separate state machine, and the crate translates the
|
|
|
|
external network protocol into a stateless, request/response-oriented
|
|
|
|
protocol for internal use. The crate provides two interfaces:
|
|
|
|
- an auto-managed connection pool that load-balances local node requests
|
|
|
|
over available peers, and sends peer requests to a local inbound service,
|
|
|
|
and
|
|
|
|
- a `connect_isolated` method that produces a peer connection completely
|
|
|
|
isolated from all other node state. This can be used, for instance, to
|
|
|
|
safely relay data over Tor, without revealing distinguishing information.
|
|
|
|
|
|
|
|
- [`zebra-script`](https://doc.zebra.zfnd.org/zebra_script/index.html) provides
|
|
|
|
script validation. Currently, this is implemented by linking to the C++
|
|
|
|
script verification code from `zcashd`, but in the future we may implement a
|
|
|
|
pure-Rust script implementation.
|
|
|
|
|
|
|
|
- [`zebra-consensus`](https://doc.zebra.zfnd.org/zebra_consensus/index.html)
|
|
|
|
performs [*semantic validation*](https://zebra.zfnd.org/dev/rfcs/0002-parallel-verification.html#verification-stages)
|
|
|
|
of blocks and transactions: all consensus
|
|
|
|
rules that can be checked independently of the chain state, such as
|
|
|
|
verification of signatures, proofs, and scripts. Internally, the library
|
|
|
|
uses [`tower-batch`](https://doc.zebra.zfnd.org/tower_batch/index.html) to
|
|
|
|
perform automatic, transparent batch processing of contemporaneous
|
|
|
|
verification requests.
|
|
|
|
|
|
|
|
- [`zebra-state`](https://doc.zebra.zfnd.org/zebra_state/index.html) is
|
|
|
|
responsible for storing, updating, and querying the chain state. The state
|
|
|
|
service is responsible for [*contextual verification*](https://zebra.zfnd.org/dev/rfcs/0002-parallel-verification.html#verification-stages):
|
|
|
|
all consensus rules
|
|
|
|
that check whether a new block is a valid extension of an existing chain,
|
|
|
|
such as updating the nullifier set or checking that transaction inputs remain
|
|
|
|
unspent.
|
|
|
|
|
|
|
|
- [`zebrad`](https://doc.zebra.zfnd.org/zebrad/index.html) contains the full
|
|
|
|
node, which connects these components together and implements logic to handle
|
|
|
|
inbound requests from peers and the chain sync process.
|
|
|
|
|
|
|
|
- `zebra-rpc` and `zebra-client` will eventually contain the RPC and wallet
|
|
|
|
functionality, but as mentioned above, our goal is to implement replication
|
|
|
|
of chain state first before asking users to entrust Zebra with their funds.
|
|
|
|
|
|
|
|
All of these components can be reused as independent libraries, and all
|
|
|
|
communication between stateful components is handled internally by
|
|
|
|
[internal asynchronous RPC abstraction](https://docs.rs/tower/)
|
|
|
|
("microservices in one process").
|
|
|
|
|
|
|
|
### `zebra-chain`
|
|
|
|
|
|
|
|
#### Internal Dependencies
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2020-06-03 20:02:03 -07:00
|
|
|
None: these are the core data structure definitions.
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Responsible for
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- definitions of commonly used data structures, e.g.,
|
|
|
|
- `Block`,
|
|
|
|
- `Transaction`,
|
|
|
|
- `Address`,
|
|
|
|
- `KeyPair`...
|
2020-06-07 21:19:32 -07:00
|
|
|
- parsing bytes into these data structures
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- definitions of core traits, e.g.,
|
|
|
|
- `ZcashSerialize` and `ZcashDeserialize`, which perform
|
|
|
|
consensus-critical serialization logic.
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Exported types
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- [...]
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
### `zebra-network`
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Internal Dependencies
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- `zebra-chain`
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Responsible for
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2020-06-15 17:07:55 -07:00
|
|
|
- definition of a well structured, internal request/response protocol
|
2019-10-21 11:04:28 -07:00
|
|
|
- provides an abstraction for "this node" and "the network" using the
|
|
|
|
internal protocol
|
|
|
|
- dynamic, backpressure-driven peer set management
|
|
|
|
- per-peer state machine that translates the internal protocol to the
|
|
|
|
Bitcoin/Zcash protocol
|
|
|
|
- tokio codec for Bitcoin/Zcash message encoding.
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Exported types
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- `Request`, an enum representing all possible requests in the internal protocol;
|
|
|
|
- `Response`, an enum representing all possible responses in the internal protocol;
|
|
|
|
- `AddressBook`, a data structure for storing peer addresses;
|
|
|
|
- `Config`, a configuration object for all networking-related parameters;
|
|
|
|
- `init<S: Service>(Config, S) -> (impl Service,
|
|
|
|
Arc<Mutex<AddressBook>>)`, the main entry-point.
|
|
|
|
|
|
|
|
The `init` entrypoint constructs a dynamically-sized pool of peers
|
|
|
|
sending inbound requests to the provided `S: tower::Service`
|
|
|
|
representing "this node", and returns a `Service` that can be used to
|
|
|
|
send requests to "the network", together with an `AddressBook` updated
|
|
|
|
with liveness information from the peer pool. The `AddressBook` can
|
|
|
|
be used to respond to inbound requests for peers.
|
|
|
|
|
|
|
|
All peerset management (finding new peers, creating new outbound
|
|
|
|
connections, etc) is completely encapsulated, as is responsibility for
|
|
|
|
routing outbound requests to appropriate peers.
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
### `zebra-state`
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Internal Dependencies
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- `zebra-chain` for data structure definitions.
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Responsible for
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2020-06-07 21:21:25 -07:00
|
|
|
- block storage API
|
|
|
|
- operates on parsed block structs
|
2020-06-07 21:19:32 -07:00
|
|
|
- these structs can be converted from and into raw bytes
|
2019-10-21 11:04:28 -07:00
|
|
|
- primarily aimed at network replication, not at processing
|
|
|
|
- can be used to rebuild the database below
|
|
|
|
- maintaining a database of tx, address, etc data
|
|
|
|
- this database can be blown away and rebuilt from the blocks, which
|
|
|
|
are otherwise unused.
|
|
|
|
- threadsafe, typed lookup API that completely encapsulates the
|
|
|
|
database logic
|
|
|
|
- handles stuff like "transactions are reference counted by outputs"
|
|
|
|
etc.
|
|
|
|
- providing `tower::Service` interfaces for all of the above to
|
|
|
|
support backpressure.
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Exported types
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2020-06-07 21:21:25 -07:00
|
|
|
- `Request`, an enum representing all possible requests in the internal protocol;
|
|
|
|
- blocks can be accessed via their chain height or hash
|
|
|
|
- confirmed transactions can be accessed via their block, or directly via their hash
|
|
|
|
- `Response`, an enum representing all possible responses in the internal protocol;
|
|
|
|
- `init() -> impl Service`, the main entry-point.
|
|
|
|
|
|
|
|
The `init` entrypoint returns a `Service` that can be used to
|
|
|
|
send requests for the chain state.
|
|
|
|
|
|
|
|
All state management (adding blocks, getting blocks by index or hash) is completely
|
|
|
|
encapsulated.
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
### `zebra-script`
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Internal Dependencies
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- ??? depends on how it's implemented internally
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Responsible for
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- the minimal Bitcoin script implementation required for Zcash
|
2020-06-07 21:21:25 -07:00
|
|
|
- script parsing
|
|
|
|
- context-free script validation
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Notes
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
This can wrap an existing script implementation at the beginning.
|
|
|
|
|
|
|
|
If this existed in a "good" way, we could use it to implement tooling
|
|
|
|
for Zcash script inspection, debugging, etc.
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Questions
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- How does this interact with NU4 script changes?
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Exported types
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- [...]
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
### `zebra-consensus`
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Internal Dependencies
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2020-06-07 21:21:25 -07:00
|
|
|
- `zebra-chain` for data structures and parsing.
|
|
|
|
- `zebra-state` to read and update the state database.
|
|
|
|
- `zebra-script` for script parsing and validation.
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Responsible for
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- consensus-specific parameters (network magics, genesis block, pow
|
|
|
|
parameters, etc) that determine the network consensus
|
|
|
|
- consensus logic to decide which block is the current block
|
2020-06-07 21:21:25 -07:00
|
|
|
- block and transaction verification
|
|
|
|
- context-free validation, e.g., signature, proof verification, etc.
|
|
|
|
- context-dependent validation, e.g., determining whether a
|
|
|
|
transaction is accepted in a particular chain state context.
|
|
|
|
- verifying mempool (unconfirmed) transactions
|
|
|
|
- block checkpoints
|
2021-03-14 17:39:47 -07:00
|
|
|
- mandatory checkpoints (genesis block, canopy activation)
|
2020-06-07 21:21:25 -07:00
|
|
|
- optional regular checkpoints (every Nth block)
|
|
|
|
- modifying the chain state
|
|
|
|
- adding new blocks to `ZebraState`, including chain reorganisation
|
|
|
|
- adding new transactions to `ZebraMempoolState`
|
|
|
|
- storing the transaction mempool state
|
|
|
|
- mempool transactions can be accessed via their hash
|
|
|
|
- providing `tower::Service` interfaces for all of the above to
|
|
|
|
support backpressure and batch validation.
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Exported types
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2020-06-07 21:21:25 -07:00
|
|
|
- `block::init() -> impl Service`, the main entry-point for block
|
|
|
|
verification.
|
|
|
|
- `ZebraMempoolState`
|
|
|
|
- all state management (adding transactions, getting transactions
|
|
|
|
by hash) is completely encapsulated.
|
|
|
|
- `mempool::init() -> impl Service`, the main entry-point for
|
|
|
|
mempool transaction verification.
|
|
|
|
|
|
|
|
The `init` entrypoints return `Service`s that can be used to
|
|
|
|
verify blocks or transactions, and add them to the relevant state.
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
### `zebra-rpc`
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Internal Dependencies
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- `zebra-chain` for data structure definitions
|
|
|
|
- `zebra-network` possibly? for definitions of network messages?
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Responsible for
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- rpc interface
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Exported types
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- [...]
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
### `zebra-client`
|
2019-10-21 11:04:28 -07:00
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Internal Dependencies
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- `zebra-chain` for structure definitions
|
2020-06-02 16:16:17 -07:00
|
|
|
- `zebra-state` for transaction queries and client/wallet state storage
|
2019-10-21 11:04:28 -07:00
|
|
|
- `zebra-script` possibly? for constructing transactions
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Responsible for
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- implementation of some event a user might trigger
|
2020-06-03 15:24:52 -07:00
|
|
|
- would be used to implement a full wallet
|
2019-10-21 11:04:28 -07:00
|
|
|
- create transactions, monitors shielded wallet state, etc.
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Notes
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
Communication between the client code and the rest of the node should be done
|
|
|
|
by a tower service interface. Since the `Service` trait can abstract from a
|
|
|
|
function call to RPC, this means that it will be possible for us to isolate
|
|
|
|
all client code to a subprocess.
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Exported types
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- [...]
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
### `zebrad`
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
Abscissa-based application which loads configs, all application components,
|
|
|
|
and connects them to each other.
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Responsible for
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- actually running the server
|
|
|
|
- connecting functionality in dependencies
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
#### Internal Dependencies
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
- `zebra-chain`
|
|
|
|
- `zebra-network`
|
2020-06-02 16:16:17 -07:00
|
|
|
- `zebra-state`
|
2019-10-21 11:04:28 -07:00
|
|
|
- `zebra-consensus`
|
|
|
|
- `zebra-client`
|
|
|
|
- `zebra-rpc`
|
|
|
|
|
2021-10-29 12:25:30 -07:00
|
|
|
### Unassigned functionality
|
2019-10-21 11:04:28 -07:00
|
|
|
|
|
|
|
Responsibility for this functionality needs to be assigned to one of
|
|
|
|
the modules above (subject to discussion):
|
|
|
|
|
|
|
|
- [ ... add to this list ... ]
|