design: initial state sync proposal

Change-Id: I57f65cefb95e72fbe9dc6738aeba3d4945202e47

design/0005_data_availability.md

@@ -30,9 +30,7 @@ Our data availability requirements do not actually require messages to be posted
 
 - The mechanism must enable any client - native, app or web - to wait for and retrieve the signed VAA message for a specific message, identified by its unique (chain, emitter, sequence) identifier.
 
-- Signed VAAs must be available to clients and guardian nodes at least until the message was posted on the target chain. Ideally, our design would enable an optional full archive of signed VAAs to be maintained.
-
-- Signed VAAs must be available in a decentralized fashion to at least every guardian node.
+- Signed VAAs must be available to clients at least until the message was posted on the target chain. Ideally, our design would enable an optional full archive of signed VAAs to be maintained.
 
 - The design's performance characteristics must be sufficient to persist and retrieve all signed VAAs from all networks within at most 100-200ms per message, with all networks publishing messages concurrently at full capacity.
 
@@ -48,6 +46,10 @@ Our data availability requirements do not actually require messages to be posted
 
 - Resolving the fee payer issue - users need existing tokens on the target chain in order to pay fees for the client-side message submission transaction. This is a problem for users who want to use a token bridge to get such native tokens in the first place.
 
+- Denial of service prevention for the underlying gossip network.
+
+- Optimization for very large state (dozens to hundreds of GB).
+
 ## Overview
 
 Instead of submitting signed VAAs to Solana, guardians instead broadcast them on the gossip network and persist the signed VAAs locally.
@@ -56,7 +58,11 @@ Guardians that failed to observe the message (and therefore cannot reconstruct t
 
 A public API endpoint is added to guardiand, exposing an API which allows clients to retrieve the signed VAA for any (chain, emitter, sequence) tuple. Guardians can use this API to serve a public, load-balanced public service for web wallets and other clients to use.
 
-If a node has no local copy of a requested VAA, it will broadcast a message to the gossip network, requesting retransmission. Any node that stores a copy of the requested VAA will rebroadcast it, allowing the requesting node to backfill the missing data and serve the client's request.
+Since all transactions per (chain, emitter) are ordered by a gapless sequence number, we can implement an efficient state synchronization protocol for a finite set of well-known emitter addresses. Nodes will publish the highest slot for each emitter address in their signed heartbeat messages. When a node encounters missing data, either by observing a higher slot number published by a majority of the network or by observing gaps, it can send unicast sync requests to other nodes on the network and efficiently download ranges of missing state data.
+
+Well-known emitter addresses are hardcoded in the node software. Emitters that aren't part of this well-known list can still use the protocol, but with lesser data availability guarantees. In this initial design, this list is subject to off-protocol governance processes, and can be migrated to on-chain governance with proper incentives in the future. Only some uses cases like token bridges would benefit from stronger data availability - it would be unnecessary for others, like short-lived price feeds. Nodes can choose to prune local data for these emphemeral emitters.
+
+State synchronization is not possible across guardian set boundaries, since nodes won't be able to trust data signed by non-current guardian sets.
 
 ## Detailed Design
 
@@ -66,13 +72,11 @@ Depending on the order of receipt and network topology, the aggregated set of si
 
 In v1, a node would submit the VAA directly to Solana, with complex logic for fault tolerance and retries. The first signed VAA would "win" a race and be persisted on-chain as the canonical signed VAA for this message.
 
-Instead, each node will now locally persist the full signed VAA and broadcast it to the gossip network, where it can be received both by guardian nodes and unprivileged nodes (like future relayer services) that joined the gossip network. If a guardian receives a VAA for a tuple it has no state for, it will verify the signature and persist it using the same logic - the first valid response to be received "wins".
+Instead, each node will now locally persist the full signed VAA and broadcast it to the gossip network, where it can be received both by guardian nodes and unprivileged nodes (like future relayer services) that joined the gossip network.
 
 Locally persisted state is crucial to maintain data availability across the network - it is used to serve API queries (if enabled) and rebroadcast signed VAAs to other guardians that missed them.
 
-We can't rely on gossip to provide atomic or reliable broadcast - messages may be lost, or nodes may be down. We need to assume that nodes can and will lose all of their local state, and be down for maintenance, including nodes used to serve a public API. We therefore need a mechanism for API nodes to backfill missing data when such data is requested.
-
-A new gossip message type will be implemented - `RetransmissionRequest`, specifying the tuple of the missing message - which is signed using the node's guardian private key. A node that receives such a request, signed by a guardian in the current guardian set, will look up the requested the message key in its local store and rebroadcast it using the signed VAA distribution mechanism described above.
+We can't rely on gossip to provide atomic or reliable broadcast - messages may be lost, or nodes may be down. We need to assume that nodes can and will lose all of their local state, and be down for maintenance, including nodes used to serve a public API. We therefore need a mechanism for API nodes to backfill missing data.
 
 We use the (chain, emitter, sequence) tuple as global identifier. The digest is not suitable as a global identifier, since it is not known at message publication time. Instead, all contracts make the sequence number available to the caller when publishing a message, which the caller then surfaces to the client. Chain and emitter address are static.
 
@@ -86,15 +90,15 @@ No changes are required to smart contracts.
 
 ## Alternatives considered
 
-### Provider-side redundancy instead of retransmissions
+### Provider-side redundancy instead of state sync
 
-Instead of implementing the retransmission mechanism, we could instead make it the API provider's responsibility to maintain a complete record of VAAs by running multiple nodes listening to the gossip network.
+Instead of implementing the state synchronization mechanism, we could instead make it the API provider's responsibility to maintain a complete record of VAAs by running multiple nodes listening to the gossip network.
 
 Nodes could do idempotent writes to a single shared K/V store (like Bigtable or Redis), doing fallthrough API requests against other nodes in the cluster, or retry on the LB level.
 
-While such features will likely be implemented in the future to improve scalability, we decided to design and implement retransmissions first:
+While such features will likely be implemented in the future to improve scalability, we decided to design and implement state synchronization first:
 
-- We want to mitigate gossip message propagation issues or operational mistakes that could affect many nodes. With retransmissions, a message can be retrieved as long as at least one node has a copy.
+- We want to mitigate gossip message propagation issues or operational mistakes that could affect many nodes. With state synchronization, a message can be retrieved as long as at least one node has a copy.
 
 - For decentralization reasons, it should be possible to serve a fully-functional public API using a single node without requiring complex external dependencies or multiple nodes in separate failure domains.
 
@@ -112,23 +116,15 @@ Directly connecting to the gossip network remains a possible design for future f
 
 ## Caveats
 
-### Invalid requests to the API
-
-Nodes can't know whether they missed a message, or if the client requested an invalid message. Such requests for random invalid tuples can be made easily and cheaply.
-
-Nodes have to send gossip messages in these cases, which can present a denial of service risk. Operators might choose to run an internal redundancy layer (as described above) and only do gossip requests when the client completed a proof of work or when users request support in the rare case of lost messages.
-
 ### "Leechers"
 
-We do not specify an explicit incentive for nodes to maintain historic state. If a large fraction of the network fails to properly persist local state (like by running in ephemeral containers), we risk relying on insufficiently small number of nodes to serve retransmissions to others.
+We do not specify an explicit incentive for nodes to maintain historic state. If a large fraction of the network fails to properly persist local state (like by running in ephemeral containers), we risk relying on insufficiently small number of nodes to serve state to others.
 
-We believe that this is not an issue at this time due to the small amount of storage required (~1KiB per VAA) and data loss will occur infrequently, like when a node fails. The retransmission protocol is much slower than listening to the initial gossip transmissions, providing little incentive for API operators to misuse the mechanism.
+We believe that this is not an issue at this time due to the small amount of storage required (~1KiB per VAA) and data loss will occur infrequently, like when a node fails. The state synchronization protocol is much slower than listening to the initial gossip transmissions, providing little incentive for API operators to misuse the mechanism.
 
 ### Gossip performance
 
-This proposal significantly increases the amount of data broadcasted via the gossip network (signed message broadcast and retransmissions), which may affect latency and throughput of signature broadcast.
-
-Gossip performance may be insufficient to serve retransmission requests at high rates (see also: https://github.com/certusone/wormhole/issues/40).
+This proposal significantly increases the amount of data broadcasted via the gossip network (signed message broadcast and state synchronization), which may affect latency and throughput of signature broadcast. Unicast libp2p communication for state sync may cause head-of-line blocking and affect consensus. 
 
 ### Decentralization concerns
 
@@ -140,16 +136,10 @@ We believe that our proposal instead improves decentralization: Solana RPC nodes
 
 This proposal affects only data availability of data that was already validated and signed. If the in-band data availability mechanism fails, out-of-band methods can be used to ensure data availability (like manually fetching and posting signed VAAs).
 
-### Guardian secret key usage
+### Denial of service
 
-We introduce a second usage of the guardian signing key for signings payloads other than VAAs. This is new, and potentially dangerous. In order to prevent potential confused deputy vulnerabilities, we append a prefix when signing and verifying non-VAA messages (i.e `retransmission|<payload>`) to distinguish non-VAA key usage.
-
-The same approach will be used when signing heartbeats (https://github.com/certusone/wormhole/issues/267).
-
-### Byzantine fault tolerance
-
-Wormhole is designed to tolerate malicious behavior of up to 1/3 of guardian nodes. We allow any guardian node to request retransmissions. A byzantine node could abuse this behavior by sending a very large number of requests, overwhelming the gossip network or the guardian nodes.
+We allow any node on the network to request state synchronization, which could be abused for denial-of-service attacks against the network.
 
 Rate-limiting and blacklisting mechanism can be implemented to enable guardians to respond to such attacks.
 
-(this assumes that libp2p itself is safe against pubsub flooding by non-guardian nodes - this an open question tracked in https://github.com/certusone/wormhole/issues/22)
+(this assumes that libp2p itself is safe against pubsub flooding by non-guardian nodes - this an open question tracked in https://github.com/certusone/wormhole/issues/22 as well as the [official libp2p docs](https://docs.libp2p.io/concepts/security-considerations/)).