Replace blob with shred in book (#6004)
This commit is contained in:
Pankaj Garg
GitHub
parent
19ae556857
commit
558a362c46
|
|
@ -9,29 +9,29 @@ naturally form as a result of leader rotation is described in
|
|||
described here is how a validator copes with those forks until blocks
|
||||
are finalized.
|
||||
|
||||
The blocktree allows a validator to record every blob it observes
|
||||
on the network, in any order, as long as the blob is signed by the expected
|
||||
The blocktree allows a validator to record every shred it observes
|
||||
on the network, in any order, as long as the shred is signed by the expected
|
||||
leader for a given slot.
|
||||
|
||||
Blobs are moved to a fork-able key space the tuple of `leader slot` + `blob
|
||||
Shreds are moved to a fork-able key space the tuple of `leader slot` + `shred
|
||||
index` (within the slot). This permits the skip-list structure of the Solana
|
||||
protocol to be stored in its entirety, without a-priori choosing which fork to
|
||||
follow, which Entries to persist or when to persist them.
|
||||
|
||||
Repair requests for recent blobs are served out of RAM or recent files and out
|
||||
of deeper storage for less recent blobs, as implemented by the store backing
|
||||
Repair requests for recent shreds are served out of RAM or recent files and out
|
||||
of deeper storage for less recent shreds, as implemented by the store backing
|
||||
Blocktree.
|
||||
|
||||
### Functionalities of Blocktree
|
||||
|
||||
1. Persistence: the Blocktree lives in the front of the nodes verification
|
||||
pipeline, right behind network receive and signature verification. If the
|
||||
blob received is consistent with the leader schedule (i.e. was signed by the
|
||||
shred received is consistent with the leader schedule (i.e. was signed by the
|
||||
leader for the indicated slot), it is immediately stored.
|
||||
2. Repair: repair is the same as window repair above, but able to serve any
|
||||
blob that's been received. Blocktree stores blobs with signatures,
|
||||
shred that's been received. Blocktree stores shreds with signatures,
|
||||
preserving the chain of origination.
|
||||
3. Forks: Blocktree supports random access of blobs, so can support a
|
||||
3. Forks: Blocktree supports random access of shreds, so can support a
|
||||
validator's need to rollback and replay from a Bank checkpoint.
|
||||
4. Restart: with proper pruning/culling, the Blocktree can be replayed by
|
||||
ordered enumeration of entries from slot 0. The logic of the replay stage
|
||||
|
|
@ -41,22 +41,22 @@ the Blocktree.
|
|||
### Blocktree Design
|
||||
|
||||
1. Entries in the Blocktree are stored as key-value pairs, where the key is the concatenated
|
||||
slot index and blob index for an entry, and the value is the entry data. Note blob indexes are zero-based for each slot (i.e. they're slot-relative).
|
||||
slot index and shred index for an entry, and the value is the entry data. Note shred indexes are zero-based for each slot (i.e. they're slot-relative).
|
||||
|
||||
2. The Blocktree maintains metadata for each slot, in the `SlotMeta` struct containing:
|
||||
* `slot_index` - The index of this slot
|
||||
* `num_blocks` - The number of blocks in the slot (used for chaining to a previous slot)
|
||||
* `consumed` - The highest blob index `n`, such that for all `m < n`, there exists a blob in this slot with blob index equal to `n` (i.e. the highest consecutive blob index).
|
||||
* `received` - The highest received blob index for the slot
|
||||
* `consumed` - The highest shred index `n`, such that for all `m < n`, there exists a shred in this slot with shred index equal to `n` (i.e. the highest consecutive shred index).
|
||||
* `received` - The highest received shred index for the slot
|
||||
* `next_slots` - A list of future slots this slot could chain to. Used when rebuilding
|
||||
the ledger to find possible fork points.
|
||||
* `last_index` - The index of the blob that is flagged as the last blob for this slot. This flag on a blob will be set by the leader for a slot when they are transmitting the last blob for a slot.
|
||||
* `last_index` - The index of the shred that is flagged as the last shred for this slot. This flag on a shred will be set by the leader for a slot when they are transmitting the last shred for a slot.
|
||||
* `is_rooted` - True iff every block from 0...slot forms a full sequence without any holes. We can derive is_rooted for each slot with the following rules. Let slot(n) be the slot with index `n`, and slot(n).is_full() is true if the slot with index `n` has all the ticks expected for that slot. Let is_rooted(n) be the statement that "the slot(n).is_rooted is true". Then:
|
||||
|
||||
is_rooted(0)
|
||||
is_rooted(n+1) iff (is_rooted(n) and slot(n).is_full()
|
||||
|
||||
3. Chaining - When a blob for a new slot `x` arrives, we check the number of blocks (`num_blocks`) for that new slot (this information is encoded in the blob). We then know that this new slot chains to slot `x - num_blocks`.
|
||||
3. Chaining - When a shred for a new slot `x` arrives, we check the number of blocks (`num_blocks`) for that new slot (this information is encoded in the shred). We then know that this new slot chains to slot `x - num_blocks`.
|
||||
|
||||
4. Subscriptions - The Blocktree records a set of slots that have been "subscribed" to. This means entries that chain to these slots will be sent on the Blocktree channel for consumption by the ReplayStage. See the `Blocktree APIs` for details.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,14 +1,14 @@
|
|||
# Repair Service
|
||||
|
||||
The RepairService is in charge of retrieving missing blobs that failed to be delivered by primary communication protocols like Avalanche. It is in charge of managing the protocols described below in the `Repair Protocols` section below.
|
||||
The RepairService is in charge of retrieving missing shreds that failed to be delivered by primary communication protocols like Avalanche. It is in charge of managing the protocols described below in the `Repair Protocols` section below.
|
||||
|
||||
# Challenges:
|
||||
|
||||
1) Validators can fail to receive particular blobs due to network failures
|
||||
1) Validators can fail to receive particular shreds due to network failures
|
||||
|
||||
2) Consider a scenario where blocktree contains the set of slots {1, 3, 5}. Then Blocktree receives blobs for some slot 7, where for each of the blobs b, b.parent == 6, so then the parent-child relation 6 -> 7 is stored in blocktree. However, there is no way to chain these slots to any of the existing banks in Blocktree, and thus the `Blob Repair` protocol will not repair these slots. If these slots happen to be part of the main chain, this will halt replay progress on this node.
|
||||
2) Consider a scenario where blocktree contains the set of slots {1, 3, 5}. Then Blocktree receives shreds for some slot 7, where for each of the shreds b, b.parent == 6, so then the parent-child relation 6 -> 7 is stored in blocktree. However, there is no way to chain these slots to any of the existing banks in Blocktree, and thus the `Shred Repair` protocol will not repair these slots. If these slots happen to be part of the main chain, this will halt replay progress on this node.
|
||||
|
||||
3) Validators that find themselves behind the cluster by an entire epoch struggle/fail to catch up because they do not have a leader schedule for future epochs. If nodes were to blindly accept repair blobs in these future epochs, this exposes nodes to spam.
|
||||
3) Validators that find themselves behind the cluster by an entire epoch struggle/fail to catch up because they do not have a leader schedule for future epochs. If nodes were to blindly accept repair shreds in these future epochs, this exposes nodes to spam.
|
||||
|
||||
# Repair Protocols
|
||||
|
||||
|
|
@ -16,10 +16,10 @@ The repair protocol makes best attempts to progress the forking structure of Blo
|
|||
|
||||
The different protocol strategies to address the above challenges:
|
||||
|
||||
1. Blob Repair (Addresses Challenge #1):
|
||||
This is the most basic repair protocol, with the purpose of detecting and filling "holes" in the ledger. Blocktree tracks the latest root slot. RepairService will then periodically iterate every fork in blocktree starting from the root slot, sending repair requests to validators for any missing blobs. It will send at most some `N` repair reqeusts per iteration.
|
||||
1. Shred Repair (Addresses Challenge #1):
|
||||
This is the most basic repair protocol, with the purpose of detecting and filling "holes" in the ledger. Blocktree tracks the latest root slot. RepairService will then periodically iterate every fork in blocktree starting from the root slot, sending repair requests to validators for any missing shreds. It will send at most some `N` repair reqeusts per iteration.
|
||||
|
||||
Note: Validators will only accept blobs within the current verifiable epoch (epoch the validator has a leader schedule for).
|
||||
Note: Validators will only accept shreds within the current verifiable epoch (epoch the validator has a leader schedule for).
|
||||
|
||||
2. Preemptive Slot Repair (Addresses Challenge #2):
|
||||
The goal of this protocol is to discover the chaining relationship of "orphan" slots that do not currently chain to any known fork.
|
||||
|
|
@ -29,23 +29,23 @@ The different protocol strategies to address the above challenges:
|
|||
* RepairService will periodically make `RequestOrphan` requests for each of the orphans in blocktree.
|
||||
|
||||
`RequestOrphan(orphan)` request - `orphan` is the orphan slot that the requestor wants to know the parents of
|
||||
`RequestOrphan(orphan)` response - The highest blobs for each of the first `N` parents of the requested `orphan`
|
||||
`RequestOrphan(orphan)` response - The highest shreds for each of the first `N` parents of the requested `orphan`
|
||||
|
||||
On receiving the responses `p`, where `p` is some blob in a parent slot, validators will:
|
||||
On receiving the responses `p`, where `p` is some shred in a parent slot, validators will:
|
||||
* Insert an empty `SlotMeta` in blocktree for `p.slot` if it doesn't already exist.
|
||||
* If `p.slot` does exist, update the parent of `p` based on `parents`
|
||||
|
||||
Note: that once these empty slots are added to blocktree, the `Blob Repair` protocol should attempt to fill those slots.
|
||||
Note: that once these empty slots are added to blocktree, the `Shred Repair` protocol should attempt to fill those slots.
|
||||
|
||||
Note: Validators will only accept responses containing blobs within the current verifiable epoch (epoch the validator has a leader schedule for).
|
||||
Note: Validators will only accept responses containing shreds within the current verifiable epoch (epoch the validator has a leader schedule for).
|
||||
|
||||
3. Repairmen (Addresses Challenge #3):
|
||||
This part of the repair protocol is the primary mechanism by which new nodes joining the cluster catch up after loading a snapshot. This protocol works in a "forward" fashion, so validators can verify every blob that they receive against a known leader schedule.
|
||||
This part of the repair protocol is the primary mechanism by which new nodes joining the cluster catch up after loading a snapshot. This protocol works in a "forward" fashion, so validators can verify every shred that they receive against a known leader schedule.
|
||||
|
||||
Each validator advertises in gossip:
|
||||
* Current root
|
||||
* The set of all completed slots in the confirmed epochs (an epoch that was calculated based on a bank <= current root) past the current root
|
||||
|
||||
Observers of this gossip message with higher epochs (repairmen) send blobs to catch the lagging node up with the rest of the cluster. The repairmen are responsible for sending the slots within the epochs that are confrimed by the advertised `root` in gossip. The repairmen divide the responsibility of sending each of the missing slots in these epochs based on a random seed (simple blob.index iteration by N, seeded with the repairman's node_pubkey). Ideally, each repairman in an N node cluster (N nodes whose epochs are higher than that of the repairee) sends 1/N of the missing blobs. Both data and coding blobs for missing slots are sent. Repairmen do not send blobs again to the same validator until they see the message in gossip updated, at which point they perform another iteration of this protocol.
|
||||
Observers of this gossip message with higher epochs (repairmen) send shreds to catch the lagging node up with the rest of the cluster. The repairmen are responsible for sending the slots within the epochs that are confrimed by the advertised `root` in gossip. The repairmen divide the responsibility of sending each of the missing slots in these epochs based on a random seed (simple shred.index iteration by N, seeded with the repairman's node_pubkey). Ideally, each repairman in an N node cluster (N nodes whose epochs are higher than that of the repairee) sends 1/N of the missing shreds. Both data and coding shreds for missing slots are sent. Repairmen do not send shreds again to the same validator until they see the message in gossip updated, at which point they perform another iteration of this protocol.
|
||||
|
||||
Gossip messages are updated every time a validator receives a complete slot within the epoch. Completed slots are detected by blocktree and sent over a channel to RepairService. It is important to note that we know that by the time a slot X is complete, the epoch schedule must exist for the epoch that contains slot X because WindowService will reject blobs for unconfirmed epochs. When a newly completed slot is detected, we also update the current root if it has changed since the last update. The root is made available to RepairService through Blocktree, which holds the latest root.
|
||||
Gossip messages are updated every time a validator receives a complete slot within the epoch. Completed slots are detected by blocktree and sent over a channel to RepairService. It is important to note that we know that by the time a slot X is complete, the epoch schedule must exist for the epoch that contains slot X because WindowService will reject shreds for unconfirmed epochs. When a newly completed slot is detected, we also update the current root if it has changed since the last update. The root is made available to RepairService through Blocktree, which holds the latest root.
|
||||
|
|
@ -11,7 +11,7 @@ A persistent file addressed by [public key](#public-key) and with
|
|||
|
||||
A front-end application that interacts with a Solana cluster.
|
||||
|
||||
#### blob
|
||||
#### shred
|
||||
|
||||
A fraction of a [block](#block); the smallest unit sent between
|
||||
[fullnodes](#fullnode).
|
||||
|
|
|
|||
|
|
@ -1,19 +1,19 @@
|
|||
# Turbine Block Propagation
|
||||
|
||||
A Solana cluster uses a multi-layer block propagation mechanism called *Turbine*
|
||||
to broadcast transaction blobs to all nodes with minimal amount of duplicate
|
||||
to broadcast transaction shreds to all nodes with minimal amount of duplicate
|
||||
messages. The cluster divides itself into small collections of nodes, called
|
||||
*neighborhoods*. Each node is responsible for sharing any data it receives with
|
||||
the other nodes in its neighborhood, as well as propagating the data on to a
|
||||
small set of nodes in other neighborhoods. This way each node only has to
|
||||
communicate with a small number of nodes.
|
||||
|
||||
During its slot, the leader node distributes blobs between the validator nodes
|
||||
During its slot, the leader node distributes shreds between the validator nodes
|
||||
in the first neighborhood (layer 0). Each validator shares its data within its
|
||||
neighborhood, but also retransmits the blobs to one node in some neighborhoods
|
||||
neighborhood, but also retransmits the shreds to one node in some neighborhoods
|
||||
in the next layer (layer 1). The layer-1 nodes each share their data with their
|
||||
neighborhood peers, and retransmit to nodes in the next layer, etc, until all
|
||||
nodes in the cluster have received all the blobs.
|
||||
nodes in the cluster have received all the shreds.
|
||||
|
||||
## Neighborhood Assignment - Weighted Selection
|
||||
|
||||
|
|
@ -26,10 +26,10 @@ make up layer 0. These will automatically be the highest stake holders, allowing
|
|||
the heaviest votes to come back to the leader first. Layer-0 and lower-layer
|
||||
nodes use the same logic to find their neighbors and next layer peers.
|
||||
|
||||
To reduce the possibility of attack vectors, each blob is transmitted over a
|
||||
To reduce the possibility of attack vectors, each shred is transmitted over a
|
||||
random tree of neighborhoods. Each node uses the same set of nodes representing
|
||||
the cluster. A random tree is generated from the set for each blob using
|
||||
randomness derived from the blob itself. Since the random seed is not known in
|
||||
the cluster. A random tree is generated from the set for each shred using
|
||||
randomness derived from the shred itself. Since the random seed is not known in
|
||||
advance, attacks that try to eclipse neighborhoods from certain leaders or
|
||||
blocks become very difficult, and should require almost complete control of the
|
||||
stake in the cluster.
|
||||
|
|
@ -43,7 +43,7 @@ these constraints to determine layer-capacity: Each neighborhood contains
|
|||
`DATA_PLANE_FANOUT` nodes. Layer-0 starts with 1 neighborhood with fanout nodes.
|
||||
The number of nodes in each additional layer grows by a factor of fanout.
|
||||
|
||||
As mentioned above, each node in a layer only has to broadcast its blobs to its
|
||||
As mentioned above, each node in a layer only has to broadcast its shreds to its
|
||||
neighbors and to exactly 1 node in some next-layer neighborhoods,
|
||||
instead of to every TVU peer in the cluster. A good way to think about this is,
|
||||
layer-0 starts with 1 neighborhood with fanout nodes, layer-1 adds "fanout"
|
||||
|
|
@ -52,11 +52,11 @@ neighborhoods, each with fanout nodes and layer-2 will have
|
|||
|
||||
This way each node only has to communicate with a maximum of `2 * DATA_PLANE_FANOUT - 1` nodes.
|
||||
|
||||
The following diagram shows how the Leader sends blobs with a Fanout of 2 to
|
||||
The following diagram shows how the Leader sends shreds with a Fanout of 2 to
|
||||
Neighborhood 0 in Layer 0 and how the nodes in Neighborhood 0 share their data
|
||||
with each other.
|
||||
|
||||
<img alt="Leader sends blobs to Neighborhood 0 in Layer 0" src="img/data-plane-seeding.svg" class="center"/>
|
||||
<img alt="Leader sends shreds to Neighborhood 0 in Layer 0" src="img/data-plane-seeding.svg" class="center"/>
|
||||
|
||||
The following diagram shows how Neighborhood 0 fans out to Neighborhoods 1 and 2.
|
||||
|
||||
|
|
@ -82,7 +82,7 @@ cluster sizes change.
|
|||
|
||||
The following diagram shows how two neighborhoods in different layers interact.
|
||||
To cripple a neighborhood, enough nodes (erasure codes +1) from the neighborhood
|
||||
above need to fail. Since each neighborhood receives blobs from multiple nodes
|
||||
above need to fail. Since each neighborhood receives shreds from multiple nodes
|
||||
in a neighborhood in the upper layer, we'd need a big network failure in the upper
|
||||
layers to end up with incomplete data.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue