proposal: Epoch Accounts Hash (#27801)
This commit is contained in:
parent
d601252296
commit
76598dbddb
|
@ -0,0 +1,296 @@
|
||||||
|
---
|
||||||
|
title: Epoch Accounts Hash
|
||||||
|
---
|
||||||
|
|
||||||
|
*Paraphrasing from https://github.com/solana-labs/solana/issues/26847*
|
||||||
|
|
||||||
|
## Background
|
||||||
|
|
||||||
|
Rent collection checks every account at least once per epoch. At each slot, a
|
||||||
|
deterministic set of accounts (based on the pubkey range) is loaded, checked
|
||||||
|
for rent collection, and stored back to the Accounts DB at this new slot.
|
||||||
|
|
||||||
|
Accounts are stored (rewritten) _even if_ they are unchanged. This has a few
|
||||||
|
positive effects.
|
||||||
|
1. Once an account is rewritten, the previous version at an older slot is now
|
||||||
|
dead. As entire slots and AppendVecs become full of only dead accounts,
|
||||||
|
they can then be dropped/recycled.
|
||||||
|
2. Each account rewritten due to rent collection is included in that slot's
|
||||||
|
bank hash. Since the bank hash is part of what is voted on for consensus,
|
||||||
|
this means every account is verified by the network at least once per
|
||||||
|
epoch.
|
||||||
|
|
||||||
|
However, there is a big downside to rewriting unchanged accounts: performance.
|
||||||
|
Storing accounts can be very expensive. And since accounts now are required to
|
||||||
|
be rent-exempt, the majority of accounts rewritten due to rent collection are
|
||||||
|
unchanged. What if unchanged accounts were no longer rewritten? This would
|
||||||
|
minimally be a big performance win.
|
||||||
|
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
If rent collection no longer is rewriting unchanged accounts, we lose the two
|
||||||
|
positive effects. Dealing with Positive Effect 1 (from above) will be handled
|
||||||
|
by _Ancient AppendVecs_, and will not be discussed here. So how do we still
|
||||||
|
get the security from Positive Effect 2? How can we still verify every account
|
||||||
|
at least once per epoch, *as part of consensus*, but without rewriting
|
||||||
|
accounts?
|
||||||
|
|
||||||
|
|
||||||
|
## Proposed Solution
|
||||||
|
|
||||||
|
Perform a full accounts hash calculation once per epoch, and hash the result
|
||||||
|
into a bank's `hash`. This will be known as the _Epoch Accounts Hash_, or
|
||||||
|
_EAH_.
|
||||||
|
|
||||||
|
This retains the Positive Effect 2 from rent collection by checking every
|
||||||
|
account at least once per epoch. Thus, any validators with missing, corrupt,
|
||||||
|
or extra accounts will identify those issues within 1-2 epochs.
|
||||||
|
|
||||||
|
|
||||||
|
### Implementation
|
||||||
|
|
||||||
|
Performing a full accounts hash takes a relatively long time to complete. For
|
||||||
|
this reason, the EAH calculation must take place in the background.
|
||||||
|
|
||||||
|
In order for all the validators to calculate the same accounts hash, the
|
||||||
|
calculation must be based on the same view of all accounts. This means the EAH
|
||||||
|
must be based on a predetermined slot. This will be known as the `start slot`.
|
||||||
|
The `start slot` is calculated as an offset into the epoch. This offset will
|
||||||
|
be known as the `start offset`. Formally, the `start slot` is the first root
|
||||||
|
*greater-than-or-equal-to* `first slot in epoch + start offset`.
|
||||||
|
|
||||||
|
Similarly, all the validators must save the EAH into a bank at a predetermined
|
||||||
|
slot, and offset from the first slot of an epoch. This will be known as the
|
||||||
|
`stop slot` and `stop offset`, respectively.
|
||||||
|
|
||||||
|
* The `start offset` will be set at one-quarter into the epoch.
|
||||||
|
* The `stop offset` will be set at three-quarters into the epoch.
|
||||||
|
* For epochs with 432,000 slots, the `start offset` will be 108,000 and the
|
||||||
|
`stop offset` will be 324,000.
|
||||||
|
|
||||||
|
These constants may be changed in the future, or may be determined at runtime.
|
||||||
|
The main justifications for these values are:
|
||||||
|
1. Do not start the EAH calculation at the beginning of an epoch, as the
|
||||||
|
beginning of an epoch is already a time of contention and stress. There is
|
||||||
|
no reason to make this worse.
|
||||||
|
2. The bank to save the EAH into—the `stop offset`—should be sufficiently far
|
||||||
|
in the future to guarantee all validators are able to complete the accounts
|
||||||
|
hash calculation in time.
|
||||||
|
3. The `start offset` should be *after* the `rewarding interval`
|
||||||
|
(from [Partitioned Inflationary Rewards Distribution](https://github.com/solana-labs/solana/pull/27455)).
|
||||||
|
This ensures stake rewards have been distributed and stored into the
|
||||||
|
accounts for this epoch.
|
||||||
|
|
||||||
|
Once the EAH calculation is complete, it must be saved somewhere. Since this
|
||||||
|
occurs in the background, there is not an associated `Bank` that would make
|
||||||
|
sense to save into. Instead, a new field will be added to `AccountsDb` that
|
||||||
|
will store the EAH. Later, the bank at slot `stop slot`†¹ will read the EAH from
|
||||||
|
`AccountsDb` and hash it into its own hash (aka _bank hash_).
|
||||||
|
|
||||||
|
EAH calculation will use the existing _accounts background services_ (_ABS_) to
|
||||||
|
perform the actual calculation. Requests for EAH calculation will be sent from
|
||||||
|
`bank_forks::set_root()`†², with a new request type to distinguish an EAH request
|
||||||
|
from a Snapshot request. Since the EAH will be part of consensus, it is not
|
||||||
|
optional; EAH requests will have the highest priority in ABS, and will be
|
||||||
|
processed first/instead of other requests.
|
||||||
|
|
||||||
|
†¹: More precisely, all banks where `bank slot >= stop slot` and `parent slot <
|
||||||
|
stop slot` will include the EAH in their _bank hash_. This ensures EAH
|
||||||
|
handles forking around `stop slot`, since only one of these banks will end
|
||||||
|
up rooted.
|
||||||
|
|
||||||
|
†²: An EAH calculation will be requested when `root bank slot >= start slot`
|
||||||
|
and `root parent slot < start slot`. This handles the scenario where
|
||||||
|
validators call `bank_forks::set_root()` at different intervals.
|
||||||
|
|
||||||
|
|
||||||
|
#### Details
|
||||||
|
|
||||||
|
#### Snapshots
|
||||||
|
|
||||||
|
A snapshot contains all the state necessary to reconstruct the cluster as of a
|
||||||
|
certain slot. A snapshot may then need to contain the EAH so that the `stop
|
||||||
|
slot` can include it. Consider the following scenarios within an epoch where a
|
||||||
|
snapshot is requested for slot `X`:
|
||||||
|
|
||||||
|
|
||||||
|
##### 1. `X >= first slot in epoch` and `X < start slot`
|
||||||
|
|
||||||
|
Since the `start slot` has not been reached yet, there is nothing special to do
|
||||||
|
in order to take a snapshot in this scenario.
|
||||||
|
|
||||||
|
|
||||||
|
##### 2. `X == start slot`
|
||||||
|
|
||||||
|
The EAH *must* be included in the snapshot. Since the snapshot process always
|
||||||
|
calculates the accounts hash, no additional calculations are required. The
|
||||||
|
accounts hash calculation result will be used both to store in the snapshot as
|
||||||
|
the EAH, and for the snapshot hash (which is used at load-time for verification).
|
||||||
|
|
||||||
|
|
||||||
|
##### 3. `X > start slot` and `X < stop slot`
|
||||||
|
|
||||||
|
If a snapshot is requested to be created *after* the `start slot` but *before*
|
||||||
|
the EAH calculation has completed, then it will be impossible to create a
|
||||||
|
snapshot with the correct EAH. The snapshot process will wait until the EAH
|
||||||
|
calculation has completed before proceeding.
|
||||||
|
|
||||||
|
|
||||||
|
##### 4. `X == stop slot`
|
||||||
|
|
||||||
|
The EAH has been calculated for this epoch, and has been included in the `stop
|
||||||
|
slot` bank. No further handling is required; the snapshot does not need to
|
||||||
|
contain the EAH.
|
||||||
|
|
||||||
|
|
||||||
|
##### 5. `X > stop slot` and `X <= last slot in epoch`
|
||||||
|
|
||||||
|
Same as (4).
|
||||||
|
|
||||||
|
|
||||||
|
#### Corner Cases
|
||||||
|
|
||||||
|
#### Minimum Slots per Epoch
|
||||||
|
|
||||||
|
An EAH is requested by `BankForks::set_root()`, which happens while setting
|
||||||
|
*roots*. The EAH is stored into `Bank`s when they are *frozen*. Banks are
|
||||||
|
frozen 32 slots before they are rooted. For the expected behavior, the EAH
|
||||||
|
start slot really should be 32 slots before the stop slot. If the number of
|
||||||
|
slots per epoch is small, this can result in surprising behavior.
|
||||||
|
|
||||||
|
Example 1: Assume there are 64 slots per epoch. The EAH start offset is 16
|
||||||
|
and the EAH stop offset is 48. The difference is 32. So when Bank 48 is
|
||||||
|
frozen before Bank 16 is rooted, a new EAH request has not yet been requested;
|
||||||
|
the EAH from the previous epoch is still valid and will be used by Bank 48.
|
||||||
|
|
||||||
|
Example 2: Assume there are 66 slots per epoch, then the EAH start offset is
|
||||||
|
still 16 and the EAH stop offset is now 49. The difference is now 33. When
|
||||||
|
Bank 49 is frozen, Bank 16 will already have been rooted, and thus sent an EAH
|
||||||
|
request; Bank 49 will wait for the new EAH calculation to complete.
|
||||||
|
|
||||||
|
Example 3: Assume there are 32 slots per epoch (the minimum allowed). The EAH
|
||||||
|
start offset is 8, and the EAH stop offset is 24. Similar to Example 1, Bank
|
||||||
|
24 is frozen around when Bank 24 *of the previous epoch* is rooted. This
|
||||||
|
ensures that when the EAH is stored, it'll be for the previous epoch.
|
||||||
|
|
||||||
|
In these examples the observed behavior of the EAH is different than when using
|
||||||
|
the normal 432,000 slots per epoch. The EAH is still valid and correct with a
|
||||||
|
small number of slots per epoch; it now has a delay of one epoch. Since the
|
||||||
|
epochs themselves can be much faster, security is not reduced.
|
||||||
|
|
||||||
|
|
||||||
|
#### Warping
|
||||||
|
|
||||||
|
Warping introduces corner cases into EAH because many slots may be skipped,
|
||||||
|
including the entire range of `start slot` to `stop slot`.
|
||||||
|
|
||||||
|
When warping from before `stop slot` to after, the new bank will include the
|
||||||
|
existing EAH in its hash during `freeze()`. If the bank's parent is from
|
||||||
|
before `start slot`, then a new EAH calculation will not have been requested.
|
||||||
|
This is safe because warping cannot be used on a live cluster; only for a new
|
||||||
|
cluster or tests/debugging. This means _when_ the EAH was calculated is not
|
||||||
|
germane.
|
||||||
|
|
||||||
|
When warping from before `start slot` to after, an EAH calculation will be
|
||||||
|
requested the next time `set_root()` is called. Therefore the EAH will be
|
||||||
|
based on this new bank. This is also safe and correct.
|
||||||
|
|
||||||
|
For specific examples, refer to Appendix A.
|
||||||
|
|
||||||
|
|
||||||
|
#### Implementation Alternatives
|
||||||
|
|
||||||
|
##### Perform the EAH calculation in the foreground
|
||||||
|
|
||||||
|
The accounts hash calculation takes around 15 seconds (median, on Mainnet-Beta
|
||||||
|
today). This is far beyond the slot time; this would be bad UX, and also
|
||||||
|
decrease network stability.
|
||||||
|
|
||||||
|
|
||||||
|
##### Remove `stop offset`
|
||||||
|
|
||||||
|
Instead of having two offsets—one for `start` and one for `stop`—use a single
|
||||||
|
offset for both. This delays when the EAH is saved into a Bank and voted on.
|
||||||
|
The saved EAH is now the EAH from the previous epoch. This could work; would
|
||||||
|
reduce the number of "special" slots from two to one. No significant
|
||||||
|
advantages observed.
|
||||||
|
|
||||||
|
|
||||||
|
##### Send EAH requests when making a new bank, instead of a new root
|
||||||
|
|
||||||
|
When a bank is created, we don't yet know if it will be finalized until it is
|
||||||
|
rooted, which could result in multiple EAH requests due to forking. This would
|
||||||
|
be bad for performance.
|
||||||
|
|
||||||
|
|
||||||
|
### Appendix A: All Warping Scenarios
|
||||||
|
|
||||||
|
To enumerate how EAH interacts with warping, refer to the following diagram for
|
||||||
|
the scenarios below:
|
||||||
|
|
||||||
|
```text
|
||||||
|
+---------+-----------------+-----------+---------+-----------------+-----------+
|
||||||
|
| > < | > < |
|
||||||
|
| A > B < C | D > E < F |
|
||||||
|
| > < | > < |
|
||||||
|
+---------+-----------------+-----------+---------+-----------------+-----------+
|
||||||
|
| | | | | | |
|
||||||
|
v v v v v v v
|
||||||
|
epoch 1 start slot 1 stop slot 1 epoch 2 start slot 2 stop slot 2 epoch 3
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
#### parent slot: `A`, warp slot: `A`
|
||||||
|
|
||||||
|
No slots important to the EAH have been skipped, so no change in behavior.
|
||||||
|
|
||||||
|
|
||||||
|
#### parent slot: `A`, warp slot: `B`
|
||||||
|
|
||||||
|
An EAH calculation will be requested at the warp slot, and then will be
|
||||||
|
included in the Bank at `slot slot 1`; behavior is unchanged.
|
||||||
|
|
||||||
|
|
||||||
|
#### parent slot: `A`, warp slot: `C` or `D`
|
||||||
|
|
||||||
|
The entire EAH range has been skipped; no new EAH calculation will have been
|
||||||
|
requested for epoch 1. The warp slot will include the EAH from `epoch 0`.
|
||||||
|
This is different from the normal behavior.
|
||||||
|
|
||||||
|
|
||||||
|
#### parent slot: `A`, warp slot: `E`
|
||||||
|
|
||||||
|
Similar to `A -> B`, an EAH calculation will be requested at the warp slot, and
|
||||||
|
then will be included in the Bank at `stop slot 2`. Behavior appears normal.
|
||||||
|
|
||||||
|
|
||||||
|
#### parent slot: `A`, warp slot: `F`
|
||||||
|
|
||||||
|
Similar to `A -> C`, no new EAH calculation will be requested. The warp slot
|
||||||
|
will include the EAH from `epoch 0`. This is different from the normal
|
||||||
|
behavior.
|
||||||
|
|
||||||
|
|
||||||
|
#### parent slot: `B`, warp slot: `B`
|
||||||
|
|
||||||
|
Similar to `A -> A`, no slots important to the EAH have been skipped, so no
|
||||||
|
change in behavior.
|
||||||
|
|
||||||
|
|
||||||
|
#### parent slot: `B`, warp slot: `C` or `D`
|
||||||
|
|
||||||
|
This will be observed as normal behavior; the warp slot will include the EAH
|
||||||
|
that was calculated based on `start slot 1`.
|
||||||
|
|
||||||
|
|
||||||
|
#### parent slot: `B`, warp slot: `E`
|
||||||
|
|
||||||
|
Similar to `A -> B`, an EAH calculation will be requested at the warp slot, and
|
||||||
|
then will be included in the Bank at `stop slot 2`. Behavior appears normal.
|
||||||
|
|
||||||
|
|
||||||
|
#### parent slot: `B`, warp slot: `F`
|
||||||
|
|
||||||
|
The warp slot will include the EAH from `start slot 1`. Behavior appears
|
||||||
|
different.
|
Loading…
Reference in New Issue