136 lines
5.3 KiB
Markdown
136 lines
5.3 KiB
Markdown
---
|
|
title: "Restarting a Solana Cluster"
|
|
# really high number to ensure it is listed last in the sidebar
|
|
sidebar_position: 999
|
|
sidebar_label: Restart a Cluster
|
|
pagination_label: "Validator Guides: Restart a Cluster"
|
|
---
|
|
|
|
### Step 1. Identify the latest optimistically confirmed slot for the cluster
|
|
|
|
In Solana 1.14 or greater, run the following command to output the latest
|
|
optimistically confirmed slot your validator observed:
|
|
```bash
|
|
agave-ledger-tool -l ledger latest-optimistic-slots
|
|
```
|
|
|
|
In Solana 1.13 or less, the latest optimistically confirmed can be found by looking for the more recent occurrence of
|
|
[this](https://github.com/solana-labs/solana/blob/0264147d42d506fb888f5c4c021a998e231a3e74/core/src/optimistic_confirmation_verifier.rs#L71)
|
|
metrics datapoint.
|
|
|
|
Call this slot `SLOT_X`
|
|
|
|
Note that it's possible that some validators observed an optimistically
|
|
confirmed slot that's greater than others before the outage. Survey the other
|
|
validators on the cluster to ensure that a greater optimistically confirmed slot
|
|
does not exist before proceeding. If a greater slot value is found use it
|
|
instead.
|
|
|
|
|
|
### Step 2. Stop the validator(s)
|
|
|
|
### Step 3. Optionally install the new solana version
|
|
|
|
### Step 4. Create a new snapshot for slot `SLOT_X` with a hard fork at slot `SLOT_X`
|
|
|
|
```bash
|
|
$ agave-ledger-tool -l <LEDGER_PATH> --snapshot-archive-path <SNAPSHOTS_PATH> --incremental-snapshot-archive-path <INCREMENTAL_SNAPSHOTS_PATH> create-snapshot SLOT_X <SNAPSHOTS_PATH> --hard-fork SLOT_X
|
|
```
|
|
|
|
The snapshots directory should now contain the new snapshot.
|
|
`agave-ledger-tool create-snapshot` will also output the new shred version, and bank hash value,
|
|
call this NEW_SHRED_VERSION and NEW_BANK_HASH respectively.
|
|
|
|
Adjust your validator's arguments:
|
|
|
|
```bash
|
|
--wait-for-supermajority SLOT_X
|
|
--expected-bank-hash NEW_BANK_HASH
|
|
```
|
|
|
|
Then restart the validator.
|
|
|
|
Confirm with the log that the validator booted and is now in a holding pattern at `SLOT_X`, waiting for a super majority.
|
|
|
|
Once NEW_SHRED_VERSION is determined, nudge foundation entrypoint operators to update entrypoints.
|
|
|
|
### Step 5. Announce the restart on Discord:
|
|
|
|
Post something like the following to #announcements (adjusting the text as appropriate):
|
|
|
|
> Hi @Validators,
|
|
>
|
|
> We've released v1.1.12 and are ready to get testnet back up again.
|
|
>
|
|
> Steps:
|
|
>
|
|
> 1. Install the v1.1.12 release: https://github.com/solana-labs/solana/releases/tag/v1.1.12
|
|
> 2. a. Preferred method, start from your local ledger with:
|
|
>
|
|
> ```bash
|
|
> agave-validator
|
|
> --wait-for-supermajority SLOT_X # <-- NEW! IMPORTANT! REMOVE AFTER THIS RESTART
|
|
> --expected-bank-hash NEW_BANK_HASH # <-- NEW! IMPORTANT! REMOVE AFTER THIS RESTART
|
|
> --hard-fork SLOT_X # <-- NEW! IMPORTANT! REMOVE AFTER THIS RESTART
|
|
> --no-snapshot-fetch # <-- NEW! IMPORTANT! REMOVE AFTER THIS RESTART
|
|
> --entrypoint entrypoint.testnet.solana.com:8001
|
|
> --known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on
|
|
> --expected-genesis-hash 4uhcVJyU9pJkvQyS88uRDiswHXSCkY3zQawwpjk2NsNY
|
|
> --only-known-rpc
|
|
> --limit-ledger-size
|
|
> ... # <-- your other --identity/--vote-account/etc arguments
|
|
> ```
|
|
>
|
|
> b. If your validator doesn't have ledger up to slot SLOT_X or if you have deleted your ledger, have it instead download a snapshot with:
|
|
>
|
|
> ```bash
|
|
> agave-validator
|
|
> --wait-for-supermajority SLOT_X # <-- NEW! IMPORTANT! REMOVE AFTER THIS RESTART
|
|
> --expected-bank-hash NEW_BANK_HASH # <-- NEW! IMPORTANT! REMOVE AFTER THIS RESTART
|
|
> --entrypoint entrypoint.testnet.solana.com:8001
|
|
> --known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on
|
|
> --expected-genesis-hash 4uhcVJyU9pJkvQyS88uRDiswHXSCkY3zQawwpjk2NsNY
|
|
> --only-known-rpc
|
|
> --limit-ledger-size
|
|
> ... # <-- your other --identity/--vote-account/etc arguments
|
|
> ```
|
|
>
|
|
> You can check for which slots your ledger has with: `agave-ledger-tool -l path/to/ledger bounds`
|
|
>
|
|
> 3. Wait until 80% of the stake comes online
|
|
>
|
|
> To confirm your restarted validator is correctly waiting for the 80%:
|
|
> a. Look for `N% of active stake visible in gossip` log messages
|
|
> b. Ask it over RPC what slot it's on: `solana --url http://127.0.0.1:8899 slot`. It should return `SLOT_X` until we get to 80% stake
|
|
>
|
|
> Thanks!
|
|
|
|
### Step 7. Wait and listen
|
|
|
|
Monitor the validators as they restart. Answer questions, help folks,
|
|
|
|
## Troubleshooting
|
|
|
|
### 80% of the stake didn't participate in the restart, now what?
|
|
If less than 80% of the stake join the restart after a reasonable amount of
|
|
time, it will be necessary to retry the restart attempt with the stake from the
|
|
non-responsive validators removed.
|
|
|
|
The community should identify and come to social consensus on the set of
|
|
non-responsive validators. Then all participating validators return to Step 4
|
|
and create a new snapshot with additional `--destake-vote-account <PUBKEY>`
|
|
arguments for each of the non-responsive validator's vote account address
|
|
|
|
```bash
|
|
$ agave-ledger-tool -l ledger create-snapshot SLOT_X ledger --hard-fork SLOT_X \
|
|
--destake-vote-account <VOTE_ACCOUNT_1> \
|
|
--destake-vote-account <VOTE_ACCOUNT_2> \
|
|
.
|
|
.
|
|
--destake-vote-account <VOTE_ACCOUNT_N> \
|
|
```
|
|
|
|
This will cause all stake associated with the non-responsive validators to be
|
|
immediately deactivated. All their stakers will need to re-delegate their stake
|
|
once the cluster restart is successful.
|