* high level overview of how to enable the NTT Accountant
* emphasize all guardians need to run wormchain validators
* remove the reference to checking out a v2.x branch as the main branch
is for wormholev2 now and has been for some time.
* Since wormhole supports 30+ chains now and not just 5 chains, the
solana, eth, terra, bsc bits were removed
* The solana and terra content had the urls updated to reflect reality
and moved into their own sections.
* Made a reference to docs.wormhole.com's constants reference for chains
The lcd was built directly into the terra binary a long time ago and no
longer requires a separate service for the lcd. This happened more than
a year ago.
In addition to Wormhole itself, you need to run your own verifying node for every chain that Wormhole connects to:
- **Solana**. There is no light client for Solana yet, so you'll have to run a full solana-validator node. It does not
have to actually be a validator - you can run solana-validator in non-validating mode if you are not a validator.
Refer to the [Solana documentation](https://docs.solana.com/running-validator) on how to run a validator. The validator
requirements as stated in their docs are excessive - for the current iteration for mainnet-beta, the "low end" config
with no GPU is perfectly adequate, and will have enough spare capacity.
[Solana's Discord server](https://solana.com/community) is a great resource for questions regarding validator ops.
- **Ethereum**. See below - you need at least a light client. For stability reasons, a full node is recommended.
- **Terra** requires a full node and an [LCD server](https://docs.terra.money/terracli/lcd.html#light-client-daemon)
pointing to your full node. Refer to the [Terra documentation](https://docs.terra.money/node/join-network.html)
on how to run a full node. From a security point of view, running only an LCD server with `--trust-node=false` pointed
to somebody else's full node would be sufficient, but you'd then depend on that single node for availability unless
you set up a load balancer pointing to a set of nodes.
- **Binance Smart Chain**: Same requirements as Ethereum. Note that BSC has higher throughput than Ethereum and
roughly requires twice as many compute resources.
In addition to Wormhole itself, you need to run your own verifying node for every chain that Wormhole connects to except
for newer IBC connected chains that integrate through wormhole gateway. Please refer to the [constants reference](https://docs.wormhole.com/wormhole/reference/constants)
for all chains wormhole connects to.
**Do NOT use third-party RPC service providers** for any of the chains! You'd fully trust them, and they could lie to
you on whether an event has actually been observed. The whole point of Wormhole is not to rely on centralized nodes!
@ -33,6 +16,12 @@ so you can test changes for your mainnet full nodes and gain operational experie
### Solana node requirements
Refer to the [Solana documentation](https://docs.solanalabs.com/operations/setup-an-rpc-node) on how to run an rpc
(full) node. [Solana's Discord server](https://solana.com/community) is a great resource for questions regarding
operations.
The `#rpc-server-operators` channel is especially useful for setting up Solana rpc nodes.
Your Solana RPC node needs the following parameters enabled:
```
@ -102,6 +91,143 @@ since only very few nodes support the light client protocol.
Running a full node typically requires ~500G of SSD storage, 8G of RAM and 4-8 CPU threads (depending on clock
frequency). Light clients have much lower hardware requirements.
### Terra
Refer to the [Terra documentation](https://docs.terra.money/full-node/run-a-full-terra-node/set-up-production/) on how to run a full node.
#### Terra Classic
Refer to the [Terra Classic documentation](https://classic-docs.terra.money/docs/full-node/run-a-full-terra-node/README.html) on how to run a full node.
### Wormchain
All guardians **must run validators for wormchain**, the codename of [Wormhole Gateway](https://wormhole.com/gateway/).
The ``--wormchainURL` argument to the guardian node should point to `<validator address>:9090` which is the `grpc` port
in the app.toml.
Example port setup:
<!-- cspell:disable -->
config.toml:
```toml
[rpc]
laddr = "tcp://0.0.0.0:26657"
grpc_laddr = ""
pprof_laddr = "localhost:6060"
[p2p]
laddr = "tcp://0.0.0.0:26656"
external_address = ""
```
app.toml:
```toml
[grpc]
address = "0.0.0.0:9090"
[grpc-web]
address = "0.0.0.0:9091"
```
<!-- cspell:enable -->
For signing, consider setting up a remote threshold signer such as
[horcrux](https://github.com/strangelove-ventures/horcrux) and adopting the sentry node architecture with sentry nodes
Some non-ethereum EVM compatible blockchains need to run in archive mode for [queries](https://wormhole.com/queries)
to function correctly. By default in geth, [historical state is only kept in memory for the previous 128 blocks](https://github.com/ethereum/go-ethereum/blob/4458905f261d5d9ba5fda3d664f9bb80346ab404/core/state/statedb.go#L1259-L1265).
After 128 blocks, older states are garbage collected. Many of these chains are forks of geth that maintain this
historical limitation.
* Arbitrum
* Base
* Optimism
Newer execution clients such as [reth](https://github.com/paradigmxyz/reth) lack this limitation and are worth
investigating once they are stable.
Additionally, if there is ever a scenario where the network fails to come to consensus on an EVM compatible chain due to
a hard fork or some unforeseen scenario, it might be required to run archive nodes for those chains temporarily to ensure
the transactions can be reobserved.
### Cosmos / IBC Connected nodes
All modern cosmos integrations happen by wormhole observing IBC transactions on gateway (wormchain). Guardian node operators do not need to run full nodes for these networks. For cosmos based chains that were added before this functionality, a full node is still necessary.
The following cosmos based nodes were added prior to gateway and guardians need to run full nodes:
* Injective
* Terra
* Terra Classic
* XPLA
**NOTE**: All guardians must run validators for wormchain.
## Building guardiand
For security reasons, we do not provide a pre-built binary. You need to check out the repo and build the
@ -113,7 +239,6 @@ First, check out the version of the Wormhole repo that you want to deploy:
```bash
git clone https://github.com/wormhole-foundation/wormhole && cd wormhole
git checkout v2.0.x
```
Then, compile the release binary as an unprivileged build user:
@ -157,7 +282,8 @@ We strongly recommend a separate user and systemd services for the Wormhole serv
See the separate [wormhole-networks](https://github.com/wormhole-foundation/wormhole-networks) repository for examples
on how to set up the guardiand unit for a specific network.
You need to open port 8999/udp in your firewall for the P2P network. Nothing else has to be exposed externally.
You need to open port 8999/udp in your firewall for the P2P network and 8996/udp for
[cross chain queries](../whitepapers/0013_ccq.md). Nothing else has to be exposed externally if you don't run public rpc.
journalctl can show guardiand's colored output using the `-a` flag for binary output, i.e.: `journalctl -a -f -u guardiand`.
@ -169,11 +295,18 @@ Refer to [devnet/](../devnet) for example k8s deployments as a starting point fo
have to build your own containers. Unless you already run Kubernetes in production, we strongly recommend a traditional
deployment on a dedicated instance - it's easier to understand and troubleshoot.
When running in kubernetes, or behind any kind of NAT, pass `--gossipAdvertiseAddress=external.ip.address` to the
guardiand node process to ensure the external address is advertized in p2p. If this is not done, reobservation
requests and [ccq](https://wormhole.com/queries) will not function as intended.
### Monitoring
Wormhole exposes a status server for readiness and metrics. By default, it listens on port 6060 on localhost.
You can use a command line argument to expose it publicly: `--statusAddr=[::]:6060`.
**NOTE:** Parsing the log output for monitoring is NOT recommended. Log output is meant for human consumption and is
not considered a stable API. Log messages may be added, modified or removed without notice. Use the metrics :-)
#### `/readyz`
This endpoint returns a 200 OK status code once the Wormhole node is ready to serve requests. A node is
@ -193,9 +326,92 @@ alerting will be documented here.
See [Wormhole.json](../dashboards/Wormhole.json) for an example Grafana dashboard.
#### Wormhole Dashboard
There is a [dashboard](https://wormhole-foundation.github.io/wormhole-dashboard) which shows the overall health of the
network and has metrics on individual guardians.
**NOTE:** Parsing the log output for monitoring is NOT recommended. Log output is meant for human consumption and is
not considered a stable API. Log messages may be added, modified or removed without notice. Use the metrics :-)
#### Wormhole Fly Healthcheck
In the [wormhole-dashboard](https://github.com/wormhole-foundation/wormhole-dashboard) repository, there is a small
which verifies that the guardian is gossiping out heartbeats, is submitting chain observations, and has a working
heartbeats api available. This is a very good way to verify a specific guardian is functioning as intended.
You can clone the repo and run the check against the [MCF Guardian](https://github.com/wormhole-foundation/wormhole-networks/blob/649dcc48f29d462fe6cb0062cb6530021d36a417/mainnetv2/guardianset/v3.prototxt#L58):
$ go run main.go --pubKey 0xDA798F6896A3331F64b48c12D1D57Fd9cbe70811 --url https://wormhole-v2-mainnet-api.mcf.rocks
✅ guardian heartbeat received {12D3KooWDZVv7BhZ8yFLkarNdaSWaB43D6UbQwExJ8nnGAEmfHcU: [/ip4/185.188.42.109/udp/8999/quic-v1]}
✅ 44 observations received
✅ /v1/heartbeats
```
If the guardian public rpc is not exposed, the `--url` flag can be omitted:
```shell
$ go run main.go --pubKey 0xDA798F6896A3331F64b48c12D1D57Fd9cbe70811
✅ guardian heartbeat received {12D3KooWDZVv7BhZ8yFLkarNdaSWaB43D6UbQwExJ8nnGAEmfHcU: [/ip4/185.188.42.109/udp/8999/quic-v1]}
✅ 41 observations received
ℹ️ --url not defined, skipping web checks
```
The bootstrap nodes and network defaults to mainnet and the values can be found in the [network constants](../node/pkg/p2p/network_consts.go).
It can also be used to test a specific bootstrap node/s:
```shell
$ go run main.go --pubKey 0xDA798F6896A3331F64b48c12D1D57Fd9cbe70811 --bootstrap /dns4/wormhole.mcf.rocks/udp/8999/quic/p2p/12D3KooWDZVv7BhZ8yFLkarNdaSWaB43D6UbQwExJ8nnGAEmfHcU
✅ guardian heartbeat received {12D3KooWDZVv7BhZ8yFLkarNdaSWaB43D6UbQwExJ8nnGAEmfHcU: [/ip4/185.188.42.109/udp/8999/quic-v1]}
✅ 44 observations received
ℹ️ --url not defined, skipping web checks
```
## Native Token Transfers
[NTT](https://github.com/wormhole-foundation/example-native-token-transfers) is an exciting feature of wormhole that builds upon the core bridge to allow mint/burn style transfers. Ensuring it runs correctly requires integrating it with the NTT Accountant. To enable this feature, create a **new** wormchain key. Do not reuse an existing global accountant key and add the following parameters:
Please remember to allowlist the new NTT Accountant key for use with Wormchain! For instructions on how to do that, speak with someone from the Wormhole Foundation.
## Cross Chain Queries
[CCQ](https://github.com/wormhole-foundation/wormhole/blob/main/whitepapers/0013_ccq.md) also known as [Wormhole Queries](https://wormhole.com/queries) is a feature to allow pulling attestations in a cross chain manner. To run ccq, a few additional flags need to be enabled on the guardian node:
Jeff Schroeder
Evan Gray
bruce-riley
Dirk Brink