228 lines
10 KiB
Markdown
228 lines
10 KiB
Markdown
# Operating the Wormhole Queries Proxy Server
|
|
|
|
The Wormhole queries proxy server (sometimes referred to as the CCQ proxy) is a server that listens on a
|
|
REST endpoint for Wormhole query requests. It validates those requests and forwards them to the guardian
|
|
CCQ P2P network for processing by the guardians. It then accumulates the responses from the guardians,
|
|
verifies quorum and forwards the response to the client.
|
|
|
|
## Building the Proxy Server
|
|
|
|
The proxy server runs as another instance of the `guardiand` process, similar to the spy. It is built exactly
|
|
the same as the spy, and requires the same dependencies. Please see the [Operations Guide](operations.md#building-guardiand) for
|
|
details on how to build `guardiand`.
|
|
|
|
## Deploying the Proxy Server
|
|
|
|
The proxy server can be deployed just like the spy, including potentially running in a container. Note that it
|
|
requires a public IP address to listen for REST requests, and it needs to be able to reach the guardian P2P network.
|
|
|
|
The proxy is not particularly resource intensive, so should run successfully on a reasonable size VM.
|
|
|
|
## Configuring the Proxy Server
|
|
|
|
There are two main parts to configuring the proxy server. The first is setting up the command line arguments,
|
|
which generally will not change after initial setup. The second part of the configuration is the permissions file,
|
|
which will change as the requirements of integrators change.
|
|
|
|
### Proxy Server Command Line Arguments
|
|
|
|
The following is a sample command line for running the proxy server in mainnet.
|
|
|
|
```shell
|
|
wormhole $build/bin/guardiand query-server \
|
|
--env "mainnet" \
|
|
--nodeKey /home/ccq/data/ccq_server.nodeKey \
|
|
--permFile "/home/ccq/data/ccq_server.perms.json" \
|
|
--signerKey "/home/ccq/data/ccq_server.signerKey" \
|
|
--listenAddr "[::]:8080" \
|
|
--ethRPC https://eth.drpc.org \
|
|
--ethContract "0x98f3c9e6E3fAce36bAAd05FE09d375Ef1464288B" \
|
|
--logLevel=info \
|
|
--telemetryLokiURL $LOKI_URL \
|
|
--telemetryNodeName "Mainnet CCQ server 1" \
|
|
--promRemoteURL $PROM_URL
|
|
```
|
|
|
|
- The `env` can be mainnet, testnet or devnet.
|
|
- The `nodeKey` should point to the file containing the P2P key. The first time the proxy runs, if the
|
|
file does not exist, it will be created. You can look in the proxy server logs to get the generated key.
|
|
- The `permFile` is the JSON permissions file, which is documented below.
|
|
- The `signerKey` should point to an armored file containing a key that will be used to sign requests received
|
|
from integrators who are configured to support auto signing and opt not to sign a request. Please see below
|
|
for how to generate this file.
|
|
- The `listenAddr` specifies the port on which the proxy listens for REST requests.
|
|
- The `ethRPC` and `ethContract` are used to read the wormhole guardian set on start up. The address
|
|
above is for mainnet. If you are running in testnet, you should point to Holesky and use `0xa10f2eF61dE1f19f586ab8B6F2EbA89bACE63F7a`.
|
|
(You can confirm these addresses [here](https://docs.wormhole.com/wormhole/reference/constants#contract-addresses).)
|
|
Note that using a public endpoint should be fine, since the proxy only does a single read of the guardian set.
|
|
- The `telemetryLokiURL`, `telemetryNodeName` and `promRemoteURL` are used for telemetry purposes and
|
|
the values will be provided by Wormhole Foundation personnel if appropriate.
|
|
|
|
Optional Parameters
|
|
|
|
- The `gossipAdvertiseAddress` argument allows you to specify an external IP to advertize on P2P (use if behind a NAT or running in k8s).
|
|
- The `monitorPeers` flag will cause the proxy server to periodically check its connectivity to the P2P bootstrap peers, and attempt to reconnect if necessary.
|
|
- The `allowAnything` flag enables defining users with the `allowAnything` flag set to true. This is only allowed in testnet and devnet.
|
|
|
|
#### Creating the Signing Key File
|
|
|
|
Do the following to create the signing key file. Note that the `block-type` must exactly match what is specified below,
|
|
but the `desc` can be anything you want.
|
|
|
|
```shell
|
|
wormhole$ build/bin/guardiand keygen --desc "Your CCQ proxy server" --block-type "CCQ SERVER SIGNING KEY" /home/ccq/data/ccq_server.signerKey
|
|
```
|
|
|
|
### Guardian Support for a New Proxy Server
|
|
|
|
The Queries P2P network is permissioned. The guardians will ignore P2P traffic from sources that are not in their configuration.
|
|
Additionally, they will only honor query requests signed using a key in their configured list. Before you can begin publishing
|
|
requests from your proxy, you must get a quorum (preferably all) of the guardians to add your values for the following to their
|
|
configurations:
|
|
|
|
- P2P key (the value from `nodeKey` file, logged on proxy start up).
|
|
- The public key associated with the signing key. See the `signerKey` file.
|
|
|
|
Please work with foundation personnel to get your proxy server added to the guardian configurations.
|
|
|
|
### Permissions Configuration
|
|
|
|
The file specified by the `permFile` parameter contains JSON that defines the set of allowed queries users, along with the
|
|
sets of requests they are allowed to make.
|
|
|
|
#### File Format
|
|
|
|
The simplest file would look something like this
|
|
|
|
```json
|
|
{
|
|
"permissions": [
|
|
{
|
|
"userName": "Monitor",
|
|
"apiKey": "insert_generated_api_key_here",
|
|
"allowUnsigned": true,
|
|
"allowedCalls": [
|
|
{
|
|
"ethCall": {
|
|
"note:": "Name of WETH on Ethereum",
|
|
"chain": 2,
|
|
"contractAddress": "0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2",
|
|
"call": "0x06fdde03"
|
|
}
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
This creates a single user called "Monitor", who will use the specified API key (more on API keys below).
|
|
This user is allowed to submit unsigned requests (which will be signed using the configured signing key).
|
|
|
|
This sample user is only allowed to make a single `ethCall` request on Ethereum (Wormhole chain ID 2),
|
|
which allows them to call the `name` method on the contract that resides at `0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2`.
|
|
The `call` parameter is the first four bytes of the hash of the ABI encoded function call to be allowed.
|
|
|
|
A given user can have any number of allowed calls (at least one), but they can only make calls that are configured here.
|
|
|
|
#### Supported Call Types
|
|
|
|
The proxy server supports all of the query types supported by the Wormhole Queries protocol. For details on those calls,
|
|
please see the [Wormhole Queries Whitepaper](../whitepapers/0013_ccq.md).
|
|
|
|
The following are the EVM call types, all of which require the `chain`, `contractAddress` and `call` arguments.
|
|
|
|
- `ethCall`
|
|
- `ethCallByTimestamp`
|
|
- `ethCallWithFinality`
|
|
|
|
The following are the Solana call types. Both require the `chain` parameter plus the extra parameter listed below.
|
|
|
|
- `solAccount`, requires the `account` parameter.
|
|
- `solPDA`, requires the `programAddress` parameter.
|
|
|
|
The Solana account and and program address can be expressed as either a 32 byte hex string starting with "0x" or as a base 58 value.
|
|
|
|
#### Creating New API Keys
|
|
|
|
Each user must have an API key. These keys only have meaning to the proxy server. They are not passed to the guardians.
|
|
The proxy requires that a key be present in each query request, and that the specified key exists in the permissions file.
|
|
Beyond that, the API keys have no special meaning. They can be generated using a site like [this](https://www.uuidgenerator.net/version4).
|
|
|
|
#### Updating the Permissions File
|
|
|
|
The proxy server monitors the permissions file for changes. Whenever a change is detected, it reads the file, validates it, and if
|
|
it passes validation, switches to the new version. Care should be taken when editing the file while the proxy server is running, because
|
|
as soon as you save the file, the changes will be picked up (whether they are logically complete or not).
|
|
|
|
#### The `allowAnything` flag
|
|
|
|
If this flag is specified for a user, then that user may make any call on any supported chain, without restriction.
|
|
This flag is only allowed if the `allowAnything` command line argument is specified.
|
|
If this flag is specified, then `allowedCalls` must not be specified.
|
|
|
|
```json
|
|
{
|
|
"permissions": [
|
|
{
|
|
"userName": "Monitor",
|
|
"apiKey": "insert_generated_api_key_here",
|
|
"allowUnsigned": true,
|
|
"allowedAnything": true
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
## Telemetry
|
|
|
|
The proxy server provides two types of telemetry data, logs and metrics.
|
|
|
|
### Logging
|
|
|
|
The proxy server uses the same logging mechanism as the guardian. It will write to a local file, but can also be configured to
|
|
publish logs to Grafana using the Loki protocol. If you will be running your proxy server in mainnet, you should contact foundation
|
|
personnel about getting a Grafana ID to be used for logging and use it to set the `--telemetryLokiURL` command line argument.
|
|
|
|
If you set the log level to `info`, the proxy server logs information on all incoming requests and output bound responses. This can
|
|
be helpful for determining when requests reach quorum, but may be too chatty as the level of queries traffic grows. If that is the
|
|
case, you can set the log level to `warn`.
|
|
|
|
### Metrics
|
|
|
|
The proxy server uses Prometheus to track various activity and can publish them to Grafana. If you will be running your proxy server in mainnet,
|
|
you should contact foundation personnel about getting a Grafana ID and use it to set the `--promRemoteURL` command line argument.
|
|
|
|
For the set of available metrics, see [here](../node/cmd/ccq/metrics.go).
|
|
|
|
## Troubleshooting
|
|
|
|
### P2P Health
|
|
|
|
If you think you are having trouble with your access to the P2P network, you can add `--monitorPeers` to the command line arguments,
|
|
which will cause the proxy server to periodically check its connectivity to the P2P bootstrap peers, and attempt to reconnect if necessary.
|
|
|
|
### Invalid Requests
|
|
|
|
If the proxy server determines that a request is invalid, it does the following:
|
|
|
|
- Logs an error message using the user name (not the API Key).
|
|
- Increments the appropriate Prometheus metric.
|
|
- Sends a failure response to the user.
|
|
|
|
Note that if the proxy server thinks a request is valid, but the guardians do not, the guardians silently drop the request, so it will look
|
|
like a timeout. This is to avoid a denial of service attack on the guardians. This can happen if the proxy server is not properly permissioned
|
|
on the guardians.
|
|
|
|
### Logging Request Detail.
|
|
|
|
If a given integrator is reporting problems with their queries, you may find it useful to add the following to their permissions config
|
|
(at the same level as the API Key, etc).
|
|
|
|
```json
|
|
"logResponses": true,
|
|
```
|
|
|
|
This will cause the proxy server to log every response received for that user, along with the number of responses and how many are
|
|
still needed to meet quorum.
|