2019-09-22 20:38:34 -07:00
# Monitoring a Validator
2019-08-08 10:42:17 -07:00
## Check Gossip
2019-09-22 20:38:34 -07:00
2019-08-08 10:42:17 -07:00
The **identity pubkey** for your validator can also be found by running:
2019-09-22 20:38:34 -07:00
2019-08-08 10:42:17 -07:00
```bash
2019-10-29 16:13:20 -07:00
solana-keygen pubkey ~/validator-keypair.json
2019-08-08 10:42:17 -07:00
```
2019-09-22 20:38:34 -07:00
From another console, confirm the IP address and **identity pubkey** of your validator is visible in the gossip network by running:
2019-08-08 10:42:17 -07:00
```bash
2019-10-29 16:13:20 -07:00
solana-gossip --entrypoint testnet.solana.com:8001 spy
2019-08-08 10:42:17 -07:00
```
2019-11-13 14:58:14 -08:00
## Monitoring Catch Up
2019-09-22 20:38:34 -07:00
2019-11-13 14:58:14 -08:00
It may take some time to catch up with the cluster after your validator boots.
Use the `catchup` command to monitor your validator through this process:
2019-09-22 20:38:34 -07:00
2019-08-08 10:42:17 -07:00
```bash
2019-11-13 14:58:14 -08:00
solana catchup ~/validator-keypair.json
2019-08-08 10:42:17 -07:00
```
2019-11-13 14:58:14 -08:00
Until your validator has caught up, it will not be able to vote successfully and
stake cannot be delegated to it.
2019-09-22 20:38:34 -07:00
2019-11-13 14:58:14 -08:00
Also if you find the cluster's slot advancing faster than yours, you will likely
never catch up. This typically implies some kind of networking issue between
your validator and the rest of the cluster.
2019-08-08 10:42:17 -07:00
## Check Your Balance
2019-09-22 20:38:34 -07:00
2019-11-13 14:58:14 -08:00
Your account balance should decrease by the transaction fee amount as your
validator submits votes, and increase after serving as the leader. Pass the
`--lamports` are to observe in finer detail:
2019-09-22 20:38:34 -07:00
2019-08-08 10:42:17 -07:00
```bash
2019-10-29 16:13:20 -07:00
solana balance --lamports
2019-08-08 10:42:17 -07:00
```
2019-11-13 14:58:14 -08:00
## Check Vote Activity
2019-08-08 10:42:17 -07:00
2019-11-13 14:58:14 -08:00
The `solana show-vote-account` command displays the recent voting activity from your validator:
2019-09-22 20:38:34 -07:00
2019-08-08 10:42:17 -07:00
```bash
2019-11-13 14:58:14 -08:00
solana show-vote-account ~/validator-vote-keypair.json
2019-08-08 10:42:17 -07:00
```
## Get Cluster Info
2019-09-22 20:38:34 -07:00
There are several useful JSON-RPC endpoints for monitoring your validator on the cluster, as well as the health of the cluster:
2019-08-08 10:42:17 -07:00
```bash
# Similar to solana-gossip, you should see your validator in the list of cluster nodes
2019-10-29 16:13:20 -07:00
curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1, "method":"getClusterNodes"}' http://testnet.solana.com:8899
2019-08-16 16:02:19 -07:00
# If your validator is properly voting, it should appear in the list of `current` vote accounts. If staked, `stake` should be > 0
2019-10-29 16:13:20 -07:00
curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1, "method":"getVoteAccounts"}' http://testnet.solana.com:8899
2019-08-08 10:42:17 -07:00
# Returns the current leader schedule
2019-10-29 16:13:20 -07:00
curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1, "method":"getLeaderSchedule"}' http://testnet.solana.com:8899
2019-08-08 10:42:17 -07:00
# Returns info about the current epoch. slotIndex should progress on subsequent calls.
curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1, "method":"getEpochInfo"}' http://testnet.solana.com:8899
```
2019-11-13 14:58:14 -08:00
2019-08-08 10:42:17 -07:00
## Validator Metrics
2019-09-22 20:38:34 -07:00
2019-08-08 10:42:17 -07:00
Metrics are available for local monitoring of your validator.
2019-09-22 20:38:34 -07:00
Docker must be installed and the current user added to the docker group. Then download `solana-metrics.tar.bz2` from the Github Release and run
2019-08-08 10:42:17 -07:00
```bash
2019-10-29 16:13:20 -07:00
tar jxf solana-metrics.tar.bz2
cd solana-metrics/
./start.sh
2019-08-08 10:42:17 -07:00
```
2019-09-22 20:38:34 -07:00
A local InfluxDB and Grafana instance is now running on your machine. Define `SOLANA_METRICS_CONFIG` in your environment as described at the end of the `start.sh` output and restart your validator.
2019-08-08 10:42:17 -07:00
Metrics should now be streaming and visible from your local Grafana dashboard.
## Timezone For Log Messages
2019-09-22 20:38:34 -07:00
Log messages emitted by your validator include a timestamp. When sharing logs with others to help triage issues, that timestamp can cause confusion as it does not contain timezone information.
To make it easier to compare logs between different sources we request that everybody use Pacific Time on their validator nodes. In Linux this can be accomplished by running:
2019-08-08 10:42:17 -07:00
```bash
2019-10-29 16:13:20 -07:00
sudo ln -sf /usr/share/zoneinfo/America/Los_Angeles /etc/localtime
2019-08-08 10:42:17 -07:00
```