146 lines
5.6 KiB
Markdown
146 lines
5.6 KiB
Markdown
# Terraform & Ansible
|
|
|
|
Automated deployments are done using
|
|
[Terraform](https://www.terraform.io/) to create servers on Digital
|
|
Ocean then [Ansible](http://www.ansible.com/) to create and manage
|
|
testnets on those servers.
|
|
|
|
## Install
|
|
|
|
NOTE: see the [integration bash
|
|
script](https://github.com/tendermint/tendermint/blob/develop/networks/remote/integration.sh)
|
|
that can be run on a fresh DO droplet and will automatically spin up a 4
|
|
node testnet. The script more or less does everything described below.
|
|
|
|
- Install [Terraform](https://www.terraform.io/downloads.html) and
|
|
[Ansible](http://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html)
|
|
on a Linux machine.
|
|
- Create a [DigitalOcean API
|
|
token](https://cloud.digitalocean.com/settings/api/tokens) with read
|
|
and write capability.
|
|
- Install the python dopy package (`pip install dopy`)
|
|
- Create SSH keys (`ssh-keygen`)
|
|
- Set environment variables:
|
|
|
|
export DO_API_TOKEN="abcdef01234567890abcdef01234567890"
|
|
export SSH_KEY_FILE="$HOME/.ssh/id_rsa.pub"
|
|
|
|
These will be used by both `terraform` and `ansible`.
|
|
|
|
### Terraform
|
|
|
|
This step will create four Digital Ocean droplets. First, go to the
|
|
correct directory:
|
|
|
|
cd $GOPATH/src/github.com/tendermint/tendermint/networks/remote/terraform
|
|
|
|
then:
|
|
|
|
terraform init
|
|
terraform apply -var DO_API_TOKEN="$DO_API_TOKEN" -var SSH_KEY_FILE="$SSH_KEY_FILE"
|
|
|
|
and you will get a list of IP addresses that belong to your droplets.
|
|
|
|
With the droplets created and running, let's setup Ansible.
|
|
|
|
### Ansible
|
|
|
|
The playbooks in [the ansible
|
|
directory](https://github.com/tendermint/tendermint/tree/master/networks/remote/ansible)
|
|
run ansible roles to configure the sentry node architecture. You must
|
|
switch to this directory to run ansible
|
|
(`cd $GOPATH/src/github.com/tendermint/tendermint/networks/remote/ansible`).
|
|
|
|
There are several roles that are self-explanatory:
|
|
|
|
First, we configure our droplets by specifying the paths for tendermint
|
|
(`BINARY`) and the node files (`CONFIGDIR`). The latter expects any
|
|
number of directories named `node0, node1, ...` and so on (equal to the
|
|
number of droplets created). For this example, we use pre-created files
|
|
from [this
|
|
directory](https://github.com/tendermint/tendermint/tree/master/docs/examples).
|
|
To create your own files, use either the `tendermint testnet` command or
|
|
review [manual deployments](./deploy-testnets.md).
|
|
|
|
Here's the command to run:
|
|
|
|
ansible-playbook -i inventory/digital_ocean.py -l sentrynet config.yml -e BINARY=$GOPATH/src/github.com/tendermint/tendermint/build/tendermint -e CONFIGDIR=$GOPATH/src/github.com/tendermint/tendermint/docs/examples
|
|
|
|
Voila! All your droplets now have the `tendermint` binary and required
|
|
configuration files to run a testnet.
|
|
|
|
Next, we run the install role:
|
|
|
|
ansible-playbook -i inventory/digital_ocean.py -l sentrynet install.yml
|
|
|
|
which as you'll see below, executes
|
|
`tendermint node --proxy_app=kvstore` on all droplets. Although we'll
|
|
soon be modifying this role and running it again, this first execution
|
|
allows us to get each `node_info.id` that corresponds to each
|
|
`node_info.listen_addr`. (This part will be automated in the future). In
|
|
your browser (or using `curl`), for every droplet, go to IP:46657/status
|
|
and note the two just mentioned `node_info` fields. Notice that blocks
|
|
aren't being created (`latest_block_height` should be zero and not
|
|
increasing).
|
|
|
|
Next, open `roles/install/templates/systemd.service.j2` and look for the
|
|
line `ExecStart` which should look something like:
|
|
|
|
ExecStart=/usr/bin/tendermint node --proxy_app=kvstore
|
|
|
|
and add the `--p2p.persistent_peers` flag with the relevant information
|
|
for each node. The resulting file should look something like:
|
|
|
|
[Unit]
|
|
Description={{service}}
|
|
Requires=network-online.target
|
|
After=network-online.target
|
|
|
|
[Service]
|
|
Restart=on-failure
|
|
User={{service}}
|
|
Group={{service}}
|
|
PermissionsStartOnly=true
|
|
ExecStart=/usr/bin/tendermint node --proxy_app=kvstore --p2p.persistent_peers=167b80242c300bf0ccfb3ced3dec60dc2a81776e@165.227.41.206:46656,3c7a5920811550c04bf7a0b2f1e02ab52317b5e6@165.227.43.146:46656,303a1a4312c30525c99ba66522dd81cca56a361a@159.89.115.32:46656,b686c2a7f4b1b46dca96af3a0f31a6a7beae0be4@159.89.119.125:46656
|
|
ExecReload=/bin/kill -HUP $MAINPID
|
|
KillSignal=SIGTERM
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
|
|
Then, stop the nodes:
|
|
|
|
ansible-playbook -i inventory/digital_ocean.py -l sentrynet stop.yml
|
|
|
|
Finally, we run the install role again:
|
|
|
|
ansible-playbook -i inventory/digital_ocean.py -l sentrynet install.yml
|
|
|
|
to re-run `tendermint node` with the new flag, on all droplets. The
|
|
`latest_block_hash` should now be changing and `latest_block_height`
|
|
increasing. Your testnet is now up and running :)
|
|
|
|
Peek at the logs with the status role:
|
|
|
|
ansible-playbook -i inventory/digital_ocean.py -l sentrynet status.yml
|
|
|
|
### Logging
|
|
|
|
The crudest way is the status role described above. You can also ship
|
|
logs to Logz.io, an Elastic stack (Elastic search, Logstash and Kibana)
|
|
service provider. You can set up your nodes to log there automatically.
|
|
Create an account and get your API key from the notes on [this
|
|
page](https://app.logz.io/#/dashboard/data-sources/Filebeat), then:
|
|
|
|
yum install systemd-devel || echo "This will only work on RHEL-based systems."
|
|
apt-get install libsystemd-dev || echo "This will only work on Debian-based systems."
|
|
|
|
go get github.com/mheese/journalbeat
|
|
ansible-playbook -i inventory/digital_ocean.py -l sentrynet logzio.yml -e LOGZIO_TOKEN=ABCDEFGHIJKLMNOPQRSTUVWXYZ012345
|
|
|
|
### Cleanup
|
|
|
|
To remove your droplets, run:
|
|
|
|
terraform destroy -var DO_API_TOKEN="$DO_API_TOKEN" -var SSH_KEY_FILE="$SSH_KEY_FILE"
|