From 69dd371df31ff9e584252b44064ab4926732f1d3 Mon Sep 17 00:00:00 2001 From: Zach Date: Sat, 11 Aug 2018 17:10:54 -0400 Subject: [PATCH] networks: clean up & some fixes to prepare for docs (#1986) * networks: update ports * networks: organize some files, convert rst to md * networks: rm old readme * networks: more little fixes --- networks/README-DO.rst | 69 ---------------- networks/README.md | 65 ++++++++++++++++ networks/README.rst | 78 ------------------- networks/local/README.md | 4 +- networks/local/gaiadnode/Dockerfile | 2 +- .../terraform-do/Makefile} | 1 - networks/remote/terraform-do/README.md | 58 ++++++++++++++ networks/remote/terraform-do/README.rst | 53 ------------- 8 files changed, 126 insertions(+), 204 deletions(-) delete mode 100644 networks/README-DO.rst create mode 100644 networks/README.md delete mode 100644 networks/README.rst rename networks/{Makefile.do => remote/terraform-do/Makefile} (99%) create mode 100644 networks/remote/terraform-do/README.md delete mode 100644 networks/remote/terraform-do/README.rst diff --git a/networks/README-DO.rst b/networks/README-DO.rst deleted file mode 100644 index 9e9275bc1..000000000 --- a/networks/README-DO.rst +++ /dev/null @@ -1,69 +0,0 @@ -Terraform & Ansible -=================== - -WARNING: The Digital Ocean scripts are obsolete. They are here because they might still be useful for developers. - -Automated deployments are done using `Terraform `__ to create servers on Digital Ocean then -`Ansible `__ to create and manage testnets on those servers. - -Prerequisites -------------- - -- Install `Terraform `__ and `Ansible `__ on a Linux machine. -- Create a `DigitalOcean API token `__ with read and write capability. -- Install the python dopy package (``pip install dopy``) (This is necessary for the digitalocean.py script for ansible.) -- Create SSH keys - -:: - - export DO_API_TOKEN="abcdef01234567890abcdef01234567890" - export TESTNET_NAME="remotenet" - export SSH_PRIVATE_FILE="$HOME/.ssh/id_rsa" - export SSH_PUBLIC_FILE="$HOME/.ssh/id_rsa.pub" - -These will be used by both ``terraform`` and ``ansible``. - -Create a remote network ------------------------ - -:: - - make remotenet-start - - -Optionally, you can set the number of servers you want to launch and the name of the testnet (which defaults to remotenet): - -:: - - TESTNET_NAME="mytestnet" SERVERS=7 make remotenet-start - - -Quickly see the /status endpoint --------------------------------- - -:: - - make remotenet-status - - -Delete servers --------------- - -:: - - make remotenet-stop - -Logging -------- - -You can ship logs to Logz.io, an Elastic stack (Elastic search, Logstash and Kibana) service provider. You can set up your nodes to log there automatically. Create an account and get your API key from the notes on `this page `__, then: - -:: - - yum install systemd-devel || echo "This will only work on RHEL-based systems." - apt-get install libsystemd-dev || echo "This will only work on Debian-based systems." - - go get github.com/mheese/journalbeat - ansible-playbook -i inventory/digital_ocean.py -l remotenet logzio.yml -e LOGZIO_TOKEN=ABCDEFGHIJKLMNOPQRSTUVWXYZ012345 - - diff --git a/networks/README.md b/networks/README.md new file mode 100644 index 000000000..322d37340 --- /dev/null +++ b/networks/README.md @@ -0,0 +1,65 @@ +# Terraform & Ansible + +Automated deployments are done using [Terraform](https://www.terraform.io/) to create servers on AWS then +[Ansible](http://www.ansible.com/) to create and manage testnets on those servers. + +## Prerequisites + +- Install [Terraform](https://www.terraform.io/downloads.html) and [Ansible](http://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) on a Linux machine. +- Create an [AWS API token](https://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html) with EC2 create capability. +- Create SSH keys + +``` +export AWS_ACCESS_KEY_ID="2345234jk2lh4234" +export AWS_SECRET_ACCESS_KEY="234jhkg234h52kh4g5khg34" +export TESTNET_NAME="remotenet" +export CLUSTER_NAME= "remotenetvalidators" +export SSH_PRIVATE_FILE="$HOME/.ssh/id_rsa" +export SSH_PUBLIC_FILE="$HOME/.ssh/id_rsa.pub" +``` + +These will be used by both `terraform` and `ansible`. + +## Create a remote network + +``` +SERVERS=1 REGION_LIMIT=1 make validators-start +``` + +The testnet name is what's going to be used in --chain-id, while the cluster name is the administrative tag in AWS for the servers. The code will create SERVERS amount of servers in each availability zone up to the number of REGION_LIMITs, starting at us-east-2. (us-east-1 is excluded.) The below BaSH script does the same, but sometimes it's more comfortable for input. + +``` +./new-testnet.sh "$TESTNET_NAME" "$CLUSTER_NAME" 1 1 +``` + +## Quickly see the /status endpoint + +``` +make validators-status +``` + +## Delete servers + +``` +make validators-stop +``` + +## Logging + +You can ship logs to Logz.io, an Elastic stack (Elastic search, Logstash and Kibana) service provider. You can set up your nodes to log there automatically. Create an account and get your API key from the notes on [this page](https://app.logz.io/#/dashboard/data-sources/Filebeat), then: + +``` +yum install systemd-devel || echo "This will only work on RHEL-based systems." +apt-get install libsystemd-dev || echo "This will only work on Debian-based systems." + +go get github.com/mheese/journalbeat +ansible-playbook -i inventory/digital_ocean.py -l remotenet logzio.yml -e LOGZIO_TOKEN=ABCDEFGHIJKLMNOPQRSTUVWXYZ012345 +``` + +## Monitoring + +You can install the DataDog agent with: + +``` +make datadog-install +``` diff --git a/networks/README.rst b/networks/README.rst deleted file mode 100644 index 1e2fd8724..000000000 --- a/networks/README.rst +++ /dev/null @@ -1,78 +0,0 @@ -Terraform & Ansible -=================== - -Automated deployments are done using `Terraform `__ to create servers on AWS then -`Ansible `__ to create and manage testnets on those servers. - -Prerequisites -------------- - -- Install `Terraform `__ and `Ansible `__ on a Linux machine. -- Create an `AWS API token `__ with EC2 create capability. -- Create SSH keys - -:: - - export AWS_ACCESS_KEY_ID="2345234jk2lh4234" - export AWS_SECRET_ACCESS_KEY="234jhkg234h52kh4g5khg34" - export TESTNET_NAME="remotenet" - export CLUSTER_NAME= "remotenetvalidators" - export SSH_PRIVATE_FILE="$HOME/.ssh/id_rsa" - export SSH_PUBLIC_FILE="$HOME/.ssh/id_rsa.pub" - -These will be used by both ``terraform`` and ``ansible``. - -Create a remote network ------------------------ - -:: - - SERVERS=1 REGION_LIMIT=1 make validators-start - - -The testnet name is what's going to be used in --chain-id, while the cluster name is the administrative tag in AWS for the servers. The code will create SERVERS amount of servers in each availability zone up to the number of REGION_LIMITs, starting at us-east-2. (us-east-1 is excluded.) The below BaSH script does the same, but sometimes it's more comfortable for input. - -:: - - ./new-testnet.sh "$TESTNET_NAME" "$CLUSTER_NAME" 1 1 - - -Quickly see the /status endpoint --------------------------------- - -:: - - make validators-status - - -Delete servers --------------- - -:: - - make validators-stop - -Logging -------- - -You can ship logs to Logz.io, an Elastic stack (Elastic search, Logstash and Kibana) service provider. You can set up your nodes to log there automatically. Create an account and get your API key from the notes on `this page `__, then: - -:: - - yum install systemd-devel || echo "This will only work on RHEL-based systems." - apt-get install libsystemd-dev || echo "This will only work on Debian-based systems." - - go get github.com/mheese/journalbeat - ansible-playbook -i inventory/digital_ocean.py -l remotenet logzio.yml -e LOGZIO_TOKEN=ABCDEFGHIJKLMNOPQRSTUVWXYZ012345 - - -Monitoring ----------- - -You can install DataDog agent using - -:: - - make datadog-install - - diff --git a/networks/local/README.md b/networks/local/README.md index d38e837a2..3a0f855be 100644 --- a/networks/local/README.md +++ b/networks/local/README.md @@ -31,9 +31,9 @@ To start a 4 node testnet run: make localnet-start ``` -The nodes bind their RPC servers to ports 46657, 46660, 46662, and 46664 on the host. +The nodes bind their RPC servers to ports 26657, 26660, 26662, and 26664 on the host. This file creates a 4-node network using the gaiadnode image. -The nodes of the network expose their P2P and RPC endpoints to the host machine on ports 46656-46657, 46659-46660, 46661-46662, and 46663-46664 respectively. +The nodes of the network expose their P2P and RPC endpoints to the host machine on ports 26656-26657, 26659-26660, 26661-26662, and 26663-26664 respectively. To update the binary, just rebuild it and restart the nodes: diff --git a/networks/local/gaiadnode/Dockerfile b/networks/local/gaiadnode/Dockerfile index fc2c0d4a0..d82036a46 100644 --- a/networks/local/gaiadnode/Dockerfile +++ b/networks/local/gaiadnode/Dockerfile @@ -7,7 +7,7 @@ RUN apk update && \ VOLUME [ /gaiad ] WORKDIR /gaiad -EXPOSE 46656 46657 +EXPOSE 26656 26657 ENTRYPOINT ["/usr/bin/wrapper.sh"] CMD ["start"] STOPSIGNAL SIGTERM diff --git a/networks/Makefile.do b/networks/remote/terraform-do/Makefile similarity index 99% rename from networks/Makefile.do rename to networks/remote/terraform-do/Makefile index a626bad8b..76040e208 100644 --- a/networks/Makefile.do +++ b/networks/remote/terraform-do/Makefile @@ -1,7 +1,6 @@ ######################################## ### WARNING: The DigitalOcean scripts are deprecated. They are still here because ### they might be useful for developers. -### Use -f to call this Makefile: "make -f Makefile.do target" # Name of the testnet. Used in chain-id. TESTNET_NAME?=remotenet diff --git a/networks/remote/terraform-do/README.md b/networks/remote/terraform-do/README.md new file mode 100644 index 000000000..0486a8bc4 --- /dev/null +++ b/networks/remote/terraform-do/README.md @@ -0,0 +1,58 @@ +# Terraform & Ansible + +WARNING: The Digital Ocean scripts are obsolete. They are here because they might still be useful for developers. + +Automated deployments are done using [Terraform](https://www.terraform.io/) to create servers on Digital Ocean then +[Ansible](http://www.ansible.com/) to create and manage testnets on those servers. + +## Prerequisites + +- Install [Terraform](https://www.terraform.io/downloads.html) and [Ansible](http://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) on a Linux machine. +- Create a [DigitalOcean API token](https://cloud.digitalocean.com/settings/api/tokens) with read and write capability. +- Install the python dopy package (`pip install dopy`) (This is necessary for the digitalocean.py script for ansible.) +- Create SSH keys + +``` +export DO_API_TOKEN="abcdef01234567890abcdef01234567890" +export TESTNET_NAME="remotenet" +export SSH_PRIVATE_FILE="$HOME/.ssh/id_rsa" +export SSH_PUBLIC_FILE="$HOME/.ssh/id_rsa.pub" +``` + +These will be used by both `terraform` and `ansible`. + +## Create a remote network + +``` +make remotenet-start +``` + +Optionally, you can set the number of servers you want to launch and the name of the testnet (which defaults to remotenet): + +``` +TESTNET_NAME="mytestnet" SERVERS=7 make remotenet-start +``` + +## Quickly see the /status endpoint + +``` +make remotenet-status +``` + +## Delete servers + +``` +make remotenet-stop +``` + +## Logging + +You can ship logs to Logz.io, an Elastic stack (Elastic search, Logstash and Kibana) service provider. You can set up your nodes to log there automatically. Create an account and get your API key from the notes on [this page](https://app.logz.io/#/dashboard/data-sources/Filebeat), then: + +``` +yum install systemd-devel || echo "This will only work on RHEL-based systems." +apt-get install libsystemd-dev || echo "This will only work on Debian-based systems." + +go get github.com/mheese/journalbeat +ansible-playbook -i inventory/digital_ocean.py -l remotenet logzio.yml -e LOGZIO_TOKEN=ABCDEFGHIJKLMNOPQRSTUVWXYZ012345 +``` diff --git a/networks/remote/terraform-do/README.rst b/networks/remote/terraform-do/README.rst deleted file mode 100644 index fdccb1a6a..000000000 --- a/networks/remote/terraform-do/README.rst +++ /dev/null @@ -1,53 +0,0 @@ -Using Terraform -=============== - -This is a `Terraform `__ configuration that sets up DigitalOcean droplets. - -Prerequisites -------------- - -- Install `HashiCorp Terraform `__ on a linux machine. -- Create a `DigitalOcean API token `__ with read and write capability. -- Create SSH keys - -Build ------ - -:: - - export DO_API_TOKEN="abcdef01234567890abcdef01234567890" - export TESTNET_NAME="remotenet" - export SSH_PUBLIC_FILE="$HOME/.ssh/id_rsa.pub" - export SSH_PRIVATE_FILE="$HOME/.ssh/id_rsa" - - terraform init - terraform apply -var DO_API_TOKEN="$DO_API_TOKEN" -var SSH_PUBLIC_FILE="$SSH_PUBLIC_FILE" -var SSH_PRIVATE_FILE="$SSH_PRIVATE_FILE" - -At the end you will get a list of IP addresses that belongs to your new droplets. - -Destroy -------- - -Run the below: - -:: - - terraform destroy -var DO_API_TOKEN="$DO_API_TOKEN" -var SSH_PUBLIC_FILE="$SSH_PUBLIC_FILE" -var SSH_PRIVATE_FILE="$SSH_PRIVATE_FILE" - -Good to know ------------- - -The DigitalOcean API was not very reliable for me. If you find that terraform fails to install a specific server (for example cluster[2]), check -the regions variable and remove data center names that you find unreliable. The variable is at cluster/variables.tf - -Example: - -:: - - variable "regions" { - description = "Regions to launch in" - type = "list" - default = ["TOR1", "LON1"] - } - -