260 lines
11 KiB
Markdown
260 lines
11 KiB
Markdown
# Usage
|
|
|
|
## Prerequisites
|
|
|
|
The bootstrap script included in this project expects the AWS CLI, jq, and Terraform to be installed and on the PATH.
|
|
|
|
On macOS, with Homebrew installed, just run: `brew install --with-default-names awscli gnu-sed jq terraform`
|
|
|
|
For other platforms, or if you don't have Homebrew installed, please see the following links:
|
|
|
|
- [jq](https://stedolan.github.io/jq/download/)
|
|
- [awscli](https://docs.aws.amazon.com/cli/latest/userguide/installing.html)
|
|
- [terraform](https://www.terraform.io/intro/getting-started/install.html)
|
|
|
|
You will also need the following information for the installer:
|
|
|
|
- A unique prefix to use for provisioned resources (5 alphanumeric chars or less)
|
|
- A password to use for the RDS database (at least 8 characters long)
|
|
- The name of a IAM key pair to use for EC2 instances, if you provide a name which
|
|
already exists it will be used, otherwise it will be generated for you.
|
|
|
|
## AWS
|
|
|
|
You will need to set up a new AWS account (or subaccount), and then either login
|
|
to that account using the AWS CLI (via `aws configure`) or create a user account
|
|
that you will use for provisioning, and login to that account. Set output format
|
|
to `json` for the AWS CLI. The account used
|
|
requires full access to all AWS services, as a wide variety of services are used,
|
|
a mostly complete list is as follows:
|
|
|
|
- VPCs and associated networking resources (subnets, routing tables, etc.)
|
|
- Security Groups
|
|
- EC2
|
|
- S3
|
|
- SSM
|
|
- DynamoDB
|
|
- Route53
|
|
- RDS
|
|
- ElastiCache
|
|
- CodeDeploy
|
|
|
|
Given the large number of services involved, and the unpredictability of which
|
|
specific API calls will be needed during provisioning, it is recommended that
|
|
you provide a user account with full access. You do not need to keep this user
|
|
around (or enabled) except during the initial provisioning, and any subsequent
|
|
runs to update the infrastructure. How you choose to handle this user is up to you.
|
|
|
|
## Usage
|
|
|
|
Once the prerequisites are out of the way, you are ready to spin up your new infrastructure!
|
|
|
|
From the root of the project:
|
|
|
|
```
|
|
$ bin/infra help
|
|
```
|
|
|
|
This will show you the tasks and options available to you with this script.
|
|
|
|
The infra script will request any information it needs to proceed, and then call Terraform to bootstrap the necessary infrastructure
|
|
for its own state management. This state management infra is needed to ensure that Terraforms state is stored in a centralized location,
|
|
so that multiple people can use Terraform on the same infra without stepping on each others toes. Terraform prevents this from happening by
|
|
holding locks (via DynamoDB) against the state data (stored in S3). Generating the S3 bucket and DynamoDB table has to be done using local state
|
|
the first time, but once provisioned, the local state is migrated to S3, and all further invocations of `terraform` will use the state stored in S3.
|
|
|
|
The infra created, at a high level, is as follows:
|
|
|
|
- An SSH keypair (or you can choose to use one which was already created), this is used with any EC2 hosts
|
|
- A VPC containing all of the resources provisioned
|
|
- A public subnet for the app servers, and a private subnet for the database (and Redis for now)
|
|
- An internet gateway to provide internet access for the VPC
|
|
- An ALB which exposes the app server HTTPS endpoints to the world
|
|
- A security group to lock down ingress to the app servers to 80/443 + SSH
|
|
- A security group to allow the ALB to talk to the app servers
|
|
- A security group to allow the app servers access to the database
|
|
- An internal DNS zone
|
|
- A DNS record for the database
|
|
- An autoscaling group and launch configuration for each chain
|
|
- A CodeDeploy application and deployment group targeting the corresponding autoscaling groups
|
|
|
|
Each configured chain will receive its own ASG (autoscaling group) and deployment group, when application updates
|
|
are pushed to CodeDeploy, all autoscaling groups will deploy the new version using a blue/green strategy. Currently,
|
|
there is only one EC2 host to run, and the ASG is configured to allow scaling up, but no triggers are set up to actually perform the
|
|
scaling yet. This is something that may come in the future.
|
|
|
|
**IMPORTANT**: This repository's `.gitignore` prevents the storage of several files generated during provisioning, but it is important
|
|
that you keep them around in your own fork, so that subsequent runs of the `infra` script are using the same configuration and state.
|
|
These files are `backend.tfvars`, `main.tfvars`, and the Terraform state directories. If you generated
|
|
a private key for EC2 (the default), then you will also have a `*.privkey** file in your project root, you need to store this securely out of
|
|
band once created, but does not need to be in the repository.
|
|
|
|
## Migration Prompt
|
|
|
|
The installer will prompt during its initial run to ask if you want to migrate
|
|
the Terraform state to S3, this is a necessary step, and is only prompted due to
|
|
a bug in the Terraform CLI, in a future release, this shouldn't occur, but in
|
|
the meantime, you will need to answer yes to this prompt.
|
|
|
|
## Configuring Installer
|
|
|
|
The `infra` script generates config files for storing the values provided for
|
|
future runs. You can provide overrides to this configuration in
|
|
`terraform.tfvars` or any file with the `.tfvars` extension.
|
|
|
|
An example `terraform.tfvars` configuration file looks like:
|
|
|
|
```
|
|
region = "us-east-1"
|
|
bucket = "poa-terraform-state"
|
|
dynamodb_table = "poa-terraform-lock"
|
|
key_name = "sokol-test"
|
|
prefix = "sokol"
|
|
db_password = "qwerty12345"
|
|
db_instance_class = "db.m4.xlarge"
|
|
db_storage = "120"
|
|
alb_ssl_policy = "ELBSecurityPolicy-2016-08"
|
|
alb_certificate_arn = "arn:aws:acm:us-east-1:290379793816:certificate/6d1bab74-fb46-4244-aab2-832bf519ab24"
|
|
root_block_size = 120
|
|
pool_size = 30
|
|
```
|
|
|
|
- The region should be left at `us-east-1` as some of the other regions fail for different reasons.
|
|
- The `bucket` and `dynamodb_table` can be edited but should have an identical prefix.
|
|
- The `key_name` should start with the `prefix` and can only contain 5 characters and must start with a letter.
|
|
- The `db_password` can be a changed to any alphanumeric value.
|
|
- The `db_instance_class` and `db_storage` are not required but are defaulted to `db.m4.large` and `100`GB respectively.
|
|
- If you don't plan to use SSL, set variable `use_ssl = "false"`
|
|
- The `alb_ssl_policy` and `alb_certificate_arn` are required in order to force SSL usage.
|
|
- The `root_block_size` is the amount of storage on your EC2 instance. This value can be adjusted by how frequently logs are rotated. Logs are located in `/opt/app/logs` of your EC2 instance.
|
|
- The `pool_size` defines the number of connections allowed by the RDS instance.
|
|
|
|
## Database Storage Required
|
|
|
|
The configuration variable `db_storage` can be used to define the amount of storage allocated to your RDS instance. The chart below shows an estimated amount of storage that is required to index individual chains. The `db_storage` can only be adjusted 1 time in a 24 hour period on AWS.
|
|
|
|
| Chain | Storage (GiB) |
|
|
| --- | --- |
|
|
| POA Core | 200 |
|
|
| POA Sokol | 400 |
|
|
| Ethereum Classic | 1000 |
|
|
| Ethereum Mainnet | 4000 |
|
|
| Kovan Testnet | 800 |
|
|
| Ropsten Testnet | 1500 |
|
|
|
|
## Defining Chains/Adding Chains
|
|
|
|
The default of this repo is to build infra for the `sokol` chain, but you may not want that, or want a different set, so you need to create/edit `terraform.tfvars` and add the following configuration:
|
|
|
|
```terraform
|
|
chains = {
|
|
"mychain" = "url/to/endpoint"
|
|
}
|
|
chain_trace_endpoint = {
|
|
"mychain" = "url/to/debug/endpoint/or/the/main/chain/endpoint"
|
|
}
|
|
chain_ws_endpoint = {
|
|
"mychain" = "url/to/websocket/endpoint"
|
|
}
|
|
chain_jsonrpc_variant = {
|
|
"mychain" = "parity"
|
|
}
|
|
chain_logo = {
|
|
"mychain" = "url/to/logo"
|
|
}
|
|
chain_coin = {
|
|
"mychain" = "coin"
|
|
}
|
|
chain_network = {
|
|
"mychain" = "network name"
|
|
}
|
|
chain_subnetwork = {
|
|
"mychain" = "subnetwork name"
|
|
}
|
|
chain_network_path = {
|
|
"mychain" = path/to/root"
|
|
}
|
|
chain_network_icon = {
|
|
"mychain" = "_test_network_icon.html"
|
|
}
|
|
chain_heart_beat_timeout = {
|
|
"mychain" = 30
|
|
}
|
|
chain_heart_command = {
|
|
"mychain" = "systemctl restart explorer.service"
|
|
}
|
|
chain_blockscout_version = {
|
|
"mychain" = "v1.3.0-beta"
|
|
}
|
|
```
|
|
|
|
This will ensure that those chains are used when provisioning the infrastructure.
|
|
|
|
## Configuration
|
|
|
|
Config is stored in the Systems Manager Parameter Store, each chain has its own set of config values. If you modify one of these values,
|
|
you will need to go and terminate the instances for that chain so that they are reprovisioned with the new configuration.
|
|
|
|
You will need to make sure to import the changes into the Terraform state though, or you run the risk of getting out of sync.
|
|
|
|
## Destroying Provisioned Infrastructure
|
|
|
|
You can use `bin/infra destroy` to remove any generated infrastructure. It is
|
|
important to note though that if you run this script on partially generated
|
|
infrastructure, or if an error occurs during the destroy process, that you may
|
|
need to manually check for, and remove, any resources that were not able to be
|
|
deleted for you. You can use the `bin/infra resources` command to list all ARNs
|
|
that are tagged with the unique prefix you supplied to the installer, but not
|
|
all AWS resources support tags, and so will not be listed. Here's a list of such
|
|
resources I am aware of:
|
|
|
|
- Route53 hosted zone and records
|
|
- ElastiCache/RDS subnet groups
|
|
- CodeDeploy applications
|
|
|
|
If the `destroy` command succeeds, then everything has been removed, and you do
|
|
not have to worry about leftover resources hanging around.
|
|
|
|
## Common Errors and Questions
|
|
|
|
### Error Applying Plan (diffs didn't match)
|
|
|
|
If you see something like the following:
|
|
|
|
```
|
|
Error: Error applying plan:
|
|
|
|
1 error(s) occurred:
|
|
|
|
* module.stack.aws_autoscaling_group.explorer: aws_autoscaling_group.explorer: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.
|
|
|
|
Please include the following information in your report:
|
|
|
|
Terraform Version: 0.11.7
|
|
Resource ID: aws_autoscaling_group.explorer
|
|
Mismatch reason: attribute mismatch: availability_zones.1252502072
|
|
```
|
|
|
|
This is due to a bug in Terraform, however the fix is to just rerun `bin/infra
|
|
provision` again, and Terraform will pick up where it left off. This does not
|
|
always happen, but this is the current workaround if you see it.
|
|
|
|
### Error inspecting states in the "s3" backend
|
|
|
|
If you see the following:
|
|
|
|
```
|
|
Error inspecting states in the "s3" backend:
|
|
NoSuchBucket: The specified bucket does not exist
|
|
status code: 404, request id: xxxxxxxx, host id: xxxxxxxx
|
|
|
|
Prior to changing backends, Terraform inspects the source and destination
|
|
states to determine what kind of migration steps need to be taken, if any.
|
|
Terraform failed to load the states. The data in both the source and the
|
|
destination remain unmodified. Please resolve the above error and try again.
|
|
```
|
|
|
|
This is due to mismatched variables in `terraform.tfvars` and `main.tfvars` files. Update the `terraform.tfvars` file to match the `main.tfvars` file. Delete the `.terraform` and `terraform.dfstate.d` folders, run `bin/infra destroy_setup`, and restart provision by running `bin/infra provision`.
|
|
|
|
|