# Cloud Composer version 2 private instance, supporting Shared VPC and external CMEK key
This blueprint creates a Private instance of [Cloud Composer version 2](https://cloud.google.com/composer/docs/composer-2/composer-versioning-overview) on a VPC with a dedicated service account. Cloud Composer 2 is the new major version for Cloud Composer that supports:
- environment autoscaling
- workloads configuration: CPU, memory, and storage parameters for Airflow workers, schedulers, web server, and database.
Please consult the [documentation page](https://cloud.google.com/composer/docs/composer-2/composer-versioning-overview) for an exhaustive comparison between Composer Version 1 and Version 2.
The solution will use:
- Cloud Composer
- VPC with Private Service Access to deploy resources, if no Shared VPC configuration provided.
- Google Cloud NAT to access internet resources, if no Shared VPC configuration provided.
The solution supports as inputs:
- Shared VPC
- Cloud KMS CMEK keys
This is the high level diagram:
## Requirements
This blueprint will deploy all its resources into the project defined by the project_id variable. Please note that we assume this project already exists. However, if you provide the appropriate values to the `project_create` variable, the project will be created as part of the deployment.
If `project_create` is left to null, the identity performing the deployment needs the owner role on the project defined by the `project_id` variable. Otherwise, the identity performing the deployment needs `resourcemanager.projectCreator` on the resource hierarchy node specified by `project_create.parent` and `billing.user` on the billing account specified by `project_create.billing_account_id`.
## Deployment
Run Terraform init:
```bash
terraform init
```
Configure the Terraform variable in your terraform.tfvars file. You need to specify at least the following variables:
```tfvars
project_id = "lcaggioni-sandbox"
prefix = "lc"
region = "europe-west1"
```
You can run now:
```bash
terraform apply
```
You can now connect to your instance.
## Customizations
### VPC
If a shared VPC is not configured, a VPC will be created within the project. The following IP ranges will be used:
- Cloudsql: `10.20.10.0/24`
- GKE: `10.20.11.0/28`
Change the code as needed to match your needed configuration, remember that these addresses should not overlap with any other range used in network.
### Shared VPC
As is often the case in real-world configurations, this blueprint accepts as input an existing [`Shared-VPC`](https://cloud.google.com/vpc/docs/shared-vpc) via the `network_config` variable.
Example:
```tfvars
network_config = {
host_project = "PROJECT"
network_self_link = "projects/PROJECT/global/networks/VPC_NAME"
region = "europe-west3"
subnet_self_link = "projects/PROJECT/regions/REGION/subnetworks/VPC_NAME"
composer_secondary_ranges = {
pods = "pods"
services = "services"
}
}
# tftest skip
```
Make sure that:
- The GKE API (`container.googleapis.com`) is enabled in the VPC host project.
- The subnet has secondary ranges configured with 2 ranges:
- pods: `/22` example: `10.10.8.0/22`
- services = `/24` example: 10.10.12.0/24`
- Firewall rules are set, as described in the [documentation](https://cloud.google.com/composer/docs/composer-2/configure-private-ip#step_3_configure_firewall_rules)
In order to run the example and deploy Cloud Composer on a shared VPC the identity running Terraform must have the following IAM role on the Shared VPC Host project.
- Compute Network Admin (roles/compute.networkAdmin)
- Compute Shared VPC Admin (roles/compute.xpnAdmin)
## Encryption
As is often the case in real-world configurations, this blueprint accepts as input an existing [`Cloud KMS keys`](https://cloud.google.com/kms/docs/cmek) via the `service_encryption_keys` variable.
Example:
```tfvars
service_encryption_keys = {
`europe/west1` = `projects/PROJECT/locations/REGION/keyRings/KR_NAME/cryptoKeys/KEY_NAME`
}
# tftest skip
```
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [prefix](variables.tf#L81) | Prefix used for resource names. | string | ✓ | |
| [project_id](variables.tf#L99) | Project id, references existing project if `project_create` is null. | string | ✓ | |
| [region](variables.tf#L104) | Region where instances will be deployed. | string | ✓ | |
| [composer_config](variables.tf#L17) | Composer environment configuration. It accepts only following attributes: `environment_size`, `software_config` and `workloads_config`. See [attribute reference](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/composer_environment#argument-reference---cloud-composer-2) for details on settings variables. | object({…}) | | {…} |
| [iam_bindings_additive](variables.tf#L56) | Map of Role => principal in IAM format (`group:foo@example.org`) to be added on the project. | map(list(string)) | | {} |
| [network_config](variables.tf#L63) | Shared VPC network configurations to use. If null networks will be created in projects with preconfigured values. | object({…}) | | null |
| [project_create](variables.tf#L90) | Provide values if project creation is needed, uses existing project if null. Parent is in 'folders/nnn' or 'organizations/nnn' format. | object({…}) | | null |
| [service_encryption_keys](variables.tf#L109) | Cloud KMS keys to use to encrypt resources. Provide a key for each region in use. | map(string) | | null |
## Outputs
| name | description | sensitive |
|---|---|:---:|
| [composer_airflow_uri](outputs.tf#L17) | The URI of the Apache Airflow Web UI hosted within the Cloud Composer environment.. | |
| [composer_dag_gcs](outputs.tf#L22) | The Cloud Storage prefix of the DAGs for the Cloud Composer environment. | |
| [composer_service_account](outputs.tf#L27) | Cloud Composer nodes Service Account email. | |
## Test
```hcl
module "test" {
source = "./fabric/blueprints/data-solutions/composer-2/"
project_id = "composer"
project_create = {
billing_account_id = var.billing_account_id
parent = var.folder_id
}
prefix = var.prefix
region = var.region
}
# tftest modules=5 resources=29 e2e
```