cloud-foundation-fabric/blueprints/data-solutions/composer-2

Cloud Composer version 2 private instance, supporting Shared VPC and external CMEK key

This blueprint creates a Private instance of Cloud Composer version 2 on a VPC with a dedicated service account. Cloud Composer 2 is the new major verion for Cloud Composer that supports:

• environment autoscaling
• workloads configuration: CPU, memory, and storage parameters for Airflow workers, schedulers, web server, and database.

Please consult the documentation page for an exaustive comparison between Composer Version 1 and Version 2.

The solution will use:

• Cloud Composer
• VPC with Private Service Access to deploy resources, if no Shared VPC configuration provided.
• Google Cloud NAT to access internet resources, if no Shared VPC configuration provided.

The solution supports as inputs:

• Shared VPC
• Cloud KMS CMEK keys

This is the high level diagram:

Requirements

This blueprint will deploy all its resources into the project defined by the project_id variable. Please note that we assume this project already exists. However, if you provide the appropriate values to the project_create variable, the project will be created as part of the deployment.

If project_create is left to null, the identity performing the deployment needs the owner role on the project defined by the project_id variable. Otherwise, the identity performing the deployment needs resourcemanager.projectCreator on the resource hierarchy node specified by project_create.parent and billing.user on the billing account specified by project_create.billing_account_id.

Deployment

Run Terraform init:

$ terraform init

Configure the Terraform variable in your terraform.tfvars file. You need to spefify at least the following variables:

project_id          = "lcaggioni-sandbox"
prefix              = "lc"

You can run now:

$ terraform apply

You can now connect to your instance.

Customizations

Shared VPC

As is often the case in real-world configurations, this blueprint accepts as input an existing Shared-VPC via the network_config variable.

Example:

network_config = {
  host_project      = "PROJECT"
  network_self_link = "projects/PROJECT/global/networks/VPC_NAME"
  subnet_self_link  = "projects/PROJECT/regions/REGION/subnetworks/VPC_NAME"
  composer_secondary_ranges = {
    pods     = "pods"
    services = "services"
  }
}

Make sure that:

• The GKE API (container.googleapis.com) is enabled in the VPC host project.
• The subnet has secondary ranges configured with 2 ranges:
  • pods: /22 example: 10.10.8.0/22
  • services = /24 example: 10.10.12.0/24`
• Firewall rules are set, as described in the documentation

In order to run the example and deploy Cloud Composer on a shared VPC the identity running Terraform must have the following IAM role on the Shared VPC Host project.

• Compute Network Admin (roles/compute.networkAdmin)
• Compute Shared VPC Admin (roles/compute.xpnAdmin)

Encryption

As is often the case in real-world configurations, this blueprint accepts as input an existing Cloud KMS keys via the service_encryption_keys variable.

Example:

service_encryption_keys = {
  `europe/west1` = `projects/PROJECT/locations/REGION/keyRings/KR_NAME/cryptoKeys/KEY_NAME`
}

Variables

name	description	type	required	default
prefix	Unique prefix used for resource names. Not used for project if 'project_create' is null.	string	✓
project_id	Project id, references existing project if project_create is null.	string	✓
composer_config	Composer environemnt configuration. See attribute reference for details on settings variables.	object({…})		{…}
iam_groups_map	Map of Role => groups to be added on the project. Example: { "roles/composer.admin" = ["group:gcp-data-engineers@example.com"]}.	map(list(string))		null
network_config	Shared VPC network configurations to use. If null networks will be created in projects with preconfigured values.	object({…})		null
project_create	Provide values if project creation is needed, uses existing project if null. Parent is in 'folders/nnn' or 'organizations/nnn' format.	object({…})		null
region	Region where instances will be deployed.	string		"europe-west1"
service_encryption_keys	Cloud KMS keys to use to encrypt resources. Provide a key for each reagion in use.	map(string)		null

Outputs

name	description	sensitive
composer_airflow_uri	The URI of the Apache Airflow Web UI hosted within the Cloud Composer environment..
composer_dag_gcs	The Cloud Storage prefix of the DAGs for the Cloud Composer environment.