diff --git a/.gitignore b/.gitignore index 373948e7..4fb44601 100644 --- a/.gitignore +++ b/.gitignore @@ -28,3 +28,4 @@ fast/stages/**/terraform-*.auto.tfvars.json fast/stages/**/0*.auto.tfvars* **/node_modules fast/stages/**/globals.auto.tfvars.json +cloud_sql_proxy \ No newline at end of file diff --git a/examples/README.md b/examples/README.md index ab8a8c79..da15a85c 100644 --- a/examples/README.md +++ b/examples/README.md @@ -5,7 +5,7 @@ This section contains **[foundational examples](./foundations/)** that bootstrap Currently available examples: - **cloud operations** - [Resource tracking and remediation via Cloud Asset feeds](./cloud-operations/asset-inventory-feed-remediation), [Granular Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam), [Granular Cloud DNS IAM for Shared VPC](./cloud-operations/dns-shared-vpc), [Compute Engine quota monitoring](./cloud-operations/quota-monitoring), [Scheduled Cloud Asset Inventory Export to Bigquery](./cloud-operations/scheduled-asset-inventory-export-bq), [Packer image builder](./cloud-operations/packer-image-builder), [On-prem SA key management](./cloud-operations/onprem-sa-key-management), [TCP healthcheck for unmanaged GCE instances](./cloud-operations/unmanaged-instances-healthcheck) -- **data solutions** - [GCE/GCS CMEK via centralized Cloud KMS](./data-solutions/gcs-to-bq-with-least-privileges/), [Cloud Storage to Bigquery with Cloud Dataflow with least privileges](./data-solutions/gcs-to-bq-with-least-privileges/), [Data Platform Foundations](./data-solutions/data-platform-foundations/), [SQL Server AlwaysOn availability groups example](./data-solutions/sqlserver-alwayson) +- **data solutions** - [GCE/GCS CMEK via centralized Cloud KMS](./data-solutions/gcs-to-bq-with-least-privileges/), [Cloud Storage to Bigquery with Cloud Dataflow with least privileges](./data-solutions/gcs-to-bq-with-least-privileges/), [Data Platform Foundations](./data-solutions/data-platform-foundations/), [SQL Server AlwaysOn availability groups example](./data-solutions/sqlserver-alwayson), [Cloud SQL instance with multi-region read replicas](./data-solutions/cloudsql-multiregion/) - **factories** - [The why and the how of resource factories](./factories/README.md) - **foundations** - [single level hierarchy](./foundations/environments/) (environments), [multiple level hierarchy](./foundations/business-units/) (business units + environments) - **networking** - [hub and spoke via peering](./networking/hub-and-spoke-peering/), [hub and spoke via VPN](./networking/hub-and-spoke-vpn/), [DNS and Google Private Access for on-premises](./networking/onprem-google-access-dns/), [Shared VPC with GKE support](./networking/shared-vpc-gke/), [ILB as next hop](./networking/ilb-next-hop), [PSC for on-premises Cloud Function invocation](./networking/private-cloud-function-from-onprem/), [decentralized firewall](./networking/decentralized-firewall) diff --git a/examples/data-solutions/README.md b/examples/data-solutions/README.md index fad34ff0..ec2cfd08 100644 --- a/examples/data-solutions/README.md +++ b/examples/data-solutions/README.md @@ -24,6 +24,12 @@ This [example](./data-platform-foundations/) implements a robust and flexible Da ### SQL Server Always On Availability Groups - + This [example](./data-platform-foundations/) implements SQL Server Always On Availability Groups using Fabric modules. It builds a two node cluster with a fileshare witness instance in an existing VPC and adds the necessary firewalling. The actual setup process (apart from Active Directory operations) has been scripted, so that least amount of manual works needs to performed.
+ +### Cloud SQL instance with multi-region read replicas + + +This [example](./cloudsql-multiregion/) creates a [Cloud SQL instance](https://cloud.google.com/sql) with multi-region read replicas as described in the [Cloud SQL for PostgreSQL disaster recovery](https://cloud.google.com/architecture/cloud-sql-postgres-disaster-recovery-complete-failover-fallback) article. +
\ No newline at end of file diff --git a/examples/data-solutions/cloudsql-multiregion/README.md b/examples/data-solutions/cloudsql-multiregion/README.md new file mode 100644 index 00000000..ca1598b6 --- /dev/null +++ b/examples/data-solutions/cloudsql-multiregion/README.md @@ -0,0 +1,88 @@ +# Cloud SQL instance with multi-region read replicas + +This example creates a [Cloud SQL instance](https://cloud.google.com/sql) with multi-region read replicas as described in the [Cloud SQL for PostgreSQL disaster recovery](https://cloud.google.com/architecture/cloud-sql-postgres-disaster-recovery-complete-failover-fallback) article. + +The solution is resilient to a regional outage. To get familiar with the procedure needed in the unfortunate case of a disaster recovery, please follow steps described in [part two](https://cloud.google.com/architecture/cloud-sql-postgres-disaster-recovery-complete-failover-fallback#phase-2) of the aforementioned article. + +The solution will use: +- VPC with Private Service Access to deploy the instances and VM +- Cloud SQL - Postgre SQL instanced with Private IP +- Goocle Cloud Storage bucket to handle database import/export +- Google Cloud Engine instance to connect to the Posgre SQL instance +- Google Cloud NAT to access internet resources + +This is the high level diagram: + +![Cloud SQL multi-region.](diagram.png "Cloud SQL multi-region") + +# Requirements + +This example will deploy all its resources into the project defined by the `project_id` variable. Please note that we assume this project already exists. However, if you provide the appropriate values to the `project_create` variable, the project will be created as part of the deployment. + +If `project_create` is left to `null`, the identity performing the deployment needs the `owner` role on the project defined by the `project_id` variable. Otherwise, the identity performing the deployment needs `resourcemanager.projectCreator` on the resource hierarchy node specified by `project_create.parent` and `billing.user` on the billing account specified by `project_create.billing_account_id`. + + +## Deployment + +Configure the Terraform variables in your `terraform.tfvars` file. You need to specify at least the `project_id` and `prefix` variables. See [`terraform.tfvars.sample`](terraform.tfvars.sample) as starting point. + +Run Terraform init: + +``` +$ terraform init +$ terraform apply +``` + +You should see the output of the Terraform script with resources created and some commands that you'll need in the following steps below. + +## Move to real use case consideration + +This implementation is intentionally minimal and easy to read. A real world use case should consider: + - Using a Shared VPC + - Using VPC-SC to mitigate data exfiltration + +## Test your environment +We assume all those steps are run using a user listed on `data_eng_principals`. You can authenticate as the user using the following command: + +``` +$ gcloud init +$ gcloud auth application-default login +``` + +Below you can find commands to connect to the VM instance and Cloud SQL instance. + +``` + $ gcloud compute ssh sql-test --project PROJECT_ID --zone ZONE + sql-test:~$ cloud_sql_proxy -instances=CLOUDSQL_INSTANCE=tcp:5432 + sql-test:~$ psql 'host=127.0.0.1 port=5432 sslmode=disable dbname=DATABASE user=USER' +``` + +You can find computed commands on the Terraform `demo_commands` output. + + +## Variables + +| name | description | type | required | default | +|---|---|:---:|:---:|:---:| +| [postgres_user_password](variables.tf#L29) | `postgres` user password. | string | ✓ | | +| [prefix](variables.tf#L40) | Unique prefix used for resource names. Not used for project if 'project_create' is null. | string | ✓ | | +| [project_id](variables.tf#L54) | Project id, references existing project if `project_create` is null. | string | ✓ | | +| [cmek_encryption](variables.tf#L17) | Flag to enable CMEK on GCP resources created. | bool | | false | +| [data_eng_principals](variables.tf#L23) | Groups with Service Account Token creator role on service accounts in IAM format, only user supported on CloudSQL, eg 'user@domain.com'. | list(string) | | [] | +| [postgres_database](variables.tf#L34) | `postgres` database. | string | | "guestbook" | +| [project_create](variables.tf#L45) | Provide values if project creation is needed, uses existing project if null. Parent is in 'folders/nnn' or 'organizations/nnn' format. | object({…}) | | null | +| [regions](variables.tf#L59) | Map of instance_name => location where instances will be deployed. | map(string) | | {…} | +| [sql_configuration](variables.tf#L73) | Cloud SQL configuration | object({…}) | | {…} | + +## Outputs + +| name | description | sensitive | +|---|---|:---:| +| [bucket](outputs.tf#L22) | Cloud storage bucket to import/export data from Cloud SQL. | | +| [connection_names](outputs.tf#L17) | Connection name of each instance. | | +| [demo_commands](outputs.tf#L37) | Demo commands. | | +| [ips](outputs.tf#L27) | IP address of each instance. | | +| [project_id](outputs.tf#L32) | ID of the project containing all the instances. | | +| [service_accounts](outputs.tf#L46) | Service Accounts. | | + + diff --git a/examples/data-solutions/cloudsql-multiregion/backend.tf.sample b/examples/data-solutions/cloudsql-multiregion/backend.tf.sample new file mode 100644 index 00000000..49a0883d --- /dev/null +++ b/examples/data-solutions/cloudsql-multiregion/backend.tf.sample @@ -0,0 +1,30 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The `impersonate_service_account` option require the identity launching terraform +# role `roles/iam.serviceAccountTokenCreator` on the Service Account specified. + +terraform { + backend "gcs" { + bucket = "BUCKET_NAME" + prefix = "PREFIX" + impersonate_service_account = "SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com" + } +} +provider "google" { + impersonate_service_account = "SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com" +} +provider "google-beta" { + impersonate_service_account = "SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com" +} \ No newline at end of file diff --git a/examples/data-solutions/cloudsql-multiregion/cloudsql.tf b/examples/data-solutions/cloudsql-multiregion/cloudsql.tf new file mode 100644 index 00000000..01c30565 --- /dev/null +++ b/examples/data-solutions/cloudsql-multiregion/cloudsql.tf @@ -0,0 +1,62 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +module "db" { + source = "../../../modules/cloudsql-instance" + project_id = module.project.project_id + availability_type = var.sql_configuration.availability_type + encryption_key_name = var.cmek_encryption ? module.kms[var.regions.primary].keys.key.id : null + network = module.vpc.self_link + name = "${var.prefix}-db" + region = var.regions.primary + database_version = var.sql_configuration.database_version + tier = var.sql_configuration.tier + flags = { + "cloudsql.iam_authentication" = "on" + } + replicas = { + for k, v in var.regions : + k => { + region = v, + encryption_key_name = var.cmek_encryption ? module.kms[v].keys.key.id : null + } if k != "primary" + } + databases = [var.postgres_database] + users = { + postgres = var.postgres_user_password + } +} + +resource "google_sql_user" "users" { + for_each = toset(var.data_eng_principals) + project = module.project.project_id + name = each.value + instance = module.db.name + type = "CLOUD_IAM_USER" +} + +resource "google_sql_user" "service-account" { + for_each = toset(var.data_eng_principals) + project = module.project.project_id + # Omit the .gserviceaccount.com suffix in the email + name = regex("(.+)(gserviceaccount)", module.service-account-sql.email)[0] + instance = module.db.name + type = "CLOUD_IAM_SERVICE_ACCOUNT" +} + +module "service-account-sql" { + source = "../../../modules/iam-service-account" + project_id = module.project.project_id + name = "${var.prefix}-sql" +} diff --git a/examples/data-solutions/cloudsql-multiregion/datastorage.tf b/examples/data-solutions/cloudsql-multiregion/datastorage.tf new file mode 100644 index 00000000..1b75deb0 --- /dev/null +++ b/examples/data-solutions/cloudsql-multiregion/datastorage.tf @@ -0,0 +1,30 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +module "gcs" { + source = "../../../modules/gcs" + project_id = module.project.project_id + prefix = var.prefix + name = "data" + location = var.regions.primary + storage_class = "REGIONAL" + encryption_key = var.cmek_encryption ? module.kms[var.regions.primary].keys["key"].id : null + force_destroy = true +} + +module "service-account-gcs" { + source = "../../../modules/iam-service-account" + project_id = module.project.project_id + name = "${var.prefix}-gcs" +} diff --git a/examples/data-solutions/cloudsql-multiregion/diagram.png b/examples/data-solutions/cloudsql-multiregion/diagram.png new file mode 100644 index 00000000..35c8ff8e Binary files /dev/null and b/examples/data-solutions/cloudsql-multiregion/diagram.png differ diff --git a/examples/data-solutions/cloudsql-multiregion/gce.tf b/examples/data-solutions/cloudsql-multiregion/gce.tf new file mode 100644 index 00000000..435d0495 --- /dev/null +++ b/examples/data-solutions/cloudsql-multiregion/gce.tf @@ -0,0 +1,60 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +locals { + startup-script = < 0, false) + // Enable backup if the user asks for it or if the user is deploying + // MySQL with replicas + enable_backup = var.backup_configuration.enabled || (local.is_mysql && local.has_replicas) + users = { for user, password in coalesce(var.users, {}) : (user) => ( @@ -67,24 +71,25 @@ resource "google_sql_database_instance" "primary" { } } - backup_configuration { - // Enable backup if the user asks for it or if the user is - // deploying MySQL with replicas - enabled = var.backup_configuration.enabled || (local.is_mysql && local.has_replicas) + dynamic "backup_configuration" { + for_each = local.enable_backup ? { 1 = 1 } : {} + content { + enabled = true - // enable binary log if the user asks for it or we have replicas, - // but only form MySQL - binary_log_enabled = ( - local.is_mysql - ? var.backup_configuration.binary_log_enabled || local.has_replicas - : null - ) - start_time = var.backup_configuration.start_time - location = var.backup_configuration.location - transaction_log_retention_days = var.backup_configuration.log_retention_days - backup_retention_settings { - retained_backups = var.backup_configuration.retention_count - retention_unit = "COUNT" + // enable binary log if the user asks for it or we have replicas, + // but only for MySQL + binary_log_enabled = ( + local.is_mysql + ? var.backup_configuration.binary_log_enabled || local.has_replicas + : null + ) + start_time = var.backup_configuration.start_time + location = var.backup_configuration.location + transaction_log_retention_days = var.backup_configuration.log_retention_days + backup_retention_settings { + retained_backups = var.backup_configuration.retention_count + retention_unit = "COUNT" + } } } diff --git a/tests/examples/data_solutions/cloudsql-multiregion/__init__.py b/tests/examples/data_solutions/cloudsql-multiregion/__init__.py new file mode 100644 index 00000000..6d6d1266 --- /dev/null +++ b/tests/examples/data_solutions/cloudsql-multiregion/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tests/examples/data_solutions/cloudsql-multiregion/fixture/main.tf b/tests/examples/data_solutions/cloudsql-multiregion/fixture/main.tf new file mode 100644 index 00000000..7c709550 --- /dev/null +++ b/tests/examples/data_solutions/cloudsql-multiregion/fixture/main.tf @@ -0,0 +1,27 @@ +/** + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +module "test" { + source = "../../../../../examples/data-solutions/cloudsql-multiregion/" + data_eng_principals = ["dataeng@example.com"] + postgres_user_password = "my-root-password" + project_id = "project" + project_create = { + billing_account_id = "123456-123456-123456" + parent = "folders/12345678" + } + prefix = "prefix" +} diff --git a/tests/examples/data_solutions/cloudsql-multiregion/test_plan.py b/tests/examples/data_solutions/cloudsql-multiregion/test_plan.py new file mode 100644 index 00000000..e97e22d3 --- /dev/null +++ b/tests/examples/data_solutions/cloudsql-multiregion/test_plan.py @@ -0,0 +1,19 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +def test_resources(e2e_plan_runner): + "Test that plan works and the numbers of resources is as expected." + modules, resources = e2e_plan_runner() + assert len(modules) == 11 + assert len(resources) == 53