Merge pull request #801 from GoogleCloudPlatform/lcaggio/cloudsql-shared
Update Cloud SQL example
This commit is contained in:
commit
33d05ae952
|
@ -30,7 +30,7 @@ This [example](./data-platform-foundations/) implements SQL Server Always On Ava
|
||||||
|
|
||||||
### Cloud SQL instance with multi-region read replicas
|
### Cloud SQL instance with multi-region read replicas
|
||||||
|
|
||||||
<a href="./cloudsql-multiregion/" title="Cloud SQL instance with multi-region read replicas"><img src="./cloudsql-multiregion/diagram.png" align="left" width="280px"></a>
|
<a href="./cloudsql-multiregion/" title="Cloud SQL instance with multi-region read replicas"><img src="./cloudsql-multiregion/images/diagram.png" align="left" width="280px"></a>
|
||||||
This [example](./cloudsql-multiregion/) creates a [Cloud SQL instance](https://cloud.google.com/sql) with multi-region read replicas as described in the [Cloud SQL for PostgreSQL disaster recovery](https://cloud.google.com/architecture/cloud-sql-postgres-disaster-recovery-complete-failover-fallback) article.
|
This [example](./cloudsql-multiregion/) creates a [Cloud SQL instance](https://cloud.google.com/sql) with multi-region read replicas as described in the [Cloud SQL for PostgreSQL disaster recovery](https://cloud.google.com/architecture/cloud-sql-postgres-disaster-recovery-complete-failover-fallback) article.
|
||||||
<br clear="left">
|
<br clear="left">
|
||||||
|
|
||||||
|
|
|
@ -80,6 +80,27 @@ This implementation is intentionally minimal and easy to read. A real world use
|
||||||
- Using a Shared VPC
|
- Using a Shared VPC
|
||||||
- Using VPC-SC to mitigate data exfiltration
|
- Using VPC-SC to mitigate data exfiltration
|
||||||
|
|
||||||
|
### Shared VPC
|
||||||
|
The example supports the configuration of a Shared VPC as an input variable.
|
||||||
|
To deploy the solution on a Shared VPC, you have to configure the `network_config` variable:
|
||||||
|
|
||||||
|
```
|
||||||
|
network_config = {
|
||||||
|
host_project = "PROJECT_ID"
|
||||||
|
network_self_link = "https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/networks/VPC_NAME"
|
||||||
|
subnet_self_link = "https://www.googleapis.com/compute/v1/projects/PROJECT_ID/regions/$REGION/subnetworks/SUBNET_NAME"
|
||||||
|
cloudsql_psa_range = "10.60.0.0/24"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
To run this example, the Shared VPC project needs to have:
|
||||||
|
- A Private Service Connect with a range of `/24` (example: `10.60.0.0/24`) to deploy the Cloud SQL instance.
|
||||||
|
- Internet access configured (for example Cloud NAT) to let the Test VM download packages.
|
||||||
|
|
||||||
|
In order to run the example and deploy Cloud SQL on a shared VPC the identity running Terraform must have the following IAM role on the Shared VPC Host project.
|
||||||
|
- Compute Network Admin (roles/compute.networkAdmin)
|
||||||
|
- Compute Shared VPC Admin (roles/compute.xpnAdmin)
|
||||||
|
|
||||||
## Test your environment
|
## Test your environment
|
||||||
|
|
||||||
We assume all those steps are run using a user listed on `data_eng_principals`. You can authenticate as the user using the following command:
|
We assume all those steps are run using a user listed on `data_eng_principals`. You can authenticate as the user using the following command:
|
||||||
|
@ -118,15 +139,16 @@ The above command will delete the associated resources so there will be no billa
|
||||||
|
|
||||||
| name | description | type | required | default |
|
| name | description | type | required | default |
|
||||||
|---|---|:---:|:---:|:---:|
|
|---|---|:---:|:---:|:---:|
|
||||||
| [postgres_user_password](variables.tf#L29) | `postgres` user password. | <code>string</code> | ✓ | |
|
| [postgres_user_password](variables.tf#L40) | `postgres` user password. | <code>string</code> | ✓ | |
|
||||||
| [prefix](variables.tf#L40) | Unique prefix used for resource names. Not used for project if 'project_create' is null. | <code>string</code> | ✓ | |
|
| [prefix](variables.tf#L51) | Unique prefix used for resource names. Not used for project if 'project_create' is null. | <code>string</code> | ✓ | |
|
||||||
| [project_id](variables.tf#L54) | Project id, references existing project if `project_create` is null. | <code>string</code> | ✓ | |
|
| [project_id](variables.tf#L65) | Project id, references existing project if `project_create` is null. | <code>string</code> | ✓ | |
|
||||||
| [cmek_encryption](variables.tf#L17) | Flag to enable CMEK on GCP resources created. | <code>bool</code> | | <code>false</code> |
|
|
||||||
| [data_eng_principals](variables.tf#L23) | Groups with Service Account Token creator role on service accounts in IAM format, only user supported on CloudSQL, eg 'user@domain.com'. | <code>list(string)</code> | | <code>[]</code> |
|
| [data_eng_principals](variables.tf#L23) | Groups with Service Account Token creator role on service accounts in IAM format, only user supported on CloudSQL, eg 'user@domain.com'. | <code>list(string)</code> | | <code>[]</code> |
|
||||||
| [postgres_database](variables.tf#L34) | `postgres` database. | <code>string</code> | | <code>"guestbook"</code> |
|
| [network_config](variables.tf#L29) | Shared VPC network configurations to use. If null networks will be created in projects with preconfigured values. | <code title="object({ host_project = string network_self_link = string subnet_self_link = string cloudsql_psa_range = string })">object({…})</code> | | <code>null</code> |
|
||||||
| [project_create](variables.tf#L45) | Provide values if project creation is needed, uses existing project if null. Parent is in 'folders/nnn' or 'organizations/nnn' format. | <code title="object({ billing_account_id = string parent = string })">object({…})</code> | | <code>null</code> |
|
| [postgres_database](variables.tf#L45) | `postgres` database. | <code>string</code> | | <code>"guestbook"</code> |
|
||||||
| [regions](variables.tf#L59) | Map of instance_name => location where instances will be deployed. | <code>map(string)</code> | | <code title="{ primary = "europe-west1" replica = "europe-west3" }">{…}</code> |
|
| [project_create](variables.tf#L56) | Provide values if project creation is needed, uses existing project if null. Parent is in 'folders/nnn' or 'organizations/nnn' format. | <code title="object({ billing_account_id = string parent = string })">object({…})</code> | | <code>null</code> |
|
||||||
| [sql_configuration](variables.tf#L73) | Cloud SQL configuration | <code title="object({ availability_type = string database_version = string psa_range = string tier = string })">object({…})</code> | | <code title="{ availability_type = "REGIONAL" database_version = "POSTGRES_13" psa_range = "10.60.0.0/16" tier = "db-g1-small" }">{…}</code> |
|
| [regions](variables.tf#L70) | Map of instance_name => location where instances will be deployed. | <code>map(string)</code> | | <code title="{ primary = "europe-west1" replica = "europe-west3" }">{…}</code> |
|
||||||
|
| [service_encryption_keys](variables.tf#L17) | Cloud KMS keys to use to encrypt resources. Provide a key for each reagion configured. | <code>map(string)</code> | | <code>null</code> |
|
||||||
|
| [sql_configuration](variables.tf#L84) | Cloud SQL configuration | <code title="object({ availability_type = string database_version = string psa_range = string tier = string })">object({…})</code> | | <code title="{ availability_type = "REGIONAL" database_version = "POSTGRES_13" psa_range = "10.60.0.0/16" tier = "db-g1-small" }">{…}</code> |
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
|
|
|
@ -16,8 +16,8 @@ module "db" {
|
||||||
source = "../../../modules/cloudsql-instance"
|
source = "../../../modules/cloudsql-instance"
|
||||||
project_id = module.project.project_id
|
project_id = module.project.project_id
|
||||||
availability_type = var.sql_configuration.availability_type
|
availability_type = var.sql_configuration.availability_type
|
||||||
encryption_key_name = var.cmek_encryption ? module.kms[var.regions.primary].keys.key.id : null
|
encryption_key_name = var.service_encryption_keys != null ? try(var.service_encryption_keys[var.regions.primary], null) : null
|
||||||
network = module.vpc.self_link
|
network = local.vpc_self_link
|
||||||
name = "${var.prefix}-db"
|
name = "${var.prefix}-db"
|
||||||
region = var.regions.primary
|
region = var.regions.primary
|
||||||
database_version = var.sql_configuration.database_version
|
database_version = var.sql_configuration.database_version
|
||||||
|
@ -29,7 +29,7 @@ module "db" {
|
||||||
for k, v in var.regions :
|
for k, v in var.regions :
|
||||||
k => {
|
k => {
|
||||||
region = v,
|
region = v,
|
||||||
encryption_key_name = var.cmek_encryption ? module.kms[v].keys.key.id : null
|
encryption_key_name = var.service_encryption_keys != null ? try(var.service_encryption_keys[v], null) : null
|
||||||
} if k != "primary"
|
} if k != "primary"
|
||||||
}
|
}
|
||||||
databases = [var.postgres_database]
|
databases = [var.postgres_database]
|
||||||
|
|
|
@ -19,7 +19,7 @@ module "gcs" {
|
||||||
name = "data"
|
name = "data"
|
||||||
location = var.regions.primary
|
location = var.regions.primary
|
||||||
storage_class = "REGIONAL"
|
storage_class = "REGIONAL"
|
||||||
encryption_key = var.cmek_encryption ? module.kms[var.regions.primary].keys["key"].id : null
|
encryption_key = var.service_encryption_keys != null ? try(var.service_encryption_keys[var.regions.primary], null) : null
|
||||||
force_destroy = true
|
force_destroy = true
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
@ -29,8 +29,8 @@ module "test-vm" {
|
||||||
zone = "${var.regions.primary}-b"
|
zone = "${var.regions.primary}-b"
|
||||||
name = "sql-test"
|
name = "sql-test"
|
||||||
network_interfaces = [{
|
network_interfaces = [{
|
||||||
network = module.vpc.self_link
|
network = local.vpc_self_link
|
||||||
subnetwork = module.vpc.subnets["${var.regions.primary}/subnet"].self_link
|
subnetwork = local.subnet
|
||||||
nat = false
|
nat = false
|
||||||
addresses = null
|
addresses = null
|
||||||
}]
|
}]
|
||||||
|
@ -50,10 +50,10 @@ module "test-vm" {
|
||||||
type = "pd-ssd"
|
type = "pd-ssd"
|
||||||
size = 10
|
size = 10
|
||||||
}
|
}
|
||||||
encryption = var.cmek_encryption ? {
|
encryption = var.service_encryption_keys != null ? {
|
||||||
encrypt_boot = true
|
encrypt_boot = true
|
||||||
disk_encryption_key_raw = null
|
disk_encryption_key_raw = null
|
||||||
kms_key_self_link = var.cmek_encryption ? module.kms[var.regions.primary].keys["key"].id : null
|
kms_key_self_link = var.service_encryption_keys != null ? try(var.service_encryption_keys[var.regions.primary], null) : null
|
||||||
} : null
|
} : null
|
||||||
metadata = { startup-script = local.startup-script }
|
metadata = { startup-script = local.startup-script }
|
||||||
tags = ["ssh"]
|
tags = ["ssh"]
|
||||||
|
|
Binary file not shown.
Before Width: | Height: | Size: 33 KiB After Width: | Height: | Size: 24 KiB |
|
@ -1,37 +0,0 @@
|
||||||
# Copyright 2022 Google LLC
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# https://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
module "kms" {
|
|
||||||
for_each = toset(distinct(values(var.regions)))
|
|
||||||
source = "../../../modules/kms"
|
|
||||||
project_id = module.project.project_id
|
|
||||||
|
|
||||||
keyring = {
|
|
||||||
name = "${var.prefix}-keyring-${each.value}"
|
|
||||||
location = each.value
|
|
||||||
}
|
|
||||||
|
|
||||||
keys = {
|
|
||||||
key = null
|
|
||||||
}
|
|
||||||
key_iam = {
|
|
||||||
key = {
|
|
||||||
"roles/cloudkms.cryptoKeyEncrypterDecrypter" = [
|
|
||||||
"serviceAccount:${module.project.service_accounts.robots.compute}",
|
|
||||||
"serviceAccount:${module.project.service_accounts.robots.sql}",
|
|
||||||
"serviceAccount:${module.project.service_accounts.robots.storage}"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
|
@ -54,6 +54,20 @@ locals {
|
||||||
"serviceAccount:${module.project.service_accounts.robots.sql}"
|
"serviceAccount:${module.project.service_accounts.robots.sql}"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
shared_vpc_project = try(var.network_config.host_project, null)
|
||||||
|
use_shared_vpc = var.network_config != null
|
||||||
|
|
||||||
|
subnet = (
|
||||||
|
local.use_shared_vpc
|
||||||
|
? var.network_config.subnet_self_link
|
||||||
|
: values(module.vpc.0.subnet_self_links)[0]
|
||||||
|
)
|
||||||
|
vpc_self_link = (
|
||||||
|
local.use_shared_vpc
|
||||||
|
? var.network_config.network_self_link
|
||||||
|
: module.vpc.0.self_link
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
module "project" {
|
module "project" {
|
||||||
|
@ -67,6 +81,7 @@ module "project" {
|
||||||
iam_additive = var.project_create == null ? local.iam : {}
|
iam_additive = var.project_create == null ? local.iam : {}
|
||||||
services = [
|
services = [
|
||||||
"cloudkms.googleapis.com",
|
"cloudkms.googleapis.com",
|
||||||
|
"compute.googleapis.com",
|
||||||
"iap.googleapis.com",
|
"iap.googleapis.com",
|
||||||
"logging.googleapis.com",
|
"logging.googleapis.com",
|
||||||
"monitoring.googleapis.com",
|
"monitoring.googleapis.com",
|
||||||
|
@ -77,10 +92,26 @@ module "project" {
|
||||||
"storage.googleapis.com",
|
"storage.googleapis.com",
|
||||||
"storage-component.googleapis.com",
|
"storage-component.googleapis.com",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
shared_vpc_service_config = local.shared_vpc_project == null ? null : {
|
||||||
|
attach = true
|
||||||
|
host_project = local.shared_vpc_project
|
||||||
|
service_identity_iam = {}
|
||||||
|
}
|
||||||
|
|
||||||
|
service_encryption_key_ids = {
|
||||||
|
compute = try(values(var.service_encryption_keys), [])
|
||||||
|
sql = try(values(var.service_encryption_keys), [])
|
||||||
|
storage = try(values(var.service_encryption_keys), [])
|
||||||
|
}
|
||||||
|
service_config = {
|
||||||
|
disable_on_destroy = false, disable_dependent_services = false
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
module "vpc" {
|
module "vpc" {
|
||||||
source = "../../../modules/net-vpc"
|
source = "../../../modules/net-vpc"
|
||||||
|
count = local.use_shared_vpc ? 0 : 1
|
||||||
project_id = module.project.project_id
|
project_id = module.project.project_id
|
||||||
name = "vpc"
|
name = "vpc"
|
||||||
subnets = [
|
subnets = [
|
||||||
|
@ -100,15 +131,17 @@ module "vpc" {
|
||||||
|
|
||||||
module "firewall" {
|
module "firewall" {
|
||||||
source = "../../../modules/net-vpc-firewall"
|
source = "../../../modules/net-vpc-firewall"
|
||||||
|
count = local.use_shared_vpc ? 0 : 1
|
||||||
project_id = module.project.project_id
|
project_id = module.project.project_id
|
||||||
network = module.vpc.name
|
network = module.vpc.0.name
|
||||||
admin_ranges = ["10.0.0.0/20"]
|
admin_ranges = ["10.0.0.0/20"]
|
||||||
}
|
}
|
||||||
|
|
||||||
module "nat" {
|
module "nat" {
|
||||||
source = "../../../modules/net-cloudnat"
|
source = "../../../modules/net-cloudnat"
|
||||||
|
count = local.use_shared_vpc ? 0 : 1
|
||||||
project_id = module.project.project_id
|
project_id = module.project.project_id
|
||||||
region = var.regions.primary
|
region = var.regions.primary
|
||||||
name = "${var.prefix}-default"
|
name = "${var.prefix}-default"
|
||||||
router_network = module.vpc.name
|
router_network = module.vpc.0.name
|
||||||
}
|
}
|
||||||
|
|
|
@ -14,10 +14,10 @@
|
||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
variable "cmek_encryption" {
|
variable "service_encryption_keys" {
|
||||||
description = "Flag to enable CMEK on GCP resources created."
|
description = "Cloud KMS keys to use to encrypt resources. Provide a key for each reagion configured."
|
||||||
type = bool
|
type = map(string)
|
||||||
default = false
|
default = null
|
||||||
}
|
}
|
||||||
|
|
||||||
variable "data_eng_principals" {
|
variable "data_eng_principals" {
|
||||||
|
@ -26,6 +26,17 @@ variable "data_eng_principals" {
|
||||||
default = []
|
default = []
|
||||||
}
|
}
|
||||||
|
|
||||||
|
variable "network_config" {
|
||||||
|
description = "Shared VPC network configurations to use. If null networks will be created in projects with preconfigured values."
|
||||||
|
type = object({
|
||||||
|
host_project = string
|
||||||
|
network_self_link = string
|
||||||
|
subnet_self_link = string
|
||||||
|
cloudsql_psa_range = string
|
||||||
|
})
|
||||||
|
default = null
|
||||||
|
}
|
||||||
|
|
||||||
variable "postgres_user_password" {
|
variable "postgres_user_password" {
|
||||||
description = "`postgres` user password."
|
description = "`postgres` user password."
|
||||||
type = string
|
type = string
|
||||||
|
|
|
@ -15,5 +15,5 @@
|
||||||
def test_resources(e2e_plan_runner):
|
def test_resources(e2e_plan_runner):
|
||||||
"Test that plan works and the numbers of resources is as expected."
|
"Test that plan works and the numbers of resources is as expected."
|
||||||
modules, resources = e2e_plan_runner()
|
modules, resources = e2e_plan_runner()
|
||||||
assert len(modules) == 11
|
assert len(modules) == 9
|
||||||
assert len(resources) == 53
|
assert len(resources) == 48
|
||||||
|
|
Loading…
Reference in New Issue