zecfoundation
/
cloud-foundation-fabric
mirror of https://github.com/ZcashFoundation/cloud-foundation-fabric.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into workload-identity-federation

This commit is contained in:
Julio Castillo 2022-02-16 12:20:56 +01:00 committed by GitHub
parent 4f057a7933 dda86119f9
commit a07e187c31
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
105 changed files with 2722 additions and 1894 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
.github/workflows/linting.yml vendored
View File

@ -41,7 +41,7 @@ jobs:
- name: Install dependencies
run: |
pip install -r tools/REQUIREMENTS.txt
pip install -r tools/requirements.txt
- name: Boilerplate
id: boilerplate
@ -67,3 +67,8 @@ jobs:
id: name-length-fast
run: |
python3 tools/check_names.py --prefix-length=10 fast/stages
- name: Check python formatting
id: yapf
run: |
yapf --style="{based_on_style: google, indent_width: 2, SPLIT_BEFORE_NAMED_ASSIGNS: false}" -p -d tools/*.py

3
.gitignore vendored
View File

@ -21,7 +21,8 @@ bundle.zip
**/*.pkrvars.hcl
fixture_*
fast/configs
fast/stages/**/providers.tf
fast/stages/**/*providers.tf
fast/stages/**/terraform.tfvars
fast/stages/**/terraform.tfvars.json
fast/stages/**/terraform-*.auto.tfvars.json
fast/stages/**/0*.auto.tfvars*

1
CHANGELOG.md
View File

@ -12,6 +12,7 @@ All notable changes to this project will be documented in this file.
- **incompatible change** the variable for service identities IAM has changed in the project factory
- the `net-vpc` and `project` modules now use the beta provider for shared VPC-related resources
- add `data-catalog-policy-tag` module
- **incompatible change** the `psn_ranges` variable has been renamed to `psa_ranges` in the `net-vpc` module and its type changed from `list(string)` to `map(string)`
## [13.0.0] - 2022-01-27

2
examples/README.md
View File

@ -5,7 +5,7 @@ This section contains **[foundational examples](./foundations/)** that bootstrap
Currently available examples:
- **cloud operations** - [Resource tracking and remediation via Cloud Asset feeds](./cloud-operations/asset-inventory-feed-remediation), [Granular Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam), [Granular Cloud DNS IAM for Shared VPC](./cloud-operations/dns-shared-vpc), [Compute Engine quota monitoring](./cloud-operations/quota-monitoring), [Scheduled Cloud Asset Inventory Export to Bigquery](./cloud-operations/scheduled-asset-inventory-export-bq), [Packer image builder](./cloud-operations/packer-image-builder), [On-prem SA key management](./cloud-operations/onprem-sa-key-management)
- **data solutions** - [GCE/GCS CMEK via centralized Cloud KMS](./data-solutions/gcs-to-bq-with-least-privileges/), [Cloud Storage to Bigquery with Cloud Dataflow with least privileges](./data-solutions/gcs-to-bq-with-least-privileges/)
- **data solutions** - [GCE/GCS CMEK via centralized Cloud KMS](./data-solutions/gcs-to-bq-with-least-privileges/), [Cloud Storage to Bigquery with Cloud Dataflow with least privileges](./data-solutions/gcs-to-bq-with-least-privileges/), [Data Platform Foundations](./data-solutions/data-platform-foundations/)
- **factories** - [The why and the how of resource factories](./factories/README.md)
- **foundations** - [single level hierarchy](./foundations/environments/) (environments), [multiple level hierarchy](./foundations/business-units/) (business units + environments)
- **networking** - [hub and spoke via peering](./networking/hub-and-spoke-peering/), [hub and spoke via VPN](./networking/hub-and-spoke-vpn/), [DNS and Google Private Access for on-premises](./networking/onprem-google-access-dns/), [Shared VPC with GKE support](./networking/shared-vpc-gke/), [ILB as next hop](./networking/ilb-next-hop), [PSC for on-premises Cloud Function invocation](./networking/private-cloud-function-from-onprem/), [decentralized firewall](./networking/decentralized-firewall)

2
examples/data-solutions/README.md
View File

@ -18,6 +18,6 @@ They are meant to be used as minimal but complete starting points to create actu
### Data Platform Foundations
<a href="./data-platform-foundations/" title="Data Platform Foundations"><img src="./data-platform-foundations/02-resources/diagram.png" align="left" width="280px"></a>
<a href="./data-platform-foundations/" title="Data Platform Foundations"><img src="./data-platform-foundations/images/overview_diagram.png" align="left" width="280px"></a>
This [example](./data-platform-foundations/) implements a robust and flexible Data Foundation on GCP that provides opinionated defaults, allowing customers to build and scale out additional data pipelines quickly and reliably.
<br clear="left">

72
examples/data-solutions/data-platform-foundations/01-environment/README.md
View File

@ -1,72 +0,0 @@
# Data Platform Foundations - Environment (Step 1)
This is the first step needed to deploy Data Platform Foundations, which creates projects and service accounts. Please refer to the [top-level Data Platform README](../README.md) for prerequisites.
The projects that will be created are:
- Common services
- Landing
- Orchestration & Transformation
- DWH
- Datamart
A main service account named `projects-editor-sa` will be created under the common services project, and it will be granted editor permissions on all the projects in scope.
This is a high level diagram of the created resources:
![Environment - Phase 1](./diagram.png "High-level Environment diagram")
## Running the example
To create the infrastructure:
- specify your variables in a `terraform.tvars`
```tfm
billing_account = "1234-1234-1234"
parent = "folders/12345678"
admins = ["user:xxxxx@yyyyy.com"]
```
- make sure you have the right authentication setup (application default credentials, or a service account key) with the right permissions
- **The output of this stage contains the values for the resources stage**
- the `admins` variable contain a list of principals allowed to impersonate the service accounts. These principals will be given the `iam.serviceAccountTokenCreator` role
- run `terraform init` and `terraform apply`
Once done testing, you can clean up resources by running `terraform destroy`.
### CMEK configuration
You can configure GCP resources to use existing CMEK keys configuring the 'service_encryption_key_ids' variable. You need to specify a 'global' and a 'multiregional' key.
### VPC-SC configuration
You can assign projects to an existing VPC-SC standard perimeter configuring the 'service_perimeter_standard' variable. You can retrieve the list of existing perimeters from the GCP console or using the following command:
'''
gcloud access-context-manager perimeters list --format="json" | grep name
'''
The script use 'google_access_context_manager_service_perimeter_resource' terraform resource. If this resource is used alongside the 'vpc-sc' module, remember to uncomment the lifecycle block in the 'vpc-sc' module so they don't fight over which resources should be in the perimeter.
<!-- BEGIN TFDOC -->
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [billing_account_id](variables.tf#L21) | Billing account id. | <code>string</code> | ✓ | |
| [root_node](variables.tf#L50) | Parent folder or organization in 'folders/folder_id' or 'organizations/org_id' format. | <code>string</code> | ✓ | |
| [admins](variables.tf#L15) | List of users allowed to impersonate the service account. | <code>list&#40;string&#41;</code> | | <code>null</code> |
| [prefix](variables.tf#L26) | Prefix used to generate project id and name. | <code>string</code> | | <code>null</code> |
| [project_names](variables.tf#L32) | Override this variable if you need non-standard names. | <code title="object&#40;&#123;&#10; datamart &#61; string&#10; dwh &#61; string&#10; landing &#61; string&#10; services &#61; string&#10; transformation &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; datamart &#61; &#34;datamart&#34;&#10; dwh &#61; &#34;datawh&#34;&#10; landing &#61; &#34;landing&#34;&#10; services &#61; &#34;services&#34;&#10; transformation &#61; &#34;transformation&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
| [service_account_names](variables.tf#L55) | Override this variable if you need non-standard names. | <code title="object&#40;&#123;&#10; main &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; main &#61; &#34;data-platform-main&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
| [service_encryption_key_ids](variables.tf#L65) | Cloud KMS encryption key in {LOCATION => [KEY_URL]} format. Keys belong to existing project. | <code title="object&#40;&#123;&#10; multiregional &#61; string&#10; global &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; multiregional &#61; null&#10; global &#61; null&#10;&#125;">&#123;&#8230;&#125;</code> |
| [service_perimeter_standard](variables.tf#L78) | VPC Service control standard perimeter name in the form of 'accessPolicies/ACCESS_POLICY_NAME/servicePerimeters/PERIMETER_NAME'. All projects will be added to the perimeter in enforced mode. | <code>string</code> | | <code>null</code> |
## Outputs
| name | description | sensitive |
|---|---|:---:|
| [project_ids](outputs.tf#L17) | Project ids for created projects. | |
| [service_account](outputs.tf#L28) | Main service account. | |
| [service_encryption_key_ids](outputs.tf#L33) | Cloud KMS encryption keys in {LOCATION => [KEY_URL]} format. | |
<!-- END TFDOC -->

BIN
examples/data-solutions/data-platform-foundations/01-environment/diagram.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 275 KiB

162
examples/data-solutions/data-platform-foundations/01-environment/main.tf
View File

@ -1,162 +0,0 @@
/**
* Copyright 2020 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
###############################################################################
# projects #
###############################################################################
module "project-datamart" {
source = "../../../../modules/project"
parent = var.root_node
billing_account = var.billing_account_id
prefix = var.prefix
name = var.project_names.datamart
services = [
"bigquery.googleapis.com",
"bigquerystorage.googleapis.com",
"bigqueryreservation.googleapis.com",
"storage.googleapis.com",
"storage-component.googleapis.com",
]
iam_additive = {
"roles/owner" = [module.sa-services-main.iam_email]
}
service_encryption_key_ids = {
bq = [var.service_encryption_key_ids.multiregional]
storage = [var.service_encryption_key_ids.multiregional]
}
# If used, remember to uncomment 'lifecycle' block in the
# modules/vpc-sc/google_access_context_manager_service_perimeter resource.
service_perimeter_standard = var.service_perimeter_standard
}
module "project-dwh" {
source = "../../../../modules/project"
parent = var.root_node
billing_account = var.billing_account_id
prefix = var.prefix
name = var.project_names.dwh
services = [
"bigquery.googleapis.com",
"bigquerystorage.googleapis.com",
"bigqueryreservation.googleapis.com",
"storage.googleapis.com",
"storage-component.googleapis.com",
]
iam_additive = {
"roles/owner" = [module.sa-services-main.iam_email]
}
service_encryption_key_ids = {
bq = [var.service_encryption_key_ids.multiregional]
storage = [var.service_encryption_key_ids.multiregional]
}
# If used, remember to uncomment 'lifecycle' block in the
# modules/vpc-sc/google_access_context_manager_service_perimeter resource.
service_perimeter_standard = var.service_perimeter_standard
}
module "project-landing" {
source = "../../../../modules/project"
parent = var.root_node
billing_account = var.billing_account_id
prefix = var.prefix
name = var.project_names.landing
services = [
"pubsub.googleapis.com",
"storage.googleapis.com",
"storage-component.googleapis.com",
]
iam_additive = {
"roles/owner" = [module.sa-services-main.iam_email]
}
service_encryption_key_ids = {
pubsub = [var.service_encryption_key_ids.global]
storage = [var.service_encryption_key_ids.multiregional]
}
# If used, remember to uncomment 'lifecycle' block in the
# modules/vpc-sc/google_access_context_manager_service_perimeter resource.
service_perimeter_standard = var.service_perimeter_standard
}
module "project-services" {
source = "../../../../modules/project"
parent = var.root_node
billing_account = var.billing_account_id
prefix = var.prefix
name = var.project_names.services
services = [
"bigquery.googleapis.com",
"cloudresourcemanager.googleapis.com",
"iam.googleapis.com",
"pubsub.googleapis.com",
"storage.googleapis.com",
"storage-component.googleapis.com",
"sourcerepo.googleapis.com",
"stackdriver.googleapis.com",
"cloudasset.googleapis.com",
"cloudkms.googleapis.com"
]
iam_additive = {
"roles/owner" = [module.sa-services-main.iam_email]
}
service_encryption_key_ids = {
storage = [var.service_encryption_key_ids.multiregional]
}
# If used, remember to uncomment 'lifecycle' block in the
# modules/vpc-sc/google_access_context_manager_service_perimeter resource.
service_perimeter_standard = var.service_perimeter_standard
}
module "project-transformation" {
source = "../../../../modules/project"
parent = var.root_node
billing_account = var.billing_account_id
prefix = var.prefix
name = var.project_names.transformation
services = [
"bigquery.googleapis.com",
"cloudbuild.googleapis.com",
"compute.googleapis.com",
"dataflow.googleapis.com",
"servicenetworking.googleapis.com",
"storage.googleapis.com",
"storage-component.googleapis.com",
]
iam_additive = {
"roles/owner" = [module.sa-services-main.iam_email]
}
service_encryption_key_ids = {
compute = [var.service_encryption_key_ids.global]
storage = [var.service_encryption_key_ids.multiregional]
dataflow = [var.service_encryption_key_ids.global]
}
# If used, remember to uncomment 'lifecycle' block in the
# modules/vpc-sc/google_access_context_manager_service_perimeter resource.
service_perimeter_standard = var.service_perimeter_standard
}
###############################################################################
# service accounts #
###############################################################################
module "sa-services-main" {
source = "../../../../modules/iam-service-account"
project_id = module.project-services.project_id
name = var.service_account_names.main
iam = var.admins != null ? { "roles/iam.serviceAccountTokenCreator" = var.admins } : {}
}

36
examples/data-solutions/data-platform-foundations/01-environment/outputs.tf
View File

@ -1,36 +0,0 @@
/**
* Copyright 2020 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
output "project_ids" {
description = "Project ids for created projects."
value = {
datamart = module.project-datamart.project_id
dwh = module.project-dwh.project_id
landing = module.project-landing.project_id
services = module.project-services.project_id
transformation = module.project-transformation.project_id
}
}
output "service_account" {
description = "Main service account."
value = module.sa-services-main.email
}
output "service_encryption_key_ids" {
description = "Cloud KMS encryption keys in {LOCATION => [KEY_URL]} format."
value = var.service_encryption_key_ids
}

82
examples/data-solutions/data-platform-foundations/01-environment/variables.tf
View File

@ -1,82 +0,0 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
variable "admins" {
description = "List of users allowed to impersonate the service account."
type = list(string)
default = null
}
variable "billing_account_id" {
description = "Billing account id."
type = string
}
variable "prefix" {
description = "Prefix used to generate project id and name."
type = string
default = null
}
variable "project_names" {
description = "Override this variable if you need non-standard names."
type = object({
datamart = string
dwh = string
landing = string
services = string
transformation = string
})
default = {
datamart = "datamart"
dwh = "datawh"
landing = "landing"
services = "services"
transformation = "transformation"
}
}
variable "root_node" {
description = "Parent folder or organization in 'folders/folder_id' or 'organizations/org_id' format."
type = string
}
variable "service_account_names" {
description = "Override this variable if you need non-standard names."
type = object({
main = string
})
default = {
main = "data-platform-main"
}
}
variable "service_encryption_key_ids" {
description = "Cloud KMS encryption key in {LOCATION => [KEY_URL]} format. Keys belong to existing project."
type = object({
multiregional = string
global = string
})
default = {
multiregional = null
global = null
}
}
variable "service_perimeter_standard" {
description = "VPC Service control standard perimeter name in the form of 'accessPolicies/ACCESS_POLICY_NAME/servicePerimeters/PERIMETER_NAME'. All projects will be added to the perimeter in enforced mode."
type = string
default = null
}

135
examples/data-solutions/data-platform-foundations/01-landing.tf Normal file
View File

@ -0,0 +1,135 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# tfdoc:file:description land project and resources.
locals {
land_orch_service_accounts = [
module.load-sa-df-0.iam_email, module.orch-sa-cmp-0.iam_email
]
}
module "land-project" {
source = "../../../modules/project"
parent = var.folder_id
billing_account = var.billing_account_id
prefix = var.prefix
name = "lnd"
group_iam = {
(local.groups.data-engineers) = [
"roles/bigquery.dataEditor",
"roles/pubsub.editor",
"roles/storage.admin",
]
}
iam = {
"roles/bigquery.dataEditor" = [module.land-sa-bq-0.iam_email]
"roles/bigquery.user" = [module.load-sa-df-0.iam_email]
"roles/pubsub.publisher" = [module.land-sa-ps-0.iam_email]
"roles/pubsub.subscriber" = concat(
local.land_orch_service_accounts, [module.load-sa-df-0.iam_email]
)
"roles/storage.objectAdmin" = [module.load-sa-df-0.iam_email]
"roles/storage.objectCreator" = [module.land-sa-cs-0.iam_email]
"roles/storage.objectViewer" = [module.orch-sa-cmp-0.iam_email]
"roles/storage.admin" = [module.load-sa-df-0.iam_email]
}
services = concat(var.project_services, [
"bigquery.googleapis.com",
"bigqueryreservation.googleapis.com",
"bigquerystorage.googleapis.com",
"cloudkms.googleapis.com",
"pubsub.googleapis.com",
"storage.googleapis.com",
"storage-component.googleapis.com",
])
service_encryption_key_ids = {
bq = [try(local.service_encryption_keys.bq, null)]
pubsub = [try(local.service_encryption_keys.pubsub, null)]
storage = [try(local.service_encryption_keys.storage, null)]
}
}
# Cloud Storage
module "land-sa-cs-0" {
source = "../../../modules/iam-service-account"
project_id = module.land-project.project_id
prefix = var.prefix
name = "lnd-cs-0"
display_name = "Data platform GCS landing service account."
iam = {
"roles/iam.serviceAccountTokenCreator" = [
local.groups_iam.data-engineers
]
}
}
module "land-cs-0" {
source = "../../../modules/gcs"
project_id = module.land-project.project_id
prefix = var.prefix
name = "lnd-cs-0"
location = var.location
storage_class = "MULTI_REGIONAL"
encryption_key = try(local.service_encryption_keys.storage, null)
force_destroy = var.data_force_destroy
# retention_policy = {
# retention_period = 7776000 # 90 * 24 * 60 * 60
# is_locked = false
# }
}
# PubSub
module "land-sa-ps-0" {
source = "../../../modules/iam-service-account"
project_id = module.land-project.project_id
prefix = var.prefix
name = "lnd-ps-0"
display_name = "Data platform PubSub landing service account"
iam = {
"roles/iam.serviceAccountTokenCreator" = [
local.groups_iam.data-engineers
]
}
}
module "land-ps-0" {
source = "../../../modules/pubsub"
project_id = module.land-project.project_id
name = "${var.prefix}-lnd-ps-0"
kms_key = try(local.service_encryption_keys.pubsub, null)
}
# BigQuery
module "land-sa-bq-0" {
source = "../../../modules/iam-service-account"
project_id = module.land-project.project_id
prefix = var.prefix
name = "lnd-bq-0"
display_name = "Data platform BigQuery landing service account"
iam = {
"roles/iam.serviceAccountTokenCreator" = [local.groups_iam.data-engineers]
}
}
module "land-bq-0" {
source = "../../../modules/bigquery-dataset"
project_id = module.land-project.project_id
id = "${replace(var.prefix, "-", "_")}lnd_bq_0"
location = var.location
encryption_key = try(local.service_encryption_keys.bq, null)
}

137
examples/data-solutions/data-platform-foundations/02-load.tf Normal file
View File

@ -0,0 +1,137 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# tfdoc:file:description Load project and VPC.
locals {
load_service_accounts = [
"serviceAccount:${module.load-project.service_accounts.robots.dataflow}",
module.load-sa-df-0.iam_email
]
load_subnet = (
local.use_shared_vpc
? var.network_config.subnet_self_links.orchestration
: values(module.load-vpc.0.subnet_self_links)[0]
)
load_vpc = (
local.use_shared_vpc
? var.network_config.network_self_link
: module.load-vpc.0.self_link
)
}
# Project
module "load-project" {
source = "../../../modules/project"
parent = var.folder_id
billing_account = var.billing_account_id
prefix = var.prefix
name = "lod"
group_iam = {
(local.groups.data-engineers) = [
"roles/compute.viewer",
"roles/dataflow.admin",
"roles/dataflow.developer",
"roles/viewer",
]
}
iam = {
"roles/bigquery.jobUser" = [module.load-sa-df-0.iam_email]
"roles/dataflow.admin" = [
module.orch-sa-cmp-0.iam_email, module.load-sa-df-0.iam_email
]
"roles/dataflow.worker" = [module.load-sa-df-0.iam_email]
"roles/storage.objectAdmin" = local.load_service_accounts
}
services = concat(var.project_services, [
"bigquery.googleapis.com",
"bigqueryreservation.googleapis.com",
"bigquerystorage.googleapis.com",
"cloudkms.googleapis.com",
"compute.googleapis.com",
"dataflow.googleapis.com",
"dlp.googleapis.com",
"pubsub.googleapis.com",
"servicenetworking.googleapis.com",
"storage.googleapis.com",
"storage-component.googleapis.com"
])
service_encryption_key_ids = {
pubsub = [try(local.service_encryption_keys.pubsub, null)]
dataflow = [try(local.service_encryption_keys.dataflow, null)]
storage = [try(local.service_encryption_keys.storage, null)]
}
shared_vpc_service_config = local.shared_vpc_project == null ? null : {
attach = true
host_project = local.shared_vpc_project
service_identity_iam = {}
}
}
module "load-sa-df-0" {
source = "../../../modules/iam-service-account"
project_id = module.load-project.project_id
prefix = var.prefix
name = "load-df-0"
display_name = "Data platform Dataflow load service account"
iam = {
"roles/iam.serviceAccountTokenCreator" = [local.groups_iam.data-engineers]
"roles/iam.serviceAccountUser" = [module.orch-sa-cmp-0.iam_email]
}
}
module "load-cs-df-0" {
source = "../../../modules/gcs"
project_id = module.load-project.project_id
prefix = var.prefix
name = "load-cs-0"
location = var.location
storage_class = "MULTI_REGIONAL"
encryption_key = try(local.service_encryption_keys.storage, null)
}
# internal VPC resources
module "load-vpc" {
source = "../../../modules/net-vpc"
count = local.use_shared_vpc ? 0 : 1
project_id = module.load-project.project_id
name = "${var.prefix}-default"
subnets = [
{
ip_cidr_range = "10.10.0.0/24"
name = "default"
region = var.region
secondary_ip_range = {}
}
]
}
module "load-vpc-firewall" {
source = "../../../modules/net-vpc-firewall"
count = local.use_shared_vpc ? 0 : 1
project_id = module.load-project.project_id
network = module.load-vpc.0.name
admin_ranges = ["10.10.0.0/24"]
}
module "load-nat" {
source = "../../../modules/net-cloudnat"
count = local.use_shared_vpc ? 0 : 1
project_id = module.load-project.project_id
name = "${var.prefix}-default"
region = var.region
router_network = module.load-vpc.0.name
}

83
examples/data-solutions/data-platform-foundations/02-resources/README.md
View File

@ -1,83 +0,0 @@
# Data Platform Foundations - Resources (Step 2)
This is the second step needed to deploy Data Platform Foundations, which creates resources needed to store and process the data, in the projects created in the [previous step](../01-environment/README.md). Please refer to the [top-level README](../README.md) for prerequisites and how to run the first step.
![Data Foundation - Phase 2](./diagram.png "High-level diagram")
The resources that will be create in each project are:
- Common
- Landing
- [x] GCS
- [x] Pub/Sub
- Orchestration & Transformation
- [x] Dataflow
- DWH
- [x] Bigquery (L0/1/2)
- [x] GCS
- Datamart
- [x] Bigquery (views/table)
- [x] GCS
- [ ] BigTable
## Running the example
In the previous step, we created the environment (projects and service account) which we are going to use in this step.
To create the resources, copy the output of the environment step (**project_ids**) and paste it into the `terraform.tvars`:
- Specify your variables in a `terraform.tvars`, you can use the output from the environment stage
```tfm
project_ids = {
datamart = "datamart-project_id"
dwh = "dwh-project_id"
landing = "landing-project_id"
services = "services-project_id"
transformation = "transformation-project_id"
}
```
- The providers.tf file has been configured to impersonate the **main** service account
- To launch terraform:
```bash
terraform plan
terraform apply
```
Once done testing, you can clean up resources by running `terraform destroy`.
### CMEK configuration
You can configure GCP resources to use existing CMEK keys configuring the 'service_encryption_key_ids' variable. You need to specify a 'global' and a 'multiregional' key.
<!-- BEGIN TFDOC -->
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [project_ids](variables.tf#L108) | Project IDs. | <code title="object&#40;&#123;&#10; datamart &#61; string&#10; dwh &#61; string&#10; landing &#61; string&#10; services &#61; string&#10; transformation &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | |
| [admins](variables.tf#L16) | List of users allowed to impersonate the service account. | <code>list&#40;string&#41;</code> | | <code>null</code> |
| [datamart_bq_datasets](variables.tf#L22) | Datamart Bigquery datasets. | <code title="map&#40;object&#40;&#123;&#10; iam &#61; map&#40;list&#40;string&#41;&#41;&#10; location &#61; string&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code title="&#123;&#10; bq_datamart_dataset &#61; &#123;&#10; location &#61; &#34;EU&#34;&#10; iam &#61; &#123;&#10; &#125;&#10; &#125;&#10;&#125;">&#123;&#8230;&#125;</code> |
| [dwh_bq_datasets](variables.tf#L40) | DWH Bigquery datasets. | <code title="map&#40;object&#40;&#123;&#10; location &#61; string&#10; iam &#61; map&#40;list&#40;string&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code title="&#123;&#10; bq_raw_dataset &#61; &#123;&#10; iam &#61; &#123;&#125;&#10; location &#61; &#34;EU&#34;&#10; &#125;&#10;&#125;">&#123;&#8230;&#125;</code> |
| [landing_buckets](variables.tf#L54) | List of landing buckets to create. | <code title="map&#40;object&#40;&#123;&#10; location &#61; string&#10; name &#61; string&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code title="&#123;&#10; raw-data &#61; &#123;&#10; location &#61; &#34;EU&#34;&#10; name &#61; &#34;raw-data&#34;&#10; &#125;&#10; data-schema &#61; &#123;&#10; location &#61; &#34;EU&#34;&#10; name &#61; &#34;data-schema&#34;&#10; &#125;&#10;&#125;">&#123;&#8230;&#125;</code> |
| [landing_pubsub](variables.tf#L72) | List of landing pubsub topics and subscriptions to create. | <code title="map&#40;map&#40;object&#40;&#123;&#10; iam &#61; map&#40;list&#40;string&#41;&#41;&#10; labels &#61; map&#40;string&#41;&#10; options &#61; object&#40;&#123;&#10; ack_deadline_seconds &#61; number&#10; message_retention_duration &#61; number&#10; retain_acked_messages &#61; bool&#10; expiration_policy_ttl &#61; number&#10; &#125;&#41;&#10;&#125;&#41;&#41;&#41;">map&#40;map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;&#41;</code> | | <code title="&#123;&#10; landing-1 &#61; &#123;&#10; sub1 &#61; &#123;&#10; iam &#61; &#123;&#10; &#125;&#10; labels &#61; &#123;&#125;&#10; options &#61; null&#10; &#125;&#10; sub2 &#61; &#123;&#10; iam &#61; &#123;&#125;&#10; labels &#61; &#123;&#125;,&#10; options &#61; null&#10; &#125;,&#10; &#125;&#10;&#125;">&#123;&#8230;&#125;</code> |
| [landing_service_account](variables.tf#L102) | landing service accounts list. | <code>string</code> | | <code>&#34;sa-landing&#34;</code> |
| [service_account_names](variables.tf#L119) | Project service accounts list. | <code title="object&#40;&#123;&#10; datamart &#61; string&#10; dwh &#61; string&#10; landing &#61; string&#10; services &#61; string&#10; transformation &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; datamart &#61; &#34;sa-datamart&#34;&#10; dwh &#61; &#34;sa-datawh&#34;&#10; landing &#61; &#34;sa-landing&#34;&#10; services &#61; &#34;sa-services&#34;&#10; transformation &#61; &#34;sa-transformation&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
| [service_encryption_key_ids](variables.tf#L137) | Cloud KMS encryption key in {LOCATION => [KEY_URL]} format. Keys belong to existing project. | <code title="object&#40;&#123;&#10; multiregional &#61; string&#10; global &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; multiregional &#61; null&#10; global &#61; null&#10;&#125;">&#123;&#8230;&#125;</code> |
| [transformation_buckets](variables.tf#L149) | List of transformation buckets to create. | <code title="map&#40;object&#40;&#123;&#10; location &#61; string&#10; name &#61; string&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code title="&#123;&#10; temp &#61; &#123;&#10; location &#61; &#34;EU&#34;&#10; name &#61; &#34;temp&#34;&#10; &#125;,&#10; templates &#61; &#123;&#10; location &#61; &#34;EU&#34;&#10; name &#61; &#34;templates&#34;&#10; &#125;,&#10;&#125;">&#123;&#8230;&#125;</code> |
| [transformation_subnets](variables.tf#L167) | List of subnets to create in the transformation Project. | <code title="list&#40;object&#40;&#123;&#10; ip_cidr_range &#61; string&#10; name &#61; string&#10; region &#61; string&#10; secondary_ip_range &#61; map&#40;string&#41;&#10;&#125;&#41;&#41;">list&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code title="&#91;&#10; &#123;&#10; ip_cidr_range &#61; &#34;10.1.0.0&#47;20&#34;&#10; name &#61; &#34;transformation-subnet&#34;&#10; region &#61; &#34;europe-west3&#34;&#10; secondary_ip_range &#61; &#123;&#125;&#10; &#125;,&#10;&#93;">&#91;&#8230;&#93;</code> |
| [transformation_vpc_name](variables.tf#L185) | Name of the VPC created in the transformation Project. | <code>string</code> | | <code>&#34;transformation-vpc&#34;</code> |
## Outputs
| name | description | sensitive |
|---|---|:---:|
| [datamart-datasets](outputs.tf#L17) | List of bigquery datasets created for the datamart project. | |
| [dwh-datasets](outputs.tf#L24) | List of bigquery datasets created for the dwh project. | |
| [landing-buckets](outputs.tf#L29) | List of buckets created for the landing project. | |
| [landing-pubsub](outputs.tf#L34) | List of pubsub topics and subscriptions created for the landing project. | |
| [transformation-buckets](outputs.tf#L44) | List of buckets created for the transformation project. | |
| [transformation-vpc](outputs.tf#L49) | Transformation VPC details. | |
<!-- END TFDOC -->

BIN
examples/data-solutions/data-platform-foundations/02-resources/diagram.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 470 KiB

211
examples/data-solutions/data-platform-foundations/02-resources/main.tf
View File

@ -1,211 +0,0 @@
/**
* Copyright 2020 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
###############################################################################
# IAM #
###############################################################################
module "datamart-sa" {
source = "../../../../modules/iam-service-account"
project_id = var.project_ids.datamart
name = var.service_account_names.datamart
iam_project_roles = {
"${var.project_ids.datamart}" = ["roles/editor"]
}
iam = var.admins != null ? { "roles/iam.serviceAccountTokenCreator" = var.admins } : {}
}
module "dwh-sa" {
source = "../../../../modules/iam-service-account"
project_id = var.project_ids.dwh
name = var.service_account_names.dwh
iam_project_roles = {
"${var.project_ids.dwh}" = ["roles/bigquery.admin"]
}
iam = var.admins != null ? { "roles/iam.serviceAccountTokenCreator" = var.admins } : {}
}
module "landing-sa" {
source = "../../../../modules/iam-service-account"
project_id = var.project_ids.landing
name = var.service_account_names.landing
iam_project_roles = {
"${var.project_ids.landing}" = [
"roles/pubsub.publisher",
"roles/storage.objectCreator"]
}
iam = var.admins != null ? { "roles/iam.serviceAccountTokenCreator" = var.admins } : {}
}
module "services-sa" {
source = "../../../../modules/iam-service-account"
project_id = var.project_ids.services
name = var.service_account_names.services
iam_project_roles = {
"${var.project_ids.services}" = ["roles/editor"]
}
iam = var.admins != null ? { "roles/iam.serviceAccountTokenCreator" = var.admins } : {}
}
module "transformation-sa" {
source = "../../../../modules/iam-service-account"
project_id = var.project_ids.transformation
name = var.service_account_names.transformation
iam_project_roles = {
"${var.project_ids.transformation}" = [
"roles/logging.logWriter",
"roles/monitoring.metricWriter",
"roles/dataflow.admin",
"roles/iam.serviceAccountUser",
"roles/bigquery.dataOwner",
"roles/bigquery.jobUser",
"roles/dataflow.worker",
"roles/bigquery.metadataViewer",
"roles/storage.objectViewer",
],
"${var.project_ids.landing}" = [
"roles/storage.objectViewer",
],
"${var.project_ids.dwh}" = [
"roles/bigquery.dataOwner",
"roles/bigquery.jobUser",
"roles/bigquery.metadataViewer",
]
}
iam = var.admins != null ? { "roles/iam.serviceAccountTokenCreator" = var.admins } : {}
}
###############################################################################
# GCS #
###############################################################################
module "landing-buckets" {
source = "../../../../modules/gcs"
for_each = var.landing_buckets
project_id = var.project_ids.landing
prefix = var.project_ids.landing
name = each.value.name
location = each.value.location
iam = {
"roles/storage.objectCreator" = [module.landing-sa.iam_email]
"roles/storage.admin" = [module.transformation-sa.iam_email]
}
encryption_key = var.service_encryption_key_ids.multiregional
}
module "transformation-buckets" {
source = "../../../../modules/gcs"
for_each = var.transformation_buckets
project_id = var.project_ids.transformation
prefix = var.project_ids.transformation
name = each.value.name
location = each.value.location
iam = {
"roles/storage.admin" = [module.transformation-sa.iam_email]
}
encryption_key = var.service_encryption_key_ids.multiregional
}
###############################################################################
# Bigquery #
###############################################################################
module "datamart-bq" {
source = "../../../../modules/bigquery-dataset"
for_each = var.datamart_bq_datasets
project_id = var.project_ids.datamart
id = each.key
location = each.value.location
iam = {
for k, v in each.value.iam : k => (
k == "roles/bigquery.dataOwner"
? concat(v, [module.datamart-sa.iam_email])
: v
)
}
encryption_key = var.service_encryption_key_ids.multiregional
}
module "dwh-bq" {
source = "../../../../modules/bigquery-dataset"
for_each = var.dwh_bq_datasets
project_id = var.project_ids.dwh
id = each.key
location = each.value.location
iam = {
for k, v in each.value.iam : k => (
k == "roles/bigquery.dataOwner"
? concat(v, [module.dwh-sa.iam_email])
: v
)
}
encryption_key = var.service_encryption_key_ids.multiregional
}
###############################################################################
# Network #
###############################################################################
module "vpc-transformation" {
source = "../../../../modules/net-vpc"
project_id = var.project_ids.transformation
name = var.transformation_vpc_name
subnets = var.transformation_subnets
}
module "firewall" {
source = "../../../../modules/net-vpc-firewall"
project_id = var.project_ids.transformation
network = module.vpc-transformation.name
admin_ranges = []
http_source_ranges = []
https_source_ranges = []
ssh_source_ranges = []
custom_rules = {
iap-svc = {
description = "Dataflow service."
direction = "INGRESS"
action = "allow"
sources = ["dataflow"]
targets = ["dataflow"]
ranges = []
use_service_accounts = false
rules = [{ protocol = "tcp", ports = ["12345-12346"] }]
extra_attributes = {}
}
}
}
###############################################################################
# Pub/Sub #
###############################################################################
module "landing-pubsub" {
source = "../../../../modules/pubsub"
for_each = var.landing_pubsub
project_id = var.project_ids.landing
name = each.key
subscriptions = {
for k, v in each.value : k => { labels = v.labels, options = v.options }
}
subscription_iam = {
for k, v in each.value : k => merge(v.iam, {
"roles/pubsub.subscriber" = [module.transformation-sa.iam_email]
})
}
kms_key = var.service_encryption_key_ids.global
}

60
examples/data-solutions/data-platform-foundations/02-resources/outputs.tf
View File

@ -1,60 +0,0 @@
/**
* Copyright 2020 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
output "datamart-datasets" {
description = "List of bigquery datasets created for the datamart project."
value = [
for k, datasets in module.datamart-bq : datasets.dataset_id
]
}
output "dwh-datasets" {
description = "List of bigquery datasets created for the dwh project."
value = [for k, datasets in module.dwh-bq : datasets.dataset_id]
}
output "landing-buckets" {
description = "List of buckets created for the landing project."
value = [for k, bucket in module.landing-buckets : bucket.name]
}
output "landing-pubsub" {
description = "List of pubsub topics and subscriptions created for the landing project."
value = {
for t in module.landing-pubsub : t.topic.name => {
id = t.topic.id
subscriptions = { for s in t.subscriptions : s.name => s.id }
}
}
}
output "transformation-buckets" {
description = "List of buckets created for the transformation project."
value = [for k, bucket in module.transformation-buckets : bucket.name]
}
output "transformation-vpc" {
description = "Transformation VPC details."
value = {
name = module.vpc-transformation.name
subnets = {
for k, s in module.vpc-transformation.subnets : k => {
ip_cidr_range = s.ip_cidr_range
region = s.region
}
}
}
}

23
examples/data-solutions/data-platform-foundations/02-resources/providers.tf
View File

@ -1,23 +0,0 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
provider "google" {
impersonate_service_account = "data-platform-main@${var.project_ids.services}.iam.gserviceaccount.com"
}
provider "google-beta" {
impersonate_service_account = "data-platform-main@${var.project_ids.services}.iam.gserviceaccount.com"
}

189
examples/data-solutions/data-platform-foundations/02-resources/variables.tf
View File

@ -1,189 +0,0 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
variable "admins" {
description = "List of users allowed to impersonate the service account."
type = list(string)
default = null
}
variable "datamart_bq_datasets" {
description = "Datamart Bigquery datasets."
type = map(object({
iam = map(list(string))
location = string
}))
default = {
bq_datamart_dataset = {
location = "EU"
iam = {
# "roles/bigquery.dataOwner" = []
# "roles/bigquery.dataEditor" = []
# "roles/bigquery.dataViewer" = []
}
}
}
}
variable "dwh_bq_datasets" {
description = "DWH Bigquery datasets."
type = map(object({
location = string
iam = map(list(string))
}))
default = {
bq_raw_dataset = {
iam = {}
location = "EU"
}
}
}
variable "landing_buckets" {
description = "List of landing buckets to create."
type = map(object({
location = string
name = string
}))
default = {
raw-data = {
location = "EU"
name = "raw-data"
}
data-schema = {
location = "EU"
name = "data-schema"
}
}
}
variable "landing_pubsub" {
description = "List of landing pubsub topics and subscriptions to create."
type = map(map(object({
iam = map(list(string))
labels = map(string)
options = object({
ack_deadline_seconds = number
message_retention_duration = number
retain_acked_messages = bool
expiration_policy_ttl = number
})
})))
default = {
landing-1 = {
sub1 = {
iam = {
# "roles/pubsub.subscriber" = []
}
labels = {}
options = null
}
sub2 = {
iam = {}
labels = {},
options = null
},
}
}
}
variable "landing_service_account" {
description = "landing service accounts list."
type = string
default = "sa-landing"
}
variable "project_ids" {
description = "Project IDs."
type = object({
datamart = string
dwh = string
landing = string
services = string
transformation = string
})
}
variable "service_account_names" {
description = "Project service accounts list."
type = object({
datamart = string
dwh = string
landing = string
services = string
transformation = string
})
default = {
datamart = "sa-datamart"
dwh = "sa-datawh"
landing = "sa-landing"
services = "sa-services"
transformation = "sa-transformation"
}
}
variable "service_encryption_key_ids" {
description = "Cloud KMS encryption key in {LOCATION => [KEY_URL]} format. Keys belong to existing project."
type = object({
multiregional = string
global = string
})
default = {
multiregional = null
global = null
}
}
variable "transformation_buckets" {
description = "List of transformation buckets to create."
type = map(object({
location = string
name = string
}))
default = {
temp = {
location = "EU"
name = "temp"
},
templates = {
location = "EU"
name = "templates"
},
}
}
variable "transformation_subnets" {
description = "List of subnets to create in the transformation Project."
type = list(object({
ip_cidr_range = string
name = string
region = string
secondary_ip_range = map(string)
}))
default = [
{
ip_cidr_range = "10.1.0.0/20"
name = "transformation-subnet"
region = "europe-west3"
secondary_ip_range = {}
},
]
}
variable "transformation_vpc_name" {
description = "Name of the VPC created in the transformation Project."
type = string
default = "transformation-vpc"
}

121
examples/data-solutions/data-platform-foundations/03-composer.tf Normal file
View File

@ -0,0 +1,121 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# tfdoc:file:description Orchestration Cloud Composer definition.
module "orch-sa-cmp-0" {
source = "../../../modules/iam-service-account"
project_id = module.orch-project.project_id
prefix = var.prefix
name = "orc-cmp-0"
display_name = "Data platform Composer service account"
iam = {
"roles/iam.serviceAccountTokenCreator" = [local.groups_iam.data-engineers]
"roles/iam.serviceAccountUser" = [module.orch-sa-cmp-0.iam_email]
}
}
resource "google_composer_environment" "orch-cmp-0" {
provider = google-beta
project = module.orch-project.project_id
name = "${var.prefix}-orc-cmp-0"
region = var.region
config {
node_count = var.composer_config.node_count
node_config {
zone = "${var.region}-b"
service_account = module.orch-sa-cmp-0.email
network = local.orch_vpc
subnetwork = local.orch_subnet
tags = ["composer-worker", "http-server", "https-server"]
ip_allocation_policy {
use_ip_aliases = "true"
cluster_secondary_range_name = try(
var.network_config.composer_secondary_ranges.pods, "pods"
)
services_secondary_range_name = try(
var.network_config.composer_secondary_ranges.services, "services"
)
}
}
software_config {
image_version = var.composer_config.airflow_version
env_variables = merge(
var.composer_config.env_variables, {
BQ_LOCATION = var.location
DTL_L0_PRJ = module.lake-0-project.project_id
DTL_L0_BQ_DATASET = module.lake-0-bq-0.dataset_id
DTL_L0_GCS = module.lake-0-cs-0.url
DTL_L1_PRJ = module.lake-1-project.project_id
DTL_L1_BQ_DATASET = module.lake-1-bq-0.dataset_id
DTL_L1_GCS = module.lake-1-cs-0.url
DTL_L2_PRJ = module.lake-2-project.project_id
DTL_L2_BQ_DATASET = module.lake-2-bq-0.dataset_id
DTL_L2_GCS = module.lake-2-cs-0.url
DTL_PLG_PRJ = module.lake-plg-project.project_id
DTL_PLG_BQ_DATASET = module.lake-plg-bq-0.dataset_id
DTL_PLG_GCS = module.lake-plg-cs-0.url
GCP_REGION = var.region
LND_PRJ = module.land-project.project_id
LND_BQ = module.land-bq-0.dataset_id
LND_GCS = module.land-cs-0.url
LND_PS = module.land-ps-0.id
LOD_PRJ = module.load-project.project_id
LOD_GCS_STAGING = module.load-cs-df-0.url
LOD_NET_VPC = local.load_vpc
LOD_NET_SUBNET = local.load_subnet
LOD_SA_DF = module.load-sa-df-0.email
ORC_PRJ = module.orch-project.project_id
ORC_GCS = module.orch-cs-0.url
TRF_PRJ = module.transf-project.project_id
TRF_GCS_STAGING = module.transf-cs-df-0.url
TRF_NET_VPC = local.transf_vpc
TRF_NET_SUBNET = local.transf_subnet
TRF_SA_DF = module.transf-sa-df-0.email
TRF_SA_BQ = module.transf-sa-bq-0.email
}
)
}
private_environment_config {
enable_private_endpoint = "true"
cloud_sql_ipv4_cidr_block = try(
var.network_config.composer_ip_ranges.cloudsql, "10.20.10.0/24"
)
master_ipv4_cidr_block = try(
var.network_config.composer_ip_ranges.gke_master, "10.20.11.0/28"
)
web_server_ipv4_cidr_block = try(
var.network_config.composer_ip_ranges.web_server, "10.20.11.16/28"
)
}
dynamic "encryption_config" {
for_each = (
try(local.service_encryption_keys.composer != null, false)
? { 1 = 1 }
: {}
)
content {
kms_key_name = try(local.service_encryption_keys.composer, null)
}
}
# web_server_network_access_control {
# allowed_ip_range {
# value = "172.16.0.0/12"
# description = "Allowed ip range"
# }
# }
}
}

148
examples/data-solutions/data-platform-foundations/03-orchestration.tf Normal file
View File

@ -0,0 +1,148 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# tfdoc:file:description Orchestration project and VPC.
locals {
orch_subnet = (
local.use_shared_vpc
? var.network_config.subnet_self_links.orchestration
: values(module.orch-vpc.0.subnet_self_links)[0]
)
orch_vpc = (
local.use_shared_vpc
? var.network_config.network_self_link
: module.orch-vpc.0.self_link
)
}
module "orch-project" {
source = "../../../modules/project"
parent = var.folder_id
billing_account = var.billing_account_id
prefix = var.prefix
name = "orc"
group_iam = {
(local.groups.data-engineers) = [
"roles/bigquery.dataEditor",
"roles/bigquery.jobUser",
"roles/cloudbuild.builds.editor",
"roles/composer.admin",
"roles/composer.environmentAndStorageObjectAdmin",
"roles/iap.httpsResourceAccessor",
"roles/iam.serviceAccountUser",
"roles/storage.objectAdmin",
"roles/storage.admin",
]
}
iam = {
"roles/bigquery.dataEditor" = [
module.load-sa-df-0.iam_email,
module.transf-sa-df-0.iam_email,
]
"roles/bigquery.jobUser" = [
module.orch-sa-cmp-0.iam_email,
]
"roles/composer.worker" = [
module.orch-sa-cmp-0.iam_email
]
"roles/iam.serviceAccountUser" = [
module.orch-sa-cmp-0.iam_email
]
"roles/storage.objectAdmin" = [
module.orch-sa-cmp-0.iam_email,
"serviceAccount:${module.orch-project.service_accounts.robots.composer}",
]
"roles/storage.objectViewer" = [module.load-sa-df-0.iam_email]
}
oslogin = false
policy_boolean = {
"constraints/compute.requireOsLogin" = false
}
services = concat(var.project_services, [
"artifactregistry.googleapis.com",
"bigquery.googleapis.com",
"bigqueryreservation.googleapis.com",
"bigquerystorage.googleapis.com",
"cloudbuild.googleapis.com",
"cloudkms.googleapis.com",
"composer.googleapis.com",
"compute.googleapis.com",
"container.googleapis.com",
"containerregistry.googleapis.com",
"dataflow.googleapis.com",
"pubsub.googleapis.com",
"servicenetworking.googleapis.com",
"storage.googleapis.com",
"storage-component.googleapis.com"
])
service_encryption_key_ids = {
composer = [try(local.service_encryption_keys.composer, null)]
storage = [try(local.service_encryption_keys.storage, null)]
}
shared_vpc_service_config = local.shared_vpc_project == null ? null : {
attach = true
host_project = local.shared_vpc_project
service_identity_iam = {}
}
}
# Cloud Storage
module "orch-cs-0" {
source = "../../../modules/gcs"
project_id = module.orch-project.project_id
prefix = var.prefix
name = "orc-cs-0"
location = var.location
storage_class = "MULTI_REGIONAL"
encryption_key = try(local.service_encryption_keys.storage, null)
}
# internal VPC resources
module "orch-vpc" {
source = "../../../modules/net-vpc"
count = local.use_shared_vpc ? 0 : 1
project_id = module.orch-project.project_id
name = "${var.prefix}-default"
subnets = [
{
ip_cidr_range = "10.10.0.0/24"
name = "default"
region = var.region
secondary_ip_range = {
pods = "10.10.8.0/22"
services = "10.10.12.0/24"
}
}
]
}
module "orch-vpc-firewall" {
source = "../../../modules/net-vpc-firewall"
count = local.use_shared_vpc ? 0 : 1
project_id = module.orch-project.project_id
network = module.orch-vpc.0.name
admin_ranges = ["10.10.0.0/24"]
}
module "orch-nat" {
count = local.use_shared_vpc ? 0 : 1
source = "../../../modules/net-cloudnat"
project_id = module.orch-project.project_id
name = "${var.prefix}-default"
region = var.region
router_network = module.orch-vpc.0.name
}

8
examples/data-solutions/data-platform-foundations/03-pipeline/README.md
View File

@ -1,8 +0,0 @@
# Manual pipeline Example
Once you deployed projects [step 1](../01-environment/README.md) and resources [step 2](../02-resources/README.md) you can use it to run your data pipeline.
Here we will demo 2 pipelines:
* [GCS to Bigquery](./gcs_to_bigquery.md)
* [PubSub to Bigquery](./pubsub_to_bigquery.md)

140
examples/data-solutions/data-platform-foundations/03-pipeline/gcs_to_bigquery.md
View File

@ -1,140 +0,0 @@
# Manual pipeline Example: GCS to Bigquery
In this example we will publish person message in the following format:
```bash
name,surname,1617898199
```
A Dataflow pipeline will read those messages and import them into a Bigquery table in the DWH project.
[TODO] An autorized view will be created in the datamart project to expose the table.
[TODO] Further automation is expected in future.
## Set up the env vars
```bash
export DWH_PROJECT_ID=**dwh_project_id**
export LANDING_PROJECT_ID=**landing_project_id**
export TRANSFORMATION_PROJECT_ID=*transformation_project_id*
```
## Create BQ table
Those steps should be done as DWH Service Account.
You can run the command to create a table:
```bash
gcloud --impersonate-service-account=sa-datawh@$DWH_PROJECT_ID.iam.gserviceaccount.com \
alpha bq tables create person \
--project=$DWH_PROJECT_ID --dataset=bq_raw_dataset \
--description "This is a Test Person table" \
--schema name=STRING,surname=STRING,timestamp=TIMESTAMP
```
## Produce CSV data file, JSON schema file and UDF JS file
Those steps should be done as landing Service Account:
Let's now create a series of messages we can use to import:
```bash
for i in {0..10}
do
echo "Lorenzo,Caggioni,$(date +%s)" >> person.csv
done
```
and copy files to the GCS bucket:
```bash
gsutil -i sa-landing@$LANDING_PROJECT_ID.iam.gserviceaccount.com cp person.csv gs://$LANDING_PROJECT_ID-eu-raw-data
```
Let's create the data JSON schema:
```bash
cat <<'EOF' >> person_schema.json
{
"BigQuery Schema": [
{
"name": "name",
"type": "STRING"
},
{
"name": "surname",
"type": "STRING"
},
{
"name": "timestamp",
"type": "TIMESTAMP"
}
]
}
EOF
```
and copy files to the GCS bucket:
```bash
gsutil -i sa-landing@$LANDING_PROJECT_ID.iam.gserviceaccount.com cp person_schema.json gs://$LANDING_PROJECT_ID-eu-data-schema
```
Let's create the data UDF function to transform message data:
```bash
cat <<'EOF' >> person_udf.js
function transform(line) {
var values = line.split(',');
var obj = new Object();
obj.name = values[0];
obj.surname = values[1];
obj.timestamp = values[2];
var jsonString = JSON.stringify(obj);
return jsonString;
}
EOF
```
and copy files to the GCS bucket:
```bash
gsutil -i sa-landing@$LANDING_PROJECT_ID.iam.gserviceaccount.com cp person_udf.js gs://$LANDING_PROJECT_ID-eu-data-schema
```
if you want to check files copied to GCS, you can use the Transformation service account:
```bash
gsutil -i sa-transformation@$TRANSFORMATION_PROJECT_ID.iam.gserviceaccount.com ls gs://$LANDING_PROJECT_ID-eu-raw-data
gsutil -i sa-transformation@$TRANSFORMATION_PROJECT_ID.iam.gserviceaccount.com ls gs://$LANDING_PROJECT_ID-eu-data-schema
```
## Dataflow
Those steps should be done as transformation Service Account.
Let's than start a Dataflow batch pipeline using a Google provided template using internal only IPs, the created network and subnetwork, the appropriate service account and requested parameters:
```bash
gcloud --impersonate-service-account=sa-transformation@$TRANSFORMATION_PROJECT_ID.iam.gserviceaccount.com dataflow jobs run test_batch_01 \
--gcs-location gs://dataflow-templates/latest/GCS_Text_to_BigQuery \
--project $TRANSFORMATION_PROJECT_ID \
--region europe-west3 \
--disable-public-ips \
--network transformation-vpc \
--subnetwork regions/europe-west3/subnetworks/transformation-subnet \
--staging-location gs://$TRANSFORMATION_PROJECT_ID-eu-temp \
--service-account-email sa-transformation@$TRANSFORMATION_PROJECT_ID.iam.gserviceaccount.com \
--parameters \
javascriptTextTransformFunctionName=transform,\
JSONPath=gs://$LANDING_PROJECT_ID-eu-data-schema/person_schema.json,\
javascriptTextTransformGcsPath=gs://$LANDING_PROJECT_ID-eu-data-schema/person_udf.js,\
inputFilePattern=gs://$LANDING_PROJECT_ID-eu-raw-data/person.csv,\
outputTable=$DWH_PROJECT_ID:bq_raw_dataset.person,\
bigQueryLoadingTemporaryDirectory=gs://$TRANSFORMATION_PROJECT_ID-eu-temp
```

75
examples/data-solutions/data-platform-foundations/03-pipeline/pubsub_to_bigquery.md
View File

@ -1,75 +0,0 @@
# Manual pipeline Example: PubSub to Bigquery
In this example we will publish person message in the following format:
```txt
name: Name
surname: Surname
timestamp: 1617898199
```
a Dataflow pipeline will read those messages and import them into a Bigquery table in the DWH project.
An autorized view will be created in the datamart project to expose the table.
[TODO] Further automation is expected in future.
## Set up the env vars
```bash
export DWH_PROJECT_ID=**dwh_project_id**
export LANDING_PROJECT_ID=**landing_project_id**
export TRANSFORMATION_PROJECT_ID=*transformation_project_id*
```
## Create BQ table
Those steps should be done as DWH Service Account.
You can run the command to create a table:
```bash
gcloud --impersonate-service-account=sa-datawh@$DWH_PROJECT_ID.iam.gserviceaccount.com \
alpha bq tables create person \
--project=$DWH_PROJECT_ID --dataset=bq_raw_dataset \
--description "This is a Test Person table" \
--schema name=STRING,surname=STRING,timestamp=TIMESTAMP
```
## Produce PubSub messages
Those steps should be done as landing Service Account:
Let's now create a series of messages we can use to import:
```bash
for i in {0..10}
do
gcloud --impersonate-service-account=sa-landing@$LANDING_PROJECT_ID.iam.gserviceaccount.com pubsub topics publish projects/$LANDING_PROJECT_ID/topics/landing-1 --message="{\"name\": \"Lorenzo\", \"surname\": \"Caggioni\", \"timestamp\": \"$(date +%s)\"}"
done
```
if you want to check messages published, you can use the Transformation service account and read a message (message won't be acked and will stay in the subscription):
```bash
gcloud --impersonate-service-account=sa-transformation@$TRANSFORMATION_PROJECT_ID.iam.gserviceaccount.com pubsub subscriptions pull projects/$LANDING_PROJECT_ID/subscriptions/sub1
```
## Dataflow
Those steps should be done as transformation Service Account:
Let's than start a Dataflow streaming pipeline using a Google provided template using internal only IPs, the created network and subnetwork, the appropriate service account and requested parameters:
```bash
gcloud dataflow jobs run test_streaming01 \
--gcs-location gs://dataflow-templates/latest/PubSub_Subscription_to_BigQuery \
--project $TRANSFORMATION_PROJECT_ID \
--region europe-west3 \
--disable-public-ips \
--network transformation-vpc \
--subnetwork regions/europe-west3/subnetworks/transformation-subnet \
--staging-location gs://$TRANSFORMATION_PROJECT_ID-eu-temp \
--service-account-email sa-transformation@$TRANSFORMATION_PROJECT_ID.iam.gserviceaccount.com \
--parameters \
inputSubscription=projects/$LANDING_PROJECT_ID/subscriptions/sub1,\
outputTableSpec=$DWH_PROJECT_ID:bq_raw_dataset.person
```

26
examples/data-solutions/data-platform-foundations/03-pipeline/resource/raw_data.json
View File

@ -1,26 +0,0 @@
{
"schema": {
"fields": [
{
"mode": "NULLABLE",
"name": "name",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "surname",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "age",
"type": "INTEGER"
},
{
"mode": "NULLABLE",
"name": "boolean_val",
"type": "BOOLEAN"
}
]
}
}

161
examples/data-solutions/data-platform-foundations/04-transformation.tf Normal file
View File

@ -0,0 +1,161 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# tfdoc:file:description Trasformation project and VPC.
locals {
transf_subnet = (
local.use_shared_vpc
? var.network_config.subnet_self_links.orchestration
: values(module.transf-vpc.0.subnet_self_links)[0]
)
transf_vpc = (
local.use_shared_vpc
? var.network_config.network_self_link
: module.transf-vpc.0.self_link
)
}
module "transf-project" {
source = "../../../modules/project"
parent = var.folder_id
billing_account = var.billing_account_id
prefix = var.prefix
name = "trf"
group_iam = {
(local.groups.data-engineers) = [
"roles/bigquery.jobUser",
"roles/dataflow.admin",
]
}
iam = {
"roles/bigquery.jobUser" = [
module.transf-sa-bq-0.iam_email,
]
"roles/dataflow.admin" = [
module.orch-sa-cmp-0.iam_email,
]
"roles/dataflow.worker" = [
module.transf-sa-df-0.iam_email
]
"roles/storage.objectAdmin" = [
module.transf-sa-df-0.iam_email,
"serviceAccount:${module.transf-project.service_accounts.robots.dataflow}"
]
}
services = concat(var.project_services, [
"bigquery.googleapis.com",
"bigqueryreservation.googleapis.com",
"bigquerystorage.googleapis.com",
"cloudkms.googleapis.com",
"compute.googleapis.com",
"dataflow.googleapis.com",
"dlp.googleapis.com",
"pubsub.googleapis.com",
"servicenetworking.googleapis.com",
"storage.googleapis.com",
"storage-component.googleapis.com"
])
service_encryption_key_ids = {
dataflow = [try(local.service_encryption_keys.dataflow, null)]
storage = [try(local.service_encryption_keys.storage, null)]
}
shared_vpc_service_config = local.shared_vpc_project == null ? null : {
attach = true
host_project = local.shared_vpc_project
service_identity_iam = {}
}
}
# Cloud Storage
module "transf-sa-df-0" {
source = "../../../modules/iam-service-account"
project_id = module.transf-project.project_id
prefix = var.prefix
name = "trf-df-0"
display_name = "Data platform Dataflow transformation service account"
iam = {
"roles/iam.serviceAccountTokenCreator" = [
local.groups_iam.data-engineers,
module.orch-sa-cmp-0.iam_email
],
"roles/iam.serviceAccountUser" = [
module.orch-sa-cmp-0.iam_email
]
}
}
module "transf-cs-df-0" {
source = "../../../modules/gcs"
project_id = module.transf-project.project_id
prefix = var.prefix
name = "trf-cs-0"
location = var.location
storage_class = "MULTI_REGIONAL"
encryption_key = try(local.service_encryption_keys.storage, null)
}
# BigQuery
module "transf-sa-bq-0" {
source = "../../../modules/iam-service-account"
project_id = module.transf-project.project_id
prefix = var.prefix
name = "trf-bq-0"
display_name = "Data platform BigQuery transformation service account"
iam = {
"roles/iam.serviceAccountTokenCreator" = [
local.groups_iam.data-engineers,
module.orch-sa-cmp-0.iam_email
],
"roles/iam.serviceAccountUser" = [
module.orch-sa-cmp-0.iam_email
]
}
}
# internal VPC resources
module "transf-vpc" {
source = "../../../modules/net-vpc"
count = local.use_shared_vpc ? 0 : 1
project_id = module.transf-project.project_id
name = "${var.prefix}-default"
subnets = [
{
ip_cidr_range = "10.10.0.0/24"
name = "default"
region = var.region
secondary_ip_range = {}
}
]
}
module "transf-vpc-firewall" {
source = "../../../modules/net-vpc-firewall"
count = local.use_shared_vpc ? 0 : 1
project_id = module.transf-project.project_id
network = module.transf-vpc.0.name
admin_ranges = ["10.10.0.0/24"]
}
module "transf-nat" {
source = "../../../modules/net-cloudnat"
count = local.use_shared_vpc ? 0 : 1
project_id = module.transf-project.project_id
name = "${var.prefix}-default"
region = var.region
router_network = module.transf-vpc.0.name
}

228
examples/data-solutions/data-platform-foundations/05-datalake.tf Normal file
View File

@ -0,0 +1,228 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# tfdoc:file:description Datalake projects.
locals {
lake_group_iam = {
(local.groups.data-engineers) = [
"roles/bigquery.dataEditor",
"roles/storage.admin",
],
(local.groups.data-analysts) = [
"roles/bigquery.dataViewer",
"roles/bigquery.jobUser",
"roles/bigquery.user",
"roles/datacatalog.viewer",
"roles/datacatalog.tagTemplateViewer",
"roles/storage.objectViewer",
]
}
lake_plg_group_iam = {
(local.groups.data-engineers) = [
"roles/bigquery.dataEditor",
"roles/storage.admin",
],
(local.groups.data-analysts) = [
"roles/bigquery.dataEditor",
"roles/bigquery.jobUser",
"roles/bigquery.user",
"roles/datacatalog.viewer",
"roles/datacatalog.tagTemplateViewer",
"roles/storage.objectAdmin",
]
}
lake_0_iam = {
"roles/bigquery.dataEditor" = [
module.load-sa-df-0.iam_email,
module.transf-sa-df-0.iam_email,
module.transf-sa-bq-0.iam_email,
]
"roles/bigquery.jobUser" = [
module.load-sa-df-0.iam_email,
]
"roles/storage.objectCreator" = [
module.load-sa-df-0.iam_email,
]
}
lake_iam = {
"roles/bigquery.dataEditor" = [
module.transf-sa-df-0.iam_email,
module.transf-sa-bq-0.iam_email,
]
"roles/bigquery.jobUser" = [
module.transf-sa-bq-0.iam_email,
]
"roles/storage.objectCreator" = [
module.transf-sa-df-0.iam_email,
]
"roles/storage.objectViewer" = [
module.transf-sa-df-0.iam_email,
]
}
lake_services = concat(var.project_services, [
"bigquery.googleapis.com",
"bigqueryreservation.googleapis.com",
"bigquerystorage.googleapis.com",
"cloudkms.googleapis.com",
"compute.googleapis.com",
"dataflow.googleapis.com",
"pubsub.googleapis.com",
"servicenetworking.googleapis.com",
"storage.googleapis.com",
"storage-component.googleapis.com"
])
}
# Project
module "lake-0-project" {
source = "../../../modules/project"
parent = var.folder_id
billing_account = var.billing_account_id
prefix = var.prefix
name = "dtl-0"
group_iam = local.lake_group_iam
iam = local.lake_0_iam
services = local.lake_services
service_encryption_key_ids = {
bq = [try(local.service_encryption_keys.bq, null)]
storage = [try(local.service_encryption_keys.storage, null)]
}
}
module "lake-1-project" {
source = "../../../modules/project"
parent = var.folder_id
billing_account = var.billing_account_id
prefix = var.prefix
name = "dtl-1"
group_iam = local.lake_group_iam
iam = local.lake_iam
services = local.lake_services
service_encryption_key_ids = {
bq = [try(local.service_encryption_keys.bq, null)]
storage = [try(local.service_encryption_keys.storage, null)]
}
}
module "lake-2-project" {
source = "../../../modules/project"
parent = var.folder_id
billing_account = var.billing_account_id
prefix = var.prefix
name = "dtl-2"
group_iam = local.lake_group_iam
iam = local.lake_iam
services = local.lake_services
service_encryption_key_ids = {
bq = [try(local.service_encryption_keys.bq, null)]
storage = [try(local.service_encryption_keys.storage, null)]
}
}
module "lake-plg-project" {
source = "../../../modules/project"
parent = var.folder_id
billing_account = var.billing_account_id
prefix = var.prefix
name = "dtl-plg"
group_iam = local.lake_plg_group_iam
iam = {}
services = local.lake_services
service_encryption_key_ids = {
bq = [try(local.service_encryption_keys.bq, null)]
storage = [try(local.service_encryption_keys.storage, null)]
}
}
# Bigquery
module "lake-0-bq-0" {
source = "../../../modules/bigquery-dataset"
project_id = module.lake-0-project.project_id
id = "${replace(var.prefix, "-", "_")}_dtl_0_bq_0"
location = var.location
encryption_key = try(local.service_encryption_keys.bq, null)
}
module "lake-1-bq-0" {
source = "../../../modules/bigquery-dataset"
project_id = module.lake-1-project.project_id
id = "${replace(var.prefix, "-", "_")}_dtl_1_bq_0"
location = var.location
encryption_key = try(local.service_encryption_keys.bq, null)
}
module "lake-2-bq-0" {
source = "../../../modules/bigquery-dataset"
project_id = module.lake-2-project.project_id
id = "${replace(var.prefix, "-", "_")}_dtl_2_bq_0"
location = var.location
encryption_key = try(local.service_encryption_keys.bq, null)
}
module "lake-plg-bq-0" {
source = "../../../modules/bigquery-dataset"
project_id = module.lake-plg-project.project_id
id = "${replace(var.prefix, "-", "_")}_dtl_plg_bq_0"
location = var.location
encryption_key = try(local.service_encryption_keys.bq, null)
}
# Cloud storage
module "lake-0-cs-0" {
source = "../../../modules/gcs"
project_id = module.lake-0-project.project_id
prefix = var.prefix
name = "dtl-0-cs-0"
location = var.location
storage_class = "MULTI_REGIONAL"
encryption_key = try(local.service_encryption_keys.storage, null)
force_destroy = var.data_force_destroy
}
module "lake-1-cs-0" {
source = "../../../modules/gcs"
project_id = module.lake-1-project.project_id
prefix = var.prefix
name = "dtl-1-cs-0"
location = var.location
storage_class = "MULTI_REGIONAL"
encryption_key = try(local.service_encryption_keys.storage, null)
force_destroy = var.data_force_destroy
}
module "lake-2-cs-0" {
source = "../../../modules/gcs"
project_id = module.lake-2-project.project_id
prefix = var.prefix
name = "dtl-2-cs-0"
location = var.location
storage_class = "MULTI_REGIONAL"
encryption_key = try(local.service_encryption_keys.storage, null)
force_destroy = var.data_force_destroy
}
module "lake-plg-cs-0" {
source = "../../../modules/gcs"
project_id = module.lake-plg-project.project_id
prefix = var.prefix
name = "dtl-plg-cs-0"
location = var.location
storage_class = "MULTI_REGIONAL"
encryption_key = try(local.service_encryption_keys.storage, null)
force_destroy = var.data_force_destroy
}

83
examples/data-solutions/data-platform-foundations/06-common.tf Normal file
View File

@ -0,0 +1,83 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# tfdoc:file:description common project.
module "common-project" {
source = "../../../modules/project"
parent = var.folder_id
billing_account = var.billing_account_id
prefix = var.prefix
name = "cmn"
group_iam = {
(local.groups.data-engineers) = [
"roles/dlp.reader",
"roles/dlp.user",
"roles/dlp.estimatesAdmin",
]
(local.groups.data-security) = [
"roles/dlp.admin",
]
}
iam = {
"roles/dlp.user" = [
module.load-sa-df-0.iam_email,
module.transf-sa-df-0.iam_email
]
}
services = concat(var.project_services, [
"datacatalog.googleapis.com",
"dlp.googleapis.com",
])
}
# To create KMS keys in the common projet: uncomment this section and assigne key links accondingly in local.service_encryption_keys variable
# module "cmn-kms-0" {
# source = "../../../modules/kms"
# project_id = module.common-project.project_id
# keyring = {
# name = "${var.prefix}-kr-global",
# location = "global"
# }
# keys = {
# pubsub = null
# }
# }
# module "cmn-kms-1" {
# source = "../../../modules/kms"
# project_id = module.common-project.project_id
# keyring = {
# name = "${var.prefix}-kr-mregional",
# location = var.location
# }
# keys = {
# bq = null
# storage = null
# }
# }
# module "cmn-kms-2" {
# source = "../../../modules/kms"
# project_id = module.cmn-prj.project_id
# keyring = {
# name = "${var.prefix}-kr-regional",
# location = var.region
# }
# keys = {
# composer = null
# dataflow = null
# }
# }

22
examples/data-solutions/data-platform-foundations/01-environment/versions.tf → examples/data-solutions/data-platform-foundations/07-exposure.tf
View File

@ -12,18 +12,12 @@
# See the License for the specific language governing permissions and
# limitations under the License.
terraform {
required_version = ">= 1.0.0"
required_providers {
google = {
source = "hashicorp/google"
version = ">= 4.0.0"
}
google-beta = {
source = "hashicorp/google-beta"
version = ">= 4.0.0"
}
}
# tfdoc:file:description common project.
module "exp-project" {
source = "../../../modules/project"
parent = var.folder_id
billing_account = var.billing_account_id
prefix = var.prefix
name = "exp"
}

84
examples/data-solutions/data-platform-foundations/IAM.md Normal file
View File

@ -0,0 +1,84 @@
# IAM bindings reference
Legend: <code>+</code> additive, <code>•</code> conditional.
## Project <i>cmn</i>
| members | roles |
|---|---|
|<b>gcp-data-engineers</b><br><small><i>group</i></small>|[roles/dlp.estimatesAdmin](https://cloud.google.com/iam/docs/understanding-roles#dlp.estimatesAdmin) <br>[roles/dlp.reader](https://cloud.google.com/iam/docs/understanding-roles#dlp.reader) <br>[roles/dlp.user](https://cloud.google.com/iam/docs/understanding-roles#dlp.user) |
|<b>gcp-data-security</b><br><small><i>group</i></small>|[roles/dlp.admin](https://cloud.google.com/iam/docs/understanding-roles#dlp.admin) |
|<b>load-df-0</b><br><small><i>serviceAccount</i></small>|[roles/dlp.user](https://cloud.google.com/iam/docs/understanding-roles#dlp.user) |
|<b>trf-df-0</b><br><small><i>serviceAccount</i></small>|[roles/dlp.user](https://cloud.google.com/iam/docs/understanding-roles#dlp.user) |
## Project <i>dtl-0</i>
| members | roles |
|---|---|
|<b>gcp-data-analysts</b><br><small><i>group</i></small>|[roles/bigquery.dataViewer](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataViewer) <br>[roles/bigquery.jobUser](https://cloud.google.com/iam/docs/understanding-roles#bigquery.jobUser) <br>[roles/bigquery.user](https://cloud.google.com/iam/docs/understanding-roles#bigquery.user) <br>[roles/datacatalog.tagTemplateViewer](https://cloud.google.com/iam/docs/understanding-roles#datacatalog.tagTemplateViewer) <br>[roles/datacatalog.viewer](https://cloud.google.com/iam/docs/understanding-roles#datacatalog.viewer) <br>[roles/storage.objectViewer](https://cloud.google.com/iam/docs/understanding-roles#storage.objectViewer) |
|<b>gcp-data-engineers</b><br><small><i>group</i></small>|[roles/bigquery.dataEditor](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor) <br>[roles/storage.admin](https://cloud.google.com/iam/docs/understanding-roles#storage.admin) |
|<b>load-df-0</b><br><small><i>serviceAccount</i></small>|[roles/bigquery.jobUser](https://cloud.google.com/iam/docs/understanding-roles#bigquery.jobUser) <br>[roles/storage.objectCreator](https://cloud.google.com/iam/docs/understanding-roles#storage.objectCreator) |
|<b>trf-bq-0</b><br><small><i>serviceAccount</i></small>|[roles/bigquery.dataEditor](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor) |
|<b>trf-df-0</b><br><small><i>serviceAccount</i></small>|[roles/bigquery.dataEditor](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor) |
## Project <i>dtl-1</i>
| members | roles |
|---|---|
|<b>gcp-data-analysts</b><br><small><i>group</i></small>|[roles/bigquery.dataViewer](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataViewer) <br>[roles/bigquery.jobUser](https://cloud.google.com/iam/docs/understanding-roles#bigquery.jobUser) <br>[roles/bigquery.user](https://cloud.google.com/iam/docs/understanding-roles#bigquery.user) <br>[roles/datacatalog.tagTemplateViewer](https://cloud.google.com/iam/docs/understanding-roles#datacatalog.tagTemplateViewer) <br>[roles/datacatalog.viewer](https://cloud.google.com/iam/docs/understanding-roles#datacatalog.viewer) <br>[roles/storage.objectViewer](https://cloud.google.com/iam/docs/understanding-roles#storage.objectViewer) |
|<b>gcp-data-engineers</b><br><small><i>group</i></small>|[roles/bigquery.dataEditor](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor) <br>[roles/storage.admin](https://cloud.google.com/iam/docs/understanding-roles#storage.admin) |
|<b>trf-bq-0</b><br><small><i>serviceAccount</i></small>|[roles/bigquery.dataEditor](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor) <br>[roles/bigquery.jobUser](https://cloud.google.com/iam/docs/understanding-roles#bigquery.jobUser) |
|<b>trf-df-0</b><br><small><i>serviceAccount</i></small>|[roles/bigquery.dataEditor](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor) <br>[roles/storage.objectCreator](https://cloud.google.com/iam/docs/understanding-roles#storage.objectCreator) <br>[roles/storage.objectViewer](https://cloud.google.com/iam/docs/understanding-roles#storage.objectViewer) |
## Project <i>dtl-2</i>
| members | roles |
|---|---|
|<b>gcp-data-analysts</b><br><small><i>group</i></small>|[roles/bigquery.dataViewer](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataViewer) <br>[roles/bigquery.jobUser](https://cloud.google.com/iam/docs/understanding-roles#bigquery.jobUser) <br>[roles/bigquery.user](https://cloud.google.com/iam/docs/understanding-roles#bigquery.user) <br>[roles/datacatalog.tagTemplateViewer](https://cloud.google.com/iam/docs/understanding-roles#datacatalog.tagTemplateViewer) <br>[roles/datacatalog.viewer](https://cloud.google.com/iam/docs/understanding-roles#datacatalog.viewer) <br>[roles/storage.objectViewer](https://cloud.google.com/iam/docs/understanding-roles#storage.objectViewer) |
|<b>gcp-data-engineers</b><br><small><i>group</i></small>|[roles/bigquery.dataEditor](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor) <br>[roles/storage.admin](https://cloud.google.com/iam/docs/understanding-roles#storage.admin) |
|<b>trf-bq-0</b><br><small><i>serviceAccount</i></small>|[roles/bigquery.dataEditor](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor) <br>[roles/bigquery.jobUser](https://cloud.google.com/iam/docs/understanding-roles#bigquery.jobUser) |
|<b>trf-df-0</b><br><small><i>serviceAccount</i></small>|[roles/bigquery.dataEditor](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor) <br>[roles/storage.objectCreator](https://cloud.google.com/iam/docs/understanding-roles#storage.objectCreator) <br>[roles/storage.objectViewer](https://cloud.google.com/iam/docs/understanding-roles#storage.objectViewer) |
## Project <i>dtl-plg</i>
| members | roles |
|---|---|
|<b>gcp-data-analysts</b><br><small><i>group</i></small>|[roles/bigquery.dataEditor](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor) <br>[roles/bigquery.jobUser](https://cloud.google.com/iam/docs/understanding-roles#bigquery.jobUser) <br>[roles/bigquery.user](https://cloud.google.com/iam/docs/understanding-roles#bigquery.user) <br>[roles/datacatalog.tagTemplateViewer](https://cloud.google.com/iam/docs/understanding-roles#datacatalog.tagTemplateViewer) <br>[roles/datacatalog.viewer](https://cloud.google.com/iam/docs/understanding-roles#datacatalog.viewer) <br>[roles/storage.objectAdmin](https://cloud.google.com/iam/docs/understanding-roles#storage.objectAdmin) |
|<b>gcp-data-engineers</b><br><small><i>group</i></small>|[roles/bigquery.dataEditor](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor) <br>[roles/storage.admin](https://cloud.google.com/iam/docs/understanding-roles#storage.admin) |
## Project <i>lnd</i>
| members | roles |
|---|---|
|<b>gcp-data-engineers</b><br><small><i>group</i></small>|[roles/bigquery.dataEditor](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor) <br>[roles/pubsub.editor](https://cloud.google.com/iam/docs/understanding-roles#pubsub.editor) <br>[roles/storage.admin](https://cloud.google.com/iam/docs/understanding-roles#storage.admin) |
|<b>lnd-bq-0</b><br><small><i>serviceAccount</i></small>|[roles/bigquery.dataEditor](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor) |
|<b>lnd-cs-0</b><br><small><i>serviceAccount</i></small>|[roles/storage.objectCreator](https://cloud.google.com/iam/docs/understanding-roles#storage.objectCreator) |
|<b>lnd-ps-0</b><br><small><i>serviceAccount</i></small>|[roles/pubsub.publisher](https://cloud.google.com/iam/docs/understanding-roles#pubsub.publisher) |
|<b>load-df-0</b><br><small><i>serviceAccount</i></small>|[roles/bigquery.user](https://cloud.google.com/iam/docs/understanding-roles#bigquery.user) <br>[roles/pubsub.subscriber](https://cloud.google.com/iam/docs/understanding-roles#pubsub.subscriber) <br>[roles/storage.admin](https://cloud.google.com/iam/docs/understanding-roles#storage.admin) <br>[roles/storage.objectAdmin](https://cloud.google.com/iam/docs/understanding-roles#storage.objectAdmin) |
|<b>orc-cmp-0</b><br><small><i>serviceAccount</i></small>|[roles/pubsub.subscriber](https://cloud.google.com/iam/docs/understanding-roles#pubsub.subscriber) <br>[roles/storage.objectViewer](https://cloud.google.com/iam/docs/understanding-roles#storage.objectViewer) |
## Project <i>lod</i>
| members | roles |
|---|---|
|<b>gcp-data-engineers</b><br><small><i>group</i></small>|[roles/compute.viewer](https://cloud.google.com/iam/docs/understanding-roles#compute.viewer) <br>[roles/dataflow.admin](https://cloud.google.com/iam/docs/understanding-roles#dataflow.admin) <br>[roles/dataflow.developer](https://cloud.google.com/iam/docs/understanding-roles#dataflow.developer) <br>[roles/viewer](https://cloud.google.com/iam/docs/understanding-roles#viewer) |
|<b>load-df-0</b><br><small><i>serviceAccount</i></small>|[roles/bigquery.jobUser](https://cloud.google.com/iam/docs/understanding-roles#bigquery.jobUser) <br>[roles/dataflow.admin](https://cloud.google.com/iam/docs/understanding-roles#dataflow.admin) <br>[roles/dataflow.worker](https://cloud.google.com/iam/docs/understanding-roles#dataflow.worker) <br>[roles/storage.objectAdmin](https://cloud.google.com/iam/docs/understanding-roles#storage.objectAdmin) |
|<b>orc-cmp-0</b><br><small><i>serviceAccount</i></small>|[roles/dataflow.admin](https://cloud.google.com/iam/docs/understanding-roles#dataflow.admin) |
## Project <i>orc</i>
| members | roles |
|---|---|
|<b>gcp-data-engineers</b><br><small><i>group</i></small>|[roles/bigquery.dataEditor](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor) <br>[roles/bigquery.jobUser](https://cloud.google.com/iam/docs/understanding-roles#bigquery.jobUser) <br>[roles/cloudbuild.builds.editor](https://cloud.google.com/iam/docs/understanding-roles#cloudbuild.builds.editor) <br>[roles/composer.admin](https://cloud.google.com/iam/docs/understanding-roles#composer.admin) <br>[roles/composer.environmentAndStorageObjectAdmin](https://cloud.google.com/iam/docs/understanding-roles#composer.environmentAndStorageObjectAdmin) <br>[roles/compute.networkUser](https://cloud.google.com/iam/docs/understanding-roles#compute.networkUser) <br>[roles/iam.serviceAccountUser](https://cloud.google.com/iam/docs/understanding-roles#iam.serviceAccountUser) <br>[roles/iap.httpsResourceAccessor](https://cloud.google.com/iam/docs/understanding-roles#iap.httpsResourceAccessor) <br>[roles/storage.admin](https://cloud.google.com/iam/docs/understanding-roles#storage.admin) <br>[roles/storage.objectAdmin](https://cloud.google.com/iam/docs/understanding-roles#storage.objectAdmin) |
|<b>load-df-0</b><br><small><i>serviceAccount</i></small>|[roles/bigquery.dataEditor](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor) <br>[roles/storage.objectViewer](https://cloud.google.com/iam/docs/understanding-roles#storage.objectViewer) |
|<b>orc-cmp-0</b><br><small><i>serviceAccount</i></small>|[roles/bigquery.jobUser](https://cloud.google.com/iam/docs/understanding-roles#bigquery.jobUser) <br>[roles/composer.worker](https://cloud.google.com/iam/docs/understanding-roles#composer.worker) <br>[roles/iam.serviceAccountUser](https://cloud.google.com/iam/docs/understanding-roles#iam.serviceAccountUser) <br>[roles/storage.objectAdmin](https://cloud.google.com/iam/docs/understanding-roles#storage.objectAdmin) |
|<b>trf-df-0</b><br><small><i>serviceAccount</i></small>|[roles/bigquery.dataEditor](https://cloud.google.com/iam/docs/understanding-roles#bigquery.dataEditor) |
## Project <i>trf</i>
| members | roles |
|---|---|
|<b>gcp-data-engineers</b><br><small><i>group</i></small>|[roles/bigquery.jobUser](https://cloud.google.com/iam/docs/understanding-roles#bigquery.jobUser) <br>[roles/dataflow.admin](https://cloud.google.com/iam/docs/understanding-roles#dataflow.admin) |
|<b>orc-cmp-0</b><br><small><i>serviceAccount</i></small>|[roles/dataflow.admin](https://cloud.google.com/iam/docs/understanding-roles#dataflow.admin) |
|<b>trf-bq-0</b><br><small><i>serviceAccount</i></small>|[roles/bigquery.jobUser](https://cloud.google.com/iam/docs/understanding-roles#bigquery.jobUser) |
|<b>trf-df-0</b><br><small><i>serviceAccount</i></small>|[roles/dataflow.worker](https://cloud.google.com/iam/docs/understanding-roles#dataflow.worker) <br>[roles/storage.objectAdmin](https://cloud.google.com/iam/docs/understanding-roles#storage.objectAdmin) |

294
examples/data-solutions/data-platform-foundations/README.md
View File

@ -1,61 +1,265 @@
# Data Foundation Platform
# Data Platform
The goal of this example is to Build a robust and flexible Data Foundation on GCP, providing opinionated defaults while still allowing customers to quickly and reliably build and scale out additional data pipelines.
This module implements an opinionated Data Platform Architecture that creates and setup projects and related resources that compose an end-to-end data environment.
The example is composed of three separate provisioning workflows, which are deisgned to be plugged together and create end to end Data Foundations, that support multiple data pipelines on top.
The code is intentionally simple, as it's intended to provide a generic initial setup and then allow easy customizations to complete the implementation of the intended design.
1. **[Environment Setup](./01-environment/)**
*(once per environment)*
* projects
* VPC configuration
* Composer environment and identity
* shared buckets and datasets
1. **[Data Source Setup](./02-resources)**
*(once per data source)*
* landing and archive bucket
* internal and external identities
* domain specific datasets
1. **[Pipeline Setup](./03-pipeline)**
*(once per pipeline)*
* pipeline-specific tables and views
* pipeline code
* Composer DAG
The following diagram is a high-level reference of the resources created and managed here:
The resulting GCP architecture is outlined in this diagram
![Target architecture](./02-resources/diagram.png)
![Data Platform architecture overview](./images/overview_diagram.png "Data Platform architecture overview")
A demo pipeline is also part of this example: it can be built and run on top of the foundational infrastructure to quickly verify or test the setup.
A demo Airflow pipeline is also part of this example: it can be built and run on top of the foundational infrastructure to verify or test the setup quickly.
## Prerequisites
## Design overview and choices
In order to bring up this example, you will need
Despite its simplicity, this stage implements the basics of a design that we've seen working well for various customers.
The approach adapts to different high-level requirements:
- boundaries for each step
- clearly defined actors
- least privilege principle
- rely on service account impersonation
The code in this example doesn't address Organization-level configurations (Organization policy, VPC-SC, centralized logs). We expect those elements to be managed by automation stages external to this script like those in [FAST](../../../fast).
### Project structure
The Data Platform is designed to rely on several projects, one project per data stage. The stages identified are:
- landing
- load
- data lake
- orchestration
- transformation
- exposure
This separation into projects allows adhering to the least-privilege principle by using project-level roles.
The script will create the following projects:
- **Landing** Used to store temporary data. Data is pushed to Cloud Storage, BigQuery, or Cloud PubSub. Resources are configured with a customizable lifecycle policy.
- **Load** Used to load data from landing to data lake. The load is made with minimal to zero transformation logic (mainly `cast`). Anonymization or tokenization of Personally Identifiable Information (PII) can be implemented here or in the transformation stage, depending on your requirements. The use of [Cloud Dataflow templates](https://cloud.google.com/dataflow/docs/concepts/dataflow-templates) is recommended.
- **Data Lake** Several projects distributed across 3 separate layers, to host progressively processed and refined data:
- **L0 - Raw data** Structured Data, stored in relevant formats: structured data stored in BigQuery, unstructured data stored on Cloud Storage with additional metadata stored in BigQuery (for example pictures stored in Cloud Storage and analysis of the images for Cloud Vision API stored in BigQuery).
- **L1 - Cleansed, aggregated and standardized data**
- **L2 - Curated layer**
- **Playground** Temporary tables that Data Analyst may use to perform R&D on data available in other Data Lake layers.
- **Orchestration** Used to host Cloud Composer, which orchestrates all tasks that move data across layers.
- **Transformation** Used to move data between Data Lake layers. We strongly suggest relying on BigQuery Engine to perform the transformations. If BigQuery doesn't have the features needed to perform your transformations, you can use Cloud Dataflow with [Cloud Dataflow templates](https://cloud.google.com/dataflow/docs/concepts/dataflow-templates). This stage can also optionally anonymize or tokenize PII.
- **Exposure** Used to host resources that share processed data with external systems. Depending on the access pattern, data can be presented via Cloud SQL, BigQuery, or Bigtable. For BigQuery data, we strongly suggest relying on [Authorized views](https://cloud.google.com/bigquery/docs/authorized-views).
### Roles
We assign roles on resources at the project level, granting the appropriate roles via groups (humans) and service accounts (services and applications) according to best practices.
### Service accounts
Service account creation follows the least privilege principle, performing a single task which requires access to a defined set of resources. The table below shows a high level overview of roles for each service account on each data layer, using `READ` or `WRITE` access patterns for simplicity. For detailed roles please refer to the code.
|Service Account|Landing|DataLake L0|DataLake L1|DataLake L2|
|-|:-:|:-:|:-:|:-:|
|`landing-sa`|`WRITE`|-|-|-|
|`load-sa`|`READ`|`READ`/`WRITE`|-|-|
|`transformation-sa`|-|`READ`/`WRITE`|`READ`/`WRITE`|`READ`/`WRITE`|
|`orchestration-sa`|-|-|-|-|
A full reference of IAM roles managed by the Data Platform [is available here](./IAM.md).
Using of service account keys within a data pipeline exposes to several security risks deriving from a credentials leak. This example shows how to leverage impersonation to avoid the need of creating keys.
### User groups
User groups provide a stable frame of reference that allows decoupling the final set of permissions from the stage where entities and resources are created, and their IAM bindings defined.
We use three groups to control access to resources:
- *Data Engineers* They handle and run the Data Hub, with read access to all resources in order to troubleshoot possible issues with pipelines. This team can also impersonate any service account.
- *Data Analysts*. They perform analysis on datasets, with read access to the data lake L2 project, and BigQuery READ/WRITE access to the playground project.
- *Data Security*:. They handle security configurations related to the Data Hub. This team has admin access to the common project to configure Cloud DLP templates or Data Catalog policy tags.
The table below shows a high level overview of roles for each group on each project, using `READ`, `WRITE` and `ADMIN` access patterns for simplicity. For detailed roles please refer to the code.
|Group|Landing|Load|Transformation|Data Lake L0|Data Lake L1|Data Lake L2|Data Lake Playground|Orchestration|Common|
|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|Data Engineers|`ADMIN`|`ADMIN`|`ADMIN`|`ADMIN`|`ADMIN`|`ADMIN`|`ADMIN`|`ADMIN`|`ADMIN`|
|Data Analysts|-|-|-|-|-|`READ`|`READ`/`WRITE`|-|-|
|Data Security|-|-|-|-|-|-|-|-|`ADMIN`|
You can configure groups via the `groups` variable.
### Virtual Private Cloud (VPC) design
As is often the case in real-world configurations, this example accepts as input an existing [Shared-VPC](https://cloud.google.com/vpc/docs/shared-vpc) via the `network_config` variable. Make sure that the GKE API (`container.googleapis.com`) is enabled in the VPC host project.
If the `network_config` variable is not provided, one VPC will be created in each project that supports network resources (load, transformation and orchestration).
### IP ranges and subnetting
To deploy this example with self-managed VPCs you need the following ranges:
- one /24 for the load project VPC subnet used for Cloud Dataflow workers
- one /24 for the transformation VPC subnet used for Cloud Dataflow workers
- one /24 range for the orchestration VPC subnet used for Composer workers
- one /22 and one /24 ranges for the secondary ranges associated with the orchestration VPC subnet
If you are using Shared VPC, you need one subnet with one /22 and one /24 secondary range defined for Composer pods and services.
In both VPC scenarios, you also need these ranges for Composer:
- one /24 for Cloud SQL
- one /28 for the GKE control plane
- one /28 for the web server
### Resource naming conventions
Resources follow the naming convention described below.
- `prefix-layer` for projects
- `prefix-layer-prduct` for resources
- `prefix-layer[2]-gcp-product[2]-counter` for services and service accounts
### Encryption
We suggest a centralized approach to key management, where Organization Security is the only team that can access encryption material, and keyrings and keys are managed in a project external to the Data Platform.
![Centralized Cloud Key Management high-level diagram](./images/kms_diagram.png "Centralized Cloud Key Management high-level diagram")
To configure the use of Cloud KMS on resources, you have to specify the key id on the `service_encryption_keys` variable. Key locations should match resource locations. Example:
```hcl
service_encryption_keys = {
bq = "KEY_URL_MULTIREGIONAL"
composer = "KEY_URL_REGIONAL"
dataflow = "KEY_URL_REGIONAL"
storage = "KEY_URL_MULTIREGIONAL"
pubsub = "KEY_URL_MULTIREGIONAL"
}
```
This step is optional and depends on customer policies and security best practices.
## Data Anonymization
We suggest using Cloud Data Loss Prevention to identify/mask/tokenize your confidential data.
While implementing a Data Loss Prevention strategy is out of scope for this example, we enable the service in two different projects so that [Cloud Data Loss Prevention templates](https://cloud.google.com/dlp/docs/concepts-templates) can be configured in one of two ways:
- during the ingestion phase, from Dataflow
- during the transformation phase, from [BigQuery](https://cloud.google.com/bigquery/docs/scan-with-dlp) or [Cloud Dataflow](https://cloud.google.com/architecture/running-automated-dataflow-pipeline-de-identify-pii-dataset)
Cloud Data Loss Prevention resources and templates should be stored in the security project:
![Centralized Cloud Data Loss Prevention high-level diagram](./images/dlp_diagram.png "Centralized Cloud Data Loss Prevention high-level diagram")
You can find more details and best practices on using DLP to De-identification and re-identification of PII in large-scale datasets in the [GCP documentation](https://cloud.google.com/architecture/de-identification-re-identification-pii-using-cloud-dlp).
## How to run this script
To deploy this example on your GCP organization, you will need
- a folder or organization where new projects will be created
- a billing account that will be associated to new projects
- an identity (user or service account) with owner permissions on the folder or org, and billing user permissions on the billing account
- a billing account that will be associated with the new projects
## Bringing up the platform
The Data Platform is meant to be executed by a Service Account (or a regular user) having this minimal set of permission:
[![Open in Cloud Shell](https://gstatic.com/cloudssh/images/open-btn.svg)](https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2Fterraform-google-modules%2Fcloud-foundation-fabric.git&cloudshell_open_in_editor=README.md&cloudshell_workspace=examples%2Fdata-solutions%2Fdata-platform-foundations)
- **Billing account**
- `roles/billing.user`
- **Folder level**:
- `roles/resourcemanager.folderAdmin`
- `roles/resourcemanager.projectCreator`
- **KMS Keys** (If CMEK encryption in use):
- `roles/cloudkms.admin` or a custom role with `cloudkms.cryptoKeys.getIamPolicy`, `cloudkms.cryptoKeys.list`, `cloudkms.cryptoKeys.setIamPolicy` permissions
- **Shared VPC host project** (if configured):\
- `roles/compute.xpnAdmin` on the host project folder or org
- `roles/resourcemanager.projectIamAdmin` on the host project, either with no conditions or with a condition allowing [delegated role grants](https://medium.com/google-cloud/managing-gcp-service-usage-through-delegated-role-grants-a843610f2226#:~:text=Delegated%20role%20grants%20is%20a,setIamPolicy%20permission%20on%20a%20resource.) for `roles/compute.networkUser`, `roles/composer.sharedVpcAgent`, `roles/container.hostServiceAgentUser`
The end-to-end example is composed of 2 foundational, and 1 optional steps:
## Variable configuration
1. [Environment setup](./01-environment/)
1. [Data source setup](./02-resources/)
1. (Optional) [Pipeline setup](./03-pipeline/)
There are three sets of variables you will need to fill in:
The environment setup is designed to manage a single environment. Various strategies like workspaces, branching, or even separate clones can be used to support multiple environments.
```hcl
billing_account_id = "111111-222222-333333"
older_id = "folders/123456789012"
organization_domain = "domain.com"
prefix = "myco"
```
## TODO
For more fine details check variables on [`variables.tf`](./variables.tf) and update according to the desired configuration. Remember to create team groups described [below](#groups).
| Description | Priority (1:High - 5:Low ) | Status | Remarks |
|-------------|----------|:------:|---------|
| DLP best practices in the pipeline | 2 | Not Started | |
| Add Composer with a static DAG running the example | 3 | Not Started | |
| Integrate [CI/CD composer data processing workflow framework](https://github.com/jaketf/ci-cd-for-data-processing-workflow) | 3 | Not Started | |
| Schema changes, how to handle | 4 | Not Started | |
| Data lineage | 4 | Not Started | |
| Data quality checks | 4 | Not Started | |
| Shared-VPC | 5 | Not Started | |
| Logging & monitoring | TBD | Not Started | |
| Orcestration for ingestion pipeline (just in the readme) | TBD | Not Started | |
Once the configuration is complete, run the project factory by running
```bash
terraform init
terraform apply
```
## Customizations
### Create Cloud Key Management keys as part of the Data Platform
To create Cloud Key Management keys in the Data Platform you can uncomment the Cloud Key Management resources configured in the [`06-common.tf`](./06-common.tf) file and update Cloud Key Management keys pointers on `local.service_encryption_keys.*` to the local resource created.
### Assign roles at BQ Dataset level
To handle multiple groups of `data-analysts` accessing the same Data Lake layer projects but only to the dataset belonging to a specific group, you may want to assign roles at BigQuery dataset level instead of at project-level.
To do this, you need to remove IAM binging at project-level for the `data-analysts` group and give roles at BigQuery dataset level using the `iam` variable on `bigquery-dataset` modules.
## Demo pipeline
The application layer is out of scope of this script, but as a demo, it is provided with a Cloud Composer DAG to mode data from the `landing` area to the `DataLake L2` dataset.
Just follow the commands you find in the `demo_commands` Terraform output, go in the Cloud Composer UI and run the `data_pipeline_dag`.
Description of commands:
- 01: copy sample data to a `landing` Cloud Storage bucket impersonating the `load` service account.
- 02: copy sample data structure definition in the `orchestration` Cloud Storage bucket impersonating the `orchestration` service account.
- 03: copy the Cloud Composer DAG to the Cloud Composer Storage bucket impersonating the `orchestration` service account.
- 04: Open the Cloud Composer Airflow UI and run the imported DAG.
- 05: Run the BigQuery query to see results.
<!-- BEGIN TFDOC -->
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [billing_account_id](variables.tf#L17) | Billing account id. | <code>string</code> | ✓ | |
| [folder_id](variables.tf#L42) | Folder to be used for the networking resources in folders/nnnn format. | <code>string</code> | ✓ | |
| [organization_domain](variables.tf#L86) | Organization domain. | <code>string</code> | ✓ | |
| [prefix](variables.tf#L91) | Unique prefix used for resource names. | <code>string</code> | ✓ | |
| [composer_config](variables.tf#L22) | Cloud Composer config. | <code title="object&#40;&#123;&#10; node_count &#61; number&#10; airflow_version &#61; string&#10; env_variables &#61; map&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; node_count &#61; 3&#10; airflow_version &#61; &#34;composer-1.17.5-airflow-2.1.4&#34;&#10; env_variables &#61; &#123;&#125;&#10;&#125;">&#123;&#8230;&#125;</code> |
| [data_force_destroy](variables.tf#L36) | Flag to set 'force_destroy' on data services like BiguQery or Cloud Storage. | <code>bool</code> | | <code>false</code> |
| [groups](variables.tf#L53) | User groups. | <code>map&#40;string&#41;</code> | | <code title="&#123;&#10; data-analysts &#61; &#34;gcp-data-analysts&#34;&#10; data-engineers &#61; &#34;gcp-data-engineers&#34;&#10; data-security &#61; &#34;gcp-data-security&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
| [location](variables.tf#L47) | Location used for multi-regional resources. | <code>string</code> | | <code>&#34;eu&#34;</code> |
| [network_config](variables.tf#L63) | Shared VPC network configurations to use. If null networks will be created in projects with preconfigured values. | <code title="object&#40;&#123;&#10; host_project &#61; string&#10; network_self_link &#61; string&#10; subnet_self_links &#61; object&#40;&#123;&#10; load &#61; string&#10; transformation &#61; string&#10; orchestration &#61; string&#10; &#125;&#41;&#10; composer_ip_ranges &#61; object&#40;&#123;&#10; cloudsql &#61; string&#10; gke_master &#61; string&#10; web_server &#61; string&#10; &#125;&#41;&#10; composer_secondary_ranges &#61; object&#40;&#123;&#10; pods &#61; string&#10; services &#61; string&#10; &#125;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [project_services](variables.tf#L96) | List of core services enabled on all projects. | <code>list&#40;string&#41;</code> | | <code title="&#91;&#10; &#34;cloudresourcemanager.googleapis.com&#34;,&#10; &#34;iam.googleapis.com&#34;,&#10; &#34;serviceusage.googleapis.com&#34;,&#10; &#34;stackdriver.googleapis.com&#34;&#10;&#93;">&#91;&#8230;&#93;</code> |
| [region](variables.tf#L107) | Region used for regional resources. | <code>string</code> | | <code>&#34;europe-west1&#34;</code> |
## Outputs
| name | description | sensitive |
|---|---|:---:|
| [bigquery-datasets](outputs.tf#L17) | BigQuery datasets. | |
| [demo_commands](outputs.tf#L93) | Demo commands. | |
| [gcs-buckets](outputs.tf#L28) | GCS buckets. | |
| [kms_keys](outputs.tf#L42) | Cloud MKS keys. | |
| [projects](outputs.tf#L47) | GCP Projects informations. | |
| [vpc_network](outputs.tf#L75) | VPC network. | |
| [vpc_subnet](outputs.tf#L84) | VPC subnetworks. | |
<!-- END TFDOC -->
## TODOs
Features to add in future releases:
- Add support for Column level access on BigQuery
- Add example templates for Data Catalog
- Add example on how to use Cloud Data Loss Prevention
- Add solution to handle Tables, Views, and Authorized Views lifecycle
- Add solution to handle Metadata lifecycle
## To Test/Fix
- Composer require "Require OS Login" not enforced
- External Shared-VPC

25
examples/data-solutions/data-platform-foundations/02-resources/versions.tf → examples/data-solutions/data-platform-foundations/backend.tf.sample
View File

@ -12,18 +12,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# The `impersonate_service_account` option require the identity launching terraform
# role `roles/iam.serviceAccountTokenCreator` on the Service Account specified.
terraform {
required_version = ">= 1.0.0"
required_providers {
google = {
source = "hashicorp/google"
version = ">= 4.0.0"
}
google-beta = {
source = "hashicorp/google-beta"
version = ">= 4.0.0"
}
backend "gcs" {
bucket = "BUCKET_NAME"
prefix = "PREFIX"
impersonate_service_account = "SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com"
}
}
provider "google" {
impersonate_service_account = "SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com"
}
provider "google-beta" {
impersonate_service_account = "SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com"
}

3
examples/data-solutions/data-platform-foundations/demo/README.md Normal file
View File

@ -0,0 +1,3 @@
# Data ingestion Demo
In this folder you can find an example to ingest data on the `data platfoem` instantiated in [here](../). See details in the [README.m](../#demo-pipeline) to run the demo.

50
examples/data-solutions/data-platform-foundations/demo/data/customer_purchase.json Normal file
View File

@ -0,0 +1,50 @@
[
{
"mode": "REQUIRED",
"name": "id",
"type": "INTEGER",
"description": "ID"
},
{
"mode": "REQUIRED",
"name": "customer_id",
"type": "INTEGER",
"description": "ID"
},
{
"mode": "REQUIRED",
"name": "purchase_id",
"type": "INTEGER",
"description": "ID"
},
{
"mode": "REQUIRED",
"name": "customer_name",
"type": "STRING",
"description": "Name"
},
{
"mode": "REQUIRED",
"name": "customer_surname",
"type": "STRING",
"description": "Surname"
},
{
"mode": "REQUIRED",
"name": "purchase_item",
"type": "STRING",
"description": "Item Name"
},
{
"mode": "REQUIRED",
"name": "price",
"type": "FLOAT",
"description": "Item Price"
},
{
"mode": "REQUIRED",
"name": "purchase_timestamp",
"type": "TIMESTAMP",
"description": "Timestamp"
}
]

12
examples/data-solutions/data-platform-foundations/demo/data/customers.csv Normal file
View File

@ -0,0 +1,12 @@
1,Name1,Surname1,1636972001
2,Name2,Surname2,1636972002
3,Name3,Surname3,1636972003
4,Name4,Surname4,1636972004
5,Name5,Surname5,1636972005
6,Name6,Surname6,1636972006
7,Name7,Surname7,1636972007
8,Name8,Surname8,1636972008
9,Name9,Surname9,1636972009
10,Name11,Surname11,1636972010
11,Name12,Surname12,1636972011
12,Name13,Surname13,1636972012
1 1 Name1 Surname1 1636972001
2 2 Name2 Surname2 1636972002
3 3 Name3 Surname3 1636972003
4 4 Name4 Surname4 1636972004
5 5 Name5 Surname5 1636972005
6 6 Name6 Surname6 1636972006
7 7 Name7 Surname7 1636972007
8 8 Name8 Surname8 1636972008
9 9 Name9 Surname9 1636972009
10 10 Name11 Surname11 1636972010
11 11 Name12 Surname12 1636972011
12 12 Name13 Surname13 1636972012

26
examples/data-solutions/data-platform-foundations/demo/data/customers.json Normal file
View File

@ -0,0 +1,26 @@
[
{
"mode": "REQUIRED",
"name": "id",
"type": "INTEGER",
"description": "ID"
},
{
"mode": "REQUIRED",
"name": "name",
"type": "STRING",
"description": "Name"
},
{
"mode": "REQUIRED",
"name": "surname",
"type": "STRING",
"description": "Surname"
},
{
"mode": "REQUIRED",
"name": "timestamp",
"type": "TIMESTAMP",
"description": "Timestamp"
}
]

28
examples/data-solutions/data-platform-foundations/demo/data/customers_schema.json Normal file
View File

@ -0,0 +1,28 @@
{
"BigQuery Schema": [
{
"mode": "REQUIRED",
"name": "id",
"type": "INTEGER",
"description": "ID"
},
{
"mode": "REQUIRED",
"name": "name",
"type": "STRING",
"description": "Name"
},
{
"mode": "REQUIRED",
"name": "surname",
"type": "STRING",
"description": "Surname"
},
{
"mode": "REQUIRED",
"name": "timestamp",
"type": "TIMESTAMP",
"description": "Timestamp"
}
]
}

12
examples/data-solutions/data-platform-foundations/demo/data/customers_udf.js Normal file
View File

@ -0,0 +1,12 @@
function transform(line) {
var values = line.split(',');
var obj = new Object();
obj.id = values[0]
obj.name = values[1];
obj.surname = values[2];
obj.timestamp = values[3];
var jsonString = JSON.stringify(obj);
return jsonString;
}

20
examples/data-solutions/data-platform-foundations/demo/data/purchases.csv Normal file
View File

@ -0,0 +1,20 @@
1,1,Car1,5000,1636972012
1,1,Car1,7000,1636972045
1,2,Car1,6000,1636972088
1,2,Car1,8000,16369720099
1,3,Car1,10000,1636972102
1,3,Car1,50000,1636972180
1,4,Car1,13000,1636972260
1,4,Car1,5000,1636972302
1,5,Car1,2000,1636972408
1,1,Car1,77000,1636972501
1,1,Car1,64000,1636975001
1,8,Car1,2000,1636976001
1,9,Car1,4000,1636977001
1,10,Car1,18000,1636982001
1,11,Car1,21000,1636992001
1,11,Car1,33000,1636932001
1,11,Car1,37000,1636872001
1,11,Car1,26000,1636772001
1,12,Car1,22000,1636672001
1,4,Car1,11000,1636952001
1 1 1 Car1 5000 1636972012
2 1 1 Car1 7000 1636972045
3 1 2 Car1 6000 1636972088
4 1 2 Car1 8000 16369720099
5 1 3 Car1 10000 1636972102
6 1 3 Car1 50000 1636972180
7 1 4 Car1 13000 1636972260
8 1 4 Car1 5000 1636972302
9 1 5 Car1 2000 1636972408
10 1 1 Car1 77000 1636972501
11 1 1 Car1 64000 1636975001
12 1 8 Car1 2000 1636976001
13 1 9 Car1 4000 1636977001
14 1 10 Car1 18000 1636982001
15 1 11 Car1 21000 1636992001
16 1 11 Car1 33000 1636932001
17 1 11 Car1 37000 1636872001
18 1 11 Car1 26000 1636772001
19 1 12 Car1 22000 1636672001
20 1 4 Car1 11000 1636952001

32
examples/data-solutions/data-platform-foundations/demo/data/purchases.json Normal file
View File

@ -0,0 +1,32 @@
[
{
"mode": "REQUIRED",
"name": "id",
"type": "INTEGER",
"description": "ID"
},
{
"mode": "REQUIRED",
"name": "customer_id",
"type": "INTEGER",
"description": "ID"
},
{
"mode": "REQUIRED",
"name": "item",
"type": "STRING",
"description": "Item Name"
},
{
"mode": "REQUIRED",
"name": "price",
"type": "FLOAT",
"description": "Item Price"
},
{
"mode": "REQUIRED",
"name": "timestamp",
"type": "TIMESTAMP",
"description": "Timestamp"
}
]

34
examples/data-solutions/data-platform-foundations/demo/data/purchases_schema.json Normal file
View File

@ -0,0 +1,34 @@
{
"BigQuery Schema": [
{
"mode": "REQUIRED",
"name": "id",
"type": "INTEGER",
"description": "ID"
},
{
"mode": "REQUIRED",
"name": "customer_id",
"type": "INTEGER",
"description": "ID"
},
{
"mode": "REQUIRED",
"name": "item",
"type": "STRING",
"description": "Item Name"
},
{
"mode": "REQUIRED",
"name": "price",
"type": "FLOAT",
"description": "Item Price"
},
{
"mode": "REQUIRED",
"name": "timestamp",
"type": "TIMESTAMP",
"description": "Timestamp"
}
]
}

13
examples/data-solutions/data-platform-foundations/demo/data/purchases_udf.js Normal file
View File

@ -0,0 +1,13 @@
function transform(line) {
var values = line.split(',');
var obj = new Object();
obj.id = values[0];
obj.customer_id = values[1];
obj.item = values[2];
obj.price = values[3];
obj.timestamp = values[4];
var jsonString = JSON.stringify(obj);
return jsonString;
}

202
examples/data-solutions/data-platform-foundations/demo/datapipeline.py Normal file
View File

@ -0,0 +1,202 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# --------------------------------------------------------------------------------
# Load The Dependencies
# --------------------------------------------------------------------------------
import csv
import datetime
import io
import logging
import os
from airflow import models
from airflow.contrib.operators.dataflow_operator import DataflowTemplateOperator
from airflow.operators import dummy
from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
# --------------------------------------------------------------------------------
# Set variables
# ------------------------------------------------------------
BQ_LOCATION = os.environ.get("BQ_LOCATION")
DTL_L0_PRJ = os.environ.get("DTL_L0_PRJ")
DTL_L0_BQ_DATASET = os.environ.get("DTL_L0_BQ_DATASET")
DTL_L0_GCS = os.environ.get("DTL_L0_GCS")
DTL_L1_PRJ = os.environ.get("DTL_L1_PRJ")
DTL_L1_BQ_DATASET = os.environ.get("DTL_L1_BQ_DATASET")
DTL_L1_GCS = os.environ.get("DTL_L1_GCS")
DTL_L2_PRJ = os.environ.get("DTL_L2_PRJ")
DTL_L2_BQ_DATASET = os.environ.get("DTL_L2_BQ_DATASET")
DTL_L2_GCS = os.environ.get("DTL_L2_GCS")
DTL_PLG_PRJ = os.environ.get("DTL_PLG_PRJ")
DTL_PLG_BQ_DATASET = os.environ.get("DTL_PLG_BQ_DATASET")
DTL_PLG_GCS = os.environ.get("DTL_PLG_GCS")
GCP_REGION = os.environ.get("GCP_REGION")
LND_PRJ = os.environ.get("LND_PRJ")
LND_BQ = os.environ.get("LND_BQ")
LND_GCS = os.environ.get("LND_GCS")
LND_PS = os.environ.get("LND_PS")
LOD_PRJ = os.environ.get("LOD_PRJ")
LOD_GCS_STAGING = os.environ.get("LOD_GCS_STAGING")
LOD_NET_VPC = os.environ.get("LOD_NET_VPC")
LOD_NET_SUBNET = os.environ.get("LOD_NET_SUBNET")
LOD_SA_DF = os.environ.get("LOD_SA_DF")
ORC_PRJ = os.environ.get("ORC_PRJ")
ORC_GCS = os.environ.get("ORC_GCS")
TRF_PRJ = os.environ.get("TRF_PRJ")
TRF_GCS_STAGING = os.environ.get("TRF_GCS_STAGING")
TRF_NET_VPC = os.environ.get("TRF_NET_VPC")
TRF_NET_SUBNET = os.environ.get("TRF_NET_SUBNET")
TRF_SA_DF = os.environ.get("TRF_SA_DF")
TRF_SA_BQ = os.environ.get("TRF_SA_BQ")
DF_ZONE = os.environ.get("GCP_REGION") + "-b"
DF_REGION = os.environ.get("GCP_REGION")
# --------------------------------------------------------------------------------
# Set default arguments
# --------------------------------------------------------------------------------
# If you are running Airflow in more than one time zone
# see https://airflow.apache.org/docs/apache-airflow/stable/timezone.html
# for best practices
yesterday = datetime.datetime.now() - datetime.timedelta(days=1)
default_args = {
'owner': 'airflow',
'start_date': yesterday,
'depends_on_past': False,
'email': [''],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': datetime.timedelta(minutes=5),
'dataflow_default_options': {
'project': LOD_PRJ,
'location': DF_REGION,
'zone': DF_ZONE,
'stagingLocation': LOD_GCS_STAGING,
'tempLocation': LOD_GCS_STAGING + "/tmp",
'serviceAccountEmail': LOD_SA_DF,
'subnetwork': LOD_NET_SUBNET,
'ipConfiguration': "WORKER_IP_PRIVATE"
},
}
# --------------------------------------------------------------------------------
# Main DAG
# --------------------------------------------------------------------------------
with models.DAG(
'data_pipeline_dag',
default_args=default_args,
schedule_interval=None) as dag:
start = dummy.DummyOperator(
task_id='start',
trigger_rule='all_success'
)
end = dummy.DummyOperator(
task_id='end',
trigger_rule='all_success'
)
customers_import = DataflowTemplateOperator(
task_id="dataflow_customer_import",
template="gs://dataflow-templates/latest/GCS_Text_to_BigQuery",
parameters={
"javascriptTextTransformFunctionName": "transform",
"JSONPath": ORC_GCS + "/customers_schema.json",
"javascriptTextTransformGcsPath": ORC_GCS + "/customers_udf.js",
"inputFilePattern": LND_GCS + "/customers.csv",
"outputTable": DTL_L0_PRJ + ":"+DTL_L0_BQ_DATASET+".customers",
"bigQueryLoadingTemporaryDirectory": LOD_GCS_STAGING + "/tmp/bq/",
},
)
purchases_import = DataflowTemplateOperator(
task_id="dataflow_purchases_import",
template="gs://dataflow-templates/latest/GCS_Text_to_BigQuery",
parameters={
"javascriptTextTransformFunctionName": "transform",
"JSONPath": ORC_GCS + "/purchases_schema.json",
"javascriptTextTransformGcsPath": ORC_GCS + "/purchases_udf.js",
"inputFilePattern": LND_GCS + "/purchases.csv",
"outputTable": DTL_L0_PRJ + ":"+DTL_L0_BQ_DATASET+".purchases",
"bigQueryLoadingTemporaryDirectory": LOD_GCS_STAGING + "/tmp/bq/",
},
)
join_customer_purchase = BigQueryInsertJobOperator(
task_id='bq_join_customer_purchase',
gcp_conn_id='bigquery_default',
project_id=TRF_PRJ,
location=BQ_LOCATION,
configuration={
'jobType':'QUERY',
'query':{
'query':"""SELECT
c.id as customer_id,
p.id as purchase_id,
c.name as name,
c.surname as surname,
p.item as item,
p.price as price,
p.timestamp as timestamp
FROM `{dtl_0_prj}.{dtl_0_dataset}.customers` c
JOIN `{dtl_0_prj}.{dtl_0_dataset}.purchases` p ON c.id = p.customer_id
""".format(dtl_0_prj=DTL_L0_PRJ, dtl_0_dataset=DTL_L0_BQ_DATASET, ),
'destinationTable':{
'projectId': DTL_L1_PRJ,
'datasetId': DTL_L1_BQ_DATASET,
'tableId': 'customer_purchase'
},
'writeDisposition':'WRITE_TRUNCATE',
"useLegacySql": False
}
},
impersonation_chain=[TRF_SA_BQ]
)
l2_customer_purchase = BigQueryInsertJobOperator(
task_id='bq_l2_customer_purchase',
gcp_conn_id='bigquery_default',
project_id=TRF_PRJ,
location=BQ_LOCATION,
configuration={
'jobType':'QUERY',
'query':{
'query':"""SELECT
customer_id,
purchase_id,
name,
surname,
item,
price,
timestamp
FROM `{dtl_1_prj}.{dtl_1_dataset}.customer_purchase`
""".format(dtl_1_prj=DTL_L1_PRJ, dtl_1_dataset=DTL_L1_BQ_DATASET, ),
'destinationTable':{
'projectId': DTL_L2_PRJ,
'datasetId': DTL_L2_BQ_DATASET,
'tableId': 'customer_purchase'
},
'writeDisposition':'WRITE_TRUNCATE',
"useLegacySql": False
}
},
impersonation_chain=[TRF_SA_BQ]
)
start >> [customers_import, purchases_import] >> join_customer_purchase >> l2_customer_purchase >> end

BIN
examples/data-solutions/data-platform-foundations/images/dlp_diagram.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 27 KiB

BIN
examples/data-solutions/data-platform-foundations/images/kms_diagram.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

BIN
examples/data-solutions/data-platform-foundations/images/overview_diagram.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 70 KiB

67
examples/data-solutions/data-platform-foundations/main.tf Normal file
View File

@ -0,0 +1,67 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# tfdoc:file:description Core locals.
locals {
# we cannot reference service accounts directly as they are dynamic
_shared_vpc_bindings = {
"roles/compute.networkUser" = [
"load-robot-df", "load-sa-df-worker",
"orch-cloudservices", "orch-robot-df", "orch-robot-gke",
"transf-robot-df", "transf-sa-df-worker",
]
"roles/composer.sharedVpcAgent" = [
"orch-robot-cs"
]
"roles/container.hostServiceAgentUser" = [
"orch-robot-df"
]
}
groups = {
for k, v in var.groups : k => "${v}@${var.organization_domain}"
}
groups_iam = {
for k, v in local.groups : k => "group:${v}"
}
service_encryption_keys = var.service_encryption_keys
shared_vpc_project = try(var.network_config.host_project, null)
# this is needed so that for_each only uses static values
shared_vpc_role_members = {
load-robot-df = "serviceAccount:${module.load-project.service_accounts.robots.dataflow}"
load-sa-df-worker = module.load-sa-df-0.iam_email
orch-cloudservices = "serviceAccount:${module.orch-project.service_accounts.cloud_services}"
orch-robot-cs = "serviceAccount:${module.orch-project.service_accounts.robots.composer}"
orch-robot-df = "serviceAccount:${module.orch-project.service_accounts.robots.dataflow}"
orch-robot-gke = "serviceAccount:${module.orch-project.service_accounts.robots.container-engine}"
transf-robot-df = "serviceAccount:${module.transf-project.service_accounts.robots.dataflow}"
transf-sa-df-worker = module.transf-sa-df-0.iam_email
}
# reassemble in a format suitable for for_each
shared_vpc_bindings_map = {
for binding in flatten([
for role, members in local._shared_vpc_bindings : [
for member in members : { role = role, member = member }
]
]) : "${binding.role}-${binding.member}" => binding
}
use_shared_vpc = var.network_config != null
}
resource "google_project_iam_member" "shared_vpc" {
for_each = local.use_shared_vpc ? local.shared_vpc_bindings_map : {}
project = var.network_config.host_project
role = each.value.role
member = lookup(local.shared_vpc_role_members, each.value.member)
}

104
examples/data-solutions/data-platform-foundations/outputs.tf Normal file
View File

@ -0,0 +1,104 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# tfdoc:file:description Output variables.
output "bigquery-datasets" {
description = "BigQuery datasets."
value = {
land-bq-0 = module.land-bq-0.dataset_id,
lake-0-bq-0 = module.lake-0-bq-0.dataset_id,
lake-1-bq-0 = module.lake-1-bq-0.dataset_id,
lake-2-bq-0 = module.lake-2-bq-0.dataset_id,
lake-plg-bq-0 = module.lake-plg-bq-0.dataset_id,
}
}
output "gcs-buckets" {
description = "GCS buckets."
value = {
lake-0-cs-0 = module.lake-0-cs-0.name,
lake-1-cs-0 = module.lake-1-cs-0.name,
lake-2-cs-0 = module.lake-2-cs-0.name,
lake-plg-cs-0 = module.lake-plg-cs-0.name,
land-cs-0 = module.land-cs-0.name,
lod-cs-df = module.load-cs-df-0.name,
orch-cs-0 = module.orch-cs-0.name,
transf-cs-df = module.transf-cs-df-0.name,
}
}
output "kms_keys" {
description = "Cloud MKS keys."
value = local.service_encryption_keys
}
output "projects" {
description = "GCP Projects informations."
value = {
project_number = {
lake-0 = module.lake-0-project.number,
lake-1 = module.lake-1-project.number,
lake-2 = module.lake-2-project.number,
lake-plg = module.lake-plg-project.number,
exposure = module.exp-project.number,
landing = module.land-project.number,
load = module.load-project.number,
orchestration = module.orch-project.number,
transformation = module.transf-project.number,
}
project_id = {
lake-0 = module.lake-0-project.project_id,
lake-1 = module.lake-1-project.project_id,
lake-2 = module.lake-2-project.project_id,
lake-plg = module.lake-plg-project.project_id,
exposure = module.exp-project.project_id,
landing = module.land-project.project_id,
load = module.load-project.project_id,
orchestration = module.orch-project.project_id,
transformation = module.transf-project.project_id,
}
}
}
output "vpc_network" {
description = "VPC network."
value = {
load = local.load_vpc
orchestration = local.orch_vpc
transformation = local.transf_vpc
}
}
output "vpc_subnet" {
description = "VPC subnetworks."
value = {
load = local.load_subnet
orchestration = local.orch_subnet
transformation = local.transf_subnet
}
}
output "demo_commands" {
description = "Demo commands."
value = {
01 = "gsutil -i ${module.land-sa-cs-0.email} cp demo/data/*.csv gs://${module.land-cs-0.name}"
02 = "gsutil -i ${module.orch-sa-cmp-0.email} cp demo/data/*.j* gs://${module.orch-cs-0.name}"
03 = "gsutil -i ${module.orch-sa-cmp-0.email} cp demo/*.py ${google_composer_environment.orch-cmp-0.config[0].dag_gcs_prefix}/"
04 = "Open ${google_composer_environment.orch-cmp-0.config.0.airflow_uri} and run uploaded DAG."
05 = <<EOT
bq query --project_id=${module.lake-2-project.project_id} --use_legacy_sql=false 'SELECT * FROM `${module.lake-2-project.project_id}.${module.lake-2-bq-0.dataset_id}.customer_purchase` LIMIT 1000'"
EOT
}
}

4
examples/data-solutions/data-platform-foundations/terraform.tfvars.sample Normal file
View File

@ -0,0 +1,4 @@
prefix = "prefix"
folder_id = "folders/123456789012"
billing_account_id = "111111-222222-333333"
organization_domain = "example.com"

123
examples/data-solutions/data-platform-foundations/variables.tf Normal file
View File

@ -0,0 +1,123 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# tfdoc:file:description Terraform Variables.
variable "billing_account_id" {
description = "Billing account id."
type = string
}
variable "composer_config" {
description = "Cloud Composer config."
type = object({
node_count = number
airflow_version = string
env_variables = map(string)
})
default = {
node_count = 3
airflow_version = "composer-1.17.5-airflow-2.1.4"
env_variables = {}
}
}
variable "data_force_destroy" {
description = "Flag to set 'force_destroy' on data services like BiguQery or Cloud Storage."
type = bool
default = false
}
variable "folder_id" {
description = "Folder to be used for the networking resources in folders/nnnn format."
type = string
}
variable "location" {
description = "Location used for multi-regional resources."
type = string
default = "eu"
}
variable "groups" {
description = "User groups."
type = map(string)
default = {
data-analysts = "gcp-data-analysts"
data-engineers = "gcp-data-engineers"
data-security = "gcp-data-security"
}
}
variable "network_config" {
description = "Shared VPC network configurations to use. If null networks will be created in projects with preconfigured values."
type = object({
host_project = string
network_self_link = string
subnet_self_links = object({
load = string
transformation = string
orchestration = string
})
composer_ip_ranges = object({
cloudsql = string
gke_master = string
web_server = string
})
composer_secondary_ranges = object({
pods = string
services = string
})
})
default = null
}
variable "organization_domain" {
description = "Organization domain."
type = string
}
variable "prefix" {
description = "Unique prefix used for resource names."
type = string
}
variable "project_services" {
description = "List of core services enabled on all projects."
type = list(string)
default = [
"cloudresourcemanager.googleapis.com",
"iam.googleapis.com",
"serviceusage.googleapis.com",
"stackdriver.googleapis.com"
]
}
variable "region" {
description = "Region used for regional resources."
type = string
default = "europe-west1"
}
variable "service_encryption_keys" { # service encription key
description = "Cloud KMS to use to encrypt different services. Key location should match service region."
type = object({
bq = string
composer = string
dataflow = string
storage = string
pubsub = string
})
default = null
}

45
fast/stages/00-bootstrap/README.md
View File

@ -192,7 +192,7 @@ outputs_location = "../../fast-config"
### Output files and cross-stage variables
At any time during the life of this stage, you can configure it to automatically generate provider configurations and variable files for the following, to simplify exchanging inputs and outputs between stages and avoid having to edit files manually.
At any time during the life of this stage, you can configure it to automatically generate provider configurations and variable files consumed by the following stages, to simplify passing outputs to input variables by not having to edit files manually.
Automatic generation of files is disabled by default. To enable the mechanism, set the `outputs_location` variable to a valid path on a local filesystem, e.g.
@ -202,27 +202,23 @@ outputs_location = "../../config"
Once the variable is set, `apply` will generate and manage providers and variables files, including the initial one used for this stage after the first run. You can then link these files in the relevant stages, instead of manually transfering outputs from one stage, to Terraform variables in another.
Below is the outline of the output files generated by this stage:
Below is the outline of the output files generated by all stages:
```bash
[path specified in outputs_location]
├── 00-bootstrap
│   ├── providers.tf
├── 01-resman
│   ├── providers.tf
│   ├── terraform-bootstrap.auto.tfvars.json
├── 02-networking
│   ├── terraform-bootstrap.auto.tfvars.json
├── 02-security
│   ├── terraform-bootstrap.auto.tfvars.json
├── 03-gke-multitenant-dev
│   └── terraform-bootstrap.auto.tfvars.json
├── 03-gke-multitenant-prod
│   └── terraform-bootstrap.auto.tfvars.json
├── 03-project-factory-dev
│   └── terraform-bootstrap.auto.tfvars.json
├── 03-project-factory-prod
│   └── terraform-bootstrap.auto.tfvars.json
├── providers
│   ├── 00-bootstrap-providers.tf
│   ├── 01-resman-providers.tf
│   ├── 02-networking-providers.tf
│   ├── 02-security-providers.tf
│   ├── 03-project-factory-dev-providers.tf
│   ├── 03-project-factory-prod-providers.tf
│   └── 99-sandbox-providers.tf
└── tfvars
├── 00-bootstrap.auto.tfvars.json
├── 01-resman.auto.tfvars.json
├── 02-networking.auto.tfvars.json
└── 02-security.auto.tfvars.json
```
### Running the stage
@ -241,7 +237,7 @@ Once the initial `apply` completes successfully, configure a remote backend usin
```bash
# if using output files via the outputs_location and set to `../../config`
ln -s ../../config/00-bootstrap/* ./
ln -s ../../config/providers/00-bootstrap* ./
# or from outputs if not using output files
terraform output -json providers | jq -r '.["00-bootstrap"]' \
> providers.tf
@ -350,9 +346,10 @@ Names used in internal references (e.g. `module.foo-prod.id`) are only used by T
| name | description | sensitive | consumers |
|---|---|:---:|---|
| [billing_dataset](outputs.tf#L89) | BigQuery dataset prepared for billing export. | | |
| [project_ids](outputs.tf#L94) | Projects created by this stage. | | |
| [providers](outputs.tf#L105) | Terraform provider files for this stage and dependent stages. | ✓ | <code>stage-01</code> |
| [tfvars](outputs.tf#L114) | Terraform variable files for the following stages. | ✓ | |
| [billing_dataset](outputs.tf#L58) | BigQuery dataset prepared for billing export. | | |
| [custom_roles](outputs.tf#L63) | Organization-level custom roles. | | |
| [project_ids](outputs.tf#L68) | Projects created by this stage. | | |
| [providers](outputs.tf#L79) | Terraform provider files for this stage and dependent stages. | ✓ | <code>stage-01</code> |
| [tfvars](outputs.tf#L88) | Terraform variable files for the following stages. | ✓ | |
<!-- END TFDOC -->

58
fast/stages/00-bootstrap/outputs.tf
View File

@ -15,7 +15,7 @@
*/
locals {
_custom_roles = {
custom_roles = {
for k, v in var.custom_role_names :
k => module.organization.custom_role_id[v]
}
@ -32,56 +32,25 @@ locals {
})
}
tfvars = {
"01-resman" = jsonencode({
automation_project_id = module.automation-project.project_id
billing_account = var.billing_account
custom_roles = local._custom_roles
groups = var.groups
organization = var.organization
prefix = var.prefix
})
"02-networking" = jsonencode({
billing_account_id = var.billing_account.id
custom_roles = local._custom_roles
organization = var.organization
prefix = var.prefix
})
"02-security" = jsonencode({
billing_account_id = var.billing_account.id
organization = var.organization
prefix = var.prefix
})
"03-gke-multitenant-dev" = jsonencode({
billing_account_id = var.billing_account.id
prefix = var.prefix
})
"03-gke-multitenant-prod" = jsonencode({
billing_account_id = var.billing_account.id
prefix = var.prefix
})
"03-project-factory-dev" = jsonencode({
billing_account_id = var.billing_account.id
prefix = var.prefix
})
"03-project-factory-prod" = jsonencode({
billing_account_id = var.billing_account.id
prefix = var.prefix
})
automation_project_id = module.automation-project.project_id
custom_roles = local.custom_roles
}
}
# optionally generate providers and tfvars files for subsequent stages
resource "local_file" "providers" {
for_each = var.outputs_location == null ? {} : local.providers
filename = "${pathexpand(var.outputs_location)}/${each.key}/providers.tf"
content = each.value
for_each = var.outputs_location == null ? {} : local.providers
file_permission = "0644"
filename = "${pathexpand(var.outputs_location)}/providers/${each.key}-providers.tf"
content = each.value
}
resource "local_file" "tfvars" {
for_each = var.outputs_location == null ? {} : local.tfvars
filename = "${pathexpand(var.outputs_location)}/${each.key}/terraform-bootstrap.auto.tfvars.json"
content = each.value
for_each = var.outputs_location == null ? {} : { 1 = 1 }
file_permission = "0644"
filename = "${pathexpand(var.outputs_location)}/tfvars/00-bootstrap.auto.tfvars.json"
content = jsonencode(local.tfvars)
}
# outputs
@ -91,6 +60,11 @@ output "billing_dataset" {
value = try(module.billing-export-dataset.0.id, null)
}
output "custom_roles" {
description = "Organization-level custom roles."
value = local.custom_roles
}
output "project_ids" {
description = "Projects created by this stage."
value = {

34
fast/stages/01-resman/README.md
View File

@ -50,11 +50,11 @@ The default way of making sure you have the right permissions, is to use the ide
To simplify setup, the previous stage pre-configures a valid providers file in its output, and optionally writes it to a local file if the `outputs_location` variable is set to a valid path.
If you have set a valid value for `outputs_location` in the bootstrap stage, simply link the relevant `providers.tf` file from this stage's folder in the path you specified:
If you have set a valid value for `outputs_location` in the bootstrap stage (see the [bootstrap stage README](../00-bootstrap/#output-files-and-cross-stage-variables) for more details), simply link the relevant `providers.tf` file from this stage's folder in the path you specified:
```bash
# `outputs_location` is set to `../../config`
ln -s ../../config/01-resman/providers.tf
# `outputs_location` is set to `~/config`
ln -s ~/config/providers/01-resman* ./
```
If you have not configured `outputs_location` in bootstrap, you can derive the providers file from that stage's outputs:
@ -76,14 +76,16 @@ There are two broad sets of variables you will need to fill in:
To avoid the tedious job of filling in the first group of variable with values derived from other stages' outputs, the same mechanism used above for the provider configuration can be used to leverage pre-configured `.tfvars` files.
If you configured a valid path for `outputs_location` in the bootstrap stage, simply link the relevant `terraform-*.auto.tfvars.json` files from this stage's outputs folder (under the path you specified), where the `*` above is set to the name of the stage that produced it. For this stage, a single `.tfvars` file is avalaible:
If you configured a valid path for `outputs_location` in the bootstrap stage, simply link the relevant `terraform-*.auto.tfvars.json` files from this stage's outputs folder. For this stage, you need the `.tfvars` file compiled manually for the bootstrap stage, and the one generated by it:
```bash
# `outputs_location` is set to `../../config`
ln -s ../../config/01-resman/terraform-bootstrap.auto.tfvars.json
# `outputs_location` is set to `~/config`
ln -s ../../config/tfvars/00*.json ./
# also copy the tfvars file used for the bootstrap stage
cp ../00-bootstrap/terraform.tfvars ./
```
A second set of variables is specific to this stage, they are all optional so if you need to customize them, create an extra `terraform.tfvars` file.
A second set of variables is specific to this stage, they are all optional so if you need to customize them, create an extra `terraform.tfvars` file or add them to the file copied from bootstrap.
Refer to the [Variables](#variables) table at the bottom of this document, for a full list of variables, their origin (e.g. a stage or specific to this one), and descriptions explaining their meaning. The sections below also describe some of the possible customizations. For billing configurations, refer to the [Bootstrap documentation on billing](../00-bootstrap/README.md#billing-account) as the `billing_account` variable is identical across all stages.
@ -163,8 +165,8 @@ Due to its simplicity, this stage lends itself easily to customizations: adding
| name | description | type | required | default | producer |
|---|---|:---:|:---:|:---:|:---:|
| [automation_project_id](variables.tf#L29) | Project id for the automation project created by the bootstrap stage. | <code>string</code> | ✓ | | <code>00-bootstrap</code> |
| [billing_account](variables.tf#L20) | Billing account id and organization id ('nnnnnnnn' or null). | <code title="object&#40;&#123;&#10; id &#61; string&#10; organization_id &#61; number&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>00-bootstrap</code> |
| [automation_project_id](variables.tf#L20) | Project id for the automation project created by the bootstrap stage. | <code>string</code> | ✓ | | <code>00-bootstrap</code> |
| [billing_account](variables.tf#L26) | Billing account id and organization id ('nnnnnnnn' or null). | <code title="object&#40;&#123;&#10; id &#61; string&#10; organization_id &#61; number&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>00-bootstrap</code> |
| [organization](variables.tf#L57) | Organization details. | <code title="object&#40;&#123;&#10; domain &#61; string&#10; id &#61; number&#10; customer_id &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>00-bootstrap</code> |
| [prefix](variables.tf#L81) | Prefix used for resources that need unique names. Use 9 characters or less. | <code>string</code> | ✓ | | <code>00-bootstrap</code> |
| [custom_roles](variables.tf#L35) | Custom roles defined at the org level, in key => id format. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> | <code>00-bootstrap</code> |
@ -177,12 +179,12 @@ Due to its simplicity, this stage lends itself easily to customizations: adding
| name | description | sensitive | consumers |
|---|---|:---:|---|
| [networking](outputs.tf#L83) | Data for the networking stage. | | <code>02-networking</code> |
| [project_factories](outputs.tf#L93) | Data for the project factories stage. | | <code>xx-teams</code> |
| [providers](outputs.tf#L110) | Terraform provider files for this stage and dependent stages. | ✓ | <code>02-networking</code> · <code>02-security</code> · <code>xx-sandbox</code> · <code>xx-teams</code> |
| [sandbox](outputs.tf#L117) | Data for the sandbox stage. | | <code>xx-sandbox</code> |
| [security](outputs.tf#L127) | Data for the networking stage. | | <code>02-security</code> |
| [teams](outputs.tf#L137) | Data for the teams stage. | | |
| [tfvars](outputs.tf#L150) | Terraform variable files for the following stages. | ✓ | |
| [networking](outputs.tf#L101) | Data for the networking stage. | | |
| [project_factories](outputs.tf#L110) | Data for the project factories stage. | | |
| [providers](outputs.tf#L126) | Terraform provider files for this stage and dependent stages. | ✓ | <code>02-networking</code> · <code>02-security</code> · <code>xx-sandbox</code> · <code>xx-teams</code> |
| [sandbox](outputs.tf#L133) | Data for the sandbox stage. | | <code>xx-sandbox</code> |
| [security](outputs.tf#L143) | Data for the networking stage. | | <code>02-security</code> |
| [teams](outputs.tf#L153) | Data for the teams stage. | | |
| [tfvars](outputs.tf#L166) | Terraform variable files for the following stages. | ✓ | |
<!-- END TFDOC -->

4
fast/stages/01-resman/branch-teams.tf
View File

@ -77,7 +77,7 @@ module "branch-teams-team-dev-folder" {
for_each = coalesce(var.team_folders, {})
parent = module.branch-teams-team-folder[each.key].id
# naming: environment descriptive name
name = "${module.branch-teams-team-folder[each.key].name} - Development"
name = "Development"
# environment-wide human permissions on the whole teams environment
group_iam = {}
iam = {
@ -127,7 +127,7 @@ module "branch-teams-team-prod-folder" {
for_each = coalesce(var.team_folders, {})
parent = module.branch-teams-team-folder[each.key].id
# naming: environment descriptive name
name = "${module.branch-teams-team-folder[each.key].name} - Production"
name = "Production"
# environment-wide human permissions on the whole teams environment
group_iam = {}
iam = {

68
fast/stages/01-resman/outputs.tf
View File

@ -15,10 +15,25 @@
*/
locals {
_project_factory_sas = {
dev = module.branch-teams-dev-projectfactory-sa.iam_email
prod = module.branch-teams-prod-projectfactory-sa.iam_email
}
folder_ids = merge(
{
networking = module.branch-network-folder.id
networking-dev = module.branch-network-dev-folder.id
networking-prod = module.branch-network-prod-folder.id
sandbox = module.branch-sandbox-folder.id
security = module.branch-security-folder.id
teams = module.branch-teams-folder.id
},
{
for k, v in module.branch-teams-team-folder : "team-${k}" => v.id
},
{
for k, v in module.branch-teams-team-dev-folder : "team-${k}-dev" => v.id
},
{
for k, v in module.branch-teams-team-prod-folder : "team-${k}-prod" => v.id
}
)
providers = {
"02-networking" = templatefile("${path.module}/../../assets/templates/providers.tpl", {
bucket = module.branch-network-gcs.name
@ -46,42 +61,44 @@ locals {
sa = module.branch-sandbox-sa.email
})
}
service_accounts = merge(
{
networking = module.branch-network-sa.email
project-factory-dev = module.branch-teams-dev-projectfactory-sa.email
project-factory-prod = module.branch-teams-prod-projectfactory-sa.email
sandbox = module.branch-sandbox-sa.email
security = module.branch-security-sa.email
teams = module.branch-teams-prod-sa.email
},
{
for k, v in module.branch-teams-team-sa : "team-${k}" => v.email
},
)
tfvars = {
"02-networking" = jsonencode({
folder_ids = {
networking = module.branch-network-folder.id
networking-dev = module.branch-network-dev-folder.id
networking-prod = module.branch-network-prod-folder.id
}
project_factory_sa = local._project_factory_sas
})
"02-security" = jsonencode({
folder_id = module.branch-security-folder.id
kms_restricted_admins = {
for k, v in local._project_factory_sas : k => [v]
}
})
folder_ids = local.folder_ids
service_accounts = local.service_accounts
}
}
# optionally generate providers and tfvars files for subsequent stages
resource "local_file" "providers" {
for_each = var.outputs_location == null ? {} : local.providers
filename = "${pathexpand(var.outputs_location)}/${each.key}/providers.tf"
content = each.value
for_each = var.outputs_location == null ? {} : local.providers
file_permission = "0644"
filename = "${pathexpand(var.outputs_location)}/providers/${each.key}-providers.tf"
content = each.value
}
resource "local_file" "tfvars" {
for_each = var.outputs_location == null ? {} : local.tfvars
filename = "${pathexpand(var.outputs_location)}/${each.key}/terraform-resman.auto.tfvars.json"
content = each.value
for_each = var.outputs_location == null ? {} : { 1 = 1 }
file_permission = "0644"
filename = "${pathexpand(var.outputs_location)}/tfvars/01-resman.auto.tfvars.json"
content = jsonencode(local.tfvars)
}
# outputs
output "networking" {
# tfdoc:output:consumers 02-networking
description = "Data for the networking stage."
value = {
folder = module.branch-network-folder.id
@ -91,7 +108,6 @@ output "networking" {
}
output "project_factories" {
# tfdoc:output:consumers xx-teams
description = "Data for the project factories stage."
value = {
dev = {

12
fast/stages/01-resman/variables.tf
View File

@ -17,6 +17,12 @@
# defaults for variables marked with global tfdoc annotations, can be set via
# the tfvars file generated in stage 00 and stored in its outputs
variable "automation_project_id" {
# tfdoc:variable:source 00-bootstrap
description = "Project id for the automation project created by the bootstrap stage."
type = string
}
variable "billing_account" {
# tfdoc:variable:source 00-bootstrap
description = "Billing account id and organization id ('nnnnnnnn' or null)."
@ -26,12 +32,6 @@ variable "billing_account" {
})
}
variable "automation_project_id" {
# tfdoc:variable:source 00-bootstrap
description = "Project id for the automation project created by the bootstrap stage."
type = string
}
variable "custom_roles" {
# tfdoc:variable:source 00-bootstrap
description = "Custom roles defined at the org level, in key => id format."

44
fast/stages/02-networking-nva/README.md
View File

@ -353,8 +353,8 @@ Don't forget to add a peering zone in the landing project and point it to the ne
| [monitoring.tf](./monitoring.tf) | Network monitoring dashboards. | | <code>google_monitoring_dashboard</code> |
| [nva.tf](./nva.tf) | None | <code>compute-mig</code> · <code>compute-vm</code> · <code>net-ilb</code> | |
| [outputs.tf](./outputs.tf) | Module outputs. | | <code>local_file</code> |
| [spoke-dev.tf](./spoke-dev.tf) | Dev spoke VPC and related resources. | <code>net-address</code> · <code>net-vpc</code> · <code>net-vpc-firewall</code> · <code>net-vpc-peering</code> · <code>project</code> | |
| [spoke-prod.tf](./spoke-prod.tf) | Production spoke VPC and related resources. | <code>net-address</code> · <code>net-vpc</code> · <code>net-vpc-firewall</code> · <code>net-vpc-peering</code> · <code>project</code> | |
| [spoke-dev.tf](./spoke-dev.tf) | Dev spoke VPC and related resources. | <code>net-vpc</code> · <code>net-vpc-firewall</code> · <code>net-vpc-peering</code> · <code>project</code> | |
| [spoke-prod.tf](./spoke-prod.tf) | Production spoke VPC and related resources. | <code>net-vpc</code> · <code>net-vpc-firewall</code> · <code>net-vpc-peering</code> · <code>project</code> | |
| [test-resources.tf](./test-resources.tf) | temporary instances for testing | <code>compute-vm</code> | |
| [variables.tf](./variables.tf) | Module variables. | | |
| [vpn-onprem.tf](./vpn-onprem.tf) | VPN between landing and onprem. | <code>net-vpn-ha</code> | |
@ -363,30 +363,30 @@ Don't forget to add a peering zone in the landing project and point it to the ne
| name | description | type | required | default | producer |
|---|---|:---:|:---:|:---:|:---:|
| [billing_account_id](variables.tf#L17) | Billing account id. | <code>string</code> | ✓ | | <code>00-bootstrap</code> |
| [folder_ids](variables.tf#L59) | Folders to be used for the networking resources in folders/nnnnnnnnnnn format. If null, folder will be created. | <code>map&#40;string&#41;</code> | ✓ | | <code>01-resman</code> |
| [organization](variables.tf#L91) | Organization details. | <code title="object&#40;&#123;&#10; domain &#61; string&#10; id &#61; number&#10; customer_id &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>00-bootstrap</code> |
| [prefix](variables.tf#L107) | Prefix used for resources that need unique names. Use 9 characters or less. | <code>string</code> | ✓ | | <code>00-bootstrap</code> |
| [custom_adv](variables.tf#L23) | Custom advertisement definitions in name => range format. | <code>map&#40;string&#41;</code> | | <code title="&#123;&#10; cloud_dns &#61; &#34;35.199.192.0&#47;19&#34;&#10; gcp_all &#61; &#34;10.128.0.0&#47;16&#34;&#10; gcp_dev_ew1 &#61; &#34;10.128.128.0&#47;19&#34;&#10; gcp_dev_ew4 &#61; &#34;10.128.160.0&#47;19&#34;&#10; gcp_landing_trusted_ew1 &#61; &#34;10.128.64.0&#47;19&#34;&#10; gcp_landing_trusted_ew4 &#61; &#34;10.128.96.0&#47;19&#34;&#10; gcp_landing_untrusted_ew1 &#61; &#34;10.128.0.0&#47;19&#34;&#10; gcp_landing_untrusted_ew4 &#61; &#34;10.128.32.0&#47;19&#34;&#10; gcp_prod_ew1 &#61; &#34;10.128.192.0&#47;19&#34;&#10; gcp_prod_ew4 &#61; &#34;10.128.224.0&#47;19&#34;&#10; googleapis_private &#61; &#34;199.36.153.8&#47;30&#34;&#10; googleapis_restricted &#61; &#34;199.36.153.4&#47;30&#34;&#10; rfc_1918_10 &#61; &#34;10.0.0.0&#47;8&#34;&#10; rfc_1918_172 &#61; &#34;172.16.0.0&#47;12&#34;&#10; rfc_1918_192 &#61; &#34;192.168.0.0&#47;16&#34;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [data_dir](variables.tf#L45) | Relative path for the folder storing configuration data for network resources. | <code>string</code> | | <code>&#34;data&#34;</code> | |
| [dns](variables.tf#L51) | Onprem DNS resolvers | <code>map&#40;list&#40;string&#41;&#41;</code> | | <code title="&#123;&#10; onprem &#61; &#91;&#34;10.0.200.3&#34;&#93;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [l7ilb_subnets](variables.tf#L65) | Subnets used for L7 ILBs. | <code title="map&#40;list&#40;object&#40;&#123;&#10; ip_cidr_range &#61; string&#10; region &#61; string&#10;&#125;&#41;&#41;&#41;">map&#40;list&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;&#41;</code> | | <code title="&#123;&#10; dev &#61; &#91;&#10; &#123; ip_cidr_range &#61; &#34;10.128.159.0&#47;24&#34;, region &#61; &#34;europe-west1&#34; &#125;,&#10; &#123; ip_cidr_range &#61; &#34;10.128.191.0&#47;24&#34;, region &#61; &#34;europe-west4&#34; &#125;&#10; &#93;&#10; prod &#61; &#91;&#10; &#123; ip_cidr_range &#61; &#34;10.128.223.0&#47;24&#34;, region &#61; &#34;europe-west1&#34; &#125;,&#10; &#123; ip_cidr_range &#61; &#34;10.128.255.0&#47;24&#34;, region &#61; &#34;europe-west4&#34; &#125;&#10; &#93;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [onprem_cidr](variables.tf#L83) | Onprem addresses in name => range format. | <code>map&#40;string&#41;</code> | | <code title="&#123;&#10; main &#61; &#34;10.0.0.0&#47;24&#34;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [outputs_location](variables.tf#L101) | Path where providers and tfvars files for the following stages are written. Leave empty to disable. | <code>string</code> | | <code>null</code> | |
| [project_factory_sa](variables.tf#L118) | IAM emails for project factory service accounts | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> | <code>01-resman</code> |
| [psa_ranges](variables.tf#L125) | IP ranges used for Private Service Access (e.g. CloudSQL). | <code>map&#40;map&#40;string&#41;&#41;</code> | | <code title="&#123;&#10; dev &#61; &#123;&#10; cloudsql-mysql-ew1 &#61; &#34;10.128.157.0&#47;24&#34;&#10; cloudsql-mysql-ew4 &#61; &#34;10.128.189.0&#47;24&#34;&#10; cloudsql-sqlserver-ew1 &#61; &#34;10.128.158.0&#47;24&#34;&#10; cloudsql-sqlserver-ew4 &#61; &#34;10.128.190.0&#47;24&#34;&#10; &#125;&#10; prod &#61; &#123;&#10; cloudsql-mysql-ew1 &#61; &#34;10.128.221.0&#47;24&#34;&#10; cloudsql-mysql-ew4 &#61; &#34;10.128.253.0&#47;24&#34;&#10; cloudsql-sqlserver-ew1 &#61; &#34;10.128.222.0&#47;24&#34;&#10; cloudsql-sqlserver-ew4 &#61; &#34;10.128.254.0&#47;24&#34;&#10; &#125;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [router_configs](variables.tf#L144) | Configurations for CRs and onprem routers. | <code title="map&#40;object&#40;&#123;&#10; adv &#61; object&#40;&#123;&#10; custom &#61; list&#40;string&#41;&#10; default &#61; bool&#10; &#125;&#41;&#10; asn &#61; number&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code title="&#123;&#10; landing-trusted-ew1 &#61; &#123;&#10; asn &#61; &#34;64512&#34;&#10; adv &#61; null&#10; &#125;&#10; landing-trusted-ew4 &#61; &#123;&#10; asn &#61; &#34;64512&#34;&#10; adv &#61; null&#10; &#125;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [vpn_onprem_configs](variables.tf#L167) | VPN gateway configuration for onprem interconnection. | <code title="map&#40;object&#40;&#123;&#10; adv &#61; object&#40;&#123;&#10; default &#61; bool&#10; custom &#61; list&#40;string&#41;&#10; &#125;&#41;&#10; peer_external_gateway &#61; object&#40;&#123;&#10; redundancy_type &#61; string&#10; interfaces &#61; list&#40;object&#40;&#123;&#10; id &#61; number&#10; ip_address &#61; string&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#10; tunnels &#61; list&#40;object&#40;&#123;&#10; peer_asn &#61; number&#10; peer_external_gateway_interface &#61; number&#10; secret &#61; string&#10; session_range &#61; string&#10; vpn_gateway_interface &#61; number&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code title="&#123;&#10; landing-trusted-ew1 &#61; &#123;&#10; adv &#61; &#123;&#10; default &#61; false&#10; custom &#61; &#91;&#10; &#34;cloud_dns&#34;, &#34;googleapis_private&#34;, &#34;googleapis_restricted&#34;, &#34;gcp_all&#34;&#10; &#93;&#10; &#125;&#10; peer_external_gateway &#61; &#123;&#10; redundancy_type &#61; &#34;SINGLE_IP_INTERNALLY_REDUNDANT&#34;&#10; interfaces &#61; &#91;&#10; &#123; id &#61; 0, ip_address &#61; &#34;8.8.8.8&#34; &#125;,&#10; &#93;&#10; &#125;&#10; tunnels &#61; &#91;&#10; &#123;&#10; peer_asn &#61; 65534&#10; peer_external_gateway_interface &#61; 0&#10; secret &#61; &#34;foobar&#34;&#10; session_range &#61; &#34;169.254.1.0&#47;30&#34;&#10; vpn_gateway_interface &#61; 0&#10; &#125;,&#10; &#123;&#10; peer_asn &#61; 65534&#10; peer_external_gateway_interface &#61; 0&#10; secret &#61; &#34;foobar&#34;&#10; session_range &#61; &#34;169.254.1.4&#47;30&#34;&#10; vpn_gateway_interface &#61; 1&#10; &#125;&#10; &#93;&#10; &#125;&#10; landing-trusted-ew4 &#61; &#123;&#10; adv &#61; &#123;&#10; default &#61; false&#10; custom &#61; &#91;&#10; &#34;cloud_dns&#34;, &#34;googleapis_private&#34;, &#34;googleapis_restricted&#34;, &#34;gcp_all&#34;&#10; &#93;&#10; &#125;&#10; peer_external_gateway &#61; &#123;&#10; redundancy_type &#61; &#34;SINGLE_IP_INTERNALLY_REDUNDANT&#34;&#10; interfaces &#61; &#91;&#10; &#123; id &#61; 0, ip_address &#61; &#34;8.8.8.8&#34; &#125;,&#10; &#93;&#10; &#125;&#10; tunnels &#61; &#91;&#10; &#123;&#10; peer_asn &#61; 65534&#10; peer_external_gateway_interface &#61; 0&#10; secret &#61; &#34;foobar&#34;&#10; session_range &#61; &#34;169.254.1.0&#47;30&#34;&#10; vpn_gateway_interface &#61; 0&#10; &#125;,&#10; &#123;&#10; peer_asn &#61; 65534&#10; peer_external_gateway_interface &#61; 0&#10; secret &#61; &#34;foobar&#34;&#10; session_range &#61; &#34;169.254.1.4&#47;30&#34;&#10; vpn_gateway_interface &#61; 1&#10; &#125;&#10; &#93;&#10; &#125;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [billing_account](variables.tf#L17) | Billing account id and organization id ('nnnnnnnn' or null). | <code title="object&#40;&#123;&#10; id &#61; string&#10; organization_id &#61; number&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>00-bootstrap</code> |
| [folder_ids](variables.tf#L71) | Folders to be used for the networking resources in folders/nnnnnnnnnnn format. If null, folder will be created. | <code title="object&#40;&#123;&#10; networking &#61; string&#10; networking-dev &#61; string&#10; networking-prod &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>01-resman</code> |
| [organization](variables.tf#L107) | Organization details. | <code title="object&#40;&#123;&#10; domain &#61; string&#10; id &#61; number&#10; customer_id &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>00-bootstrap</code> |
| [prefix](variables.tf#L123) | Prefix used for resources that need unique names. Use 9 characters or less. | <code>string</code> | ✓ | | <code>00-bootstrap</code> |
| [custom_adv](variables.tf#L26) | Custom advertisement definitions in name => range format. | <code>map&#40;string&#41;</code> | | <code title="&#123;&#10; cloud_dns &#61; &#34;35.199.192.0&#47;19&#34;&#10; gcp_all &#61; &#34;10.128.0.0&#47;16&#34;&#10; gcp_dev_ew1 &#61; &#34;10.128.128.0&#47;19&#34;&#10; gcp_dev_ew4 &#61; &#34;10.128.160.0&#47;19&#34;&#10; gcp_landing_trusted_ew1 &#61; &#34;10.128.64.0&#47;19&#34;&#10; gcp_landing_trusted_ew4 &#61; &#34;10.128.96.0&#47;19&#34;&#10; gcp_landing_untrusted_ew1 &#61; &#34;10.128.0.0&#47;19&#34;&#10; gcp_landing_untrusted_ew4 &#61; &#34;10.128.32.0&#47;19&#34;&#10; gcp_prod_ew1 &#61; &#34;10.128.192.0&#47;19&#34;&#10; gcp_prod_ew4 &#61; &#34;10.128.224.0&#47;19&#34;&#10; googleapis_private &#61; &#34;199.36.153.8&#47;30&#34;&#10; googleapis_restricted &#61; &#34;199.36.153.4&#47;30&#34;&#10; rfc_1918_10 &#61; &#34;10.0.0.0&#47;8&#34;&#10; rfc_1918_172 &#61; &#34;172.16.0.0&#47;12&#34;&#10; rfc_1918_192 &#61; &#34;192.168.0.0&#47;16&#34;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [custom_roles](variables.tf#L48) | Custom roles defined at the org level, in key => id format. | <code title="object&#40;&#123;&#10; service_project_network_admin &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> | <code>00-bootstrap</code> |
| [data_dir](variables.tf#L57) | Relative path for the folder storing configuration data for network resources. | <code>string</code> | | <code>&#34;data&#34;</code> | |
| [dns](variables.tf#L63) | Onprem DNS resolvers | <code>map&#40;list&#40;string&#41;&#41;</code> | | <code title="&#123;&#10; onprem &#61; &#91;&#34;10.0.200.3&#34;&#93;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [l7ilb_subnets](variables.tf#L81) | Subnets used for L7 ILBs. | <code title="map&#40;list&#40;object&#40;&#123;&#10; ip_cidr_range &#61; string&#10; region &#61; string&#10;&#125;&#41;&#41;&#41;">map&#40;list&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;&#41;</code> | | <code title="&#123;&#10; dev &#61; &#91;&#10; &#123; ip_cidr_range &#61; &#34;10.128.159.0&#47;24&#34;, region &#61; &#34;europe-west1&#34; &#125;,&#10; &#123; ip_cidr_range &#61; &#34;10.128.191.0&#47;24&#34;, region &#61; &#34;europe-west4&#34; &#125;&#10; &#93;&#10; prod &#61; &#91;&#10; &#123; ip_cidr_range &#61; &#34;10.128.223.0&#47;24&#34;, region &#61; &#34;europe-west1&#34; &#125;,&#10; &#123; ip_cidr_range &#61; &#34;10.128.255.0&#47;24&#34;, region &#61; &#34;europe-west4&#34; &#125;&#10; &#93;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [onprem_cidr](variables.tf#L99) | Onprem addresses in name => range format. | <code>map&#40;string&#41;</code> | | <code title="&#123;&#10; main &#61; &#34;10.0.0.0&#47;24&#34;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [outputs_location](variables.tf#L117) | Path where providers and tfvars files for the following stages are written. Leave empty to disable. | <code>string</code> | | <code>null</code> | |
| [psa_ranges](variables.tf#L134) | IP ranges used for Private Service Access (e.g. CloudSQL). | <code>map&#40;map&#40;string&#41;&#41;</code> | | <code title="&#123;&#10; dev &#61; &#123;&#10; cloudsql-mysql-ew1 &#61; &#34;10.128.157.0&#47;24&#34;&#10; cloudsql-mysql-ew4 &#61; &#34;10.128.189.0&#47;24&#34;&#10; cloudsql-sqlserver-ew1 &#61; &#34;10.128.158.0&#47;24&#34;&#10; cloudsql-sqlserver-ew4 &#61; &#34;10.128.190.0&#47;24&#34;&#10; &#125;&#10; prod &#61; &#123;&#10; cloudsql-mysql-ew1 &#61; &#34;10.128.221.0&#47;24&#34;&#10; cloudsql-mysql-ew4 &#61; &#34;10.128.253.0&#47;24&#34;&#10; cloudsql-sqlserver-ew1 &#61; &#34;10.128.222.0&#47;24&#34;&#10; cloudsql-sqlserver-ew4 &#61; &#34;10.128.254.0&#47;24&#34;&#10; &#125;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [router_configs](variables.tf#L153) | Configurations for CRs and onprem routers. | <code title="map&#40;object&#40;&#123;&#10; adv &#61; object&#40;&#123;&#10; custom &#61; list&#40;string&#41;&#10; default &#61; bool&#10; &#125;&#41;&#10; asn &#61; number&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code title="&#123;&#10; landing-trusted-ew1 &#61; &#123;&#10; asn &#61; &#34;64512&#34;&#10; adv &#61; null&#10; &#125;&#10; landing-trusted-ew4 &#61; &#123;&#10; asn &#61; &#34;64512&#34;&#10; adv &#61; null&#10; &#125;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [service_accounts](variables.tf#L176) | Automation service accounts in name => email format. | <code title="object&#40;&#123;&#10; project-factory-dev &#61; string&#10; project-factory-prod &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> | <code>01-resman</code> |
| [vpn_onprem_configs](variables.tf#L186) | VPN gateway configuration for onprem interconnection. | <code title="map&#40;object&#40;&#123;&#10; adv &#61; object&#40;&#123;&#10; default &#61; bool&#10; custom &#61; list&#40;string&#41;&#10; &#125;&#41;&#10; peer_external_gateway &#61; object&#40;&#123;&#10; redundancy_type &#61; string&#10; interfaces &#61; list&#40;object&#40;&#123;&#10; id &#61; number&#10; ip_address &#61; string&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#10; tunnels &#61; list&#40;object&#40;&#123;&#10; peer_asn &#61; number&#10; peer_external_gateway_interface &#61; number&#10; secret &#61; string&#10; session_range &#61; string&#10; vpn_gateway_interface &#61; number&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code title="&#123;&#10; landing-trusted-ew1 &#61; &#123;&#10; adv &#61; &#123;&#10; default &#61; false&#10; custom &#61; &#91;&#10; &#34;cloud_dns&#34;, &#34;googleapis_private&#34;, &#34;googleapis_restricted&#34;, &#34;gcp_all&#34;&#10; &#93;&#10; &#125;&#10; peer_external_gateway &#61; &#123;&#10; redundancy_type &#61; &#34;SINGLE_IP_INTERNALLY_REDUNDANT&#34;&#10; interfaces &#61; &#91;&#10; &#123; id &#61; 0, ip_address &#61; &#34;8.8.8.8&#34; &#125;,&#10; &#93;&#10; &#125;&#10; tunnels &#61; &#91;&#10; &#123;&#10; peer_asn &#61; 65534&#10; peer_external_gateway_interface &#61; 0&#10; secret &#61; &#34;foobar&#34;&#10; session_range &#61; &#34;169.254.1.0&#47;30&#34;&#10; vpn_gateway_interface &#61; 0&#10; &#125;,&#10; &#123;&#10; peer_asn &#61; 65534&#10; peer_external_gateway_interface &#61; 0&#10; secret &#61; &#34;foobar&#34;&#10; session_range &#61; &#34;169.254.1.4&#47;30&#34;&#10; vpn_gateway_interface &#61; 1&#10; &#125;&#10; &#93;&#10; &#125;&#10; landing-trusted-ew4 &#61; &#123;&#10; adv &#61; &#123;&#10; default &#61; false&#10; custom &#61; &#91;&#10; &#34;cloud_dns&#34;, &#34;googleapis_private&#34;, &#34;googleapis_restricted&#34;, &#34;gcp_all&#34;&#10; &#93;&#10; &#125;&#10; peer_external_gateway &#61; &#123;&#10; redundancy_type &#61; &#34;SINGLE_IP_INTERNALLY_REDUNDANT&#34;&#10; interfaces &#61; &#91;&#10; &#123; id &#61; 0, ip_address &#61; &#34;8.8.8.8&#34; &#125;,&#10; &#93;&#10; &#125;&#10; tunnels &#61; &#91;&#10; &#123;&#10; peer_asn &#61; 65534&#10; peer_external_gateway_interface &#61; 0&#10; secret &#61; &#34;foobar&#34;&#10; session_range &#61; &#34;169.254.1.0&#47;30&#34;&#10; vpn_gateway_interface &#61; 0&#10; &#125;,&#10; &#123;&#10; peer_asn &#61; 65534&#10; peer_external_gateway_interface &#61; 0&#10; secret &#61; &#34;foobar&#34;&#10; session_range &#61; &#34;169.254.1.4&#47;30&#34;&#10; vpn_gateway_interface &#61; 1&#10; &#125;&#10; &#93;&#10; &#125;&#10;&#125;">&#123;&#8230;&#125;</code> | |
## Outputs
| name | description | sensitive | consumers |
|---|---|:---:|---|
| [project_ids](outputs.tf#L42) | Network project ids. | | |
| [project_numbers](outputs.tf#L51) | Network project numbers. | | |
| [shared_vpc_host_projects](outputs.tf#L60) | Shared VPC host projects. | | |
| [shared_vpc_self_links](outputs.tf#L69) | Shared VPC host projects. | | |
| [tfvars](outputs.tf#L93) | Network-related variables used in other stages. | ✓ | |
| [vpn_gateway_endpoints](outputs.tf#L79) | External IP Addresses for the GCP VPN gateways. | | |
| [host_project_ids](outputs.tf#L52) | Network project ids. | | |
| [host_project_numbers](outputs.tf#L57) | Network project numbers. | | |
| [shared_vpc_self_links](outputs.tf#L62) | Shared VPC host projects. | | |
| [tfvars](outputs.tf#L81) | Terraform variables file for the following stages. | ✓ | |
| [vpn_gateway_endpoints](outputs.tf#L67) | External IP Addresses for the GCP VPN gateways. | | |
<!-- END TFDOC -->

9
fast/stages/02-networking-nva/landing.tf
View File

@ -18,7 +18,7 @@
module "landing-project" {
source = "../../../modules/project"
billing_account = var.billing_account_id
billing_account = var.billing_account.id
name = "prod-net-landing-0"
parent = var.folder_ids.networking-prod
prefix = var.prefix
@ -37,6 +37,13 @@ module "landing-project" {
enabled = true
service_projects = []
}
metric_scopes = [module.landing-project.project_id]
iam = {
"roles/dns.admin" = [local.service_accounts.project-factory-prod]
(local.custom_roles.service_project_network_admin) = [
local.service_accounts.project-factory-prod
]
}
}
# Untrusted VPC

4
fast/stages/02-networking-nva/main.tf
View File

@ -17,12 +17,16 @@
# tfdoc:file:description Networking folder and hierarchical policy.
locals {
custom_roles = coalesce(var.custom_roles, {})
l7ilb_subnets = { for env, v in var.l7ilb_subnets : env => [
for s in v : merge(s, {
active = true
name = "${env}-l7ilb-${s.region}"
})]
}
service_accounts = {
for k, v in coalesce(var.service_accounts, {}) : k => "serviceAccount:${v}"
}
}
module "folder" {

76
fast/stages/02-networking-nva/outputs.tf
View File

@ -14,66 +14,54 @@
* limitations under the License.
*/
# Optionally, generate providers and tfvars files for subsequent stages
locals {
host_project_ids = {
dev-spoke-0 = module.dev-spoke-project.project_id
prod-landing = module.landing-project.project_id
prod-spoke-0 = module.prod-spoke-project.project_id
}
host_project_numbers = {
dev-spoke-0 = module.dev-spoke-project.number
prod-landing = module.landing-project.number
prod-spoke-0 = module.prod-spoke-project.number
}
tfvars = {
"03-project-factory-dev" = jsonencode({
environment_dns_zone = module.dev-dns-private-zone.domain
shared_vpc_self_link = module.dev-spoke-vpc.self_link
vpc_host_project = module.dev-spoke-project.project_id
})
"03-project-factory-prod" = jsonencode({
environment_dns_zone = module.prod-dns-private-zone.domain
shared_vpc_self_link = module.prod-spoke-vpc.self_link
vpc_host_project = module.prod-spoke-project.project_id
})
host_project_ids = local.host_project_ids
host_project_numbers = local.host_project_numbers
vpc_self_links = local.vpc_self_links
}
vpc_self_links = {
prod-landing-trusted = module.landing-trusted-vpc.self_link
prod-landing-untrusted = module.landing-untrusted-vpc.self_link
dev-spoke-0 = module.dev-spoke-vpc.self_link
prod-spoke-0 = module.prod-spoke-vpc.self_link
}
}
# optionally generate tfvars file for subsequent stages
resource "local_file" "tfvars" {
for_each = var.outputs_location == null ? {} : local.tfvars
filename = "${pathexpand(var.outputs_location)}/${each.key}/terraform-networking.auto.tfvars.json"
content = each.value
for_each = var.outputs_location == null ? {} : { 1 = 1 }
file_permission = "0644"
filename = "${pathexpand(var.outputs_location)}/tfvars/02-networking.auto.tfvars.json"
content = jsonencode(local.tfvars)
}
# Outputs
# outputs
output "project_ids" {
output "host_project_ids" {
description = "Network project ids."
value = {
dev = module.dev-spoke-project.project_id
landing = module.landing-project.project_id
prod = module.prod-spoke-project.project_id
}
value = local.host_project_ids
}
output "project_numbers" {
output "host_project_numbers" {
description = "Network project numbers."
value = {
dev = "projects/${module.dev-spoke-project.number}"
landing = "projects/${module.landing-project.number}"
prod = "projects/${module.prod-spoke-project.number}"
}
}
output "shared_vpc_host_projects" {
description = "Shared VPC host projects."
value = {
dev = module.dev-spoke-project.project_id
landing = module.landing-project.project_id
prod = module.prod-spoke-project.project_id
}
value = local.host_project_numbers
}
output "shared_vpc_self_links" {
description = "Shared VPC host projects."
value = {
dev = module.dev-spoke-vpc.self_link
landing-trusted = module.landing-trusted-vpc.self_link
landing-untrusted = module.landing-untrusted-vpc.self_link
prod = module.prod-spoke-vpc.self_link
}
value = local.vpc_self_links
}
output "vpn_gateway_endpoints" {
@ -91,7 +79,7 @@ output "vpn_gateway_endpoints" {
}
output "tfvars" {
description = "Network-related variables used in other stages."
description = "Terraform variables file for the following stages."
sensitive = true
value = local.tfvars
}

19
fast/stages/02-networking-nva/spoke-dev.tf
View File

@ -18,7 +18,7 @@
module "dev-spoke-project" {
source = "../../../modules/project"
billing_account = var.billing_account_id
billing_account = var.billing_account.id
name = "dev-net-spoke-0"
parent = var.folder_ids.networking-dev
prefix = var.prefix
@ -39,7 +39,10 @@ module "dev-spoke-project" {
}
metric_scopes = [module.landing-project.project_id]
iam = {
"roles/dns.admin" = [var.project_factory_sa.dev]
"roles/dns.admin" = [local.service_accounts.project-factory-dev]
(local.custom_roles.service_project_network_admin) = [
local.service_accounts.project-factory-dev
]
}
}
@ -50,6 +53,7 @@ module "dev-spoke-vpc" {
mtu = 1500
data_folder = "${var.data_dir}/subnets/dev"
delete_default_routes_on_create = true
psa_ranges = var.psa_ranges.dev
subnets_l7ilb = local.l7ilb_subnets.dev
# Set explicit routes for googleapis; send everything else to NVAs
routes = {
@ -110,17 +114,6 @@ module "dev-spoke-firewall" {
cidr_template_file = "${var.data_dir}/cidrs.yaml"
}
module "dev-spoke-psa-addresses" {
source = "../../../modules/net-address"
project_id = module.dev-spoke-project.project_id
psa_addresses = { for r, v in var.psa_ranges.dev : r => {
address = cidrhost(v, 0)
network = module.dev-spoke-vpc.self_link
prefix_length = split("/", v)[1]
}
}
}
module "peering-dev" {
source = "../../../modules/net-vpc-peering"
prefix = "dev-peering-0"

19
fast/stages/02-networking-nva/spoke-prod.tf
View File

@ -18,7 +18,7 @@
module "prod-spoke-project" {
source = "../../../modules/project"
billing_account = var.billing_account_id
billing_account = var.billing_account.id
name = "prod-net-spoke-0"
parent = var.folder_ids.networking-prod
prefix = var.prefix
@ -39,7 +39,10 @@ module "prod-spoke-project" {
}
metric_scopes = [module.landing-project.project_id]
iam = {
"roles/dns.admin" = [var.project_factory_sa.prod]
"roles/dns.admin" = [local.service_accounts.project-factory-prod]
(local.custom_roles.service_project_network_admin) = [
local.service_accounts.project-factory-prod
]
}
}
@ -50,6 +53,7 @@ module "prod-spoke-vpc" {
mtu = 1500
data_folder = "${var.data_dir}/subnets/prod"
delete_default_routes_on_create = true
psa_ranges = var.psa_ranges.prod
subnets_l7ilb = local.l7ilb_subnets.prod
# Set explicit routes for googleapis; send everything else to NVAs
routes = {
@ -110,17 +114,6 @@ module "prod-spoke-firewall" {
cidr_template_file = "${var.data_dir}/cidrs.yaml"
}
module "prod-spoke-psa-addresses" {
source = "../../../modules/net-address"
project_id = module.prod-spoke-project.project_id
psa_addresses = { for r, v in var.psa_ranges.prod : r => {
address = cidrhost(v, 0)
network = module.prod-spoke-vpc.self_link
prefix_length = split("/", v)[1]
}
}
}
module "peering-prod" {
source = "../../../modules/net-vpc-peering"
prefix = "prod-peering-0"

41
fast/stages/02-networking-nva/variables.tf
View File

@ -14,10 +14,13 @@
* limitations under the License.
*/
variable "billing_account_id" {
variable "billing_account" {
# tfdoc:variable:source 00-bootstrap
description = "Billing account id."
type = string
description = "Billing account id and organization id ('nnnnnnnn' or null)."
type = object({
id = string
organization_id = number
})
}
variable "custom_adv" {
@ -42,6 +45,15 @@ variable "custom_adv" {
}
}
variable "custom_roles" {
# tfdoc:variable:source 00-bootstrap
description = "Custom roles defined at the org level, in key => id format."
type = object({
service_project_network_admin = string
})
default = null
}
variable "data_dir" {
description = "Relative path for the folder storing configuration data for network resources."
type = string
@ -59,7 +71,11 @@ variable "dns" {
variable "folder_ids" {
# tfdoc:variable:source 01-resman
description = "Folders to be used for the networking resources in folders/nnnnnnnnnnn format. If null, folder will be created."
type = map(string)
type = object({
networking = string
networking-dev = string
networking-prod = string
})
}
variable "l7ilb_subnets" {
@ -115,13 +131,6 @@ variable "prefix" {
}
}
variable "project_factory_sa" {
# tfdoc:variable:source 01-resman
description = "IAM emails for project factory service accounts"
type = map(string)
default = {}
}
variable "psa_ranges" {
description = "IP ranges used for Private Service Access (e.g. CloudSQL)."
type = map(map(string))
@ -164,6 +173,16 @@ variable "router_configs" {
}
}
variable "service_accounts" {
# tfdoc:variable:source 01-resman
description = "Automation service accounts in name => email format."
type = object({
project-factory-dev = string
project-factory-prod = string
})
default = null
}
variable "vpn_onprem_configs" {
description = "VPN gateway configuration for onprem interconnection."
type = map(object({

47
fast/stages/02-networking-vpn/README.md
View File

@ -296,8 +296,8 @@ DNS configurations are centralised in the `dns.tf` file. Spokes delegate DNS res
| [main.tf](./main.tf) | Networking folder and hierarchical policy. | <code>folder</code> | |
| [monitoring.tf](./monitoring.tf) | Network monitoring dashboards. | | <code>google_monitoring_dashboard</code> |
| [outputs.tf](./outputs.tf) | Module outputs. | | <code>local_file</code> |
| [spoke-dev.tf](./spoke-dev.tf) | Dev spoke VPC and related resources. | <code>net-address</code> · <code>net-cloudnat</code> · <code>net-vpc</code> · <code>net-vpc-firewall</code> · <code>project</code> | <code>google_project_iam_binding</code> |
| [spoke-prod.tf](./spoke-prod.tf) | Production spoke VPC and related resources. | <code>net-address</code> · <code>net-cloudnat</code> · <code>net-vpc</code> · <code>net-vpc-firewall</code> · <code>project</code> | <code>google_project_iam_binding</code> |
| [spoke-dev.tf](./spoke-dev.tf) | Dev spoke VPC and related resources. | <code>net-cloudnat</code> · <code>net-vpc</code> · <code>net-vpc-firewall</code> · <code>project</code> | <code>google_project_iam_binding</code> |
| [spoke-prod.tf](./spoke-prod.tf) | Production spoke VPC and related resources. | <code>net-cloudnat</code> · <code>net-vpc</code> · <code>net-vpc-firewall</code> · <code>project</code> | <code>google_project_iam_binding</code> |
| [test-resources.tf](./test-resources.tf) | temporary instances for testing | <code>compute-vm</code> | |
| [variables.tf](./variables.tf) | Module variables. | | |
| [vpn-onprem.tf](./vpn-onprem.tf) | VPN between landing and onprem. | <code>net-vpn-ha</code> | |
@ -308,32 +308,31 @@ DNS configurations are centralised in the `dns.tf` file. Spokes delegate DNS res
| name | description | type | required | default | producer |
|---|---|:---:|:---:|:---:|:---:|
| [billing_account_id](variables.tf#L17) | Billing account id. | <code>string</code> | ✓ | | <code>00-bootstrap</code> |
| [folder_ids](variables.tf#L61) | Folders to be used for the networking resources in folders/nnnnnnnnnnn format. If null, folder will be created. | <code>map&#40;string&#41;</code> | ✓ | | <code>01-resman</code> |
| [organization](variables.tf#L85) | Organization details. | <code title="object&#40;&#123;&#10; domain &#61; string&#10; id &#61; number&#10; customer_id &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>00-bootstrap</code> |
| [prefix](variables.tf#L101) | Prefix used for resources that need unique names. Use 9 characters or less. | <code>string</code> | ✓ | | <code>00-bootstrap</code> |
| [custom_adv](variables.tf#L23) | Custom advertisement definitions in name => range format. | <code>map&#40;string&#41;</code> | | <code title="&#123;&#10; cloud_dns &#61; &#34;35.199.192.0&#47;19&#34;&#10; gcp_all &#61; &#34;10.128.0.0&#47;16&#34;&#10; gcp_dev &#61; &#34;10.128.32.0&#47;19&#34;&#10; gcp_landing &#61; &#34;10.128.0.0&#47;19&#34;&#10; gcp_prod &#61; &#34;10.128.64.0&#47;19&#34;&#10; googleapis_private &#61; &#34;199.36.153.8&#47;30&#34;&#10; googleapis_restricted &#61; &#34;199.36.153.4&#47;30&#34;&#10; rfc_1918_10 &#61; &#34;10.0.0.0&#47;8&#34;&#10; rfc_1918_172 &#61; &#34;172.16.0.0&#47;12&#34;&#10; rfc_1918_192 &#61; &#34;192.168.0.0&#47;16&#34;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [custom_roles](variables.tf#L40) | Custom roles defined at the org level, in key => id format. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> | <code>00-bootstrap</code> |
| [data_dir](variables.tf#L47) | Relative path for the folder storing configuration data for network resources. | <code>string</code> | | <code>&#34;data&#34;</code> | |
| [dns](variables.tf#L53) | Onprem DNS resolvers. | <code>map&#40;list&#40;string&#41;&#41;</code> | | <code title="&#123;&#10; onprem &#61; &#91;&#34;10.0.200.3&#34;&#93;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [l7ilb_subnets](variables.tf#L67) | Subnets used for L7 ILBs. | <code title="map&#40;list&#40;object&#40;&#123;&#10; ip_cidr_range &#61; string&#10; region &#61; string&#10;&#125;&#41;&#41;&#41;">map&#40;list&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;&#41;</code> | | <code title="&#123;&#10; prod &#61; &#91;&#10; &#123; ip_cidr_range &#61; &#34;10.128.92.0&#47;24&#34;, region &#61; &#34;europe-west1&#34; &#125;,&#10; &#123; ip_cidr_range &#61; &#34;10.128.93.0&#47;24&#34;, region &#61; &#34;europe-west4&#34; &#125;&#10; &#93;&#10; dev &#61; &#91;&#10; &#123; ip_cidr_range &#61; &#34;10.128.60.0&#47;24&#34;, region &#61; &#34;europe-west1&#34; &#125;,&#10; &#123; ip_cidr_range &#61; &#34;10.128.61.0&#47;24&#34;, region &#61; &#34;europe-west4&#34; &#125;&#10; &#93;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [outputs_location](variables.tf#L95) | Path where providers and tfvars files for the following stages are written. Leave empty to disable. | <code>string</code> | | <code>null</code> | |
| [project_factory_sa](variables.tf#L112) | IAM emails for project factory service accounts. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> | <code>01-resman</code> |
| [psa_ranges](variables.tf#L119) | IP ranges used for Private Service Access (e.g. CloudSQL). | <code>map&#40;map&#40;string&#41;&#41;</code> | | <code title="&#123;&#10; prod &#61; &#123;&#10; cloudsql-mysql &#61; &#34;10.128.94.0&#47;24&#34;&#10; cloudsql-sqlserver &#61; &#34;10.128.95.0&#47;24&#34;&#10; &#125;&#10; dev &#61; &#123;&#10; cloudsql-mysql &#61; &#34;10.128.62.0&#47;24&#34;&#10; cloudsql-sqlserver &#61; &#34;10.128.63.0&#47;24&#34;&#10; &#125;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [router_configs](variables.tf#L134) | Configurations for CRs and onprem routers. | <code title="map&#40;object&#40;&#123;&#10; adv &#61; object&#40;&#123;&#10; custom &#61; list&#40;string&#41;&#10; default &#61; bool&#10; &#125;&#41;&#10; asn &#61; number&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code title="&#123;&#10; onprem-ew1 &#61; &#123;&#10; asn &#61; &#34;65534&#34;&#10; adv &#61; null&#10; &#125;&#10; landing-ew1 &#61; &#123; asn &#61; &#34;64512&#34;, adv &#61; null &#125;&#10; landing-ew4 &#61; &#123; asn &#61; &#34;64512&#34;, adv &#61; null &#125;&#10; spoke-dev-ew1 &#61; &#123; asn &#61; &#34;64513&#34;, adv &#61; null &#125;&#10; spoke-dev-ew4 &#61; &#123; asn &#61; &#34;64513&#34;, adv &#61; null &#125;&#10; spoke-prod-ew1 &#61; &#123; asn &#61; &#34;64514&#34;, adv &#61; null &#125;&#10; spoke-prod-ew4 &#61; &#123; asn &#61; &#34;64514&#34;, adv &#61; null &#125;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [vpn_onprem_configs](variables.tf#L158) | VPN gateway configuration for onprem interconnection. | <code title="map&#40;object&#40;&#123;&#10; adv &#61; object&#40;&#123;&#10; default &#61; bool&#10; custom &#61; list&#40;string&#41;&#10; &#125;&#41;&#10; peer_external_gateway &#61; object&#40;&#123;&#10; redundancy_type &#61; string&#10; interfaces &#61; list&#40;object&#40;&#123;&#10; id &#61; number&#10; ip_address &#61; string&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#10; tunnels &#61; list&#40;object&#40;&#123;&#10; peer_asn &#61; number&#10; peer_external_gateway_interface &#61; number&#10; secret &#61; string&#10; session_range &#61; string&#10; vpn_gateway_interface &#61; number&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code title="&#123;&#10; landing-ew1 &#61; &#123;&#10; adv &#61; &#123;&#10; default &#61; false&#10; custom &#61; &#91;&#10; &#34;cloud_dns&#34;, &#34;googleapis_private&#34;, &#34;googleapis_restricted&#34;, &#34;gcp_all&#34;&#10; &#93;&#10; &#125;&#10; peer_external_gateway &#61; &#123;&#10; redundancy_type &#61; &#34;SINGLE_IP_INTERNALLY_REDUNDANT&#34;&#10; interfaces &#61; &#91;&#10; &#123; id &#61; 0, ip_address &#61; &#34;8.8.8.8&#34; &#125;,&#10; &#93;&#10; &#125;&#10; tunnels &#61; &#91;&#10; &#123;&#10; peer_asn &#61; 65534&#10; peer_external_gateway_interface &#61; 0&#10; secret &#61; &#34;foobar&#34;&#10; session_range &#61; &#34;169.254.1.0&#47;30&#34;&#10; vpn_gateway_interface &#61; 0&#10; &#125;,&#10; &#123;&#10; peer_asn &#61; 65534&#10; peer_external_gateway_interface &#61; 0&#10; secret &#61; &#34;foobar&#34;&#10; session_range &#61; &#34;169.254.1.4&#47;30&#34;&#10; vpn_gateway_interface &#61; 1&#10; &#125;&#10; &#93;&#10; &#125;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [vpn_spoke_configs](variables.tf#L214) | VPN gateway configuration for spokes. | <code title="map&#40;object&#40;&#123;&#10; adv &#61; object&#40;&#123;&#10; default &#61; bool&#10; custom &#61; list&#40;string&#41;&#10; &#125;&#41;&#10; session_range &#61; string&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code title="&#123;&#10; landing-ew1 &#61; &#123;&#10; adv &#61; &#123;&#10; default &#61; false&#10; custom &#61; &#91;&#34;rfc_1918_10&#34;, &#34;rfc_1918_172&#34;, &#34;rfc_1918_192&#34;&#93;&#10; &#125;&#10; session_range &#61; null&#10; &#125;&#10; landing-ew4 &#61; &#123;&#10; adv &#61; &#123;&#10; default &#61; false&#10; custom &#61; &#91;&#34;rfc_1918_10&#34;, &#34;rfc_1918_172&#34;, &#34;rfc_1918_192&#34;&#93;&#10; &#125;&#10; session_range &#61; null&#10; &#125;&#10; dev-ew1 &#61; &#123;&#10; adv &#61; &#123;&#10; default &#61; false&#10; custom &#61; &#91;&#34;gcp_dev&#34;&#93;&#10; &#125;&#10; session_range &#61; &#34;169.254.0.0&#47;27&#34;&#10; &#125;&#10; prod-ew1 &#61; &#123;&#10; adv &#61; &#123;&#10; default &#61; false&#10; custom &#61; &#91;&#34;gcp_prod&#34;&#93;&#10; &#125;&#10; session_range &#61; &#34;169.254.0.64&#47;27&#34;&#10; &#125;&#10; prod-ew4 &#61; &#123;&#10; adv &#61; &#123;&#10; default &#61; false&#10; custom &#61; &#91;&#34;gcp_prod&#34;&#93;&#10; &#125;&#10; session_range &#61; &#34;169.254.0.96&#47;27&#34;&#10; &#125;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [billing_account](variables.tf#L17) | Billing account id and organization id ('nnnnnnnn' or null). | <code title="object&#40;&#123;&#10; id &#61; string&#10; organization_id &#61; number&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>00-bootstrap</code> |
| [folder_ids](variables.tf#L66) | Folders to be used for the networking resources in folders/nnnnnnnnnnn format. If null, folder will be created. | <code title="object&#40;&#123;&#10; networking &#61; string&#10; networking-dev &#61; string&#10; networking-prod &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>01-resman</code> |
| [organization](variables.tf#L94) | Organization details. | <code title="object&#40;&#123;&#10; domain &#61; string&#10; id &#61; number&#10; customer_id &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>00-bootstrap</code> |
| [prefix](variables.tf#L110) | Prefix used for resources that need unique names. Use 9 characters or less. | <code>string</code> | ✓ | | <code>00-bootstrap</code> |
| [custom_adv](variables.tf#L26) | Custom advertisement definitions in name => range format. | <code>map&#40;string&#41;</code> | | <code title="&#123;&#10; cloud_dns &#61; &#34;35.199.192.0&#47;19&#34;&#10; gcp_all &#61; &#34;10.128.0.0&#47;16&#34;&#10; gcp_dev &#61; &#34;10.128.32.0&#47;19&#34;&#10; gcp_landing &#61; &#34;10.128.0.0&#47;19&#34;&#10; gcp_prod &#61; &#34;10.128.64.0&#47;19&#34;&#10; googleapis_private &#61; &#34;199.36.153.8&#47;30&#34;&#10; googleapis_restricted &#61; &#34;199.36.153.4&#47;30&#34;&#10; rfc_1918_10 &#61; &#34;10.0.0.0&#47;8&#34;&#10; rfc_1918_172 &#61; &#34;172.16.0.0&#47;12&#34;&#10; rfc_1918_192 &#61; &#34;192.168.0.0&#47;16&#34;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [custom_roles](variables.tf#L43) | Custom roles defined at the org level, in key => id format. | <code title="object&#40;&#123;&#10; service_project_network_admin &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> | <code>00-bootstrap</code> |
| [data_dir](variables.tf#L52) | Relative path for the folder storing configuration data for network resources. | <code>string</code> | | <code>&#34;data&#34;</code> | |
| [dns](variables.tf#L58) | Onprem DNS resolvers. | <code>map&#40;list&#40;string&#41;&#41;</code> | | <code title="&#123;&#10; onprem &#61; &#91;&#34;10.0.200.3&#34;&#93;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [l7ilb_subnets](variables.tf#L76) | Subnets used for L7 ILBs. | <code title="map&#40;list&#40;object&#40;&#123;&#10; ip_cidr_range &#61; string&#10; region &#61; string&#10;&#125;&#41;&#41;&#41;">map&#40;list&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;&#41;</code> | | <code title="&#123;&#10; prod &#61; &#91;&#10; &#123; ip_cidr_range &#61; &#34;10.128.92.0&#47;24&#34;, region &#61; &#34;europe-west1&#34; &#125;,&#10; &#123; ip_cidr_range &#61; &#34;10.128.93.0&#47;24&#34;, region &#61; &#34;europe-west4&#34; &#125;&#10; &#93;&#10; dev &#61; &#91;&#10; &#123; ip_cidr_range &#61; &#34;10.128.60.0&#47;24&#34;, region &#61; &#34;europe-west1&#34; &#125;,&#10; &#123; ip_cidr_range &#61; &#34;10.128.61.0&#47;24&#34;, region &#61; &#34;europe-west4&#34; &#125;&#10; &#93;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [outputs_location](variables.tf#L104) | Path where providers and tfvars files for the following stages are written. Leave empty to disable. | <code>string</code> | | <code>null</code> | |
| [psa_ranges](variables.tf#L121) | IP ranges used for Private Service Access (e.g. CloudSQL). | <code>map&#40;map&#40;string&#41;&#41;</code> | | <code title="&#123;&#10; prod &#61; &#123;&#10; cloudsql-mysql &#61; &#34;10.128.94.0&#47;24&#34;&#10; cloudsql-sqlserver &#61; &#34;10.128.95.0&#47;24&#34;&#10; &#125;&#10; dev &#61; &#123;&#10; cloudsql-mysql &#61; &#34;10.128.62.0&#47;24&#34;&#10; cloudsql-sqlserver &#61; &#34;10.128.63.0&#47;24&#34;&#10; &#125;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [router_configs](variables.tf#L136) | Configurations for CRs and onprem routers. | <code title="map&#40;object&#40;&#123;&#10; adv &#61; object&#40;&#123;&#10; custom &#61; list&#40;string&#41;&#10; default &#61; bool&#10; &#125;&#41;&#10; asn &#61; number&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code title="&#123;&#10; onprem-ew1 &#61; &#123;&#10; asn &#61; &#34;65534&#34;&#10; adv &#61; null&#10; &#125;&#10; landing-ew1 &#61; &#123; asn &#61; &#34;64512&#34;, adv &#61; null &#125;&#10; landing-ew4 &#61; &#123; asn &#61; &#34;64512&#34;, adv &#61; null &#125;&#10; spoke-dev-ew1 &#61; &#123; asn &#61; &#34;64513&#34;, adv &#61; null &#125;&#10; spoke-dev-ew4 &#61; &#123; asn &#61; &#34;64513&#34;, adv &#61; null &#125;&#10; spoke-prod-ew1 &#61; &#123; asn &#61; &#34;64514&#34;, adv &#61; null &#125;&#10; spoke-prod-ew4 &#61; &#123; asn &#61; &#34;64514&#34;, adv &#61; null &#125;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [service_accounts](variables.tf#L160) | Automation service accounts in name => email format. | <code title="object&#40;&#123;&#10; project-factory-dev &#61; string&#10; project-factory-prod &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> | <code>01-resman</code> |
| [vpn_onprem_configs](variables.tf#L170) | VPN gateway configuration for onprem interconnection. | <code title="map&#40;object&#40;&#123;&#10; adv &#61; object&#40;&#123;&#10; default &#61; bool&#10; custom &#61; list&#40;string&#41;&#10; &#125;&#41;&#10; peer_external_gateway &#61; object&#40;&#123;&#10; redundancy_type &#61; string&#10; interfaces &#61; list&#40;object&#40;&#123;&#10; id &#61; number&#10; ip_address &#61; string&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#10; tunnels &#61; list&#40;object&#40;&#123;&#10; peer_asn &#61; number&#10; peer_external_gateway_interface &#61; number&#10; secret &#61; string&#10; session_range &#61; string&#10; vpn_gateway_interface &#61; number&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code title="&#123;&#10; landing-ew1 &#61; &#123;&#10; adv &#61; &#123;&#10; default &#61; false&#10; custom &#61; &#91;&#10; &#34;cloud_dns&#34;, &#34;googleapis_private&#34;, &#34;googleapis_restricted&#34;, &#34;gcp_all&#34;&#10; &#93;&#10; &#125;&#10; peer_external_gateway &#61; &#123;&#10; redundancy_type &#61; &#34;SINGLE_IP_INTERNALLY_REDUNDANT&#34;&#10; interfaces &#61; &#91;&#10; &#123; id &#61; 0, ip_address &#61; &#34;8.8.8.8&#34; &#125;,&#10; &#93;&#10; &#125;&#10; tunnels &#61; &#91;&#10; &#123;&#10; peer_asn &#61; 65534&#10; peer_external_gateway_interface &#61; 0&#10; secret &#61; &#34;foobar&#34;&#10; session_range &#61; &#34;169.254.1.0&#47;30&#34;&#10; vpn_gateway_interface &#61; 0&#10; &#125;,&#10; &#123;&#10; peer_asn &#61; 65534&#10; peer_external_gateway_interface &#61; 0&#10; secret &#61; &#34;foobar&#34;&#10; session_range &#61; &#34;169.254.1.4&#47;30&#34;&#10; vpn_gateway_interface &#61; 1&#10; &#125;&#10; &#93;&#10; &#125;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [vpn_spoke_configs](variables.tf#L226) | VPN gateway configuration for spokes. | <code title="map&#40;object&#40;&#123;&#10; adv &#61; object&#40;&#123;&#10; default &#61; bool&#10; custom &#61; list&#40;string&#41;&#10; &#125;&#41;&#10; session_range &#61; string&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code title="&#123;&#10; landing-ew1 &#61; &#123;&#10; adv &#61; &#123;&#10; default &#61; false&#10; custom &#61; &#91;&#34;rfc_1918_10&#34;, &#34;rfc_1918_172&#34;, &#34;rfc_1918_192&#34;&#93;&#10; &#125;&#10; session_range &#61; null&#10; &#125;&#10; landing-ew4 &#61; &#123;&#10; adv &#61; &#123;&#10; default &#61; false&#10; custom &#61; &#91;&#34;rfc_1918_10&#34;, &#34;rfc_1918_172&#34;, &#34;rfc_1918_192&#34;&#93;&#10; &#125;&#10; session_range &#61; null&#10; &#125;&#10; dev-ew1 &#61; &#123;&#10; adv &#61; &#123;&#10; default &#61; false&#10; custom &#61; &#91;&#34;gcp_dev&#34;&#93;&#10; &#125;&#10; session_range &#61; &#34;169.254.0.0&#47;27&#34;&#10; &#125;&#10; prod-ew1 &#61; &#123;&#10; adv &#61; &#123;&#10; default &#61; false&#10; custom &#61; &#91;&#34;gcp_prod&#34;&#93;&#10; &#125;&#10; session_range &#61; &#34;169.254.0.64&#47;27&#34;&#10; &#125;&#10; prod-ew4 &#61; &#123;&#10; adv &#61; &#123;&#10; default &#61; false&#10; custom &#61; &#91;&#34;gcp_prod&#34;&#93;&#10; &#125;&#10; session_range &#61; &#34;169.254.0.96&#47;27&#34;&#10; &#125;&#10;&#125;">&#123;&#8230;&#125;</code> | |
## Outputs
| name | description | sensitive | consumers |
|---|---|:---:|---|
| [cloud_dns_inbound_policy](outputs.tf#L41) | IP Addresses for Cloud DNS inbound policy. | | |
| [project_ids](outputs.tf#L46) | Network project ids. | | |
| [project_numbers](outputs.tf#L55) | Network project numbers. | | |
| [shared_vpc_host_projects](outputs.tf#L64) | Shared VPC host projects. | | |
| [shared_vpc_self_links](outputs.tf#L74) | Shared VPC host projects. | | |
| [tfvars](outputs.tf#L91) | Network-related variables used in other stages. | ✓ | |
| [vpn_gateway_endpoints](outputs.tf#L84) | External IP Addresses for the GCP VPN gateways. | | |
| [cloud_dns_inbound_policy](outputs.tf#L51) | IP Addresses for Cloud DNS inbound policy. | | |
| [host_project_ids](outputs.tf#L56) | Network project ids. | | |
| [host_project_numbers](outputs.tf#L61) | Network project numbers. | | |
| [shared_vpc_self_links](outputs.tf#L66) | Shared VPC host projects. | | |
| [tfvars](outputs.tf#L81) | Terraform variables file for the following stages. | ✓ | |
| [vpn_gateway_endpoints](outputs.tf#L71) | External IP Addresses for the GCP VPN gateways. | | |
<!-- END TFDOC -->

9
fast/stages/02-networking-vpn/landing.tf
View File

@ -18,7 +18,7 @@
module "landing-project" {
source = "../../../modules/project"
billing_account = var.billing_account_id
billing_account = var.billing_account.id
name = "prod-net-landing-0"
parent = var.folder_ids.networking-prod
prefix = var.prefix
@ -37,6 +37,13 @@ module "landing-project" {
enabled = true
service_projects = []
}
metric_scopes = [module.landing-project.project_id]
iam = {
"roles/dns.admin" = [local.service_accounts.project-factory-prod]
(local.custom_roles.service_project_network_admin) = [
local.service_accounts.project-factory-prod
]
}
}
module "landing-vpc" {

4
fast/stages/02-networking-vpn/main.tf
View File

@ -30,6 +30,7 @@ locals {
route_priority = null
}
}
custom_roles = coalesce(var.custom_roles, {})
l7ilb_subnets = {
for env, v in var.l7ilb_subnets : env => [
for s in v : merge(s, {
@ -47,6 +48,9 @@ locals {
"roles/container.hostServiceAgentUser",
"roles/vpcaccess.user",
]
service_accounts = {
for k, v in coalesce(var.service_accounts, {}) : k => "serviceAccount:${v}"
}
}
module "folder" {

78
fast/stages/02-networking-vpn/outputs.tf
View File

@ -13,27 +13,37 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
# optionally generate providers and tfvars files for subsequent stages
locals {
host_project_ids = {
dev-spoke-0 = module.dev-spoke-project.project_id
prod-landing = module.landing-project.project_id
prod-spoke-0 = module.prod-spoke-project.project_id
}
host_project_numbers = {
dev-spoke-0 = module.dev-spoke-project.number
prod-landing = module.landing-project.number
prod-spoke-0 = module.prod-spoke-project.number
}
tfvars = {
"03-project-factory-dev" = jsonencode({
environment_dns_zone = module.dev-dns-private-zone.domain
shared_vpc_self_link = module.dev-spoke-vpc.self_link
vpc_host_project = module.dev-spoke-project.project_id
})
"03-project-factory-prod" = jsonencode({
environment_dns_zone = module.prod-dns-private-zone.domain
shared_vpc_self_link = module.prod-spoke-vpc.self_link
vpc_host_project = module.prod-spoke-project.project_id
})
host_project_ids = local.host_project_ids
host_project_numbers = local.host_project_numbers
vpc_self_links = local.vpc_self_links
}
vpc_self_links = {
prod-landing = module.landing-vpc.self_link
dev-spoke-0 = module.dev-spoke-vpc.self_link
prod-spoke-0 = module.prod-spoke-vpc.self_link
}
}
# optionally generate tfvars file for subsequent stages
resource "local_file" "tfvars" {
for_each = var.outputs_location == null ? {} : local.tfvars
filename = "${pathexpand(var.outputs_location)}/${each.key}/terraform-networking.auto.tfvars.json"
content = each.value
for_each = var.outputs_location == null ? {} : { 1 = 1 }
file_permission = "0644"
filename = "${pathexpand(var.outputs_location)}/tfvars/02-networking.auto.tfvars.json"
content = jsonencode(local.tfvars)
}
# outputs
@ -43,53 +53,33 @@ output "cloud_dns_inbound_policy" {
value = [for s in module.landing-vpc.subnets : cidrhost(s.ip_cidr_range, 2)]
}
output "project_ids" {
output "host_project_ids" {
description = "Network project ids."
value = {
dev = module.dev-spoke-project.project_id
landing = module.landing-project.project_id
prod = module.prod-spoke-project.project_id
}
value = local.host_project_ids
}
output "project_numbers" {
output "host_project_numbers" {
description = "Network project numbers."
value = {
dev = "projects/${module.dev-spoke-project.number}"
landing = "projects/${module.landing-project.number}"
prod = "projects/${module.prod-spoke-project.number}"
}
value = local.host_project_numbers
}
output "shared_vpc_host_projects" {
description = "Shared VPC host projects."
value = {
landing = module.landing-project.project_id
dev = module.dev-spoke-project.project_id
prod = module.prod-spoke-project.project_id
}
}
output "shared_vpc_self_links" {
description = "Shared VPC host projects."
value = {
landing = module.landing-vpc.self_link
dev = module.dev-spoke-vpc.self_link
prod = module.prod-spoke-vpc.self_link
}
value = local.vpc_self_links
}
output "vpn_gateway_endpoints" {
description = "External IP Addresses for the GCP VPN gateways."
value = {
onprem-ew1 = { for v in module.landing-to-onprem-ew1-vpn.gateway.vpn_interfaces : v.id => v.ip_address }
onprem-ew1 = {
for v in module.landing-to-onprem-ew1-vpn.gateway.vpn_interfaces :
v.id => v.ip_address
}
}
}
output "tfvars" {
description = "Network-related variables used in other stages."
description = "Terraform variables file for the following stages."
sensitive = true
value = local.tfvars
}

22
fast/stages/02-networking-vpn/spoke-dev.tf
View File

@ -18,7 +18,7 @@
module "dev-spoke-project" {
source = "../../../modules/project"
billing_account = var.billing_account_id
billing_account = var.billing_account.id
name = "dev-net-spoke-0"
parent = var.folder_ids.networking-dev
prefix = var.prefix
@ -39,9 +39,9 @@ module "dev-spoke-project" {
}
metric_scopes = [module.landing-project.project_id]
iam = {
"roles/dns.admin" = [var.project_factory_sa.dev]
(var.custom_roles.service_project_network_admin) = [
var.project_factory_sa.prod
"roles/dns.admin" = [local.service_accounts.project-factory-dev]
(local.custom_roles.service_project_network_admin) = [
local.service_accounts.project-factory-dev
]
}
}
@ -52,6 +52,7 @@ module "dev-spoke-vpc" {
name = "dev-spoke-0"
mtu = 1500
data_folder = "${var.data_dir}/subnets/dev"
psa_ranges = var.psa_ranges.dev
subnets_l7ilb = local.l7ilb_subnets.dev
# set explicit routes for googleapis in case the default route is deleted
routes = {
@ -96,23 +97,12 @@ module "dev-spoke-cloudnat" {
logging_filter = "ERRORS_ONLY"
}
module "dev-spoke-psa-addresses" {
source = "../../../modules/net-address"
project_id = module.dev-spoke-project.project_id
psa_addresses = { for r, v in var.psa_ranges.dev : r => {
address = cidrhost(v, 0)
network = module.dev-spoke-vpc.self_link
prefix_length = split("/", v)[1]
}
}
}
# Create delegated grants for stage3 service accounts
resource "google_project_iam_binding" "dev_spoke_project_iam_delegated" {
project = module.dev-spoke-project.project_id
role = "roles/resourcemanager.projectIamAdmin"
members = [
var.project_factory_sa.dev
local.service_accounts.project-factory-dev
]
condition {
title = "dev_stage3_sa_delegated_grants"

22
fast/stages/02-networking-vpn/spoke-prod.tf
View File

@ -18,7 +18,7 @@
module "prod-spoke-project" {
source = "../../../modules/project"
billing_account = var.billing_account_id
billing_account = var.billing_account.id
name = "prod-net-spoke-0"
parent = var.folder_ids.networking-prod
prefix = var.prefix
@ -39,9 +39,9 @@ module "prod-spoke-project" {
}
metric_scopes = [module.landing-project.project_id]
iam = {
"roles/dns.admin" = [var.project_factory_sa.prod]
(var.custom_roles.service_project_network_admin) = [
var.project_factory_sa.prod
"roles/dns.admin" = [local.service_accounts.project-factory-prod]
(local.custom_roles.service_project_network_admin) = [
local.service_accounts.project-factory-prod
]
}
}
@ -52,6 +52,7 @@ module "prod-spoke-vpc" {
name = "prod-spoke-0"
mtu = 1500
data_folder = "${var.data_dir}/subnets/prod"
psa_ranges = var.psa_ranges.prod
subnets_l7ilb = local.l7ilb_subnets.prod
# set explicit routes for googleapis in case the default route is deleted
routes = {
@ -96,23 +97,12 @@ module "prod-spoke-cloudnat" {
logging_filter = "ERRORS_ONLY"
}
module "prod-spoke-psa-addresses" {
source = "../../../modules/net-address"
project_id = module.prod-spoke-project.project_id
psa_addresses = { for r, v in var.psa_ranges.prod : r => {
address = cidrhost(v, 0)
network = module.prod-spoke-vpc.self_link
prefix_length = split("/", v)[1]
}
}
}
# Create delegated grants for stage3 service accounts
resource "google_project_iam_binding" "prod_spoke_project_iam_delegated" {
project = module.prod-spoke-project.project_id
role = "roles/resourcemanager.projectIamAdmin"
members = [
var.project_factory_sa.prod
local.service_accounts.project-factory-prod
]
condition {
title = "prod_stage3_sa_delegated_grants"

38
fast/stages/02-networking-vpn/variables.tf
View File

@ -14,10 +14,13 @@
* limitations under the License.
*/
variable "billing_account_id" {
variable "billing_account" {
# tfdoc:variable:source 00-bootstrap
description = "Billing account id."
type = string
description = "Billing account id and organization id ('nnnnnnnn' or null)."
type = object({
id = string
organization_id = number
})
}
variable "custom_adv" {
@ -40,8 +43,10 @@ variable "custom_adv" {
variable "custom_roles" {
# tfdoc:variable:source 00-bootstrap
description = "Custom roles defined at the org level, in key => id format."
type = map(string)
default = {}
type = object({
service_project_network_admin = string
})
default = null
}
variable "data_dir" {
@ -61,7 +66,11 @@ variable "dns" {
variable "folder_ids" {
# tfdoc:variable:source 01-resman
description = "Folders to be used for the networking resources in folders/nnnnnnnnnnn format. If null, folder will be created."
type = map(string)
type = object({
networking = string
networking-dev = string
networking-prod = string
})
}
variable "l7ilb_subnets" {
@ -109,13 +118,6 @@ variable "prefix" {
}
}
variable "project_factory_sa" {
# tfdoc:variable:source 01-resman
description = "IAM emails for project factory service accounts."
type = map(string)
default = {}
}
variable "psa_ranges" {
description = "IP ranges used for Private Service Access (e.g. CloudSQL)."
type = map(map(string))
@ -155,6 +157,16 @@ variable "router_configs" {
}
}
variable "service_accounts" {
# tfdoc:variable:source 01-resman
description = "Automation service accounts in name => email format."
type = object({
project-factory-dev = string
project-factory-prod = string
})
default = null
}
variable "vpn_onprem_configs" {
description = "VPN gateway configuration for onprem interconnection."
type = map(object({

36
fast/stages/02-security/README.md
View File

@ -285,27 +285,29 @@ Some references that might be useful in setting up this stage:
| name | description | type | required | default | producer |
|---|---|:---:|:---:|:---:|:---:|
| [billing_account_id](variables.tf#L17) | Billing account id. | <code>string</code> | ✓ | | <code>bootstrap</code> |
| [folder_id](variables.tf#L23) | Folder to be used for the networking resources in folders/nnnn format. | <code>string</code> | ✓ | | <code>resman</code> |
| [organization](variables.tf#L73) | Organization details. | <code title="object&#40;&#123;&#10; domain &#61; string&#10; id &#61; number&#10; customer_id &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>bootstrap</code> |
| [prefix](variables.tf#L89) | Prefix used for resources that need unique names. Use 9 characters or less. | <code>string</code> | ✓ | | <code>00-bootstrap</code> |
| [groups](variables.tf#L29) | Group names to grant organization-level permissions. | <code>map&#40;string&#41;</code> | | <code title="&#123;&#10; gcp-billing-admins &#61; &#34;gcp-billing-admins&#34;,&#10; gcp-devops &#61; &#34;gcp-devops&#34;,&#10; gcp-network-admins &#61; &#34;gcp-network-admins&#34;&#10; gcp-organization-admins &#61; &#34;gcp-organization-admins&#34;&#10; gcp-security-admins &#61; &#34;gcp-security-admins&#34;&#10; gcp-support &#61; &#34;gcp-support&#34;&#10;&#125;">&#123;&#8230;&#125;</code> | <code>bootstrap</code> |
| [kms_defaults](variables.tf#L44) | Defaults used for KMS keys. | <code title="object&#40;&#123;&#10; locations &#61; list&#40;string&#41;&#10; rotation_period &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; locations &#61; &#91;&#34;europe&#34;, &#34;europe-west1&#34;, &#34;europe-west3&#34;, &#34;global&#34;&#93;&#10; rotation_period &#61; &#34;7776000s&#34;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [kms_keys](variables.tf#L56) | KMS keys to create, keyed by name. Null attributes will be interpolated with defaults. | <code title="map&#40;object&#40;&#123;&#10; iam &#61; map&#40;list&#40;string&#41;&#41;&#10; labels &#61; map&#40;string&#41;&#10; locations &#61; list&#40;string&#41;&#10; rotation_period &#61; string&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> | |
| [kms_restricted_admins](variables.tf#L67) | Map of environment => [identities] who can assign the encrypt/decrypt roles on keys. | <code>map&#40;list&#40;string&#41;&#41;</code> | | <code>&#123;&#125;</code> | |
| [outputs_location](variables.tf#L83) | Path where providers, tfvars files, and lists for the following stages are written. Leave empty to disable. | <code>string</code> | | <code>null</code> | |
| [vpc_sc_access_levels](variables.tf#L100) | VPC SC access level definitions. | <code title="map&#40;object&#40;&#123;&#10; combining_function &#61; string&#10; conditions &#61; list&#40;object&#40;&#123;&#10; ip_subnetworks &#61; list&#40;string&#41;&#10; members &#61; list&#40;string&#41;&#10; negate &#61; bool&#10; regions &#61; list&#40;string&#41;&#10; required_access_levels &#61; list&#40;string&#41;&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> | |
| [vpc_sc_egress_policies](variables.tf#L115) | VPC SC egress policy defnitions. | <code title="map&#40;object&#40;&#123;&#10; egress_from &#61; object&#40;&#123;&#10; identity_type &#61; string&#10; identities &#61; list&#40;string&#41;&#10; &#125;&#41;&#10; egress_to &#61; object&#40;&#123;&#10; operations &#61; list&#40;object&#40;&#123;&#10; method_selectors &#61; list&#40;string&#41;&#10; service_name &#61; string&#10; &#125;&#41;&#41;&#10; resources &#61; list&#40;string&#41;&#10; &#125;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> | |
| [vpc_sc_ingress_policies](variables.tf#L133) | VPC SC ingress policy defnitions. | <code title="map&#40;object&#40;&#123;&#10; ingress_from &#61; object&#40;&#123;&#10; identity_type &#61; string&#10; identities &#61; list&#40;string&#41;&#10; source_access_levels &#61; list&#40;string&#41;&#10; source_resources &#61; list&#40;string&#41;&#10; &#125;&#41;&#10; ingress_to &#61; object&#40;&#123;&#10; operations &#61; list&#40;object&#40;&#123;&#10; method_selectors &#61; list&#40;string&#41;&#10; service_name &#61; string&#10; &#125;&#41;&#41;&#10; resources &#61; list&#40;string&#41;&#10; &#125;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> | |
| [vpc_sc_perimeter_access_levels](variables.tf#L153) | VPC SC perimeter access_levels. | <code title="object&#40;&#123;&#10; dev &#61; list&#40;string&#41;&#10; landing &#61; list&#40;string&#41;&#10; prod &#61; list&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> | |
| [vpc_sc_perimeter_egress_policies](variables.tf#L163) | VPC SC egress policies per perimeter, values reference keys defined in the `vpc_sc_ingress_policies` variable. | <code title="object&#40;&#123;&#10; dev &#61; list&#40;string&#41;&#10; landing &#61; list&#40;string&#41;&#10; prod &#61; list&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> | |
| [vpc_sc_perimeter_ingress_policies](variables.tf#L173) | VPC SC ingress policies per perimeter, values reference keys defined in the `vpc_sc_ingress_policies` variable. | <code title="object&#40;&#123;&#10; dev &#61; list&#40;string&#41;&#10; landing &#61; list&#40;string&#41;&#10; prod &#61; list&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> | |
| [vpc_sc_perimeter_projects](variables.tf#L183) | VPC SC perimeter resources. | <code title="object&#40;&#123;&#10; dev &#61; list&#40;string&#41;&#10; landing &#61; list&#40;string&#41;&#10; prod &#61; list&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> | |
| [billing_account](variables.tf#L17) | Billing account id and organization id ('nnnnnnnn' or null). | <code title="object&#40;&#123;&#10; id &#61; string&#10; organization_id &#61; number&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>00-bootstrap</code> |
| [folder_ids](variables.tf#L26) | Folder name => id mappings, the 'security' folder name must exist. | <code title="object&#40;&#123;&#10; security &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>01-resman</code> |
| [organization](variables.tf#L81) | Organization details. | <code title="object&#40;&#123;&#10; domain &#61; string&#10; id &#61; number&#10; customer_id &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>00-bootstrap</code> |
| [prefix](variables.tf#L97) | Prefix used for resources that need unique names. Use 9 characters or less. | <code>string</code> | ✓ | | <code>00-bootstrap</code> |
| [service_accounts](variables.tf#L72) | Automation service accounts that can assign the encrypt/decrypt roles on keys. | <code title="object&#40;&#123;&#10; project-factory-dev &#61; string&#10; project-factory-prod &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>01-resman</code> |
| [groups](variables.tf#L34) | Group names to grant organization-level permissions. | <code>map&#40;string&#41;</code> | | <code title="&#123;&#10; gcp-billing-admins &#61; &#34;gcp-billing-admins&#34;,&#10; gcp-devops &#61; &#34;gcp-devops&#34;,&#10; gcp-network-admins &#61; &#34;gcp-network-admins&#34;&#10; gcp-organization-admins &#61; &#34;gcp-organization-admins&#34;&#10; gcp-security-admins &#61; &#34;gcp-security-admins&#34;&#10; gcp-support &#61; &#34;gcp-support&#34;&#10;&#125;">&#123;&#8230;&#125;</code> | <code>00-bootstrap</code> |
| [kms_defaults](variables.tf#L49) | Defaults used for KMS keys. | <code title="object&#40;&#123;&#10; locations &#61; list&#40;string&#41;&#10; rotation_period &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; locations &#61; &#91;&#34;europe&#34;, &#34;europe-west1&#34;, &#34;europe-west3&#34;, &#34;global&#34;&#93;&#10; rotation_period &#61; &#34;7776000s&#34;&#10;&#125;">&#123;&#8230;&#125;</code> | |
| [kms_keys](variables.tf#L61) | KMS keys to create, keyed by name. Null attributes will be interpolated with defaults. | <code title="map&#40;object&#40;&#123;&#10; iam &#61; map&#40;list&#40;string&#41;&#41;&#10; labels &#61; map&#40;string&#41;&#10; locations &#61; list&#40;string&#41;&#10; rotation_period &#61; string&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> | |
| [outputs_location](variables.tf#L91) | Path where providers, tfvars files, and lists for the following stages are written. Leave empty to disable. | <code>string</code> | | <code>null</code> | |
| [vpc_sc_access_levels](variables.tf#L108) | VPC SC access level definitions. | <code title="map&#40;object&#40;&#123;&#10; combining_function &#61; string&#10; conditions &#61; list&#40;object&#40;&#123;&#10; ip_subnetworks &#61; list&#40;string&#41;&#10; members &#61; list&#40;string&#41;&#10; negate &#61; bool&#10; regions &#61; list&#40;string&#41;&#10; required_access_levels &#61; list&#40;string&#41;&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> | |
| [vpc_sc_egress_policies](variables.tf#L123) | VPC SC egress policy defnitions. | <code title="map&#40;object&#40;&#123;&#10; egress_from &#61; object&#40;&#123;&#10; identity_type &#61; string&#10; identities &#61; list&#40;string&#41;&#10; &#125;&#41;&#10; egress_to &#61; object&#40;&#123;&#10; operations &#61; list&#40;object&#40;&#123;&#10; method_selectors &#61; list&#40;string&#41;&#10; service_name &#61; string&#10; &#125;&#41;&#41;&#10; resources &#61; list&#40;string&#41;&#10; &#125;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> | |
| [vpc_sc_ingress_policies](variables.tf#L141) | VPC SC ingress policy defnitions. | <code title="map&#40;object&#40;&#123;&#10; ingress_from &#61; object&#40;&#123;&#10; identity_type &#61; string&#10; identities &#61; list&#40;string&#41;&#10; source_access_levels &#61; list&#40;string&#41;&#10; source_resources &#61; list&#40;string&#41;&#10; &#125;&#41;&#10; ingress_to &#61; object&#40;&#123;&#10; operations &#61; list&#40;object&#40;&#123;&#10; method_selectors &#61; list&#40;string&#41;&#10; service_name &#61; string&#10; &#125;&#41;&#41;&#10; resources &#61; list&#40;string&#41;&#10; &#125;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> | |
| [vpc_sc_perimeter_access_levels](variables.tf#L161) | VPC SC perimeter access_levels. | <code title="object&#40;&#123;&#10; dev &#61; list&#40;string&#41;&#10; landing &#61; list&#40;string&#41;&#10; prod &#61; list&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> | |
| [vpc_sc_perimeter_egress_policies](variables.tf#L171) | VPC SC egress policies per perimeter, values reference keys defined in the `vpc_sc_ingress_policies` variable. | <code title="object&#40;&#123;&#10; dev &#61; list&#40;string&#41;&#10; landing &#61; list&#40;string&#41;&#10; prod &#61; list&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> | |
| [vpc_sc_perimeter_ingress_policies](variables.tf#L181) | VPC SC ingress policies per perimeter, values reference keys defined in the `vpc_sc_ingress_policies` variable. | <code title="object&#40;&#123;&#10; dev &#61; list&#40;string&#41;&#10; landing &#61; list&#40;string&#41;&#10; prod &#61; list&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> | |
| [vpc_sc_perimeter_projects](variables.tf#L191) | VPC SC perimeter resources. | <code title="object&#40;&#123;&#10; dev &#61; list&#40;string&#41;&#10; landing &#61; list&#40;string&#41;&#10; prod &#61; list&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> | |
## Outputs
| name | description | sensitive | consumers |
|---|---|:---:|---|
| [stage_perimeter_projects](outputs.tf#L37) | Security project numbers. They can be added to perimeter resources. | | |
| [kms_keys](outputs.tf#L53) | KMS key ids. | | |
| [stage_perimeter_projects](outputs.tf#L58) | Security project numbers. They can be added to perimeter resources. | | |
| [tfvars](outputs.tf#L68) | Terraform variable files for the following stages. | ✓ | |
<!-- END TFDOC -->

14
fast/stages/02-security/core-dev.tf
View File

@ -14,14 +14,20 @@
* limitations under the License.
*/
locals {
dev_kms_restricted_admins = [
"serviceAccount:${var.service_accounts.project-factory-dev}"
]
}
module "dev-sec-project" {
source = "../../../modules/project"
name = "dev-sec-core-0"
parent = var.folder_id
parent = var.folder_ids.security
prefix = var.prefix
billing_account = var.billing_account_id
billing_account = var.billing_account.id
iam = {
"roles/cloudkms.viewer" = try(var.kms_restricted_admins.dev, [])
"roles/cloudkms.viewer" = local.dev_kms_restricted_admins
}
labels = { environment = "dev", team = "security" }
services = local.project_services
@ -46,7 +52,7 @@ module "dev-sec-kms" {
# TODO(ludo): grant delegated role at key instead of project level
resource "google_project_iam_member" "dev_key_admin_delegated" {
for_each = toset(try(var.kms_restricted_admins.dev, []))
for_each = toset(local.dev_kms_restricted_admins)
project = module.dev-sec-project.project_id
role = "roles/cloudkms.admin"
member = each.key

14
fast/stages/02-security/core-prod.tf
View File

@ -14,14 +14,20 @@
* limitations under the License.
*/
locals {
prod_kms_restricted_admins = [
"serviceAccount:${var.service_accounts.project-factory-prod}"
]
}
module "prod-sec-project" {
source = "../../../modules/project"
name = "prod-sec-core-0"
parent = var.folder_id
parent = var.folder_ids.security
prefix = var.prefix
billing_account = var.billing_account_id
billing_account = var.billing_account.id
iam = {
"roles/cloudkms.viewer" = try(var.kms_restricted_admins.prod, [])
"roles/cloudkms.viewer" = local.prod_kms_restricted_admins
}
labels = { environment = "prod", team = "security" }
services = local.project_services
@ -45,7 +51,7 @@ module "prod-sec-kms" {
# TODO(ludo): add support for conditions to Fabric modules
resource "google_project_iam_member" "prod_key_admin_delegated" {
for_each = toset(try(var.kms_restricted_admins.prod, []))
for_each = toset(local.prod_kms_restricted_admins)
project = module.prod-sec-project.project_id
role = "roles/cloudkms.admin"
member = each.key

57
fast/stages/02-security/outputs.tf
View File

@ -14,26 +14,47 @@
* limitations under the License.
*/
# optionally generate files for subsequent stages
resource "local_file" "dev_sec_kms" {
for_each = var.outputs_location == null ? {} : { 1 = 1 }
filename = "${pathexpand(var.outputs_location)}/yamls/02-security-kms-dev-keys.yaml"
content = yamlencode({
for k, m in module.dev-sec-kms : k => m.key_ids
})
locals {
_output_kms_keys = concat(
flatten([
for location, mod in module.dev-sec-kms : [
for name, id in mod.key_ids : {
key = "dev-${name}:${location}"
id = id
}
]
]),
flatten([
for location, mod in module.prod-sec-kms : [
for name, id in mod.key_ids : {
key = "prod-${name}:${location}"
id = id
}
]
])
)
output_kms_keys = { for k in local._output_kms_keys : k.key => k.id }
tfvars = {
kms_keys = local.output_kms_keys
}
}
resource "local_file" "prod_sec_kms" {
for_each = var.outputs_location == null ? {} : { 1 = 1 }
filename = "${pathexpand(var.outputs_location)}/yamls/02-security-kms-prod-keys.yaml"
content = yamlencode({
for k, m in module.prod-sec-kms : k => m.key_ids
})
# optionally generate files for subsequent stages
resource "local_file" "tfvars" {
for_each = var.outputs_location == null ? {} : { 1 = 1 }
file_permission = "0644"
filename = "${pathexpand(var.outputs_location)}/tfvars/02-security.auto.tfvars.json"
content = jsonencode(local.tfvars)
}
# outputs
output "kms_keys" {
description = "KMS key ids."
value = local.output_kms_keys
}
output "stage_perimeter_projects" {
description = "Security project numbers. They can be added to perimeter resources."
value = {
@ -41,3 +62,11 @@ output "stage_perimeter_projects" {
prod = ["projects/${module.prod-sec-project.number}"]
}
}
# ready to use variable values for subsequent stages
output "tfvars" {
description = "Terraform variable files for the following stages."
sensitive = true
value = local.tfvars
}

36
fast/stages/02-security/variables.tf
View File

@ -14,20 +14,25 @@
* limitations under the License.
*/
variable "billing_account_id" {
# tfdoc:variable:source bootstrap
description = "Billing account id."
type = string
variable "billing_account" {
# tfdoc:variable:source 00-bootstrap
description = "Billing account id and organization id ('nnnnnnnn' or null)."
type = object({
id = string
organization_id = number
})
}
variable "folder_id" {
# tfdoc:variable:source resman
description = "Folder to be used for the networking resources in folders/nnnn format."
type = string
variable "folder_ids" {
# tfdoc:variable:source 01-resman
description = "Folder name => id mappings, the 'security' folder name must exist."
type = object({
security = string
})
}
variable "groups" {
# tfdoc:variable:source bootstrap
# tfdoc:variable:source 00-bootstrap
description = "Group names to grant organization-level permissions."
type = map(string)
# https://cloud.google.com/docs/enterprise/setup-checklist
@ -64,14 +69,17 @@ variable "kms_keys" {
default = {}
}
variable "kms_restricted_admins" {
description = "Map of environment => [identities] who can assign the encrypt/decrypt roles on keys."
type = map(list(string))
default = {}
variable "service_accounts" {
# tfdoc:variable:source 01-resman
description = "Automation service accounts that can assign the encrypt/decrypt roles on keys."
type = object({
project-factory-dev = string
project-factory-prod = string
})
}
variable "organization" {
# tfdoc:variable:source bootstrap
# tfdoc:variable:source 00-bootstrap
description = "Organization details."
type = object({
domain = string

2
fast/stages/03-project-factory/README.md
View File

@ -3,4 +3,4 @@
The Project Factory (PF) builds on top of your foundations to create and set up projects (and related resources) to be used for your workloads.
It is organized in folders representing environments (e.g. "dev", "prod"), each implemented by a stand-alone terraform [resource factory](https://medium.com/google-cloud/resource-factories-a-descriptive-approach-to-terraform-581b3ebb59c).
This directory contains a single project factory ([`prod/`](./prod/)) as an example - to implement multiple environments (e.g. "prod" and "dev") you'll need to copy the `prod` folder into one folder per environment, then customize each one following the instructions found in [`prod/README.md`](./prod/README.md).
This directory contains a single project factory ([`dev/`](./dev/)) as an example - to implement multiple environments (e.g. "prod" and "dev") you'll need to copy the `dev` folder into one folder per environment, then customize each one following the instructions found in [`dev/README.md`](./dev/README.md).

28
fast/stages/03-project-factory/prod/README.md → fast/stages/03-project-factory/dev/README.md
View File

@ -49,7 +49,7 @@ It's of course possible to run this stage in isolation, by making sure the archi
- `"roles/compute.viewer"`
- `"roles/dns.admin"`
- If networking is used (e.g., for VMs, GKE Clusters or AppEngine flex), VPC Host projects and their subnets should exist when creating projects
- If per-environment DNS sub-zones are required, one "root" zone per environment should exist when creating projects (e.g., prod.gcp.example.com.)
- If per-environment DNS sub-zones are required, one "root" zone per environment should exist when creating projects (e.g., dev.gcp.example.com.)
### Providers configuration
@ -57,8 +57,8 @@ If you're running this on top of Fast, you should run the following commands to
```bash
# Variable `outputs_location` is set to `../../../config` in stage 01-resman
$ cd fabric-fast/stages/03-project-factory/prod
ln -s ../../../config/03-project-factory-prod/providers.tf
$ cd fabric-fast/stages/03-project-factory/dev
ln -s ../../../config/03-project-factory-dev/providers.tf
```
### Variable configuration
@ -74,16 +74,16 @@ If you configured a valid path for `outputs_location` in the bootstrap and netwo
```bash
# Variable `outputs_location` is set to `../../../config` in stages 01-bootstrap and the 02-networking stage in use
ln -s ../../../config/03-project-factory-prod/terraform-bootstrap.auto.tfvars.json
ln -s ../../../config/03-project-factory-prod/terraform-networking.auto.tfvars.json
ln -s ../../../config/03-project-factory-dev/terraform-bootstrap.auto.tfvars.json
ln -s ../../../config/03-project-factory-dev/terraform-networking.auto.tfvars.json
```
If you're not using Fast, refer to the [Variables](#variables) table at the bottom of this document for a full list of variables, their origin (e.g., a stage or specific to this one), and descriptions explaining their meaning.
Besides the values above, a project factory takes 2 additional inputs:
- `data/defaults.yaml`, manually configured by adapting the [`prod/data/defaults.yaml.sample`](./prod/data/defaults.yaml.sample), which defines per-environment default values e.g., for billing alerts and labels.
- `data/projects/*.yaml`, one file per project (optionally grouped in folders), which configures each project. A [`prod/data/projects/project.yaml.sample`](./prod/data/projects/project.yaml.sample) is provided as reference and documentation for the schema. Projects will be named after the filename, e.g., `fast-prod-lab0.yaml` will create project `fast-prod-lab0`.
- `data/defaults.yaml`, manually configured by adapting the [`data/defaults.yaml`](./data/defaults.yaml), which defines per-environment default values e.g., for billing alerts and labels.
- `data/projects/*.yaml`, one file per project (optionally grouped in folders), which configures each project. A [`data/projects/project.yaml`](./data/projects/project.yaml) is provided as reference and documentation for the schema. Projects will be named after the filename, e.g., `fast-dev-lab0.yaml` will create project `fast-dev-lab0`.
Once the configuration is complete, run the project factory by running
@ -107,13 +107,13 @@ terraform apply
| name | description | type | required | default | producer |
|---|---|:---:|:---:|:---:|:---:|
| [billing_account_id](variables.tf#L19) | Billing account id. | <code>string</code> | ✓ | | <code>00-bootstrap</code> |
| [prefix](variables.tf#L44) | Prefix used for resources that need unique names. Use 9 characters or less. | <code>string</code> | ✓ | | <code>00-bootstrap</code> |
| [data_dir](variables.tf#L25) | Relative path for the folder storing configuration data. | <code>string</code> | | <code>&#34;data&#47;projects&#34;</code> | |
| [defaults_file](variables.tf#L38) | Relative path for the file storing the project factory configuration. | <code>string</code> | | <code>&#34;data&#47;defaults.yaml&#34;</code> | |
| [environment_dns_zone](variables.tf#L31) | DNS zone suffix for environment. | <code>string</code> | | <code>null</code> | <code>02-networking</code> |
| [shared_vpc_self_link](variables.tf#L55) | Self link for the shared VPC. | <code>string</code> | | <code>null</code> | <code>02-networking</code> |
| [vpc_host_project](variables.tf#L62) | Host project for the shared VPC. | <code>string</code> | | <code>null</code> | <code>02-networking</code> |
| [billing_account](variables.tf#L19) | Billing account id and organization id ('nnnnnnnn' or null). | <code title="object&#40;&#123;&#10; id &#61; string&#10; organization_id &#61; number&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>00-bootstrap</code> |
| [prefix](variables.tf#L47) | Prefix used for resources that need unique names. Use 9 characters or less. | <code>string</code> | ✓ | | <code>00-bootstrap</code> |
| [data_dir](variables.tf#L28) | Relative path for the folder storing configuration data. | <code>string</code> | | <code>&#34;data&#47;projects&#34;</code> | |
| [defaults_file](variables.tf#L41) | Relative path for the file storing the project factory configuration. | <code>string</code> | | <code>&#34;data&#47;defaults.yaml&#34;</code> | |
| [environment_dns_zone](variables.tf#L34) | DNS zone suffix for environment. | <code>string</code> | | <code>null</code> | <code>02-networking</code> |
| [shared_vpc_self_links](variables.tf#L58) | Self link for the shared VPC. | <code title="object&#40;&#123;&#10; dev-spoke-0 &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> | <code>02-networking</code> |
| [vpc_host_project_ids](variables.tf#L67) | Host project for the shared VPC. | <code title="object&#40;&#123;&#10; dev-spoke-0 &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> | <code>02-networking</code> |
## Outputs

2
fast/stages/03-project-factory/prod/data/defaults.yaml → fast/stages/03-project-factory/dev/data/defaults.yaml
View File

@ -15,7 +15,7 @@ essential_contacts: ["team-contacts@example.com"]
# [opt] Labels set for all projects
labels:
environment: prod
environment: dev
department: accounting
application: example-app
foo: bar

8
fast/stages/03-project-factory/prod/data/projects/project.yaml → fast/stages/03-project-factory/dev/data/projects/project.yaml
View File

@ -44,7 +44,7 @@ kms_service_agents:
# [opt] Labels for the project - merged with the ones defined in defaults
labels:
environment: prod
environment: dev
# [opt] Org policy overrides defined at project level
org_policies:
@ -56,7 +56,7 @@ org_policies:
status: true
suggested_value: null
values:
- projects/fast-prod-iac-core-0
- projects/fast-dev-iac-core-0
# [opt] Service account to create for the project and their roles on the project
# in name => [roles] format
@ -90,11 +90,11 @@ vpc:
enable_security_admin: true
# Host project the project will be service project of
host_project: fast-prod-net-spoke-0
host_project: fast-dev-net-spoke-0
# [opt] Subnets in the host project where principals will be granted networkUser
# in region/subnet-name => [principals]
subnets_iam:
europe-west1/prod-default-ew1:
europe-west1/dev-default-ew1:
- user:foobar@example.com
- serviceAccount:service-account1

0
fast/stages/03-project-factory/prod/diagram.png → fast/stages/03-project-factory/dev/diagram.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 56 KiB

After

Width:  |  Height:  |  Size: 56 KiB

0
fast/stages/03-project-factory/prod/diagram.svg → fast/stages/03-project-factory/dev/diagram.svg
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 590 KiB

After

Width:  |  Height:  |  Size: 590 KiB

6
fast/stages/03-project-factory/prod/main.tf → fast/stages/03-project-factory/dev/main.tf
View File

@ -20,10 +20,10 @@
locals {
_defaults = yamldecode(file(var.defaults_file))
_defaults_net = {
billing_account_id = var.billing_account_id
billing_account_id = var.billing_account.id
environment_dns_zone = var.environment_dns_zone
shared_vpc_self_link = var.shared_vpc_self_link
vpc_host_project = var.vpc_host_project
shared_vpc_self_link = try(var.shared_vpc_self_links["dev:spoke-0"], null)
vpc_host_project = try(var.vpc_host_project_ids["dev:spoke-0"], null)
}
defaults = merge(local._defaults, local._defaults_net)
projects = {

0
fast/stages/03-project-factory/prod/outputs.tf → fast/stages/03-project-factory/dev/outputs.tf
View File

25
fast/stages/03-project-factory/prod/variables.tf → fast/stages/03-project-factory/dev/variables.tf
View File

@ -16,10 +16,13 @@
#TODO: tfdoc annotations
variable "billing_account_id" {
variable "billing_account" {
# tfdoc:variable:source 00-bootstrap
description = "Billing account id."
type = string
description = "Billing account id and organization id ('nnnnnnnn' or null)."
type = object({
id = string
organization_id = number
})
}
variable "data_dir" {
@ -52,16 +55,20 @@ variable "prefix" {
}
}
variable "shared_vpc_self_link" {
variable "shared_vpc_self_links" {
# tfdoc:variable:source 02-networking
description = "Self link for the shared VPC."
type = string
default = null
type = object({
dev-spoke-0 = string
})
default = null
}
variable "vpc_host_project" {
variable "vpc_host_project_ids" {
# tfdoc:variable:source 02-networking
description = "Host project for the shared VPC."
type = string
default = null
type = object({
dev-spoke-0 = string
})
default = null
}

25
fast/stages/README.md
View File

@ -2,23 +2,38 @@
Each of the folders contained here is a separate "stage", or Terraform root module.
They are designed to be combined together, each stage leveraging the previous stage's resources and providing outputs to the following stages, but they can also be run in isolation if their specific functionality is all that is needed (e.g. only bring up a hub and spoke VPC in an existing environment).
Each stage can be run in isolation (for example to only bring up a hub and spoke VPC in an existing environment), but when combined together they form a modular setup that allows top-down configuration of a whole GCP organization.
When combined together, each stage is designed to leverage the previous stage's resources and to provide outputs to the following stages via predefined contracts, that regulate what is exchanged.
This has two important consequences
- any stage can be swapped out and replaced by different code as long as it respects the contract by providing a predefined set of outputs and optionally accepting a predefined set of variables
- data flow between stages can be partially automated, reducing the effort and pain required to compile variables by hand
One important assumption is that the flow of data is always forward looking, so no stage should depend on outputs generated further down the chain. This greatly simplifies both the logic and the implementation, and allows stages to be effectively independent.
To achieve this, we rely on specific GCP functionality like [delegated role grants](https://medium.com/google-cloud/managing-gcp-service-usage-through-delegated-role-grants-a843610f2226) that allow controlled delegation of responsibilities, for example to allow managing IAM bindings at the organization level only for specific roles.
Refer to each stage's documentation for a detailed description of its purpose, the architectural choices made in its design, and how it can be configured and wired together to terraform a whole GCP organization. The following is a brief overview of each stage.
## Organizational level (00-01)
- [Bootstrap](00-bootstrap/README.md)
Enables critical organization-level functionality that depends on broad permissions. It has two primary purposes. The first is to bootstrap the resources needed for automation of this and the following stages (service accounts, GCS buckets). And secondly, it applies the minimum amount of configuration needed at the organization level, to avoid the need of broad permissions later on, and to implement a minimum of security features like sinks and exports from the start.
Enables critical organization-level functionality that depends on broad permissions. It has two primary purposes. The first is to bootstrap the resources needed for automation of this and the following stages (service accounts, GCS buckets). And secondly, it applies the minimum amount of configuration needed at the organization level, to avoid the need of broad permissions later on, and to implement a minimum of security features like sinks and exports from the start.\
Exports: automation project id, organization-level custom roles
- [Resource Management](01-resman/README.md)
Creates the base resource hierarchy (folders) and the automation resources required later to delegate deployment of each part of the hierarchy to separate stages. This stage also configures organization-level policies and any exceptions needed by different branches of the resource hierarchy.
Creates the base resource hierarchy (folders) and the automation resources required later to delegate deployment of each part of the hierarchy to separate stages. This stage also configures organization-level policies and any exceptions needed by different branches of the resource hierarchy.\
Exports: folder ids, automation service account emails
## Shared resources (02)
- [Security](02-security/README.md)
Manages centralized security configurations in a separate stage, and is typically owned by the security team. This stage implements VPC Security Controls via separate perimeters for environments and central services, and creates projects to host centralized KMS keys used by the whole organization. It's meant to be easily extended to include other security-related resources which are required, like Secret Manager.
Manages centralized security configurations in a separate stage, and is typically owned by the security team. This stage implements VPC Security Controls via separate perimeters for environments and central services, and creates projects to host centralized KMS keys used by the whole organization. It's meant to be easily extended to include other security-related resources which are required, like Secret Manager.\
Exports: KMS key ids
- Networking ([VPN](02-networking-vpn/README.md)/[NVA](02-networking-nva/README.md))
Manages centralized network resources in a separate stage, and is typically owned by the networking team. This stage implements a hub-and-spoke design, and includes connectivity via VPN to on-premises, and YAML-based factories for firewall rules (hierarchical and VPC-level) and subnets. It's currently available in two versions: [spokes connected via VPN](02-networking-vpn/README.md), [and spokes connected via appliances](02-networking-nva/README.md).
Manages centralized network resources in a separate stage, and is typically owned by the networking team. This stage implements a hub-and-spoke design, and includes connectivity via VPN to on-premises, and YAML-based factories for firewall rules (hierarchical and VPC-level) and subnets. It's currently available in two versions: [spokes connected via VPN](02-networking-vpn/README.md), [and spokes connected via appliances](02-networking-nva/README.md).\
Exports: host project ids and numbers, vpc self links
## Environment-level resources (03)

2
modules/cloudsql-instance/README.md
View File

@ -25,7 +25,7 @@ module "vpc" {
source = "./modules/net-vpc"
project_id = module.project.project_id
name = "my-network"
psn_ranges = ["10.60.0.0/16"]
psa_ranges = {cloudsql-ew1-0="10.60.0.0/16"}
}
module "db" {

2
modules/folder/README.md
View File

@ -61,7 +61,7 @@ module "folder" {
policy_name = null
rules_file = "data/rules.yaml"
}
firewall_policy_attachments = {
firewall_policy_association = {
factory-policy = module.folder.firewall_policy_id["factory"]
}
}

6
modules/net-vpc/README.md
View File

@ -138,7 +138,7 @@ module "vpc" {
secondary_ip_range = null
}
]
psn_ranges = ["10.10.0.0/16"]
psa_ranges = {range-a = "10.10.0.0/16"}
}
# tftest modules=1 resources=4
```
@ -171,8 +171,8 @@ module "vpc" {
```
### Subnet Factory
The `net-vpc` module includes a subnet factory (see [Resource Factories](../../examples/factories/)) for the massive creation of subnets leveraging one configuration file per subnet.
The `net-vpc` module includes a subnet factory (see [Resource Factories](../../examples/factories/)) for the massive creation of subnets leveraging one configuration file per subnet.
```hcl
module "vpc" {
@ -220,7 +220,7 @@ flow_logs: # enable, set to empty map to use defaults
| [mtu](variables.tf#L80) | Maximum Transmission Unit in bytes. The minimum value for this field is 1460 and the maximum value is 1500 bytes. | <code></code> | | <code>null</code> |
| [peering_config](variables.tf#L90) | VPC peering configuration. | <code title="object&#40;&#123;&#10; peer_vpc_self_link &#61; string&#10; export_routes &#61; bool&#10; import_routes &#61; bool&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [peering_create_remote_end](variables.tf#L100) | Skip creation of peering on the remote end when using peering_config. | <code>bool</code> | | <code>true</code> |
| [psn_ranges](variables.tf#L111) | CIDR ranges used for Google services that support Private Service Networking. | <code>list&#40;string&#41;</code> | | <code>null</code> |
| [psa_ranges](variables.tf#L111) | CIDR ranges used for Google services that support Private Service Networking. | <code>map&#40;string&#41;</code> | | <code>null</code> |
| [routes](variables.tf#L124) | Network routes, keyed by name. | <code title="map&#40;object&#40;&#123;&#10; dest_range &#61; string&#10; priority &#61; number&#10; tags &#61; list&#40;string&#41;&#10; next_hop_type &#61; string &#35; gateway, instance, ip, vpn_tunnel, ilb&#10; next_hop &#61; string&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [routing_mode](variables.tf#L136) | The network routing mode (default 'GLOBAL'). | <code>string</code> | | <code>&#34;GLOBAL&#34;</code> |
| [shared_vpc_host](variables.tf#L146) | Enable shared VPC for this project. | <code>bool</code> | | <code>false</code> |

22
modules/net-vpc/main.tf
View File

@ -78,11 +78,11 @@ locals {
? null
: element(reverse(split("/", var.peering_config.peer_vpc_self_link)), 0)
)
psn_ranges = {
for r in(var.psn_ranges == null ? [] : var.psn_ranges) : r => {
address = split("/", r)[0]
name = replace(split("/", r)[0], ".", "-")
prefix_length = split("/", r)[1]
psa_ranges = {
for k, v in coalesce(var.psa_ranges, {}) : k => {
address = split("/", v)[0]
name = k
prefix_length = split("/", v)[1]
}
}
routes = {
@ -328,10 +328,10 @@ resource "google_dns_policy" "default" {
}
}
resource "google_compute_global_address" "psn_ranges" {
for_each = local.psn_ranges
resource "google_compute_global_address" "psa_ranges" {
for_each = local.psa_ranges
project = var.project_id
name = "${var.name}-psn-${each.value.name}"
name = "${var.name}-psa-${each.key}"
purpose = "VPC_PEERING"
address_type = "INTERNAL"
address = each.value.address
@ -339,11 +339,11 @@ resource "google_compute_global_address" "psn_ranges" {
network = local.network.id
}
resource "google_service_networking_connection" "psn_connection" {
for_each = toset(local.psn_ranges == {} ? [] : [""])
resource "google_service_networking_connection" "psa_connection" {
for_each = toset(local.psa_ranges == {} ? [] : [""])
network = local.network.id
service = "servicenetworking.googleapis.com"
reserved_peering_ranges = [
for k, v in google_compute_global_address.psn_ranges : v.name
for k, v in google_compute_global_address.psa_ranges : v.name
]
}

8
modules/net-vpc/outputs.tf
View File

@ -27,7 +27,7 @@ output "name" {
google_compute_network_peering.remote,
google_compute_shared_vpc_host_project.shared_vpc_host,
google_compute_shared_vpc_service_project.service_projects,
google_service_networking_connection.psn_connection
google_service_networking_connection.psa_connection
]
}
@ -39,7 +39,7 @@ output "network" {
google_compute_network_peering.remote,
google_compute_shared_vpc_host_project.shared_vpc_host,
google_compute_shared_vpc_service_project.service_projects,
google_service_networking_connection.psn_connection
google_service_networking_connection.psa_connection
]
}
@ -52,7 +52,7 @@ output "project_id" {
google_compute_network_peering.remote,
google_compute_shared_vpc_host_project.shared_vpc_host,
google_compute_shared_vpc_service_project.service_projects,
google_service_networking_connection.psn_connection
google_service_networking_connection.psa_connection
]
}
@ -64,7 +64,7 @@ output "self_link" {
google_compute_network_peering.remote,
google_compute_shared_vpc_host_project.shared_vpc_host,
google_compute_shared_vpc_service_project.service_projects,
google_service_networking_connection.psn_connection
google_service_networking_connection.psa_connection
]
}

10
modules/net-vpc/variables.tf
View File

@ -108,16 +108,16 @@ variable "project_id" {
type = string
}
variable "psn_ranges" {
variable "psa_ranges" {
description = "CIDR ranges used for Google services that support Private Service Networking."
type = list(string)
type = map(string)
default = null
validation {
condition = alltrue([
for r in(var.psn_ranges == null ? [] : var.psn_ranges) :
can(cidrnetmask(r))
for k, v in(var.psa_ranges == null ? {} : var.psa_ranges) :
can(cidrnetmask(v))
])
error_message = "Specify a valid RFC1918 CIDR range for Private Service Networking."
error_message = "Specify valid RFC1918 CIDR ranges for Private Service Networking."
}
}

15
tests/examples/data_solutions/data_platform_foundations/fixture/main.tf
View File

@ -14,13 +14,10 @@
* limitations under the License.
*/
module "test-environment" {
source = "../../../../../examples/data-solutions/data-platform-foundations/01-environment"
billing_account_id = var.billing_account
root_node = var.root_node
}
module "test-resources" {
source = "../../../../../examples/data-solutions/data-platform-foundations/02-resources"
project_ids = module.test-environment.project_ids
module "test" {
source = "../../../../../examples/data-solutions/data-platform-foundations/"
organization_domain = "example.com"
billing_account_id = "123456-123456-123456"
folder_id = "folders/12345678"
prefix = "prefix"
}

26
tests/examples/data_solutions/data_platform_foundations/fixture/variables.tf
View File

@ -1,26 +0,0 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
variable "billing_account" {
type = string
default = "123456-123456-123456"
}
variable "root_node" {
description = "The resource name of the parent Folder or Organization. Must be of the form folders/folder_id or organizations/org_id."
type = string
default = "folders/12345678"
}

14
tests/examples/data_solutions/data_platform_foundations/test_plan.py
View File

@ -12,8 +12,16 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import pytest
FIXTURES_DIR = os.path.join(os.path.dirname(__file__), 'fixture')
def test_resources(e2e_plan_runner):
"Test that plan works and the numbers of resources is as expected."
modules, resources = e2e_plan_runner()
assert len(modules) == 6
assert len(resources) == 53
modules, resources = e2e_plan_runner(FIXTURES_DIR)
assert len(modules) == 40
assert len(resources) == 282

45
tests/fast/stages/s03_project_factory/fixture/main.tf
View File

@ -14,44 +14,15 @@
* limitations under the License.
*/
# tfdoc:file:description Project factory.
locals {
_defaults = yamldecode(file(var.defaults_file))
_defaults_net = {
billing_account_id = var.billing_account_id
environment_dns_zone = var.environment_dns_zone
shared_vpc_self_link = var.shared_vpc_self_link
vpc_host_project = var.vpc_host_project
}
defaults = merge(local._defaults, local._defaults_net)
projects = {
for f in fileset("${var.data_dir}", "**/*.yaml") :
trimsuffix(f, ".yaml") => yamldecode(file("${var.data_dir}/${f}"))
}
}
module "projects" {
#TODO(sruffilli): Pin to release
source = "../../../../../examples/factories/project-factory"
for_each = local.projects
defaults = local.defaults
project_id = each.key
billing_account_id = try(each.value.billing_account_id, null)
billing_alert = try(each.value.billing_alert, null)
dns_zones = try(each.value.dns_zones, [])
essential_contacts = try(each.value.essential_contacts, [])
folder_id = each.value.folder_id
group_iam = try(each.value.group_iam, {})
iam = try(each.value.iam, {})
kms_service_agents = try(each.value.kms, {})
labels = try(each.value.labels, {})
org_policies = try(each.value.org_policies, null)
service_accounts = try(each.value.service_accounts, {})
services = try(each.value.services, [])
services_iam = try(each.value.services_iam, {})
vpc = try(each.value.vpc, null)
source = "../../../../../fast/stages/03-project-factory/dev"
data_dir = "./data/projects/"
defaults_file = "./data/defaults.yaml"
prefix = "test"
billing_account_id = "12345-67890A-BCDEF0"
environment_dns_zone = "dev"
shared_vpc_self_link = "fake_link"
vpc_host_project = "host_project"
}

2
tests/modules/net_vpc/fixture/main.tf
View File

@ -30,6 +30,6 @@ module "test" {
subnet_flow_logs = var.subnet_flow_logs
subnet_private_access = var.subnet_private_access
auto_create_subnetworks = var.auto_create_subnetworks
psn_ranges = var.psn_ranges
psa_ranges = var.psa_ranges
data_folder = var.data_folder
}

4
tests/modules/net_vpc/fixture/variables.tf
View File

@ -61,8 +61,8 @@ variable "peering_config" {
default = null
}
variable "psn_ranges" {
type = list(string)
variable "psa_ranges" {
type = map(string)
default = null
}

13
tests/modules/net_vpc/test_plan_psn.py → tests/modules/net_vpc/test_plan_psa.py
View File

@ -16,20 +16,21 @@ import tftest
def test_single_range(plan_runner):
"Test single PSN range."
_, resources = plan_runner(psn_ranges='["172.16.100.0/24"]')
"Test single PSA range."
_, resources = plan_runner(psa_ranges='{foobar="172.16.100.0/24"}')
assert len(resources) == 3
def test_multi_range(plan_runner):
"Test multiple PSN ranges."
_, resources = plan_runner(psn_ranges='["172.16.100.0/24", "172.16.101.0/24"]')
"Test multiple PSA ranges."
psa_ranges = '{foobar="172.16.100.0/24", frobniz="172.16.101.0/24"}'
_, resources = plan_runner(psa_ranges=psa_ranges)
assert len(resources) == 4
def test_validation(plan_runner):
"Test PSN variable validation."
"Test PSA variable validation."
try:
plan_runner(psn_ranges='["foobar"]')
plan_runner(psa_ranges='{foobar="foobar"}')
except tftest.TerraformTestError as e:
assert 'Invalid value for variable' in e.args[0]

14
tools/check_boilerplate.py
View File

@ -13,7 +13,6 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
'''Check that boilerplate is present in relevant files.
This tools offers a simple way of ensuring that the required boilerplate header
@ -30,17 +29,12 @@ import os
import re
import sys
_EXCLUDE_DIRS = ('.git', '.terraform')
_EXCLUDE_RE = re.compile(r'# skip boilerplate check')
_MATCH_FILES = (
'Dockerfile', '.py', '.sh', '.tf', '.yaml', '.yml'
)
_MATCH_STRING = (
r'^\s*[#\*]\sCopyright [0-9]{4} Google LLC$\s+[#\*]\s+'
r'[#\*]\sLicensed under the Apache License, Version 2.0 '
r'\(the "License"\);\s+'
)
_MATCH_FILES = ('Dockerfile', '.py', '.sh', '.tf', '.yaml', '.yml')
_MATCH_STRING = (r'^\s*[#\*]\sCopyright [0-9]{4} Google LLC$\s+[#\*]\s+'
r'[#\*]\sLicensed under the Apache License, Version 2.0 '
r'\(the "License"\);\s+')
_MATCH_RE = re.compile(_MATCH_STRING, re.M)

11
tools/check_documentation.py
View File

@ -13,7 +13,6 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
'''Recursively check freshness of tfdoc's generated tables in README files.
This tool recursively checks that the embedded variables and outputs tables in
@ -29,10 +28,8 @@ import pathlib
import click
import tfdoc
BASEDIR = pathlib.Path(__file__).resolve().parents[1]
State = enum.Enum('State', 'OK FAIL SKIP')
@ -59,11 +56,9 @@ def _check_dir(dir_name, exclude_files=None, files=False, show_extra=False):
state = State.OK
else:
state = State.FAIL
diff = '\n'.join(
[f'----- {mod_name} diff -----\n'] +
list(difflib.ndiff(
result['doc'].split('\n'), new_doc.split('\n')
)))
header = f'----- {mod_name} diff -----\n'
ndiff = difflib.ndiff(result['doc'].split('\n'), new_doc.split('\n'))
diff = '\n'.join([header] + list(ndiff))
yield mod_name, state, diff

7
tools/check_links.py
View File

@ -13,14 +13,12 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
'''Recursively check link destination validity in Markdown files.
This tool recursively checks that local links in Markdown files point to valid
destinations. Its main use is in CI pipelines triggered by pull requests.
'''
import collections
import pathlib
import urllib.parse
@ -28,7 +26,6 @@ import urllib.parse
import click
import marko
BASEDIR = pathlib.Path(__file__).resolve().parents[1]
DOC = collections.namedtuple('DOC', 'path relpath links')
LINK = collections.namedtuple('LINK', 'dest valid')
@ -57,8 +54,8 @@ def check_docs(dir_name):
yield DOC(readme_path, str(readme_path.relative_to(dir_path)), links)
@ click.command()
@ click.argument('dirs', type=str, nargs=-1)
@click.command()
@click.argument('dirs', type=str, nargs=-1)
def main(dirs):
'Check links in Markdown files contained in dirs.'
errors = 0

Some files were not shown because too many files have changed in this diff Show More