Merge pull request #258 from terraform-google-modules/ludo-dpf-20210605

Refactor environment
This commit is contained in:
Yoram Ben-Yaacov 2021-06-06 09:32:36 +03:00 committed by GitHub
commit 3967551d22
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
19 changed files with 162 additions and 642 deletions

View File

@ -1,17 +1,8 @@
# Data Platform Foundations - Phase 1
# Data Platform Foundations - Environment (Step 1)
## General
This is the first step needed to deploy Data Platform Foundations, which creates projects and service accounts. Please refer to the [top-level Data Platform README](../README.md) for prerequisites.
This is the first part deploying the Data Platform foundations. In this part we will create the infrastructure needed for the foundational build. This includes projects and service accounts.
Since this example is intended for the data infra engineers we do expect that an initial organization / folder and service account with owner privileges will be pre-created and provided as variables.
This example assume the next items were already created and provided:
- Organization / folder
- Terraform runner Service account with owner permissions on the above organization / folder
This example will create the next projects:
The projects that will be created are:
- Common services
- Landing
@ -19,118 +10,38 @@ This example will create the next projects:
- DWH
- Datamart
A master service account named projects-editor-sa will be created under common services project and will be granted editor permissions on all the projects in scope.
![Data Foundation - Phase 1](./diagram.png)
A master service account named `projects-editor-sa` will be created under the common services project, and it will be granted editor permissions on all the projects in scope.
## Running the example
To create the infrastructure:
- Specify your variables in a `terraform.tvars`
- specify your variables in a `terraform.tvars`
```tfm
billing_account = "BILLING ACCOUNT ID."
parent = "Parent folder or organization in 'folders/folder_id' or 'organizations/org_id' format."
billing_account = "1234-1234-1234"
parent = "folders/12345678"
```
- Place the service account key in the terraform folder
- Go through the following steps to create resources:
- make sure you have the right authentication setup (application default credentials, or a service account key)
- run `terraform init` and `terraform apply`
```bash
terraform init
terraform apply
```
Once done testing, you can clean up resources by running:
```bash
terraform destroy
```
Once done testing, you can clean up resources by running `terraform destroy`
<!-- BEGIN TFDOC -->
## Requirements
## Variables
| Name | Version |
|------|---------|
| terraform | >= 0.13 |
## Providers
| Name | Version |
|------|---------|
| random | n/a |
## Inputs
### General inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| billing\_account | Billing account id. | `string` | n/a | ✓ |
| parent | Parent folder or organization in 'folders/folder\_id' or 'organizations/org\_id' format. | `string` | n/a | ✓ |
| projects\_suffix | Project suffix to make resources unique. If no suffix is provided a random suffix will be created | `string` | `null` | |
### Datamart project inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| datamart\_project\_name | Project name | `string` | `"datamart"` | |
### DWH project inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| dwh\_project\_name | Project name | `string` | `"dwh"` | |
### Landing project inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| landing\_project\_name | Project name | `string` | `"landing"` | |
### Services project inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| data\_service\_account\_name | Name for the projects editor service account. | `string` | `"projects-editor-sa"` | |
| services\_project\_name | Project name | `string` | `"services"` | |
### Transformation project inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| transformation\_project\_name | Project name | `string` | `"transformation"` | |
| name | description | type | required | default |
|---|---|:---: |:---:|:---:|
| billing_account_id | Billing account id. | <code title="">string</code> | ✓ | |
| root_node | Parent folder or organization in 'folders/folder_id' or 'organizations/org_id' format. | <code title="">string</code> | ✓ | |
| *prefix* | Prefix used to generate project id and name. | <code title="">string</code> | | <code title="">null</code> |
| *project_names* | Override this variable if you need non-standard names. | <code title="object&#40;&#123;&#10;datamart &#61; string&#10;dwh &#61; string&#10;landing &#61; string&#10;services &#61; string&#10;transformation &#61; string&#10;&#125;&#41;">object({...})</code> | | <code title="&#123;&#10;datamart &#61; &#34;datamart&#34;&#10;dwh &#61; &#34;datawh&#34;&#10;landing &#61; &#34;landing&#34;&#10;services &#61; &#34;services&#34;&#10;transformation &#61; &#34;transformation&#34;&#10;&#125;">...</code> |
| *service_account_names* | Override this variable if you need non-standard names. | <code title="object&#40;&#123;&#10;main &#61; string&#10;&#125;&#41;">object({...})</code> | | <code title="&#123;&#10;main &#61; &#34;data-platform-main&#34;&#10;&#125;">...</code> |
## Outputs
### Datamart project outputs
| Name | Description |
|------|-------------|
| datamart-project | Datamart project created. |
### DWH project outputs
| Name | Description |
|------|-------------|
| dwh-project | DWH project created. |
### Landing project outputs
| Name | Description |
|------|-------------|
| landing-project | Landing project created. |
### Services project outputs
| Name | Description |
|------|-------------|
| services-project | Services project created. |
### Transformation project outputs
| Name | Description |
|------|-------------|
| transformation-project | Transformation project created. |
| name | description | sensitive |
|---|---|:---:|
| project_ids | Project ids for created projects. | |
<!-- END TFDOC -->

View File

@ -1,63 +0,0 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
##########################
# project-creation
##########################
module "datamart-project-creation" {
source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
billing_account = var.billing_account
parent = var.parent
prefix = var.projects_prefix
name = join(
"-",
[
var.datamart_project_name,
var.projects_suffix != null ? var.projects_suffix : random_id.suffix.hex
]
)
auto_create_network = false
services = [
"bigtable.googleapis.com",
"bigtableadmin.googleapis.com",
"bigquery.googleapis.com",
"bigquerystorage.googleapis.com",
"bigqueryreservation.googleapis.com",
"storage-component.googleapis.com",
]
iam_additive = {}
}
##########################
# IAM
##########################
module "set-tf-data-admin-service-account-on-datamart" {
source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
name = module.datamart-project-creation.project_id
project_create = false
iam = {
"roles/editor" = [
"serviceAccount:${module.master-service-account.email}"
]
}
depends_on = [
module.master-service-account,
]
}

View File

@ -1,20 +0,0 @@
/**
* Copyright 2020 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
output "datamart-project" {
description = "Datamart project created."
value = module.datamart-project-creation
}

View File

@ -1,20 +0,0 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
variable "datamart_project_name" {
description = "Project name"
type = string
default = "datamart"
}

View File

@ -1,60 +0,0 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
##########################
# project-creation
##########################
module "dwh-project-creation" {
source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
billing_account = var.billing_account
parent = var.parent
prefix = var.projects_prefix
name = join(
"-",
[
var.dwh_project_name,
var.projects_suffix != null ? var.projects_suffix : random_id.suffix.hex
]
)
auto_create_network = false
services = [
"bigquery.googleapis.com",
"bigquerystorage.googleapis.com",
"bigqueryreservation.googleapis.com",
"storage-component.googleapis.com",
]
iam_additive = {}
}
##########################
# IAM
##########################
module "set-tf-data-admin-service-account-on-dwh" {
source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
name = module.dwh-project-creation.project_id
project_create = false
iam = {
"roles/editor" = [
"serviceAccount:${module.master-service-account.email}"
]
}
depends_on = [
module.master-service-account,
]
}

View File

@ -1,20 +0,0 @@
/**
* Copyright 2020 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
output "dwh-project" {
description = "DWH project created."
value = module.dwh-project-creation
}

View File

@ -1,20 +0,0 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
variable "dwh_project_name" {
description = "Project name"
type = string
default = "dwh"
}

View File

@ -1,58 +0,0 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
##########################
# project-creation
##########################
module "landing-project-creation" {
source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
billing_account = var.billing_account
parent = var.parent
prefix = var.projects_prefix
name = join(
"-",
[
var.landing_project_name,
var.projects_suffix != null ? var.projects_suffix : random_id.suffix.hex
]
)
auto_create_network = false
services = [
"pubsub.googleapis.com",
"storage-component.googleapis.com",
]
iam_additive = {}
}
##########################
# IAM
##########################
module "set-tf-data-admin-service-account-on-landing" {
source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
name = module.landing-project-creation.project_id
project_create = false
iam = {
"roles/editor" = [
"serviceAccount:${module.master-service-account.email}"
]
}
depends_on = [
module.master-service-account,
]
}

View File

@ -1,20 +0,0 @@
/**
* Copyright 2020 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
output "landing-project" {
description = "Landing project created."
value = module.landing-project-creation
}

View File

@ -1,20 +0,0 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
variable "landing_project_name" {
description = "Project name"
type = string
default = "landing"
}

View File

@ -14,10 +14,102 @@
* limitations under the License.
*/
locals {
module_version = "4.2.0"
###############################################################################
# projects #
###############################################################################
module "project-datamart" {
source = "../../../modules/project"
parent = var.root_node
billing_account = var.billing_account_id
prefix = var.prefix
name = var.project_names.datamart
services = [
"bigtable.googleapis.com",
"bigtableadmin.googleapis.com",
"bigquery.googleapis.com",
"bigquerystorage.googleapis.com",
"bigqueryreservation.googleapis.com",
"storage-component.googleapis.com",
]
iam = {
"roles/editor" = [module.sa-services-main.iam_email]
}
}
resource "random_id" "suffix" {
byte_length = 4
module "project-dwh" {
source = "../../../modules/project"
parent = var.root_node
billing_account = var.billing_account_id
prefix = var.prefix
name = var.project_names.dwh
services = [
"bigquery.googleapis.com",
"bigquerystorage.googleapis.com",
"bigqueryreservation.googleapis.com",
"storage-component.googleapis.com",
]
iam = {
"roles/editor" = [module.sa-services-main.iam_email]
}
}
module "project-landing" {
source = "../../../modules/project"
parent = var.root_node
billing_account = var.billing_account_id
prefix = var.prefix
name = var.project_names.landing
services = [
"pubsub.googleapis.com",
"storage-component.googleapis.com",
]
iam = {
"roles/editor" = [module.sa-services-main.iam_email]
}
}
module "project-services" {
source = "../../../modules/project"
parent = var.root_node
billing_account = var.billing_account_id
prefix = var.prefix
name = var.project_names.services
services = [
"storage-component.googleapis.com",
"sourcerepo.googleapis.com",
"stackdriver.googleapis.com",
"cloudasset.googleapis.com",
]
iam = {
"roles/editor" = [module.sa-services-main.iam_email]
}
}
module "project-transformation" {
source = "../../../modules/project"
parent = var.root_node
billing_account = var.billing_account_id
prefix = var.prefix
name = var.project_names.transformation
services = [
"cloudbuild.googleapis.com",
"compute.googleapis.com",
"dataflow.googleapis.com",
"servicenetworking.googleapis.com",
"storage-component.googleapis.com",
]
iam = {
"roles/editor" = [module.sa-services-main.iam_email]
}
}
###############################################################################
# service accounts #
###############################################################################
module "sa-services-main" {
source = "../../../modules/iam-service-account"
project_id = module.project-services.project_id
name = var.service_account_names.main
}

View File

@ -13,3 +13,14 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
output "project_ids" {
description = "Project ids for created projects."
value = {
datamart = module.project-datamart.project_id
dwh = module.project-dwh.project_id
landing = module.project-landing.project_id
services = module.project-services.project_id
transformation = module.project-transformation.project_id
}
}

View File

@ -1,71 +0,0 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
##########################
# project-creation
##########################
module "services-project-creation" {
source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
billing_account = var.billing_account
parent = var.parent
prefix = var.projects_prefix
name = join(
"-",
[
var.services_project_name,
var.projects_suffix != null ? var.projects_suffix : random_id.suffix.hex
]
)
auto_create_network = false
services = [
"storage-component.googleapis.com",
"sourcerepo.googleapis.com",
"stackdriver.googleapis.com",
"cloudasset.googleapis.com",
]
iam_additive = {}
}
##########################
# IAM
##########################
module "master-service-account" {
source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/iam-service-account?ref=v4.2.0"
project_id = module.services-project-creation.project_id
name = var.data_service_account_name
depends_on = [
module.services-project-creation
]
}
module "set-tf-data-admin-service-account-on-services" {
source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
name = module.services-project-creation.project_id
project_create = false
iam = {
"roles/editor" = [
"serviceAccount:${module.master-service-account.email}"
]
}
depends_on = [
module.master-service-account,
]
}

View File

@ -1,20 +0,0 @@
/**
* Copyright 2020 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
output "services-project" {
description = "Services project created."
value = module.services-project-creation
}

View File

@ -1,25 +0,0 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
variable "data_service_account_name" {
description = "Name for the projects editor service account."
type = string
default = "projects-editor-sa"
}
variable "services_project_name" {
description = "Project name"
type = string
default = "services"
}

View File

@ -1,61 +0,0 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
##########################
# project-creation
##########################
module "transformation-project-creation" {
source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
billing_account = var.billing_account
parent = var.parent
prefix = var.projects_prefix
name = join(
"-",
[
var.transformation_project_name,
var.projects_suffix != null ? var.projects_suffix : random_id.suffix.hex
]
)
auto_create_network = false
services = [
"cloudbuild.googleapis.com",
"compute.googleapis.com",
"dataflow.googleapis.com",
"servicenetworking.googleapis.com",
"storage-component.googleapis.com",
]
iam_additive = {}
}
##########################
# IAM
##########################
module "set-tf-data-admin-service-account-on-transformation" {
source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
name = module.transformation-project-creation.project_id
project_create = false
iam = {
"roles/editor" = [
"serviceAccount:${module.master-service-account.email}"
]
}
depends_on = [
module.master-service-account,
]
}

View File

@ -1,18 +0,0 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
output "transformation-project" {
description = "Transformation project created."
value = module.transformation-project-creation
}

View File

@ -1,20 +0,0 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
variable "transformation_project_name" {
description = "Project name"
type = string
default = "transformation"
}

View File

@ -12,24 +12,46 @@
# See the License for the specific language governing permissions and
# limitations under the License.
variable "billing_account" {
variable "billing_account_id" {
description = "Billing account id."
type = string
}
variable "parent" {
variable "prefix" {
description = "Prefix used to generate project id and name."
type = string
default = null
}
variable "project_names" {
description = "Override this variable if you need non-standard names."
type = object({
datamart = string
dwh = string
landing = string
services = string
transformation = string
})
default = {
datamart = "datamart"
dwh = "datawh"
landing = "landing"
services = "services"
transformation = "transformation"
}
}
variable "root_node" {
description = "Parent folder or organization in 'folders/folder_id' or 'organizations/org_id' format."
type = string
}
variable "projects_prefix" {
description = "Prefix used to generate project id and name"
type = string
default = null
}
variable "projects_suffix" {
description = "Project suffix to make resources unique. If no suffix is provided a random suffix will be created"
type = string
default = null
variable "service_account_names" {
description = "Override this variable if you need non-standard names."
type = object({
main = string
})
default = {
main = "data-platform-main"
}
}