Merge pull request #260 from terraform-google-modules/ludo-dpf-20210610
Refactor resources step
This commit is contained in:
commit
d0b069a162
|
@ -19,5 +19,5 @@ All resources use CMEK hosted in Cloud KMS running in a centralized project. The
|
||||||
### Data Platform Foundations
|
### Data Platform Foundations
|
||||||
|
|
||||||
<a href="./data-platform-foundations/" title="Data Platform Foundations"><img src="./data-platform-foundations/img/Data_Foundation-phase2.png" align="left" width="280px"></a>
|
<a href="./data-platform-foundations/" title="Data Platform Foundations"><img src="./data-platform-foundations/img/Data_Foundation-phase2.png" align="left" width="280px"></a>
|
||||||
This [example](./data-platform-foundations/) implements a **robust and flexible** Data Foundation on GCP that provides **opinionated defaults** while allowing customers to **build and scale** out additional data pipelines **quickly and reliably**.
|
This [example](./data-platform-foundations/) implements a robust and flexible Data Foundation on GCP that provides opinionated defaults, allowing customers to build and scale out additional data pipelines quickly and reliably.
|
||||||
<br clear="left">
|
<br clear="left">
|
||||||
|
|
|
@ -10,7 +10,11 @@ The projects that will be created are:
|
||||||
- DWH
|
- DWH
|
||||||
- Datamart
|
- Datamart
|
||||||
|
|
||||||
A master service account named `projects-editor-sa` will be created under the common services project, and it will be granted editor permissions on all the projects in scope.
|
A main service account named `projects-editor-sa` will be created under the common services project, and it will be granted editor permissions on all the projects in scope.
|
||||||
|
|
||||||
|
This is a high level diagram of the created resources:
|
||||||
|
|
||||||
|
![Environment - Phase 1](./diagram.png)
|
||||||
|
|
||||||
## Running the example
|
## Running the example
|
||||||
|
|
||||||
|
@ -26,7 +30,7 @@ parent = "folders/12345678"
|
||||||
- make sure you have the right authentication setup (application default credentials, or a service account key)
|
- make sure you have the right authentication setup (application default credentials, or a service account key)
|
||||||
- run `terraform init` and `terraform apply`
|
- run `terraform init` and `terraform apply`
|
||||||
|
|
||||||
Once done testing, you can clean up resources by running `terraform destroy`
|
Once done testing, you can clean up resources by running `terraform destroy`.
|
||||||
|
|
||||||
<!-- BEGIN TFDOC -->
|
<!-- BEGIN TFDOC -->
|
||||||
## Variables
|
## Variables
|
||||||
|
@ -44,4 +48,5 @@ Once done testing, you can clean up resources by running `terraform destroy`
|
||||||
| name | description | sensitive |
|
| name | description | sensitive |
|
||||||
|---|---|:---:|
|
|---|---|:---:|
|
||||||
| project_ids | Project ids for created projects. | |
|
| project_ids | Project ids for created projects. | |
|
||||||
|
| service_account | Main service account. | |
|
||||||
<!-- END TFDOC -->
|
<!-- END TFDOC -->
|
||||||
|
|
|
@ -24,3 +24,8 @@ output "project_ids" {
|
||||||
transformation = module.project-transformation.project_id
|
transformation = module.project-transformation.project_id
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
output "service_account" {
|
||||||
|
description = "Main service account."
|
||||||
|
value = module.sa-services-main.email
|
||||||
|
}
|
||||||
|
|
|
@ -1,12 +1,10 @@
|
||||||
# Data Platform Foundations - Phase 2
|
# Data Platform Foundations - Resources (Step 2)
|
||||||
|
|
||||||
## General
|
This is the second step needed to deploy Data Platform Foundations, which creates resources needed to store and process the data, in the projects created in the [previous step](./../environment/). Please refer to the [top-level README](../README.md) for prerequisites and how to run the first step.
|
||||||
|
|
||||||
This is the second step needed to deploy Data Platform Foundations, which creates resources needed to store and process the data in projects created in the [environment step](./../environment/). Please refer to [top-level Data Platform README](../README.md) for prerequisites and how to run the first step.
|
|
||||||
|
|
||||||
![Data Foundation - Phase 2](./diagram.png)
|
![Data Foundation - Phase 2](./diagram.png)
|
||||||
|
|
||||||
This example will create the next resources per project:
|
The resources that will be create in each project are:
|
||||||
|
|
||||||
- Common
|
- Common
|
||||||
- Landing
|
- Landing
|
||||||
|
@ -24,59 +22,49 @@ This example will create the next resources per project:
|
||||||
|
|
||||||
## Running the example
|
## Running the example
|
||||||
|
|
||||||
You can run Teraform script with the Service Account created in the first step.
|
To create the resources, you can leverage the service account created in the previous step:
|
||||||
|
|
||||||
To create the infrastructure:
|
|
||||||
|
|
||||||
- Specify your variables in a `terraform.tvars`
|
- Specify your variables in a `terraform.tvars`
|
||||||
|
|
||||||
```tfm
|
```tfm
|
||||||
project_ids = {
|
project_ids = {
|
||||||
datamart = "DATAMART-PROJECT_ID"
|
datamart = "datamart-project_id"
|
||||||
dwh = "DWH-PROJECT_ID"
|
dwh = "dwh-project_id"
|
||||||
landing = "LANDING-PROJECT_ID"
|
landing = "landing-project_id"
|
||||||
services = "SERVICES-PROJECT_ID"
|
services = "services-project_id"
|
||||||
transformation = "TRANSFORMATION-PROJECT_ID"
|
transformation = "transformation-project_id"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
- Place the data_service_account_name service account (the service account was created in phase 1) key in the terraform folder
|
- make sure you have the right authentication setup (application default credentials, or a service account key)
|
||||||
- Go through the following steps to create resources:
|
- run `terraform init` and `terraform apply`
|
||||||
|
|
||||||
```bash
|
Once done testing, you can clean up resources by running `terraform destroy`.
|
||||||
terraform init
|
|
||||||
terraform apply
|
|
||||||
```
|
|
||||||
|
|
||||||
Once done testing, you can clean up resources by running:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
terraform destroy
|
|
||||||
```
|
|
||||||
|
|
||||||
<!-- BEGIN TFDOC -->
|
<!-- BEGIN TFDOC -->
|
||||||
## Variables
|
## Variables
|
||||||
|
|
||||||
| Name | Description | Type | Required | Default|
|
| name | description | type | required | default |
|
||||||
|------|-------------|------|:--------:|:--------:|
|
|---|---|:---: |:---:|:---:|
|
||||||
| <a name="input_datamart_bq_datasets"></a> [datamart\_bq\_datasets](#Variables\_datamart\_bq\_datasets) | Datamart Bigquery datasets | `object({...})` | | `...` |
|
| project_ids | Project IDs. | <code title="object({ datamart = string dwh = string landing = string services = string transformation = string })">object({...})</code> | ✓ | |
|
||||||
| <a name="input_dwh_bq_datasets"></a> [dwh\_bq\_datasets](#Variables\_dwh\_bq\_datasets) | DWH Bigquery datasets | `object({...})` | | `...` |
|
| *datamart_bq_datasets* | Datamart Bigquery datasets | <code title="map(object({ iam = map(list(string)) location = string }))">map(object({...}))</code> | | <code title="{ bq_datamart_dataset = { location = "EU" iam = { } } }">...</code> |
|
||||||
| <a name="input_landing_buckets"></a> [landing\_buckets](#Variables\_landing\_buckets) | List of landing buckets to create | `object({...})` | | `...` |
|
| *dwh_bq_datasets* | DWH Bigquery datasets | <code title="map(object({ location = string iam = map(list(string)) }))">map(object({...}))</code> | | <code title="{ bq_raw_dataset = { iam = {} location = "EU" } }">...</code> |
|
||||||
| <a name="input_landing_pubsub"></a> [landing\_pubsub](#Variables\_landing\_pubsub) | List of landing pubsub topics and subscriptions to create | `object({...})` | | `...` |
|
| *landing_buckets* | List of landing buckets to create | <code title="map(object({ location = string name = string }))">map(object({...}))</code> | | <code title="{ raw-data = { location = "EU" name = "raw-data" } data-schema = { location = "EU" name = "data-schema" } }">...</code> |
|
||||||
| <a name="input_landing_service_account"></a> [landing\_service\_account](#Variables\_landing\_service\_account) | landing service accounts list. | `string` | | `sa-landing` |
|
| *landing_pubsub* | List of landing pubsub topics and subscriptions to create | <code title="map(map(object({ iam = map(list(string)) labels = map(string) options = object({ ack_deadline_seconds = number message_retention_duration = number retain_acked_messages = bool expiration_policy_ttl = number }) })))">map(map(object({...})))</code> | | <code title="{ landing-1 = { sub1 = { iam = { } labels = {} options = null } sub2 = { iam = {} labels = {}, options = null }, } }">...</code> |
|
||||||
| <a name="input_project_ids"></a> [project\_ids](#Variables\_project\_ids) | Project IDs. | `object({...})` | ✓ | n/a |
|
| *landing_service_account* | landing service accounts list. | <code title="">string</code> | | <code title="">sa-landing</code> |
|
||||||
| <a name="input_project_service_account"></a> [project\_service\_account](#Variables\_project\_service\_account) | Project service accounts list. | `object({...})` | | `...` |
|
| *service_account_names* | Project service accounts list. | <code title="object({ datamart = string dwh = string landing = string services = string transformation = string })">object({...})</code> | | <code title="{ datamart = "sa-datamart" dwh = "sa-datawh" landing = "sa-landing" services = "sa-services" transformation = "sa-transformation" }">...</code> |
|
||||||
| <a name="input_transformation_buckets"></a> [transformation\_buckets](#Variables\_transformation\_buckets) | List of transformation buckets to create | `object({...})` | | `...` |
|
| *transformation_buckets* | List of transformation buckets to create | <code title="map(object({ location = string name = string }))">map(object({...}))</code> | | <code title="{ temp = { location = "EU" name = "temp" }, templates = { location = "EU" name = "templates" }, }">...</code> |
|
||||||
| <a name="input_transformation_subnets"></a> [transformation\_subnets](#Variables\_transformation\_subnets) | List of subnets to create in the transformation Project. | `object({...})` | | `...` |
|
| *transformation_subnets* | List of subnets to create in the transformation Project. | <code title="list(object({ ip_cidr_range = string name = string region = string secondary_ip_range = map(string) }))">list(object({...}))</code> | | <code title="[ { ip_cidr_range = "10.1.0.0/20" name = "transformation-subnet" region = "europe-west3" secondary_ip_range = {} }, ]">...</code> |
|
||||||
| <a name="input_transformation_vpc_name"></a> [transformation\_vpc\_name](#Variables\_transformation\_vpc\_name) | Name of the VPC created in the transformation Project. | `string` | | `transformation-vpc` |
|
| *transformation_vpc_name* | Name of the VPC created in the transformation Project. | <code title="">string</code> | | <code title="">transformation-vpc</code> |
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
| Name | Description |
|
| name | description | sensitive |
|
||||||
|------|-------------|
|
|---|---|:---:|
|
||||||
| <a name="output_datamart-bigquery-datasets-list"></a> [datamart-bigquery-datasets-list](#output\_datamart-bigquery-datasets-list) | List of bigquery datasets created for the datamart project |
|
| datamart-datasets | List of bigquery datasets created for the datamart project. | |
|
||||||
| <a name="output_dwh-bigquery-datasets-list"></a> [dwh-bigquery-datasets-list](#output\_dwh-bigquery-datasets-list) | List of bigquery datasets created for the dwh project |
|
| dwh-datasets | List of bigquery datasets created for the dwh project. | |
|
||||||
| <a name="output_landing-bucket-names"></a> [landing-bucket-names](#output\_landing-bucket-names) | List of buckets created for the landing project |
|
| landing-buckets | List of buckets created for the landing project. | |
|
||||||
| <a name="output_landing-pubsub-list"></a> [landing-pubsub-list](#output\_landing-pubsub-list) | List of pubsub topics and subscriptions created for the landing project |
|
| landing-pubsub | List of pubsub topics and subscriptions created for the landing project. | |
|
||||||
| <a name="output_transformation-bucket-names"></a> [transformation-bucket-names](#output\_transformation-bucket-names) | List of buckets created for the transformation project |
|
| transformation-buckets | List of buckets created for the transformation project. | |
|
||||||
| <a name="output_transformation-vpc-info"></a> [transformation-vpc-info](#output\_transformation-vpc-info) | Transformation VPC details |<!-- END TFDOC -->
|
| transformation-vpc | Transformation VPC details | |
|
||||||
|
<!-- END TFDOC -->
|
||||||
|
|
|
@ -14,130 +14,49 @@
|
||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
locals {
|
|
||||||
landing_pubsub = merge({
|
|
||||||
for k, v in var.landing_pubsub :
|
|
||||||
k => {
|
|
||||||
name = v.name
|
|
||||||
subscriptions = v.subscriptions
|
|
||||||
subscription_iam = merge({
|
|
||||||
for s_k, s_v in v.subscription_iam :
|
|
||||||
s_k => merge(s_v, { "roles/pubsub.subscriber" : ["serviceAccount:${module.transformation-default-service-accounts.email}"] })
|
|
||||||
})
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
datamart_bq_datasets = merge({
|
|
||||||
for k, v in var.datamart_bq_datasets :
|
|
||||||
k => {
|
|
||||||
id = v.id
|
|
||||||
location = v.location
|
|
||||||
iam = merge({
|
|
||||||
for s_k, s_v in v.iam :
|
|
||||||
s_k => s_k == "roles/bigquery.dataOwner" ? concat(s_v, ["serviceAccount:${module.datamart-default-service-accounts.email}"]) : s_v
|
|
||||||
})
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
dwh_bq_datasets = merge({
|
|
||||||
for k, v in var.dwh_bq_datasets :
|
|
||||||
k => {
|
|
||||||
id = v.id
|
|
||||||
location = v.location
|
|
||||||
iam = merge({
|
|
||||||
for s_k, s_v in v.iam :
|
|
||||||
s_k => s_k == "roles/bigquery.dataOwner" ? concat(s_v, ["serviceAccount:${module.dwh-default-service-accounts.email}", "serviceAccount:${module.transformation-default-service-accounts.email}"]) : s_v
|
|
||||||
})
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
###############################################################################
|
|
||||||
# Projects #
|
|
||||||
###############################################################################
|
|
||||||
module "project-datamart" {
|
|
||||||
source = "../../../modules/project"
|
|
||||||
name = var.project_ids.datamart
|
|
||||||
project_create = false
|
|
||||||
}
|
|
||||||
|
|
||||||
module "project-dwh" {
|
|
||||||
source = "../../../modules/project"
|
|
||||||
name = var.project_ids.dwh
|
|
||||||
project_create = false
|
|
||||||
}
|
|
||||||
|
|
||||||
module "project-landing" {
|
|
||||||
source = "../../../modules/project"
|
|
||||||
name = var.project_ids.landing
|
|
||||||
project_create = false
|
|
||||||
}
|
|
||||||
|
|
||||||
module "project-services" {
|
|
||||||
source = "../../../modules/project"
|
|
||||||
name = var.project_ids.services
|
|
||||||
project_create = false
|
|
||||||
}
|
|
||||||
|
|
||||||
module "project-transformation" {
|
|
||||||
source = "../../../modules/project"
|
|
||||||
name = var.project_ids.transformation
|
|
||||||
project_create = false
|
|
||||||
}
|
|
||||||
|
|
||||||
###############################################################################
|
###############################################################################
|
||||||
# IAM #
|
# IAM #
|
||||||
###############################################################################
|
###############################################################################
|
||||||
|
|
||||||
module "datamart-default-service-accounts" {
|
module "datamart-sa" {
|
||||||
source = "../../../modules/iam-service-account"
|
source = "../../../modules/iam-service-account"
|
||||||
project_id = module.project-datamart.project_id
|
project_id = var.project_ids.datamart
|
||||||
name = var.project_service_account.datamart
|
name = var.service_account_names.datamart
|
||||||
|
|
||||||
iam_project_roles = {
|
iam_project_roles = {
|
||||||
"${module.project-datamart.project_id}" = [
|
"${var.project_ids.datamart}" = ["roles/editor"]
|
||||||
"roles/editor",
|
|
||||||
]
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
module "dwh-default-service-accounts" {
|
module "dwh-sa" {
|
||||||
source = "../../../modules/iam-service-account"
|
source = "../../../modules/iam-service-account"
|
||||||
project_id = module.project-dwh.project_id
|
project_id = var.project_ids.dwh
|
||||||
name = var.project_service_account.dwh
|
name = var.service_account_names.dwh
|
||||||
}
|
}
|
||||||
|
|
||||||
module "landing-default-service-accounts" {
|
module "landing-sa" {
|
||||||
source = "../../../modules/iam-service-account"
|
source = "../../../modules/iam-service-account"
|
||||||
project_id = module.project-landing.project_id
|
project_id = var.project_ids.landing
|
||||||
name = var.project_service_account.landing
|
name = var.service_account_names.landing
|
||||||
|
|
||||||
iam_project_roles = {
|
iam_project_roles = {
|
||||||
"${module.project-landing.project_id}" = [
|
"${var.project_ids.landing}" = ["roles/pubsub.publisher"]
|
||||||
"roles/pubsub.publisher",
|
|
||||||
]
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
module "services-default-service-accounts" {
|
module "services-sa" {
|
||||||
source = "../../../modules/iam-service-account"
|
source = "../../../modules/iam-service-account"
|
||||||
project_id = module.project-services.project_id
|
project_id = var.project_ids.services
|
||||||
name = var.project_service_account.services
|
name = var.service_account_names.services
|
||||||
|
|
||||||
iam_project_roles = {
|
iam_project_roles = {
|
||||||
"${module.project-services.project_id}" = [
|
"${var.project_ids.services}" = ["roles/editor"]
|
||||||
"roles/editor",
|
|
||||||
]
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
module "transformation-default-service-accounts" {
|
module "transformation-sa" {
|
||||||
source = "../../../modules/iam-service-account"
|
source = "../../../modules/iam-service-account"
|
||||||
project_id = module.project-transformation.project_id
|
project_id = var.project_ids.transformation
|
||||||
name = var.project_service_account.transformation
|
name = var.service_account_names.transformation
|
||||||
|
|
||||||
iam_project_roles = {
|
iam_project_roles = {
|
||||||
"${module.project-transformation.project_id}" = [
|
"${var.project_ids.transformation}" = [
|
||||||
"roles/logging.logWriter",
|
"roles/logging.logWriter",
|
||||||
"roles/monitoring.metricWriter",
|
"roles/monitoring.metricWriter",
|
||||||
"roles/dataflow.admin",
|
"roles/dataflow.admin",
|
||||||
|
@ -155,30 +74,28 @@ module "transformation-default-service-accounts" {
|
||||||
# GCS #
|
# GCS #
|
||||||
###############################################################################
|
###############################################################################
|
||||||
|
|
||||||
module "bucket-landing" {
|
module "landing-buckets" {
|
||||||
source = "../../../modules/gcs"
|
source = "../../../modules/gcs"
|
||||||
project_id = module.project-landing.project_id
|
|
||||||
prefix = var.project_ids.landing
|
|
||||||
iam = {
|
|
||||||
"roles/storage.objectCreator" = ["serviceAccount:${module.landing-default-service-accounts.email}"],
|
|
||||||
"roles/storage.admin" = ["serviceAccount:${module.transformation-default-service-accounts.email}"],
|
|
||||||
}
|
|
||||||
|
|
||||||
for_each = var.landing_buckets
|
for_each = var.landing_buckets
|
||||||
|
project_id = var.project_ids.landing
|
||||||
|
prefix = var.project_ids.landing
|
||||||
name = each.value.name
|
name = each.value.name
|
||||||
location = each.value.location
|
location = each.value.location
|
||||||
|
iam = {
|
||||||
|
"roles/storage.objectCreator" = [module.landing-sa.iam_email]
|
||||||
|
"roles/storage.admin" = [module.transformation-sa.iam_email]
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
module "bucket-transformation" {
|
module "transformation-buckets" {
|
||||||
source = "../../../modules/gcs"
|
source = "../../../modules/gcs"
|
||||||
project_id = module.project-transformation.project_id
|
|
||||||
prefix = var.project_ids.transformation
|
|
||||||
|
|
||||||
for_each = var.transformation_buckets
|
for_each = var.transformation_buckets
|
||||||
|
project_id = var.project_ids.transformation
|
||||||
|
prefix = var.project_ids.transformation
|
||||||
name = each.value.name
|
name = each.value.name
|
||||||
location = each.value.location
|
location = each.value.location
|
||||||
iam = {
|
iam = {
|
||||||
"roles/storage.admin" = ["serviceAccount:${module.transformation-default-service-accounts.email}"],
|
"roles/storage.admin" = [module.transformation-sa.iam_email]
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -186,24 +103,34 @@ module "bucket-transformation" {
|
||||||
# Bigquery #
|
# Bigquery #
|
||||||
###############################################################################
|
###############################################################################
|
||||||
|
|
||||||
module "bigquery-datasets-datamart" {
|
module "datamart-bq" {
|
||||||
source = "../../../modules/bigquery-dataset"
|
source = "../../../modules/bigquery-dataset"
|
||||||
project_id = module.project-datamart.project_id
|
for_each = var.datamart_bq_datasets
|
||||||
|
project_id = var.project_ids.datamart
|
||||||
for_each = local.datamart_bq_datasets
|
id = each.key
|
||||||
id = each.value.id
|
|
||||||
location = each.value.location
|
location = each.value.location
|
||||||
iam = each.value.iam
|
iam = {
|
||||||
|
for k, v in each.value.iam : k => (
|
||||||
|
k == "roles/bigquery.dataOwner"
|
||||||
|
? concat(v, [module.datamart-sa.iam_email])
|
||||||
|
: v
|
||||||
|
)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
module "bigquery-datasets-dwh" {
|
module "dwh-bq" {
|
||||||
source = "../../../modules/bigquery-dataset"
|
source = "../../../modules/bigquery-dataset"
|
||||||
project_id = module.project-dwh.project_id
|
for_each = var.dwh_bq_datasets
|
||||||
|
project_id = var.project_ids.dwh
|
||||||
for_each = local.dwh_bq_datasets
|
id = each.key
|
||||||
id = each.value.id
|
|
||||||
location = each.value.location
|
location = each.value.location
|
||||||
iam = each.value.iam
|
iam = {
|
||||||
|
for k, v in each.value.iam : k => (
|
||||||
|
k == "roles/bigquery.dataOwner"
|
||||||
|
? concat(v, [module.dwh-sa.iam_email])
|
||||||
|
: v
|
||||||
|
)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
###############################################################################
|
###############################################################################
|
||||||
|
@ -211,7 +138,7 @@ module "bigquery-datasets-dwh" {
|
||||||
###############################################################################
|
###############################################################################
|
||||||
module "vpc-transformation" {
|
module "vpc-transformation" {
|
||||||
source = "../../../modules/net-vpc"
|
source = "../../../modules/net-vpc"
|
||||||
project_id = module.project-transformation.project_id
|
project_id = var.project_ids.transformation
|
||||||
name = var.transformation_vpc_name
|
name = var.transformation_vpc_name
|
||||||
subnets = var.transformation_subnets
|
subnets = var.transformation_subnets
|
||||||
}
|
}
|
||||||
|
@ -219,12 +146,18 @@ module "vpc-transformation" {
|
||||||
###############################################################################
|
###############################################################################
|
||||||
# Pub/Sub #
|
# Pub/Sub #
|
||||||
###############################################################################
|
###############################################################################
|
||||||
module "pubsub-landing" {
|
|
||||||
source = "../../../modules/pubsub"
|
|
||||||
project_id = module.project-landing.project_id
|
|
||||||
|
|
||||||
for_each = local.landing_pubsub
|
module "landing-pubsub" {
|
||||||
name = each.value.name
|
source = "../../../modules/pubsub"
|
||||||
subscriptions = each.value.subscriptions
|
for_each = var.landing_pubsub
|
||||||
subscription_iam = each.value.subscription_iam
|
project_id = var.project_ids.landing
|
||||||
|
name = each.key
|
||||||
|
subscriptions = {
|
||||||
|
for k, v in each.value : k => { labels = v.labels, options = v.options }
|
||||||
|
}
|
||||||
|
subscription_iam = {
|
||||||
|
for k, v in each.value : k => merge(v.iam, {
|
||||||
|
"roles/pubsub.subscriber" = [module.transformation-sa.iam_email]
|
||||||
|
})
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
|
@ -14,70 +14,47 @@
|
||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
output "datamart-datasets" {
|
||||||
|
description = "List of bigquery datasets created for the datamart project."
|
||||||
|
value = [
|
||||||
|
for k, datasets in module.datamart-bq : datasets.dataset_id
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
###############################################################################
|
output "dwh-datasets" {
|
||||||
# Network #
|
description = "List of bigquery datasets created for the dwh project."
|
||||||
###############################################################################
|
value = [for k, datasets in module.dwh-bq : datasets.dataset_id]
|
||||||
|
}
|
||||||
|
|
||||||
output "transformation-vpc-info" {
|
output "landing-buckets" {
|
||||||
|
description = "List of buckets created for the landing project."
|
||||||
|
value = [for k, bucket in module.landing-buckets : bucket.name]
|
||||||
|
}
|
||||||
|
|
||||||
|
output "landing-pubsub" {
|
||||||
|
description = "List of pubsub topics and subscriptions created for the landing project."
|
||||||
|
value = {
|
||||||
|
for t in module.landing-pubsub : t.topic.name => {
|
||||||
|
id = t.topic.id
|
||||||
|
subscriptions = { for s in t.subscriptions : s.name => s.id }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
output "transformation-buckets" {
|
||||||
|
description = "List of buckets created for the transformation project."
|
||||||
|
value = [for k, bucket in module.transformation-buckets : bucket.name]
|
||||||
|
}
|
||||||
|
|
||||||
|
output "transformation-vpc" {
|
||||||
description = "Transformation VPC details"
|
description = "Transformation VPC details"
|
||||||
value = {
|
value = {
|
||||||
name = module.vpc-transformation.name
|
name = module.vpc-transformation.name
|
||||||
subnets = {
|
subnets = {
|
||||||
for s in module.vpc-transformation.subnets : s.name => {
|
for k, s in module.vpc-transformation.subnets : k => {
|
||||||
gateway_address = s.gateway_address
|
|
||||||
ip_cidr_range = s.ip_cidr_range
|
ip_cidr_range = s.ip_cidr_range
|
||||||
private_ip_google_access = s.private_ip_google_access
|
|
||||||
region = s.region
|
region = s.region
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
###############################################################################
|
|
||||||
# GCS #
|
|
||||||
###############################################################################
|
|
||||||
|
|
||||||
output "landing-bucket-names" {
|
|
||||||
description = "List of buckets created for the landing project"
|
|
||||||
value = [for k, bucket in module.bucket-landing : "${bucket.name}"]
|
|
||||||
}
|
|
||||||
|
|
||||||
output "transformation-bucket-names" {
|
|
||||||
description = "List of buckets created for the transformation project"
|
|
||||||
value = [for k, bucket in module.bucket-transformation : "${bucket.name}"]
|
|
||||||
}
|
|
||||||
|
|
||||||
###############################################################################
|
|
||||||
# Bigquery #
|
|
||||||
###############################################################################
|
|
||||||
|
|
||||||
output "dwh-bigquery-datasets-list" {
|
|
||||||
description = "List of bigquery datasets created for the dwh project"
|
|
||||||
value = [for k, datasets in module.bigquery-datasets-dwh : "${datasets.dataset_id}"]
|
|
||||||
}
|
|
||||||
|
|
||||||
output "datamart-bigquery-datasets-list" {
|
|
||||||
description = "List of bigquery datasets created for the datamart project"
|
|
||||||
value = [for k, datasets in module.bigquery-datasets-datamart : "${datasets.dataset_id}"]
|
|
||||||
}
|
|
||||||
|
|
||||||
###############################################################################
|
|
||||||
# Pub/Sub #
|
|
||||||
###############################################################################
|
|
||||||
|
|
||||||
output "landing-pubsub-list" {
|
|
||||||
description = "List of pubsub topics and subscriptions created for the landing project"
|
|
||||||
value = {
|
|
||||||
for t in module.pubsub-landing : t.topic.name => {
|
|
||||||
name = t.topic.name
|
|
||||||
id = t.topic.id
|
|
||||||
subscriptions = {
|
|
||||||
for s in t.subscriptions : s.name => {
|
|
||||||
name = s.name
|
|
||||||
id = s.id
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
|
@ -14,15 +14,17 @@
|
||||||
|
|
||||||
variable "datamart_bq_datasets" {
|
variable "datamart_bq_datasets" {
|
||||||
description = "Datamart Bigquery datasets"
|
description = "Datamart Bigquery datasets"
|
||||||
type = map(any)
|
type = map(object({
|
||||||
|
iam = map(list(string))
|
||||||
|
location = string
|
||||||
|
}))
|
||||||
default = {
|
default = {
|
||||||
bq_datamart_dataset = {
|
bq_datamart_dataset = {
|
||||||
id = "bq_datamart_dataset"
|
|
||||||
location = "EU"
|
location = "EU"
|
||||||
iam = {
|
iam = {
|
||||||
"roles/bigquery.dataOwner" = []
|
# "roles/bigquery.dataOwner" = []
|
||||||
"roles/bigquery.dataEditor" = []
|
# "roles/bigquery.dataEditor" = []
|
||||||
"roles/bigquery.dataViewer" = []
|
# "roles/bigquery.dataViewer" = []
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
@ -30,69 +32,62 @@ variable "datamart_bq_datasets" {
|
||||||
|
|
||||||
variable "dwh_bq_datasets" {
|
variable "dwh_bq_datasets" {
|
||||||
description = "DWH Bigquery datasets"
|
description = "DWH Bigquery datasets"
|
||||||
type = map(any)
|
type = map(object({
|
||||||
|
location = string
|
||||||
|
iam = map(list(string))
|
||||||
|
}))
|
||||||
default = {
|
default = {
|
||||||
bq_raw_dataset = {
|
bq_raw_dataset = {
|
||||||
id = "bq_raw_dataset"
|
iam = {}
|
||||||
location = "EU"
|
location = "EU"
|
||||||
iam = {
|
|
||||||
"roles/bigquery.dataOwner" = []
|
|
||||||
"roles/bigquery.dataEditor" = []
|
|
||||||
"roles/bigquery.dataViewer" = []
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
variable "landing_buckets" {
|
variable "landing_buckets" {
|
||||||
description = "List of landing buckets to create"
|
description = "List of landing buckets to create"
|
||||||
type = map(any)
|
type = map(object({
|
||||||
|
location = string
|
||||||
|
name = string
|
||||||
|
}))
|
||||||
default = {
|
default = {
|
||||||
raw-data = {
|
raw-data = {
|
||||||
|
location = "EU"
|
||||||
name = "raw-data"
|
name = "raw-data"
|
||||||
location = "EU"
|
}
|
||||||
},
|
|
||||||
data-schema = {
|
data-schema = {
|
||||||
name = "data-schema"
|
|
||||||
location = "EU"
|
location = "EU"
|
||||||
},
|
name = "data-schema"
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
variable "landing_pubsub" {
|
variable "landing_pubsub" {
|
||||||
description = "List of landing pubsub topics and subscriptions to create"
|
description = "List of landing pubsub topics and subscriptions to create"
|
||||||
type = map(any)
|
type = map(map(object({
|
||||||
|
iam = map(list(string))
|
||||||
|
labels = map(string)
|
||||||
|
options = object({
|
||||||
|
ack_deadline_seconds = number
|
||||||
|
message_retention_duration = number
|
||||||
|
retain_acked_messages = bool
|
||||||
|
expiration_policy_ttl = number
|
||||||
|
})
|
||||||
|
})))
|
||||||
default = {
|
default = {
|
||||||
landing_1 = {
|
landing-1 = {
|
||||||
name = "landing-1"
|
|
||||||
subscriptions = {
|
|
||||||
sub1 = {
|
sub1 = {
|
||||||
labels = {},
|
iam = {
|
||||||
options = {
|
# "roles/pubsub.subscriber" = []
|
||||||
ack_deadline_seconds = null
|
|
||||||
message_retention_duration = null
|
|
||||||
retain_acked_messages = false
|
|
||||||
expiration_policy_ttl = null
|
|
||||||
}
|
}
|
||||||
},
|
labels = {}
|
||||||
sub2 = {
|
options = null
|
||||||
labels = {},
|
|
||||||
options = {
|
|
||||||
ack_deadline_seconds = null
|
|
||||||
message_retention_duration = null
|
|
||||||
retain_acked_messages = false
|
|
||||||
expiration_policy_ttl = null
|
|
||||||
}
|
|
||||||
},
|
|
||||||
}
|
|
||||||
subscription_iam = {
|
|
||||||
sub1 = {
|
|
||||||
"roles/pubsub.subscriber" = []
|
|
||||||
}
|
}
|
||||||
sub2 = {
|
sub2 = {
|
||||||
"roles/pubsub.subscriber" = []
|
iam = {}
|
||||||
}
|
labels = {},
|
||||||
}
|
options = null
|
||||||
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
@ -115,7 +110,7 @@ variable "project_ids" {
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
variable "project_service_account" {
|
variable "service_account_names" {
|
||||||
description = "Project service accounts list."
|
description = "Project service accounts list."
|
||||||
type = object({
|
type = object({
|
||||||
datamart = string
|
datamart = string
|
||||||
|
@ -135,28 +130,36 @@ variable "project_service_account" {
|
||||||
|
|
||||||
variable "transformation_buckets" {
|
variable "transformation_buckets" {
|
||||||
description = "List of transformation buckets to create"
|
description = "List of transformation buckets to create"
|
||||||
type = map(any)
|
type = map(object({
|
||||||
|
location = string
|
||||||
|
name = string
|
||||||
|
}))
|
||||||
default = {
|
default = {
|
||||||
temp = {
|
temp = {
|
||||||
name = "temp"
|
|
||||||
location = "EU"
|
location = "EU"
|
||||||
|
name = "temp"
|
||||||
},
|
},
|
||||||
templates = {
|
templates = {
|
||||||
name = "templates"
|
|
||||||
location = "EU"
|
location = "EU"
|
||||||
|
name = "templates"
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
variable "transformation_subnets" {
|
variable "transformation_subnets" {
|
||||||
description = "List of subnets to create in the transformation Project."
|
description = "List of subnets to create in the transformation Project."
|
||||||
type = list(any)
|
type = list(object({
|
||||||
|
ip_cidr_range = string
|
||||||
|
name = string
|
||||||
|
region = string
|
||||||
|
secondary_ip_range = map(string)
|
||||||
|
}))
|
||||||
default = [
|
default = [
|
||||||
{
|
{
|
||||||
name = "transformation-subnet",
|
ip_cidr_range = "10.1.0.0/20"
|
||||||
ip_cidr_range = "10.1.0.0/20",
|
name = "transformation-subnet"
|
||||||
secondary_ip_range = {},
|
|
||||||
region = "europe-west3"
|
region = "europe-west3"
|
||||||
|
secondary_ip_range = {}
|
||||||
},
|
},
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
Loading…
Reference in New Issue