zecfoundation
/
cloud-foundation-fabric
mirror of https://github.com/ZcashFoundation/cloud-foundation-fabric.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #260 from terraform-google-modules/ludo-dpf-20210610 

Refactor resources step
This commit is contained in:
Yoram Ben-Yaacov 2021-06-10 20:43:44 +03:00 committed by GitHub
parent fc987a9a5d abca488737
commit d0b069a162
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
7 changed files with 207 additions and 296 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
data-solutions/README.md
View File

@ -19,5 +19,5 @@ All resources use CMEK hosted in Cloud KMS running in a centralized project. The
### Data Platform Foundations ### Data Platform Foundations
<a href="./data-platform-foundations/" title="Data Platform Foundations"><img src="./data-platform-foundations/img/Data_Foundation-phase2.png" align="left" width="280px"></a> <a href="./data-platform-foundations/" title="Data Platform Foundations"><img src="./data-platform-foundations/img/Data_Foundation-phase2.png" align="left" width="280px"></a>
This [example](./data-platform-foundations/) implements a **robust and flexible** Data Foundation on GCP that provides **opinionated defaults** while allowing customers to **build and scale** out additional data pipelines **quickly and reliably**. This [example](./data-platform-foundations/) implements a robust and flexible Data Foundation on GCP that provides opinionated defaults, allowing customers to build and scale out additional data pipelines quickly and reliably.
<br clear="left"> <br clear="left">

9
data-solutions/data-platform-foundations/environment/README.md
View File

@ -10,7 +10,11 @@ The projects that will be created are:
- DWH - DWH
- Datamart - Datamart
A master service account named `projects-editor-sa` will be created under the common services project, and it will be granted editor permissions on all the projects in scope. A main service account named `projects-editor-sa` will be created under the common services project, and it will be granted editor permissions on all the projects in scope.
This is a high level diagram of the created resources:
![Environment - Phase 1](./diagram.png)
## Running the example ## Running the example
@ -26,7 +30,7 @@ parent = "folders/12345678"
- make sure you have the right authentication setup (application default credentials, or a service account key) - make sure you have the right authentication setup (application default credentials, or a service account key)
- run `terraform init` and `terraform apply` - run `terraform init` and `terraform apply`
Once done testing, you can clean up resources by running `terraform destroy` Once done testing, you can clean up resources by running `terraform destroy`.
<!-- BEGIN TFDOC --> <!-- BEGIN TFDOC -->
## Variables ## Variables
@ -44,4 +48,5 @@ Once done testing, you can clean up resources by running `terraform destroy`
| name | description | sensitive | | name | description | sensitive |
|---|---|:---:| |---|---|:---:|
| project_ids | Project ids for created projects. | | | project_ids | Project ids for created projects. | |
| service_account | Main service account. | |
<!-- END TFDOC --> <!-- END TFDOC -->

5
data-solutions/data-platform-foundations/environment/outputs.tf
View File

@ -24,3 +24,8 @@ output "project_ids" {
transformation = module.project-transformation.project_id transformation = module.project-transformation.project_id
} }
} }
output "service_account" {
description = "Main service account."
value = module.sa-services-main.email
}

78
data-solutions/data-platform-foundations/resources/README.md
View File

@ -1,12 +1,10 @@
# Data Platform Foundations - Phase 2 # Data Platform Foundations - Resources (Step 2)
## General This is the second step needed to deploy Data Platform Foundations, which creates resources needed to store and process the data, in the projects created in the [previous step](./../environment/). Please refer to the [top-level README](../README.md) for prerequisites and how to run the first step.
This is the second step needed to deploy Data Platform Foundations, which creates resources needed to store and process the data in projects created in the [environment step](./../environment/). Please refer to [top-level Data Platform README](../README.md) for prerequisites and how to run the first step.
![Data Foundation - Phase 2](./diagram.png) ![Data Foundation - Phase 2](./diagram.png)
This example will create the next resources per project: The resources that will be create in each project are:
- Common - Common
- Landing - Landing
@ -24,59 +22,49 @@ This example will create the next resources per project:
## Running the example ## Running the example
You can run Teraform script with the Service Account created in the first step. To create the resources, you can leverage the service account created in the previous step:
To create the infrastructure:
- Specify your variables in a `terraform.tvars` - Specify your variables in a `terraform.tvars`
```tfm ```tfm
project_ids = { project_ids = {
datamart = "DATAMART-PROJECT_ID" datamart = "datamart-project_id"
dwh = "DWH-PROJECT_ID" dwh = "dwh-project_id"
landing = "LANDING-PROJECT_ID" landing = "landing-project_id"
services = "SERVICES-PROJECT_ID" services = "services-project_id"
transformation = "TRANSFORMATION-PROJECT_ID" transformation = "transformation-project_id"
} }
``` ```
- Place the data_service_account_name service account (the service account was created in phase 1) key in the terraform folder - make sure you have the right authentication setup (application default credentials, or a service account key)
- Go through the following steps to create resources: - run `terraform init` and `terraform apply`
```bash Once done testing, you can clean up resources by running `terraform destroy`.
terraform init
terraform apply
```
Once done testing, you can clean up resources by running:
```bash
terraform destroy
```
<!-- BEGIN TFDOC --> <!-- BEGIN TFDOC -->
## Variables ## Variables
| Name | Description | Type | Required | Default| | name | description | type | required | default |
|------|-------------|------|:--------:|:--------:| |---|---|:---: |:---:|:---:|
| <a name="input_datamart_bq_datasets"></a> [datamart\_bq\_datasets](#Variables\_datamart\_bq\_datasets) | Datamart Bigquery datasets | `object({...})` | | `...` | | project_ids | Project IDs. | <code title="object&#40;&#123;&#10;datamart &#61; string&#10;dwh &#61; string&#10;landing &#61; string&#10;services &#61; string&#10;transformation &#61; string&#10;&#125;&#41;">object({...})</code> | ✓ | |
| <a name="input_dwh_bq_datasets"></a> [dwh\_bq\_datasets](#Variables\_dwh\_bq\_datasets) | DWH Bigquery datasets | `object({...})` | | `...` | | *datamart_bq_datasets* | Datamart Bigquery datasets | <code title="map&#40;object&#40;&#123;&#10;iam &#61; map&#40;list&#40;string&#41;&#41;&#10;location &#61; string&#10;&#125;&#41;&#41;">map(object({...}))</code> | | <code title="&#123;&#10;bq_datamart_dataset &#61; &#123;&#10;location &#61; &#34;EU&#34;&#10;iam &#61; &#123;&#10;&#125;&#10;&#125;&#10;&#125;">...</code> |
| <a name="input_landing_buckets"></a> [landing\_buckets](#Variables\_landing\_buckets) | List of landing buckets to create | `object({...})` | | `...` | | *dwh_bq_datasets* | DWH Bigquery datasets | <code title="map&#40;object&#40;&#123;&#10;location &#61; string&#10;iam &#61; map&#40;list&#40;string&#41;&#41;&#10;&#125;&#41;&#41;">map(object({...}))</code> | | <code title="&#123;&#10;bq_raw_dataset &#61; &#123;&#10;iam &#61; &#123;&#125;&#10;location &#61; &#34;EU&#34;&#10;&#125;&#10;&#125;">...</code> |
| <a name="input_landing_pubsub"></a> [landing\_pubsub](#Variables\_landing\_pubsub) | List of landing pubsub topics and subscriptions to create | `object({...})` | | `...` | | *landing_buckets* | List of landing buckets to create | <code title="map&#40;object&#40;&#123;&#10;location &#61; string&#10;name &#61; string&#10;&#125;&#41;&#41;">map(object({...}))</code> | | <code title="&#123;&#10;raw-data &#61; &#123;&#10;location &#61; &#34;EU&#34;&#10;name &#61; &#34;raw-data&#34;&#10;&#125;&#10;data-schema &#61; &#123;&#10;location &#61; &#34;EU&#34;&#10;name &#61; &#34;data-schema&#34;&#10;&#125;&#10;&#125;">...</code> |
| <a name="input_landing_service_account"></a> [landing\_service\_account](#Variables\_landing\_service\_account) | landing service accounts list. | `string` | | `sa-landing` | | *landing_pubsub* | List of landing pubsub topics and subscriptions to create | <code title="map&#40;map&#40;object&#40;&#123;&#10;iam &#61; map&#40;list&#40;string&#41;&#41;&#10;labels &#61; map&#40;string&#41;&#10;options &#61; object&#40;&#123;&#10;ack_deadline_seconds &#61; number&#10;message_retention_duration &#61; number&#10;retain_acked_messages &#61; bool&#10;expiration_policy_ttl &#61; number&#10;&#125;&#41;&#10;&#125;&#41;&#41;&#41;">map(map(object({...})))</code> | | <code title="&#123;&#10;landing-1 &#61; &#123;&#10;sub1 &#61; &#123;&#10;iam &#61; &#123;&#10;&#125;&#10;labels &#61; &#123;&#125;&#10;options &#61; null&#10;&#125;&#10;sub2 &#61; &#123;&#10;iam &#61; &#123;&#125;&#10;labels &#61; &#123;&#125;,&#10;options &#61; null&#10;&#125;,&#10;&#125;&#10;&#125;">...</code> |
| <a name="input_project_ids"></a> [project\_ids](#Variables\_project\_ids) | Project IDs. | `object({...})` | ✓ | n/a | | *landing_service_account* | landing service accounts list. | <code title="">string</code> | | <code title="">sa-landing</code> |
| <a name="input_project_service_account"></a> [project\_service\_account](#Variables\_project\_service\_account) | Project service accounts list. | `object({...})` | | `...` | | *service_account_names* | Project service accounts list. | <code title="object&#40;&#123;&#10;datamart &#61; string&#10;dwh &#61; string&#10;landing &#61; string&#10;services &#61; string&#10;transformation &#61; string&#10;&#125;&#41;">object({...})</code> | | <code title="&#123;&#10;datamart &#61; &#34;sa-datamart&#34;&#10;dwh &#61; &#34;sa-datawh&#34;&#10;landing &#61; &#34;sa-landing&#34;&#10;services &#61; &#34;sa-services&#34;&#10;transformation &#61; &#34;sa-transformation&#34;&#10;&#125;">...</code> |
| <a name="input_transformation_buckets"></a> [transformation\_buckets](#Variables\_transformation\_buckets) | List of transformation buckets to create | `object({...})` | | `...` | | *transformation_buckets* | List of transformation buckets to create | <code title="map&#40;object&#40;&#123;&#10;location &#61; string&#10;name &#61; string&#10;&#125;&#41;&#41;">map(object({...}))</code> | | <code title="&#123;&#10;temp &#61; &#123;&#10;location &#61; &#34;EU&#34;&#10;name &#61; &#34;temp&#34;&#10;&#125;,&#10;templates &#61; &#123;&#10;location &#61; &#34;EU&#34;&#10;name &#61; &#34;templates&#34;&#10;&#125;,&#10;&#125;">...</code> |
| <a name="input_transformation_subnets"></a> [transformation\_subnets](#Variables\_transformation\_subnets) | List of subnets to create in the transformation Project. | `object({...})` | | `...` | | *transformation_subnets* | List of subnets to create in the transformation Project. | <code title="list&#40;object&#40;&#123;&#10;ip_cidr_range &#61; string&#10;name &#61; string&#10;region &#61; string&#10;secondary_ip_range &#61; map&#40;string&#41;&#10;&#125;&#41;&#41;">list(object({...}))</code> | | <code title="&#91;&#10;&#123;&#10;ip_cidr_range &#61; &#34;10.1.0.0&#47;20&#34;&#10;name &#61; &#34;transformation-subnet&#34;&#10;region &#61; &#34;europe-west3&#34;&#10;secondary_ip_range &#61; &#123;&#125;&#10;&#125;,&#10;&#93;">...</code> |
| <a name="input_transformation_vpc_name"></a> [transformation\_vpc\_name](#Variables\_transformation\_vpc\_name) | Name of the VPC created in the transformation Project. | `string` | | `transformation-vpc` | | *transformation_vpc_name* | Name of the VPC created in the transformation Project. | <code title="">string</code> | | <code title="">transformation-vpc</code> |
## Outputs ## Outputs
| Name | Description | | name | description | sensitive |
|------|-------------| |---|---|:---:|
| <a name="output_datamart-bigquery-datasets-list"></a> [datamart-bigquery-datasets-list](#output\_datamart-bigquery-datasets-list) | List of bigquery datasets created for the datamart project | | datamart-datasets | List of bigquery datasets created for the datamart project. | |
| <a name="output_dwh-bigquery-datasets-list"></a> [dwh-bigquery-datasets-list](#output\_dwh-bigquery-datasets-list) | List of bigquery datasets created for the dwh project | | dwh-datasets | List of bigquery datasets created for the dwh project. | |
| <a name="output_landing-bucket-names"></a> [landing-bucket-names](#output\_landing-bucket-names) | List of buckets created for the landing project | | landing-buckets | List of buckets created for the landing project. | |
| <a name="output_landing-pubsub-list"></a> [landing-pubsub-list](#output\_landing-pubsub-list) | List of pubsub topics and subscriptions created for the landing project | | landing-pubsub | List of pubsub topics and subscriptions created for the landing project. | |
| <a name="output_transformation-bucket-names"></a> [transformation-bucket-names](#output\_transformation-bucket-names) | List of buckets created for the transformation project | | transformation-buckets | List of buckets created for the transformation project. | |
| <a name="output_transformation-vpc-info"></a> [transformation-vpc-info](#output\_transformation-vpc-info) | Transformation VPC details |<!-- END TFDOC --> | transformation-vpc | Transformation VPC details | |
<!-- END TFDOC -->

207
data-solutions/data-platform-foundations/resources/main.tf
View File

@ -14,130 +14,49 @@
* limitations under the License. * limitations under the License.
*/ */
locals {
landing_pubsub = merge({
for k, v in var.landing_pubsub :
k => {
name = v.name
subscriptions = v.subscriptions
subscription_iam = merge({
for s_k, s_v in v.subscription_iam :
s_k => merge(s_v, { "roles/pubsub.subscriber" : ["serviceAccount:${module.transformation-default-service-accounts.email}"] })
})
}
})
datamart_bq_datasets = merge({
for k, v in var.datamart_bq_datasets :
k => {
id = v.id
location = v.location
iam = merge({
for s_k, s_v in v.iam :
s_k => s_k == "roles/bigquery.dataOwner" ? concat(s_v, ["serviceAccount:${module.datamart-default-service-accounts.email}"]) : s_v
})
}
})
dwh_bq_datasets = merge({
for k, v in var.dwh_bq_datasets :
k => {
id = v.id
location = v.location
iam = merge({
for s_k, s_v in v.iam :
s_k => s_k == "roles/bigquery.dataOwner" ? concat(s_v, ["serviceAccount:${module.dwh-default-service-accounts.email}", "serviceAccount:${module.transformation-default-service-accounts.email}"]) : s_v
})
}
})
}
###############################################################################
# Projects #
###############################################################################
module "project-datamart" {
source = "../../../modules/project"
name = var.project_ids.datamart
project_create = false
}
module "project-dwh" {
source = "../../../modules/project"
name = var.project_ids.dwh
project_create = false
}
module "project-landing" {
source = "../../../modules/project"
name = var.project_ids.landing
project_create = false
}
module "project-services" {
source = "../../../modules/project"
name = var.project_ids.services
project_create = false
}
module "project-transformation" {
source = "../../../modules/project"
name = var.project_ids.transformation
project_create = false
}
############################################################################### ###############################################################################
# IAM # # IAM #
############################################################################### ###############################################################################
module "datamart-default-service-accounts" { module "datamart-sa" {
source = "../../../modules/iam-service-account" source = "../../../modules/iam-service-account"
project_id = module.project-datamart.project_id project_id = var.project_ids.datamart
name = var.project_service_account.datamart name = var.service_account_names.datamart
iam_project_roles = { iam_project_roles = {
"${module.project-datamart.project_id}" = [ "${var.project_ids.datamart}" = ["roles/editor"]
"roles/editor",
]
} }
} }
module "dwh-default-service-accounts" { module "dwh-sa" {
source = "../../../modules/iam-service-account" source = "../../../modules/iam-service-account"
project_id = module.project-dwh.project_id project_id = var.project_ids.dwh
name = var.project_service_account.dwh name = var.service_account_names.dwh
} }
module "landing-default-service-accounts" { module "landing-sa" {
source = "../../../modules/iam-service-account" source = "../../../modules/iam-service-account"
project_id = module.project-landing.project_id project_id = var.project_ids.landing
name = var.project_service_account.landing name = var.service_account_names.landing
iam_project_roles = { iam_project_roles = {
"${module.project-landing.project_id}" = [ "${var.project_ids.landing}" = ["roles/pubsub.publisher"]
"roles/pubsub.publisher",
]
} }
} }
module "services-default-service-accounts" { module "services-sa" {
source = "../../../modules/iam-service-account" source = "../../../modules/iam-service-account"
project_id = module.project-services.project_id project_id = var.project_ids.services
name = var.project_service_account.services name = var.service_account_names.services
iam_project_roles = { iam_project_roles = {
"${module.project-services.project_id}" = [ "${var.project_ids.services}" = ["roles/editor"]
"roles/editor",
]
} }
} }
module "transformation-default-service-accounts" { module "transformation-sa" {
source = "../../../modules/iam-service-account" source = "../../../modules/iam-service-account"
project_id = module.project-transformation.project_id project_id = var.project_ids.transformation
name = var.project_service_account.transformation name = var.service_account_names.transformation
iam_project_roles = { iam_project_roles = {
"${module.project-transformation.project_id}" = [ "${var.project_ids.transformation}" = [
"roles/logging.logWriter", "roles/logging.logWriter",
"roles/monitoring.metricWriter", "roles/monitoring.metricWriter",
"roles/dataflow.admin", "roles/dataflow.admin",
@ -155,30 +74,28 @@ module "transformation-default-service-accounts" {
# GCS # # GCS #
############################################################################### ###############################################################################
module "bucket-landing" { module "landing-buckets" {
source = "../../../modules/gcs" source = "../../../modules/gcs"
project_id = module.project-landing.project_id for_each = var.landing_buckets
project_id = var.project_ids.landing
prefix = var.project_ids.landing prefix = var.project_ids.landing
name = each.value.name
location = each.value.location
iam = { iam = {
"roles/storage.objectCreator" = ["serviceAccount:${module.landing-default-service-accounts.email}"], "roles/storage.objectCreator" = [module.landing-sa.iam_email]
"roles/storage.admin" = ["serviceAccount:${module.transformation-default-service-accounts.email}"], "roles/storage.admin" = [module.transformation-sa.iam_email]
} }
for_each = var.landing_buckets
name = each.value.name
location = each.value.location
} }
module "bucket-transformation" { module "transformation-buckets" {
source = "../../../modules/gcs" source = "../../../modules/gcs"
project_id = module.project-transformation.project_id for_each = var.transformation_buckets
project_id = var.project_ids.transformation
prefix = var.project_ids.transformation prefix = var.project_ids.transformation
name = each.value.name
for_each = var.transformation_buckets location = each.value.location
name = each.value.name
location = each.value.location
iam = { iam = {
"roles/storage.admin" = ["serviceAccount:${module.transformation-default-service-accounts.email}"], "roles/storage.admin" = [module.transformation-sa.iam_email]
} }
} }
@ -186,24 +103,34 @@ module "bucket-transformation" {
# Bigquery # # Bigquery #
############################################################################### ###############################################################################
module "bigquery-datasets-datamart" { module "datamart-bq" {
source = "../../../modules/bigquery-dataset" source = "../../../modules/bigquery-dataset"
project_id = module.project-datamart.project_id for_each = var.datamart_bq_datasets
project_id = var.project_ids.datamart
for_each = local.datamart_bq_datasets id = each.key
id = each.value.id location = each.value.location
location = each.value.location iam = {
iam = each.value.iam for k, v in each.value.iam : k => (
k == "roles/bigquery.dataOwner"
? concat(v, [module.datamart-sa.iam_email])
: v
)
}
} }
module "bigquery-datasets-dwh" { module "dwh-bq" {
source = "../../../modules/bigquery-dataset" source = "../../../modules/bigquery-dataset"
project_id = module.project-dwh.project_id for_each = var.dwh_bq_datasets
project_id = var.project_ids.dwh
for_each = local.dwh_bq_datasets id = each.key
id = each.value.id location = each.value.location
location = each.value.location iam = {
iam = each.value.iam for k, v in each.value.iam : k => (
k == "roles/bigquery.dataOwner"
? concat(v, [module.dwh-sa.iam_email])
: v
)
}
} }
############################################################################### ###############################################################################
@ -211,7 +138,7 @@ module "bigquery-datasets-dwh" {
############################################################################### ###############################################################################
module "vpc-transformation" { module "vpc-transformation" {
source = "../../../modules/net-vpc" source = "../../../modules/net-vpc"
project_id = module.project-transformation.project_id project_id = var.project_ids.transformation
name = var.transformation_vpc_name name = var.transformation_vpc_name
subnets = var.transformation_subnets subnets = var.transformation_subnets
} }
@ -219,12 +146,18 @@ module "vpc-transformation" {
############################################################################### ###############################################################################
# Pub/Sub # # Pub/Sub #
############################################################################### ###############################################################################
module "pubsub-landing" {
source = "../../../modules/pubsub"
project_id = module.project-landing.project_id
for_each = local.landing_pubsub module "landing-pubsub" {
name = each.value.name source = "../../../modules/pubsub"
subscriptions = each.value.subscriptions for_each = var.landing_pubsub
subscription_iam = each.value.subscription_iam project_id = var.project_ids.landing
name = each.key
subscriptions = {
for k, v in each.value : k => { labels = v.labels, options = v.options }
}
subscription_iam = {
for k, v in each.value : k => merge(v.iam, {
"roles/pubsub.subscriber" = [module.transformation-sa.iam_email]
})
}
} }

91
data-solutions/data-platform-foundations/resources/outputs.tf
View File

@ -14,69 +14,46 @@
* limitations under the License. * limitations under the License.
*/ */
output "datamart-datasets" {
description = "List of bigquery datasets created for the datamart project."
value = [
for k, datasets in module.datamart-bq : datasets.dataset_id
]
}
############################################################################### output "dwh-datasets" {
# Network # description = "List of bigquery datasets created for the dwh project."
############################################################################### value = [for k, datasets in module.dwh-bq : datasets.dataset_id]
}
output "transformation-vpc-info" { output "landing-buckets" {
description = "List of buckets created for the landing project."
value = [for k, bucket in module.landing-buckets : bucket.name]
}
output "landing-pubsub" {
description = "List of pubsub topics and subscriptions created for the landing project."
value = {
for t in module.landing-pubsub : t.topic.name => {
id = t.topic.id
subscriptions = { for s in t.subscriptions : s.name => s.id }
}
}
}
output "transformation-buckets" {
description = "List of buckets created for the transformation project."
value = [for k, bucket in module.transformation-buckets : bucket.name]
}
output "transformation-vpc" {
description = "Transformation VPC details" description = "Transformation VPC details"
value = { value = {
name = module.vpc-transformation.name name = module.vpc-transformation.name
subnets = { subnets = {
for s in module.vpc-transformation.subnets : s.name => { for k, s in module.vpc-transformation.subnets : k => {
gateway_address = s.gateway_address ip_cidr_range = s.ip_cidr_range
ip_cidr_range = s.ip_cidr_range region = s.region
private_ip_google_access = s.private_ip_google_access
region = s.region
}
}
}
}
###############################################################################
# GCS #
###############################################################################
output "landing-bucket-names" {
description = "List of buckets created for the landing project"
value = [for k, bucket in module.bucket-landing : "${bucket.name}"]
}
output "transformation-bucket-names" {
description = "List of buckets created for the transformation project"
value = [for k, bucket in module.bucket-transformation : "${bucket.name}"]
}
###############################################################################
# Bigquery #
###############################################################################
output "dwh-bigquery-datasets-list" {
description = "List of bigquery datasets created for the dwh project"
value = [for k, datasets in module.bigquery-datasets-dwh : "${datasets.dataset_id}"]
}
output "datamart-bigquery-datasets-list" {
description = "List of bigquery datasets created for the datamart project"
value = [for k, datasets in module.bigquery-datasets-datamart : "${datasets.dataset_id}"]
}
###############################################################################
# Pub/Sub #
###############################################################################
output "landing-pubsub-list" {
description = "List of pubsub topics and subscriptions created for the landing project"
value = {
for t in module.pubsub-landing : t.topic.name => {
name = t.topic.name
id = t.topic.id
subscriptions = {
for s in t.subscriptions : s.name => {
name = s.name
id = s.id
}
} }
} }
} }

111
data-solutions/data-platform-foundations/resources/variables.tf
View File

@ -14,15 +14,17 @@
variable "datamart_bq_datasets" { variable "datamart_bq_datasets" {
description = "Datamart Bigquery datasets" description = "Datamart Bigquery datasets"
type = map(any) type = map(object({
iam = map(list(string))
location = string
}))
default = { default = {
bq_datamart_dataset = { bq_datamart_dataset = {
id = "bq_datamart_dataset"
location = "EU" location = "EU"
iam = { iam = {
"roles/bigquery.dataOwner" = [] # "roles/bigquery.dataOwner" = []
"roles/bigquery.dataEditor" = [] # "roles/bigquery.dataEditor" = []
"roles/bigquery.dataViewer" = [] # "roles/bigquery.dataViewer" = []
} }
} }
} }
@ -30,69 +32,62 @@ variable "datamart_bq_datasets" {
variable "dwh_bq_datasets" { variable "dwh_bq_datasets" {
description = "DWH Bigquery datasets" description = "DWH Bigquery datasets"
type = map(any) type = map(object({
location = string
iam = map(list(string))
}))
default = { default = {
bq_raw_dataset = { bq_raw_dataset = {
id = "bq_raw_dataset" iam = {}
location = "EU" location = "EU"
iam = {
"roles/bigquery.dataOwner" = []
"roles/bigquery.dataEditor" = []
"roles/bigquery.dataViewer" = []
}
} }
} }
} }
variable "landing_buckets" { variable "landing_buckets" {
description = "List of landing buckets to create" description = "List of landing buckets to create"
type = map(any) type = map(object({
location = string
name = string
}))
default = { default = {
raw-data = { raw-data = {
location = "EU"
name = "raw-data" name = "raw-data"
location = "EU" }
},
data-schema = { data-schema = {
name = "data-schema"
location = "EU" location = "EU"
}, name = "data-schema"
}
} }
} }
variable "landing_pubsub" { variable "landing_pubsub" {
description = "List of landing pubsub topics and subscriptions to create" description = "List of landing pubsub topics and subscriptions to create"
type = map(any) type = map(map(object({
iam = map(list(string))
labels = map(string)
options = object({
ack_deadline_seconds = number
message_retention_duration = number
retain_acked_messages = bool
expiration_policy_ttl = number
})
})))
default = { default = {
landing_1 = { landing-1 = {
name = "landing-1" sub1 = {
subscriptions = { iam = {
sub1 = { # "roles/pubsub.subscriber" = []
labels = {},
options = {
ack_deadline_seconds = null
message_retention_duration = null
retain_acked_messages = false
expiration_policy_ttl = null
}
},
sub2 = {
labels = {},
options = {
ack_deadline_seconds = null
message_retention_duration = null
retain_acked_messages = false
expiration_policy_ttl = null
}
},
}
subscription_iam = {
sub1 = {
"roles/pubsub.subscriber" = []
}
sub2 = {
"roles/pubsub.subscriber" = []
} }
labels = {}
options = null
} }
sub2 = {
iam = {}
labels = {},
options = null
},
} }
} }
} }
@ -115,7 +110,7 @@ variable "project_ids" {
} }
variable "project_service_account" { variable "service_account_names" {
description = "Project service accounts list." description = "Project service accounts list."
type = object({ type = object({
datamart = string datamart = string
@ -135,28 +130,36 @@ variable "project_service_account" {
variable "transformation_buckets" { variable "transformation_buckets" {
description = "List of transformation buckets to create" description = "List of transformation buckets to create"
type = map(any) type = map(object({
location = string
name = string
}))
default = { default = {
temp = { temp = {
name = "temp"
location = "EU" location = "EU"
name = "temp"
}, },
templates = { templates = {
name = "templates"
location = "EU" location = "EU"
name = "templates"
}, },
} }
} }
variable "transformation_subnets" { variable "transformation_subnets" {
description = "List of subnets to create in the transformation Project." description = "List of subnets to create in the transformation Project."
type = list(any) type = list(object({
ip_cidr_range = string
name = string
region = string
secondary_ip_range = map(string)
}))
default = [ default = [
{ {
name = "transformation-subnet", ip_cidr_range = "10.1.0.0/20"
ip_cidr_range = "10.1.0.0/20", name = "transformation-subnet"
secondary_ip_range = {},
region = "europe-west3" region = "europe-west3"
secondary_ip_range = {}
}, },
] ]
} }