History

Lorenzo Caggioni 54e4155ae8 Add details on VPC-SC and CMEK in README files		2021-07-09 10:52:52 +02:00
..
README.md	Add details on VPC-SC and CMEK in README files	2021-07-09 10:52:52 +02:00
diagram.png	Folders rename	2021-06-15 16:12:20 +03:00
main.tf	Add support for CMEK keys in Data Foundation end to end example	2021-06-25 13:06:19 +02:00
outputs.tf	Folders rename	2021-06-15 16:12:20 +03:00
variables.tf	- Remove Regional KMS key, not in use at the moment	2021-07-05 07:06:21 +02:00
versions.tf	Folders rename	2021-06-15 16:12:20 +03:00

README.md

Data Platform Foundations - Resources (Step 2)

This is the second step needed to deploy Data Platform Foundations, which creates resources needed to store and process the data, in the projects created in the previous step. Please refer to the top-level README for prerequisites and how to run the first step.

The resources that will be create in each project are:

Common
Landing
- GCS
- Pub/Sub
Orchestration & Transformation
- Dataflow
DWH
- Bigquery (L0/1/2)
- GCS
Datamart
- Bigquery (views/table)
- GCS
- BigTable

Running the example

In the previous step, we created the environment (projects and service account) which we are going to use in this step.

To create the resources, copy the output of the environment step (project_ids) and paste it into the terraform.tvars:

Specify your variables in a terraform.tvars, you can use the ouptu from the environment stage

project_ids = {
  datamart       = "datamart-project_id"
  dwh            = "dwh-project_id"
  landing        = "landing-project_id"
  services       = "services-project_id"
  transformation = "transformation-project_id"
}

Get a key for the service account created in the environment stage:
- Go into services project
- Go into IAM page
- Go into the service account section
- Creaet a new key for the service account created in previeous step (service_account)
- Download the json key into the current folder
make sure you have the right authentication setup: export GOOGLE_APPLICATION_CREDENTIALS=PATH_TO_SERVICE_ACCOUT_KEY.json
run terraform init and terraform apply

Once done testing, you can clean up resources by running terraform destroy.

CMEK configuration

You can configure GCP resources to use existing CMEK keys configuring the 'service_encryption_key_ids' variable. You need to specify a 'global' and a 'multiregional' key.

Variables

name	description	type	required	default
project_ids	Project IDs.	`object({...})`	✓
datamart_bq_datasets	Datamart Bigquery datasets	`map(object({...}))`		`...`
dwh_bq_datasets	DWH Bigquery datasets	`map(object({...}))`		`...`
landing_buckets	List of landing buckets to create	`map(object({...}))`		`...`
landing_pubsub	List of landing pubsub topics and subscriptions to create	`map(map(object({...})))`		`...`
landing_service_account	landing service accounts list.	`string`		`sa-landing`
service_account_names	Project service accounts list.	`object({...})`		`...`
service_encryption_key_ids	Cloud KMS encryption key in {LOCATION => [KEY_URL]} format. Keys belong to existing project.	`object({...})`		`...`
transformation_buckets	List of transformation buckets to create	`map(object({...}))`		`...`
transformation_subnets	List of subnets to create in the transformation Project.	`list(object({...}))`		`...`
transformation_vpc_name	Name of the VPC created in the transformation Project.	`string`		`transformation-vpc`

Outputs

name	description	sensitive
datamart-datasets	List of bigquery datasets created for the datamart project.
dwh-datasets	List of bigquery datasets created for the dwh project.
landing-buckets	List of buckets created for the landing project.
landing-pubsub	List of pubsub topics and subscriptions created for the landing project.
transformation-buckets	List of buckets created for the transformation project.
transformation-vpc	Transformation VPC details