7.2 KiB
Data Platform Foundations - Resources (Step 2)
This is the second step needed to deploy Data Platform Foundations, which creates resources needed to store and process the data, in the projects created in the previous step. Please refer to the top-level README for prerequisites and how to run the first step.
The resources that will be create in each project are:
- Common
- Landing
- GCS
- Pub/Sub
- Orchestration & Transformation
- Dataflow
- DWH
- Bigquery (L0/1/2)
- GCS
- Datamart
- Bigquery (views/table)
- GCS
- BigTable
Running the example
In the previous step, we created the environment (projects and service account) which we are going to use in this step.
To create the resources, copy the output of the environment step (project_ids) and paste it into the terraform.tvars
:
- Specify your variables in a
terraform.tvars
, you can use the ouptu from the environment stage
project_ids = {
datamart = "datamart-project_id"
dwh = "dwh-project_id"
landing = "landing-project_id"
services = "services-project_id"
transformation = "transformation-project_id"
}
- Get a key for the service account created in the environment stage:
- Go into services project
- Go into IAM page
- Go into the service account section
- Creaet a new key for the service account created in previeous step (service_account)
- Download the json key into the current folder
- make sure you have the right authentication setup:
export GOOGLE_APPLICATION_CREDENTIALS=PATH_TO_SERVICE_ACCOUT_KEY.json
- run
terraform init
andterraform apply
Once done testing, you can clean up resources by running terraform destroy
.
CMEK configuration
You can configure GCP resources to use existing CMEK keys configuring the 'service_encryption_key_ids' variable. You need to specify a 'global' and a 'multiregional' key.
Variables
name | description | type | required | default |
---|---|---|---|---|
project_ids | Project IDs. | object({...}) |
✓ | |
datamart_bq_datasets | Datamart Bigquery datasets | map(object({...})) |
... |
|
dwh_bq_datasets | DWH Bigquery datasets | map(object({...})) |
... |
|
landing_buckets | List of landing buckets to create | map(object({...})) |
... |
|
landing_pubsub | List of landing pubsub topics and subscriptions to create | map(map(object({...}))) |
... |
|
landing_service_account | landing service accounts list. | string |
sa-landing |
|
service_account_names | Project service accounts list. | object({...}) |
... |
|
service_encryption_key_ids | Cloud KMS encryption key in {LOCATION => [KEY_URL]} format. Keys belong to existing project. | object({...}) |
... |
|
transformation_buckets | List of transformation buckets to create | map(object({...})) |
... |
|
transformation_subnets | List of subnets to create in the transformation Project. | list(object({...})) |
... |
|
transformation_vpc_name | Name of the VPC created in the transformation Project. | string |
transformation-vpc |
Outputs
name | description | sensitive |
---|---|---|
datamart-datasets | List of bigquery datasets created for the datamart project. | |
dwh-datasets | List of bigquery datasets created for the dwh project. | |
landing-buckets | List of buckets created for the landing project. | |
landing-pubsub | List of pubsub topics and subscriptions created for the landing project. | |
transformation-buckets | List of buckets created for the transformation project. | |
transformation-vpc | Transformation VPC details |