Adding data-platform-foundations code

data-solutions/data-platform-foundations/README.md

@@ -0,0 +1,49 @@
+# Data Platform Foundations
+
+## General
+
+The goal of this project is to Build a **robust and flexible** Data Foundation on GCP that provides **opinionated defaults** while allowing customers to **build and scale** out additional data pipelines **quickly and reliably**.
+
+There are three provisioning workflows to enable an end to end Foundational Data Platform along with Data Pipelines on top of it. This is represented in the diagram below.
+
+![Three Main Workflows](./img/three_main_workflows.png)
+
+## Target architecture
+
+![Target architecture](./img/Data_Foundation-phase2.png)
+
+In this example we will create the infrastructure needed for the foundational build and run a demo pipeline.
+
+## Before you begine
+
+Since this example is intended for the data infra engineers we do expect that an initial organization / folder and service account with owner privileges will be pre-created and provided as variables.
+
+This example assume the next items were already created and provided:
+
+- Organization / folder
+- Terraform runner Service account with owner permissions on the above organization / folder
+
+## Building the Platform
+
+Building the platform is composed of 4 steps:
+
+1) [(Optional) Managed resources and services](./manageing_multiple_environments.md)
+1) [Data Platform Foundations - Phase 1: Building the projects](./infra/tf-phase1/README.md)
+1) [Data Platform Foundations - Phase 2: Deploy assets](./infra/tf-phase2/README.md)
+1) [(Optional) Manual pipeline Example](./data-pipeline/README.md)
+
+## TODO list
+
+| Description | Priority (1:High - 5:Low ) | Status | Remarks |
+|-------------|----------|:------:|---------|
+| DLP best practices in the pipeline | 2 | Not Started |   |
+| KMS support (CMEK) | 2 | Not Started |   |
+| VPC-SC | 3 | Not Started |   |
+| Add Composer with a static DAG running the example | 3 | Not Started |   |
+| Integrate [CI/CD composer data processing workflow framework](https://github.com/jaketf/ci-cd-for-data-processing-workflow) | 3 | Not Started |   |
+| Schema changes, how to handle | 4 | Not Started |   |
+| Data lineage | 4 | Not Started |   |
+| Data quality checks | 4 | Not Started |   |
+| Shared-VPC | 5 | Not Started |   |
+| Logging & monitoring | TBD | Not Started |   |
+| Orcestration for ingestion pipeline (just in the readme) | TBD | Not Started |   |

data-solutions/data-platform-foundations/data-pipeline/README.md

@@ -0,0 +1,8 @@
+# Manual pipeline Example
+
+Once you deployed projects [step 1](../infra/tf-phase1/README.md) and resources [step 1](../infra/tf-phase2/README.md) you can use it to run your data pipeline.
+
+Here we will demo 2 pipelines:
+
+* [GCS to Bigquery](./gcs_to_bigquery.md)
+* [PubSub to Bigquery](./pubsub_to_bigquery.md)

data-solutions/data-platform-foundations/data-pipeline/gcs_to_bigquery.md

@@ -0,0 +1,151 @@
+# Manual pipeline Example: GCS to Bigquery
+
+In this example we will publish person message in the following format:
+
+```bash
+Lorenzo,Caggioni,1617898199
+```
+
+a Dataflow pipeline will read those messages and import them into a Bigquery table in the DWH project.
+
+[TODO] An autorized view will be created in the datamart project to expose the table.
+[TODO] Remove hardcoded 'lcaggio' variables and made ENV variable for it.
+[TODO] Further automation is expected in future.
+
+Create and download keys for Service accounts you created.
+
+## Create BQ table
+
+Those steps should be done as Transformation Service Account:
+
+```bash
+gcloud auth activate-service-account sa-dwh@dwh-lc01.iam.gserviceaccount.com --key-file=sa-dwh.json --project=dwh-lc01
+```
+
+and you can run the command to create a table:
+
+```bash
+bq mk \
+-t \
+--description "This is a Test Person table" \
+dwh-lc01:bq_raw_dataset.person \
+name:STRING,surname:STRING,timestamp:TIMESTAMP
+```
+
+## Produce CSV data file, JSON schema file and UDF JS file
+
+Those steps should be done as landing Service Account:
+
+```bash
+gcloud auth activate-service-account sa-landing@landing-lc01.iam.gserviceaccount.com --key-file=sa-landing.json --project=landing-lc01
+```
+
+Let's now create a series of messages we can use to import:
+
+```bash
+for i in {0..10} 
+do 
+  echo "Lorenzo,Caggioni,$(date +%s)" >> person.csv
+done
+```
+
+and copy files to the GCS bucket:
+
+```bash
+gsutil cp person.csv gs://landing-lc01-eu-raw-data
+```
+
+Let's create the data JSON schema:
+
+```bash
+cat <<'EOF' >> person_schema.json
+{
+  "BigQuery Schema": [
+    {
+      "name": "name",
+      "type": "STRING"
+    },
+    {
+      "name": "surname",
+      "type": "STRING"
+    },
+    {
+      "name": "timestamp",
+      "type": "TIMESTAMP"
+    }
+  ]
+}
+EOF
+```
+
+and copy files to the GCS bucket:
+
+```bash
+gsutil cp person_schema.json gs://landing-lc01-eu-data-schema
+```
+
+Let's create the data UDF function to transform message data:
+
+```bash
+cat <<'EOF' >> person_udf.js
+function transform(line) {
+  var values = line.split(',');
+
+  var obj = new Object();
+  obj.name = values[0];
+  obj.surname = values[1];
+  obj.timestamp = values[2];
+  var jsonString = JSON.stringify(obj);
+
+  return jsonString;
+}
+EOF
+```
+
+and copy files to the GCS bucket:
+
+```bash
+gsutil cp person_udf.js gs://landing-lc01-eu-data-schema
+```
+
+if you want to check files copied to GCS, you can use the Transformation service account:
+
+```bash
+gcloud auth activate-service-account sa-transformation@transformation-lc01.iam.gserviceaccount.com --key-file=sa-transformation.json --project=transformation-lc01
+```
+
+and read a message (message won't be acked and will stay in the subscription):
+
+```bash
+gsutil ls gs://landing-lc01-eu-raw-data
+gsutil ls gs://landing-lc01-eu-data-schema
+```
+
+## Dataflow
+
+Those steps should be done as transformation Service Account:
+
+```bash
+gcloud auth activate-service-account sa-transformation@transformation-lc01.iam.gserviceaccount.com --key-file=sa-transformation.json --project=transformation-lc01
+```
+
+Let's than start a Dataflwo batch pipeline using a Google provided template using internal only IPs, the created network and subnetwork, the appropriate service account and requested parameters:
+
+```bash
+gcloud dataflow jobs run test_batch_lcaggio01 \
+    --gcs-location gs://dataflow-templates/latest/GCS_Text_to_BigQuery \
+    --project transformation-lc01 \
+    --region europe-west3 \
+    --disable-public-ips \
+    --network transformation-vpc \
+    --subnetwork regions/europe-west3/subnetworks/transformation-subnet \
+    --staging-location gs://transformation-lc01-eu-temp \
+    --service-account-email sa-transformation@transformation-lc01.iam.gserviceaccount.com \
+    --parameters \
+javascriptTextTransformFunctionName=transform,\
+JSONPath=gs://landing-lc01-eu-data-schema/person_schema.json,\
+javascriptTextTransformGcsPath=gs://landing-lc01-eu-data-schema/person_udf.js,\
+inputFilePattern=gs://landing-lc01-eu-raw-data/person.csv,\
+outputTable=dwh-lc01:bq_raw_dataset.person,\
+bigQueryLoadingTemporaryDirectory=gs://transformation-lc01-eu-temp
+```

data-solutions/data-platform-foundations/data-pipeline/pubsub_to_bigquery.md

@@ -0,0 +1,96 @@
+# Manual pipeline Example: PubSub to Bigquery
+
+In this example we will publish person message in the following format:
+
+```txt
+name: Lorenzo
+surname: Caggioni
+timestamp: 1617898199
+```
+
+a Dataflow pipeline will read those messages and import them into a Bigquery table in the DWH project.
+
+An autorized view will be created in the datamart project to expose the table.
+
+[TODO] Remove hardcoded 'lcaggio' variables and made ENV variable for it.
+[TODO] Further automation is expected in future.
+
+Create and download keys for Service accounts you created, be sure to have `iam.serviceAccountKeys.create` permission on projects or at folder level.
+
+```bash
+gcloud iam service-accounts keys create sa-landing.json --iam-account=sa-landing@landing-lc01.iam.gserviceaccount.com
+gcloud iam service-accounts keys create sa-transformation.json --iam-account=sa-transformation@transformation-lc01.iam.gserviceaccount.com
+gcloud iam service-accounts keys create sa-dwh.json --iam-account=sa-dwh@dwh-lc01.iam.gserviceaccount.com
+```
+
+## Create BQ table
+
+Those steps should be done as Transformation Service Account:
+
+```bash
+gcloud auth activate-service-account sa-dwh@dwh-lc01.iam.gserviceaccount.com --key-file=sa-dwh.json --project=dwh-lc01
+```
+
+and you can run the command to create a table:
+
+```bash
+bq mk \
+-t \
+--description "This is a Test Person table" \
+dwh-lc01:bq_raw_dataset.person \
+name:STRING,surname:STRING,timestamp:TIMESTAMP
+```
+
+## Produce PubSub messages
+
+Those steps should be done as landing Service Account:
+
+```bash
+gcloud auth activate-service-account sa-landing@landing-lc01.iam.gserviceaccount.com --key-file=sa-landing.json --project=landing-lc01
+```
+
+and let's now create a series of messages we can use to import:
+
+```bash
+for i in {0..10} 
+do 
+  gcloud pubsub topics publish projects/landing-lc01/topics/landing-1 --message="{\"name\": \"Lorenzo\", \"surname\": \"Caggioni\", \"timestamp\": \"$(date +%s)\"}"
+done
+```
+
+if you want to check messages published, you can use the Transformation service account:
+
+```bash
+gcloud auth activate-service-account sa-transformation@transformation-lc01.iam.gserviceaccount.com --key-file=sa-transformation.json --project=transformation-lc01
+```
+
+and read a message (message won't be acked and will stay in the subscription):
+
+```bash
+gcloud pubsub subscriptions pull projects/landing-lc01/subscriptions/sub1
+```
+
+## Dataflow
+
+Those steps should be done as transformation Service Account:
+
+```bash
+gcloud auth activate-service-account sa-transformation@transformation-lc01.iam.gserviceaccount.com --key-file=sa-transformation.json --project=transformation-lc01
+```
+
+Let's than start a Dataflwo streaming pipeline using a Google provided template using internal only IPs, the created network and subnetwork, the appropriate service account and requested parameters:
+
+```bash
+gcloud dataflow jobs run test_lcaggio01 \
+    --gcs-location gs://dataflow-templates/latest/PubSub_Subscription_to_BigQuery \
+    --project transformation-lc01 \
+    --region europe-west3 \
+    --disable-public-ips \
+    --network transformation-vpc \
+    --subnetwork regions/europe-west3/subnetworks/transformation-subnet \
+    --staging-location gs://transformation-lc01-eu-temp \
+    --service-account-email sa-transformation@transformation-lc01.iam.gserviceaccount.com \
+    --parameters \
+inputSubscription=projects/landing-lc01/subscriptions/sub1,\
+outputTableSpec=dwh-lc01:bq_raw_dataset.person
+```

data-solutions/data-platform-foundations/data-pipeline/resource/raw_data.json

@@ -0,0 +1,26 @@
+{
+  "schema": {
+    "fields": [
+      {
+        "mode": "NULLABLE",
+        "name": "name",
+        "type": "STRING"
+      },
+      {
+        "mode": "NULLABLE",
+        "name": "surname",
+        "type": "STRING"
+      },
+      {
+        "mode": "NULLABLE",
+        "name": "age",
+        "type": "INTEGER"
+      },
+      {
+        "mode": "NULLABLE",
+        "name": "boolean_val",
+        "type": "BOOLEAN"
+      }
+    ]
+  }
+}

data-solutions/data-platform-foundations/infra/tf-phase1/00_generic_variables.tf

@@ -0,0 +1,30 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+variable "billing_account" {
+  description = "Billing account id."
+  type        = string
+}
+
+variable "parent" {
+  description = "Parent folder or organization in 'folders/folder_id' or 'organizations/org_id' format."
+  type        = string
+}
+
+variable "projects_suffix" {
+  description = "Project suffix to make resources unique. If no suffix is provided a random suffix will be created"
+  type        = string
+  default     = null
+}
+

data-solutions/data-platform-foundations/infra/tf-phase1/01_services_project.tf

@@ -0,0 +1,70 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+##########################
+# project-creation
+##########################
+module "services-project-creation" {
+  source          = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
+  billing_account = var.billing_account
+  parent          = var.parent
+
+  name = join(
+    "-",
+    [
+      var.services_project_name,
+      var.projects_suffix != null ? var.projects_suffix : random_id.suffix.hex
+    ]
+  )
+  auto_create_network = false
+  services = [
+    "storage-component.googleapis.com",
+    "sourcerepo.googleapis.com",
+    "stackdriver.googleapis.com",
+    "cloudasset.googleapis.com",
+  ]
+
+  iam_additive = {}
+}
+
+##########################
+# IAM
+##########################
+module "master-service-account" {
+  source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/iam-service-account?ref=v4.2.0"
+
+  project_id = module.services-project-creation.project_id
+  name       = var.data_service_account_name
+
+  depends_on = [
+    module.services-project-creation
+  ]
+}
+
+module "set-tf-data-admin-service-account-on-services" {
+  source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
+
+  name           = module.services-project-creation.project_id
+  project_create = false
+
+  iam = {
+    "roles/editor" = [
+      "serviceAccount:${module.master-service-account.email}"
+    ]
+  }
+
+  depends_on = [
+    module.master-service-account,
+  ]
+}

data-solutions/data-platform-foundations/infra/tf-phase1/01_services_variables.tf

@@ -0,0 +1,25 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+variable "services_project_name" {
+  description = "Project name"
+  type        = string
+  default     = "services"
+}
+
+variable "data_service_account_name" {
+  description = "Name for the projects editor service account."
+  type        = string
+  default     = "projects-editor-sa"
+}

data-solutions/data-platform-foundations/infra/tf-phase1/02_landing_project.tf

@@ -0,0 +1,57 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+##########################
+# project-creation
+##########################
+module "landing-project-creation" {
+  source          = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
+  billing_account = var.billing_account
+  parent          = var.parent
+
+  name = join(
+    "-",
+    [
+      var.landing_project_name,
+      var.projects_suffix != null ? var.projects_suffix : random_id.suffix.hex
+    ]
+  )
+  auto_create_network = false
+  services = [
+    "pubsub.googleapis.com",
+    "storage-component.googleapis.com",
+  ]
+
+  iam_additive = {}
+}
+
+##########################
+# IAM
+##########################
+module "set-tf-data-admin-service-account-on-landing" {
+  source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
+
+  name           = module.landing-project-creation.project_id
+  project_create = false
+
+  iam = {
+    "roles/editor" = [
+      "serviceAccount:${module.master-service-account.email}"
+    ]
+  }
+
+  depends_on = [
+    module.master-service-account,
+  ]
+}

data-solutions/data-platform-foundations/infra/tf-phase1/02_landing_variables.tf

@@ -0,0 +1,20 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+variable "landing_project_name" {
+  description = "Project name"
+  type        = string
+  default     = "landing"
+}
+

data-solutions/data-platform-foundations/infra/tf-phase1/03_transformation_project.tf

@@ -0,0 +1,60 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+##########################
+# project-creation
+##########################
+module "transformation-project-creation" {
+  source          = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
+  billing_account = var.billing_account
+  parent          = var.parent
+
+  name = join(
+    "-",
+    [
+      var.transformation_project_name,
+      var.projects_suffix != null ? var.projects_suffix : random_id.suffix.hex
+    ]
+  )
+  auto_create_network = false
+  services = [
+    "cloudbuild.googleapis.com",
+    "compute.googleapis.com",
+    "dataflow.googleapis.com",
+    "servicenetworking.googleapis.com",
+    "storage-component.googleapis.com",
+  ]
+
+  iam_additive = {}
+}
+
+##########################
+# IAM
+##########################
+module "set-tf-data-admin-service-account-on-transformation" {
+  source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
+
+  name           = module.transformation-project-creation.project_id
+  project_create = false
+
+  iam = {
+    "roles/editor" = [
+      "serviceAccount:${module.master-service-account.email}"
+    ]
+  }
+
+  depends_on = [
+    module.master-service-account,
+  ]
+}

data-solutions/data-platform-foundations/infra/tf-phase1/03_transformation_variables.tf

@@ -0,0 +1,20 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+variable "transformation_project_name" {
+  description = "Project name"
+  type        = string
+  default     = "transformation"
+}
+

data-solutions/data-platform-foundations/infra/tf-phase1/04_dwh_project.tf

@@ -0,0 +1,59 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+##########################
+# project-creation
+##########################
+module "dwh-project-creation" {
+  source          = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
+  billing_account = var.billing_account
+  parent          = var.parent
+
+  name = join(
+    "-",
+    [
+      var.dwh_project_name,
+      var.projects_suffix != null ? var.projects_suffix : random_id.suffix.hex
+    ]
+  )
+  auto_create_network = false
+  services = [
+    "bigquery.googleapis.com",
+    "bigquerystorage.googleapis.com",
+    "bigqueryreservation.googleapis.com",
+    "storage-component.googleapis.com",
+  ]
+
+  iam_additive = {}
+}
+
+##########################
+# IAM
+##########################
+module "set-tf-data-admin-service-account-on-dwh" {
+  source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
+
+  name           = module.dwh-project-creation.project_id
+  project_create = false
+
+  iam = {
+    "roles/editor" = [
+      "serviceAccount:${module.master-service-account.email}"
+    ]
+  }
+
+  depends_on = [
+    module.master-service-account,
+  ]
+}

data-solutions/data-platform-foundations/infra/tf-phase1/04_dwh_variables.tf

@@ -0,0 +1,20 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+variable "dwh_project_name" {
+  description = "Project name"
+  type        = string
+  default     = "dwh"
+}
+

data-solutions/data-platform-foundations/infra/tf-phase1/05_datamart_project.tf

@@ -0,0 +1,62 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+##########################
+# project-creation
+##########################
+module "datamart-project-creation" {
+  source          = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
+  billing_account = var.billing_account
+  parent          = var.parent
+
+  name = join(
+    "-",
+    [
+      var.datamart_project_name,
+      var.projects_suffix != null ? var.projects_suffix : random_id.suffix.hex
+    ]
+  )
+  auto_create_network = false
+  services = [
+    "bigtable.googleapis.com",
+    "bigtableadmin.googleapis.com",
+    "bigquery.googleapis.com",
+    "bigquerystorage.googleapis.com",
+    "bigqueryreservation.googleapis.com",
+    "storage-component.googleapis.com",
+  ]
+
+  iam_additive = {}
+}
+
+##########################
+# IAM
+##########################
+module "set-tf-data-admin-service-account-on-datamart" {
+  source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
+
+  name           = module.datamart-project-creation.project_id
+  project_create = false
+
+  iam = {
+    "roles/editor" = [
+      "serviceAccount:${module.master-service-account.email}"
+    ]
+  }
+
+  depends_on = [
+    module.master-service-account,
+  ]
+}

data-solutions/data-platform-foundations/infra/tf-phase1/05_datamart_variables.tf

@@ -0,0 +1,20 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+variable "datamart_project_name" {
+  description = "Project name"
+  type        = string
+  default     = "datamart"
+}
+

data-solutions/data-platform-foundations/infra/tf-phase1/README.md

@@ -0,0 +1,87 @@
+# Data Platform Foundations - Phase 1
+
+## General
+
+This is the first part deploying the Data Platform foundations. In this part we will create the infrastructure needed for the foundational build. This includes projects and service accounts.
+
+Since this example is intended for the data infra engineers we do expect that an initial organization / folder and service account with owner privileges will be pre-created and provided as variables.
+
+This example assume the next items were already created and provided:
+
+- Organization / folder
+- Terraform runner Service account with owner permissions on the above organization / folder
+
+This example will create the next projects:
+
+- Common services
+- Landing
+- Orchestration & Transformation
+- DWH
+- Datamart
+
+A master service account named projects-editor-sa will be created under common services project and will be granted editor permissions on all the projects in scope.
+
+![Data Foundation -  Phase 1](../../img/Data_Foundation-phase1.png)
+
+## Running the example
+
+To create the infrastructure:
+
+- Specify your variables in a `terraform.tvars`
+
+```tfm
+billing_account = "BILLING ACCOUNT ID."
+parent          = "Parent folder or organization in 'folders/folder_id' or 'organizations/org_id' format."
+```
+
+- Place the service account key in the terraform folder
+- Go through the following steps to create resources:
+
+```bash
+terraform init
+terraform apply
+```
+
+Once done testing, you can clean up resources by running:
+
+```bash
+terraform destroy
+```
+
+<!-- BEGIN TFDOC -->
+## Requirements
+
+| Name | Version |
+|------|---------|
+| terraform | >= 0.13 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| random | n/a |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| billing\_account | Billing account id. | `string` | n/a | ✓ |
+| data\_service\_account\_name | Name for the projects editor service account. | `string` | `"projects-editor-sa"` |  |
+| datamart\_project\_name | Project name | `string` | `"datamart"` |  |
+| dwh\_project\_name | Project name | `string` | `"dwh"` |  |
+| landing\_project\_name | Project name | `string` | `"landing"` |  |
+| parent | Parent folder or organization in 'folders/folder\_id' or 'organizations/org\_id' format. | `string` | n/a | ✓ |
+| projects\_suffix | Project suffix to make resources unique. If no suffix is provided a random suffix will be created | `string` | `null` |  |
+| services\_project\_name | Project name | `string` | `"services"` |  |
+| transformation\_project\_name | Project name | `string` | `"transformation"` |  |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| datamart-project | Datamart project created. |
+| dwh-project | DWH project created. |
+| landing-project | Landing project created. |
+| services-project | Services project created. |
+| transformation-project | Transformation project created. |
+<!-- END TFDOC -->

data-solutions/data-platform-foundations/infra/tf-phase1/main.tf

@@ -0,0 +1,23 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+locals {
+  module_version = "4.2.0"
+}
+
+resource "random_id" "suffix" {
+  byte_length = 4
+}

data-solutions/data-platform-foundations/infra/tf-phase1/outputs.tf

@@ -0,0 +1,40 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+output "services-project" {
+  description = "Services project created."
+  value       = module.services-project-creation
+}
+
+output "landing-project" {
+  description = "Landing project created."
+  value       = module.landing-project-creation
+}
+
+output "transformation-project" {
+  description = "Transformation project created."
+  value       = module.transformation-project-creation
+}
+
+output "dwh-project" {
+  description = "DWH project created."
+  value       = module.dwh-project-creation
+}
+
+output "datamart-project" {
+  description = "Datamart project created."
+  value       = module.datamart-project-creation
+}

data-solutions/data-platform-foundations/infra/tf-phase1/versions.tf

@@ -0,0 +1,17 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+terraform {
+  required_version = ">= 0.13"
+}

data-solutions/data-platform-foundations/infra/tf-phase2/01_services.tf

@@ -0,0 +1,40 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+###############################################################################
+#                                 Project                                     #
+###############################################################################
+module "project-id-services" {
+  source         = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
+  name           = var.services_project_id
+  project_create = false
+}
+
+###############################################################################
+#                                   IAM                                       #
+###############################################################################
+module "services-default-service-accounts" {
+  source     = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/iam-service-account?ref=v4.2.0"
+  project_id = var.services_project_id
+
+  name = var.services_service_account
+
+  iam_project_roles = {
+    "${var.services_project_id}" = [
+      "roles/editor",
+    ]
+  }
+}

data-solutions/data-platform-foundations/infra/tf-phase2/01_services_variables.tf

@@ -0,0 +1,29 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+###############################################################################
+#                                 services                                    #
+###############################################################################
+variable "services_project_id" {
+  description = "services project ID."
+  type        = string
+}
+
+variable "services_service_account" {
+  description = "services service accounts list."
+  type        = string
+  default     = "sa-services"
+}

data-solutions/data-platform-foundations/infra/tf-phase2/02_landing.tf

@@ -0,0 +1,86 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+locals {
+  landing_pubsub = merge({
+    for k, v in var.landing_pubsub :
+    k => {
+      name          = v.name
+      subscriptions = v.subscriptions
+      subscription_iam = merge({
+        for s_k, s_v in v.subscription_iam :
+        s_k => merge(s_v, { "roles/pubsub.subscriber" : ["serviceAccount:${module.transformation-default-service-accounts.email}"] })
+      })
+    }
+  })
+}
+
+###############################################################################
+#                                 Project                                     #
+###############################################################################
+module "project-id-landing" {
+  source         = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
+  name           = var.landing_project_id
+  project_create = false
+}
+
+###############################################################################
+#                                   IAM                                       #
+###############################################################################
+module "landing-default-service-accounts" {
+  source     = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/iam-service-account?ref=v4.2.0"
+  project_id = var.landing_project_id
+
+  name = var.landing_service_account
+
+  iam_project_roles = {
+    "${var.landing_project_id}" = [
+      "roles/pubsub.publisher",
+    ]
+  }
+}
+
+###############################################################################
+#                                   GCS                                       #
+###############################################################################
+module "bucket-landing" {
+  source     = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/gcs?ref=v4.2.0"
+  project_id = var.landing_project_id
+  prefix     = var.landing_project_id
+  iam = {
+    "roles/storage.objectCreator" = ["serviceAccount:${module.landing-default-service-accounts.email}"],
+    "roles/storage.admin"         = ["serviceAccount:${module.transformation-default-service-accounts.email}"],
+  }
+
+  for_each = var.landing_buckets
+
+  name     = each.value.name
+  location = each.value.location
+}
+
+###############################################################################
+#                                Pub/Sub                                      #
+###############################################################################
+module "pubsub-landing" {
+  source     = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/pubsub?ref=v4.2.0"
+  project_id = var.landing_project_id
+
+  for_each = local.landing_pubsub
+
+  name             = each.value.name
+  subscriptions    = each.value.subscriptions
+  subscription_iam = each.value.subscription_iam
+}

data-solutions/data-platform-foundations/infra/tf-phase2/02_landing_variables.tf

@@ -0,0 +1,82 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+###############################################################################
+#                                 landing                                     #
+###############################################################################
+variable "landing_project_id" {
+  description = "landing project ID."
+  type        = string
+}
+
+variable "landing_service_account" {
+  description = "landing service accounts list."
+  type        = string
+  default     = "sa-landing"
+}
+
+variable "landing_buckets" {
+  description = "List of landing buckets to create"
+  type        = map(any)
+  default = {
+    raw-data = {
+      name     = "raw-data"
+      location = "EU"
+    },
+    data-schema = {
+      name     = "data-schema"
+      location = "EU"
+    },
+  }
+}
+
+variable "landing_pubsub" {
+  description = "List of landing buckets to create"
+  type        = map(any)
+  default = {
+    landing_1 = {
+      name = "landing-1"
+      subscriptions = {
+        sub1 = {
+          labels = {},
+          options = {
+            ack_deadline_seconds       = null
+            message_retention_duration = null
+            retain_acked_messages      = false
+            expiration_policy_ttl      = null
+          }
+        },
+        sub2 = {
+          labels = {},
+          options = {
+            ack_deadline_seconds       = null
+            message_retention_duration = null
+            retain_acked_messages      = false
+            expiration_policy_ttl      = null
+          }
+        },
+      }
+      subscription_iam = {
+        sub1 = {
+          "roles/pubsub.subscriber" = []
+        }
+        sub2 = {
+          "roles/pubsub.subscriber" = []
+        }
+      }
+    }
+  }
+}

data-solutions/data-platform-foundations/infra/tf-phase2/03_transformation.tf

@@ -0,0 +1,76 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+###############################################################################
+#                                 Project                                     #
+###############################################################################
+module "project-id-transformation" {
+  source         = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
+  name           = var.transformation_project_id
+  project_create = false
+}
+
+###############################################################################
+#                                   IAM                                       #
+###############################################################################
+module "transformation-default-service-accounts" {
+  source     = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/iam-service-account?ref=v4.2.0"
+  project_id = var.transformation_project_id
+
+  name = var.transformation_service_account
+
+  iam_project_roles = {
+    "${var.transformation_project_id}" = [
+      "roles/logging.logWriter",
+      "roles/monitoring.metricWriter",
+      "roles/dataflow.admin",
+      "roles/iam.serviceAccountUser",
+      "roles/bigquery.dataOwner",
+      "roles/bigquery.jobUser",
+      "roles/dataflow.worker",
+      "roles/bigquery.metadataViewer",
+      "roles/storage.objectViewer",
+    ]
+  }
+}
+
+###############################################################################
+#                                  Network                                    #
+###############################################################################
+module "vpc-transformation" {
+  source = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/net-vpc?ref=v4.2.0"
+
+  project_id = var.transformation_project_id
+  name       = var.transformation_vpc_name
+  subnets    = var.transformation_subnets
+}
+
+###############################################################################
+#                                    GCS                                      #
+###############################################################################
+module "bucket-transformation" {
+  source     = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/gcs?ref=v4.2.0"
+  project_id = var.transformation_project_id
+  prefix     = var.transformation_project_id
+
+  for_each = var.transformation_buckets
+
+  name     = each.value.name
+  location = each.value.location
+  iam = {
+    "roles/storage.admin" = ["serviceAccount:${module.transformation-default-service-accounts.email}"],
+  }
+}

data-solutions/data-platform-foundations/infra/tf-phase2/03_transformation_variables.tf

@@ -0,0 +1,63 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+###############################################################################
+#                    Orchestration and Transformation                         #
+###############################################################################
+variable "transformation_project_id" {
+  description = "Orchestration and Transformation project ID."
+  type        = string
+}
+
+variable "transformation_service_account" {
+  description = "transformation service accounts list."
+  type        = string
+  default     = "sa-transformation"
+}
+
+variable "transformation_vpc_name" {
+  description = "Name of the VPC created in the transformation Project."
+  type        = string
+  default     = "transformation-vpc"
+}
+
+variable "transformation_subnets" {
+  description = "List of subnets to create in the transformation Project."
+  type        = list(any)
+  default = [
+    {
+      name               = "transformation-subnet",
+      ip_cidr_range      = "10.1.0.0/20",
+      secondary_ip_range = {},
+      region             = "europe-west3"
+    },
+  ]
+}
+
+variable "transformation_buckets" {
+  description = "List of transformation buckets to create"
+  type        = map(any)
+  default = {
+    temp = {
+      name     = "temp"
+      location = "EU"
+    },
+    templates = {
+      name     = "templates"
+      location = "EU"
+    },
+  }
+}

data-solutions/data-platform-foundations/infra/tf-phase2/04_dwh.tf

@@ -0,0 +1,59 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+###############################################################################
+#                                 Project                                     #
+###############################################################################
+module "project-id-dwh" {
+  source         = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
+  name           = var.dwh_project_id
+  project_create = false
+}
+
+###############################################################################
+#                                   IAM                                       #
+###############################################################################
+module "dwh-default-service-accounts" {
+  source     = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/iam-service-account?ref=v4.2.0"
+  project_id = var.dwh_project_id
+
+  name = var.dwh_service_account
+}
+
+###############################################################################
+#                                 Bigquery                                    #
+###############################################################################
+module "bigquery-datasets-dwh" {
+  source     = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/bigquery-dataset?ref=v4.2.0"
+  project_id = var.dwh_project_id
+
+  for_each = var.dwh_bq_datasets
+
+  id       = each.value.id
+  location = each.value.location
+
+  access = {
+    owner  = { role = "OWNER", type = "user" }
+    reader = { role = "READER", type = "user" }
+  }
+
+  access_identities = {
+    owner  = module.transformation-default-service-accounts.email
+    reader = module.dwh-default-service-accounts.email
+  }
+
+
+}

data-solutions/data-platform-foundations/infra/tf-phase2/04_dwh_variables.tf

@@ -0,0 +1,40 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+###############################################################################
+#                                   DWH                                       #
+###############################################################################
+variable "dwh_project_id" {
+  description = "dwh project ID."
+  type        = string
+}
+
+variable "dwh_service_account" {
+  description = "dwh service accounts list."
+  type        = string
+  default     = "sa-dwh"
+}
+
+variable "dwh_bq_datasets" {
+  description = "DWH Bigquery datasets"
+  type        = map(any)
+  default = {
+    bq_raw_dataset = {
+      id       = "bq_raw_dataset"
+      location = "EU",
+    }
+  }
+}

data-solutions/data-platform-foundations/infra/tf-phase2/05_datamart.tf

@@ -0,0 +1,65 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+###############################################################################
+#                                 Project                                     #
+###############################################################################
+module "project-id-datamart" {
+  source         = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/project?ref=v4.2.0"
+  name           = var.datamart_project_id
+  project_create = false
+}
+
+###############################################################################
+#                                   IAM                                       #
+###############################################################################
+module "datamart-default-service-accounts" {
+  source     = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/iam-service-account?ref=v4.2.0"
+  project_id = var.datamart_project_id
+
+  name = var.datamart_service_account
+
+  iam_project_roles = {
+    "${var.datamart_project_id}" = [
+      "roles/editor",
+    ]
+  }
+}
+
+###############################################################################
+#                                 Bigquery                                    #
+###############################################################################
+data "google_service_account" "datamart" {
+  account_id = var.datamart_service_account
+  project    = var.datamart_project_id
+}
+
+module "bigquery-datasets-datamart" {
+  source     = "github.com/terraform-google-modules/cloud-foundation-fabric//modules/bigquery-dataset?ref=v4.2.0"
+  project_id = var.datamart_project_id
+
+  for_each = var.datamart_bq_datasets
+
+  id       = each.value.id
+  location = each.value.location
+
+  access = {
+    owner = { role = "OWNER", type = "user" }
+  }
+  access_identities = {
+    owner = module.datamart-default-service-accounts.email
+  }
+}

data-solutions/data-platform-foundations/infra/tf-phase2/05_datamart_variables.tf

@@ -0,0 +1,40 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+###############################################################################
+#                                Datamart                                     #
+###############################################################################
+variable "datamart_project_id" {
+  description = "datamart project ID."
+  type        = string
+}
+
+variable "datamart_service_account" {
+  description = "datamart service accounts list."
+  type        = string
+  default     = "sa-datamart"
+}
+
+variable "datamart_bq_datasets" {
+  description = "Datamart Bigquery datasets"
+  type        = map(any)
+  default = {
+    bq_datamart_dataset = {
+      id       = "bq_datamart_dataset"
+      location = "EU",
+    }
+  }
+}

data-solutions/data-platform-foundations/infra/tf-phase2/README.md

@@ -0,0 +1,93 @@
+# Data Platform Foundations - Phase 2
+
+## General
+
+Now that we have all the needed project, we will create all the needed assets to store and process the data.
+
+![Data Foundation -  Phase 2](../../img/Data_Foundation-phase2.png)
+
+This example will create the next resources per project:
+
+- Common
+- Landing
+  - [x] GCS
+  - [x] Pub/Sub
+- Orchestration & Transformation
+  - [x] Dataflow
+- DWH
+  - [x] Bigquery (L0/1/2)
+  - [x] GCS
+- Datamart
+  - [x] Bigquery (views/table)
+  - [x] GCS
+  - [ ] BigTable
+
+## Running the example
+
+To create the infrastructure:
+
+- Specify your variables in a `terraform.tvars`
+
+```tfm
+services_project_id       = "SERVICES-PROJECT_ID"
+landing_project_id        = "LANDING-PROJECT_ID"
+transformation_project_id = "TRANSFORMATION-PROJECT_ID"
+dwh_project_id            = "DWH-PROJECT_ID"
+datamart_project_id       = "DATAMART-PROJECT_ID"
+```
+
+- Place the data_service_account_name service account (the service account was created in phase 1) key in the terraform folder
+- Go through the following steps to create resources:
+
+```bash
+terraform init
+terraform apply
+```
+
+Once done testing, you can clean up resources by running:
+
+```bash
+terraform destroy
+```
+
+<!-- BEGIN TFDOC -->
+## Requirements
+
+| Name | Version |
+|------|---------|
+| terraform | >= 0.13 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| google | n/a |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| datamart\_bq\_datasets | Datamart Bigquery datasets | `map(any)` | <pre>{<br>  "bq_datamart_dataset": {<br>    "id": "bq_datamart_dataset",<br>    "location": "EU"<br>  }<br>}</pre> |  |
+| datamart\_project\_id | datamart project ID. | `string` | n/a | ✓ |
+| datamart\_service\_account | datamart service accounts list. | `string` | `"sa-datamart"` |  |
+| dwh\_bq\_datasets | DWH Bigquery datasets | `map(any)` | <pre>{<br>  "bq_raw_dataset": {<br>    "id": "bq_raw_dataset",<br>    "location": "EU"<br>  }<br>}</pre> |  |
+| dwh\_project\_id | dwh project ID. | `string` | n/a | ✓ |
+| dwh\_service\_account | dwh service accounts list. | `string` | `"sa-dwh"` |  |
+| landing\_buckets | List of landing buckets to create | `map(any)` | <pre>{<br>  "data-schema": {<br>    "location": "EU",<br>    "name": "data-schema"<br>  },<br>  "raw-data": {<br>    "location": "EU",<br>    "name": "raw-data"<br>  }<br>}</pre> |  |
+| landing\_project\_id | landing project ID. | `string` | n/a | ✓ |
+| landing\_pubsub | List of landing buckets to create | `map(any)` | <pre>{<br>  "landing_1": {<br>    "name": "landing-1",<br>    "subscription_iam": {<br>      "sub1": {<br>        "roles/pubsub.subscriber": []<br>      },<br>      "sub2": {<br>        "roles/pubsub.subscriber": []<br>      }<br>    },<br>    "subscriptions": {<br>      "sub1": {<br>        "labels": {},<br>        "options": {<br>          "ack_deadline_seconds": null,<br>          "expiration_policy_ttl": null,<br>          "message_retention_duration": null,<br>          "retain_acked_messages": true<br>        }<br>      },<br>      "sub2": {<br>        "labels": {},<br>        "options": {<br>          "ack_deadline_seconds": null,<br>          "expiration_policy_ttl": null,<br>          "message_retention_duration": null,<br>          "retain_acked_messages": true<br>        }<br>      }<br>    }<br>  }<br>}</pre> |  |
+| landing\_service\_account | landing service accounts list. | `string` | `"sa-landing"` |  |
+| services\_project\_id | services project ID. | `string` | n/a | ✓ |
+| services\_service\_account | services service accounts list. | `string` | `"sa-services"` |  |
+| transformation\_buckets | List of transformation buckets to create | `map(any)` | <pre>{<br>  "temp": {<br>    "location": "EU",<br>    "name": "temp"<br>  },<br>  "templates": {<br>    "location": "EU",<br>    "name": "templates"<br>  }<br>}</pre> |  |
+| transformation\_project\_id | Orchestration and Transformation project ID. | `string` | n/a | ✓ |
+| transformation\_service\_account | transformation service accounts list. | `string` | `"sa-transformation"` |  |
+| transformation\_subnets | List of subnets to create in the transformation Project. | `list(any)` | <pre>[<br>  {<br>    "ip_cidr_range": "10.1.0.0/20",<br>    "name": "transformation-subnet",<br>    "region": "europe-west3",<br>    "secondary_ip_range": {}<br>  }<br>]</pre> |  |
+| transformation\_vpc\_name | Name of the VPC created in the transformation Project. | `string` | `"transformation-vpc"` |  |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| vpc-transformation-output | Transformation VPC details |
+<!-- END TFDOC -->

data-solutions/data-platform-foundations/infra/tf-phase2/main.tf

@@ -0,0 +1,19 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+locals {
+  module_version = "4.2.0"
+}

data-solutions/data-platform-foundations/infra/tf-phase2/outputs.tf

@@ -0,0 +1,20 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+output "vpc-transformation-output" {
+  description = "Transformation VPC details"
+  value       = module.vpc-transformation
+}

data-solutions/data-platform-foundations/infra/tf-phase2/versions.tf

@@ -0,0 +1,17 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+terraform {
+  required_version = ">= 0.13"
+}

data-solutions/data-platform-foundations/manageing_multiple_environments.md

@@ -0,0 +1,27 @@
+# Manageing Multiple Environments
+
+Terraform is a great tool for provisioning immutable infrastructure.
+There are several ways to get Terraform to provision different environments using one repo. Here I’m going to use the most basic and naive method - the State separation.
+State separation signals more mature usage of Terraform but with additional maturity comes additional complexity.
+There are two primary methods to separate state between environments: directories and workspaces. I’m going to use the directory method.
+
+For this example I’ll assume we have 3 environments:
+
+- Dev
+- QA
+- Prod
+
+```bash
+export data_platform_folder="dpm"
+
+mkdir ${data_platform_folder}
+cd ${data_platform_folder}
+
+git clone https://github.com/yorambenyaacov/cloud-foundation-fabric.git dev
+
+git clone https://github.com/yorambenyaacov/cloud-foundation-fabric.git prod
+
+git clone https://github.com/yorambenyaacov/cloud-foundation-fabric.git qa
+```
+
+Now you have a directory per environment in which you can do all the needed configurations (tfvars files) and provision it.