Merge pull request #275 from terraform-google-modules/data-found-vpcsc

Data Foundation: add vpc-sc support

CHANGELOG.md

@@ -6,6 +6,7 @@ All notable changes to this project will be documented in this file.
 - create `pubsub` service identity if service is enabled
 - support for creation of GKE Autopilot clusters
 - Add support for CMEK keys in Data Foundation end to end example
+- Add support for VPC-SC perimeters in Data Foundation end to end example
 
 ## [5.0.0] - 2021-06-17
 

data-solutions/data-platform-foundations/01-environment/README.md

@@ -33,6 +33,18 @@ parent          = "folders/12345678"
 
 Once done testing, you can clean up resources by running `terraform destroy`.
 
+### CMEK configuration
+You can configure GCP resources to use existing CMEK keys configuring the 'service_encryption_key_ids' variable. You need to specify a 'global' and a 'multiregional' key.
+
+### VPC-SC configuration
+You can assign projects to an existing VPC-SC standard perimeter configuring the 'service_perimeter_standard' variable. You can retrieve the list of existing perimeters from the GCP console or using the following command:
+
+'''
+gcloud access-context-manager perimeters list --format="json" | grep name
+'''
+
+The script use 'google_access_context_manager_service_perimeter_resource' terraform resource. If this resource is used alongside the 'vpc-sc' module, remember to uncomment the lifecycle block in the 'vpc-sc' module so they don't fight over which resources should be in the perimeter. 
+
 <!-- BEGIN TFDOC -->
 ## Variables
 
@@ -44,6 +56,7 @@ Once done testing, you can clean up resources by running `terraform destroy`.
 | *project_names* | Override this variable if you need non-standard names. | <code title="object&#40;&#123;&#10;datamart       &#61; string&#10;dwh            &#61; string&#10;landing        &#61; string&#10;services       &#61; string&#10;transformation &#61; string&#10;&#125;&#41;">object({...})</code> |  | <code title="&#123;&#10;datamart       &#61; &#34;datamart&#34;&#10;dwh            &#61; &#34;datawh&#34;&#10;landing        &#61; &#34;landing&#34;&#10;services       &#61; &#34;services&#34;&#10;transformation &#61; &#34;transformation&#34;&#10;&#125;">...</code> |
 | *service_account_names* | Override this variable if you need non-standard names. | <code title="object&#40;&#123;&#10;main &#61; string&#10;&#125;&#41;">object({...})</code> |  | <code title="&#123;&#10;main &#61; &#34;data-platform-main&#34;&#10;&#125;">...</code> |
 | *service_encryption_key_ids* | Cloud KMS encryption key in {LOCATION => [KEY_URL]} format. Keys belong to existing project. | <code title="object&#40;&#123;&#10;multiregional &#61; string&#10;global        &#61; string&#10;&#125;&#41;">object({...})</code> |  | <code title="&#123;&#10;multiregional &#61; null&#10;global        &#61; null&#10;&#125;">...</code> |
+| *service_perimeter_standard* | VPC Service control standard perimeter name in the form of 'accessPolicies/ACCESS_POLICY_NAME/servicePerimeters/PERIMETER_NAME'. All projects will be added to the perimeter in enforced mode. | <code title="">string</code> |  | <code title="">null</code> |
 
 ## Outputs
 

data-solutions/data-platform-foundations/01-environment/main.tf

@@ -38,6 +38,9 @@ module "project-datamart" {
     bq      = [var.service_encryption_key_ids.multiregional]
     storage = [var.service_encryption_key_ids.multiregional]
   }
+  # If used, remember to uncomment 'lifecycle' block in the
+  # modules/vpc-sc/google_access_context_manager_service_perimeter resource.  
+  service_perimeter_standard = var.service_perimeter_standard
 }
 
 module "project-dwh" {
@@ -60,6 +63,9 @@ module "project-dwh" {
     bq      = [var.service_encryption_key_ids.multiregional]
     storage = [var.service_encryption_key_ids.multiregional]
   }
+  # If used, remember to uncomment 'lifecycle' block in the
+  # modules/vpc-sc/google_access_context_manager_service_perimeter resource.  
+  service_perimeter_standard = var.service_perimeter_standard
 }
 
 module "project-landing" {
@@ -80,6 +86,9 @@ module "project-landing" {
     pubsub  = [var.service_encryption_key_ids.global]
     storage = [var.service_encryption_key_ids.multiregional]
   }
+  # If used, remember to uncomment 'lifecycle' block in the
+  # modules/vpc-sc/google_access_context_manager_service_perimeter resource.  
+  service_perimeter_standard = var.service_perimeter_standard
 }
 
 module "project-services" {
@@ -102,6 +111,9 @@ module "project-services" {
   service_encryption_key_ids = {
     storage = [var.service_encryption_key_ids.multiregional]
   }
+  # If used, remember to uncomment 'lifecycle' block in the
+  # modules/vpc-sc/google_access_context_manager_service_perimeter resource.  
+  service_perimeter_standard = var.service_perimeter_standard
 }
 
 module "project-transformation" {
@@ -126,6 +138,9 @@ module "project-transformation" {
     storage  = [var.service_encryption_key_ids.multiregional]
     dataflow = [var.service_encryption_key_ids.global]
   }
+  # If used, remember to uncomment 'lifecycle' block in the
+  # modules/vpc-sc/google_access_context_manager_service_perimeter resource.  
+  service_perimeter_standard = var.service_perimeter_standard
 }
 
 ###############################################################################

data-solutions/data-platform-foundations/01-environment/variables.tf

@@ -67,3 +67,10 @@ variable "service_encryption_key_ids" {
     global        = null
   }
 }
+
+
+variable "service_perimeter_standard" {
+  description = "VPC Service control standard perimeter name in the form of 'accessPolicies/ACCESS_POLICY_NAME/servicePerimeters/PERIMETER_NAME'. All projects will be added to the perimeter in enforced mode."
+  type = string
+  default = null
+}

data-solutions/data-platform-foundations/02-resources/README.md

@@ -49,6 +49,9 @@ project_ids = {
 
 Once done testing, you can clean up resources by running `terraform destroy`.
 
+### CMEK configuration
+You can configure GCP resources to use existing CMEK keys configuring the 'service_encryption_key_ids' variable. You need to specify a 'global' and a 'multiregional' key.
+
 <!-- BEGIN TFDOC -->
 ## Variables
 

data-solutions/data-platform-foundations/README.md

@@ -51,8 +51,6 @@ The environment setup is designed to manage a single environment. Various strate
 | Description | Priority (1:High - 5:Low ) | Status | Remarks |
 |-------------|----------|:------:|---------|
 | DLP best practices in the pipeline | 2 | Not Started |   |
-| KMS support (CMEK) | 2 | Not Started |   |
-| VPC-SC | 3 | Not Started |   |
 | Add Composer with a static DAG running the example | 3 | Not Started |   |
 | Integrate [CI/CD composer data processing workflow framework](https://github.com/jaketf/ci-cd-for-data-processing-workflow) | 3 | Not Started |   |
 | Schema changes, how to handle | 4 | Not Started |   |