Merge pull request #275 from terraform-google-modules/data-found-vpcsc
Data Foundation: add vpc-sc support
This commit is contained in:
commit
66ce91c59d
|
@ -6,6 +6,7 @@ All notable changes to this project will be documented in this file.
|
|||
- create `pubsub` service identity if service is enabled
|
||||
- support for creation of GKE Autopilot clusters
|
||||
- Add support for CMEK keys in Data Foundation end to end example
|
||||
- Add support for VPC-SC perimeters in Data Foundation end to end example
|
||||
|
||||
## [5.0.0] - 2021-06-17
|
||||
|
||||
|
|
|
@ -33,6 +33,18 @@ parent = "folders/12345678"
|
|||
|
||||
Once done testing, you can clean up resources by running `terraform destroy`.
|
||||
|
||||
### CMEK configuration
|
||||
You can configure GCP resources to use existing CMEK keys configuring the 'service_encryption_key_ids' variable. You need to specify a 'global' and a 'multiregional' key.
|
||||
|
||||
### VPC-SC configuration
|
||||
You can assign projects to an existing VPC-SC standard perimeter configuring the 'service_perimeter_standard' variable. You can retrieve the list of existing perimeters from the GCP console or using the following command:
|
||||
|
||||
'''
|
||||
gcloud access-context-manager perimeters list --format="json" | grep name
|
||||
'''
|
||||
|
||||
The script use 'google_access_context_manager_service_perimeter_resource' terraform resource. If this resource is used alongside the 'vpc-sc' module, remember to uncomment the lifecycle block in the 'vpc-sc' module so they don't fight over which resources should be in the perimeter.
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
## Variables
|
||||
|
||||
|
@ -44,6 +56,7 @@ Once done testing, you can clean up resources by running `terraform destroy`.
|
|||
| *project_names* | Override this variable if you need non-standard names. | <code title="object({ datamart = string dwh = string landing = string services = string transformation = string })">object({...})</code> | | <code title="{ datamart = "datamart" dwh = "datawh" landing = "landing" services = "services" transformation = "transformation" }">...</code> |
|
||||
| *service_account_names* | Override this variable if you need non-standard names. | <code title="object({ main = string })">object({...})</code> | | <code title="{ main = "data-platform-main" }">...</code> |
|
||||
| *service_encryption_key_ids* | Cloud KMS encryption key in {LOCATION => [KEY_URL]} format. Keys belong to existing project. | <code title="object({ multiregional = string global = string })">object({...})</code> | | <code title="{ multiregional = null global = null }">...</code> |
|
||||
| *service_perimeter_standard* | VPC Service control standard perimeter name in the form of 'accessPolicies/ACCESS_POLICY_NAME/servicePerimeters/PERIMETER_NAME'. All projects will be added to the perimeter in enforced mode. | <code title="">string</code> | | <code title="">null</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
|
|
|
@ -38,6 +38,9 @@ module "project-datamart" {
|
|||
bq = [var.service_encryption_key_ids.multiregional]
|
||||
storage = [var.service_encryption_key_ids.multiregional]
|
||||
}
|
||||
# If used, remember to uncomment 'lifecycle' block in the
|
||||
# modules/vpc-sc/google_access_context_manager_service_perimeter resource.
|
||||
service_perimeter_standard = var.service_perimeter_standard
|
||||
}
|
||||
|
||||
module "project-dwh" {
|
||||
|
@ -60,6 +63,9 @@ module "project-dwh" {
|
|||
bq = [var.service_encryption_key_ids.multiregional]
|
||||
storage = [var.service_encryption_key_ids.multiregional]
|
||||
}
|
||||
# If used, remember to uncomment 'lifecycle' block in the
|
||||
# modules/vpc-sc/google_access_context_manager_service_perimeter resource.
|
||||
service_perimeter_standard = var.service_perimeter_standard
|
||||
}
|
||||
|
||||
module "project-landing" {
|
||||
|
@ -80,6 +86,9 @@ module "project-landing" {
|
|||
pubsub = [var.service_encryption_key_ids.global]
|
||||
storage = [var.service_encryption_key_ids.multiregional]
|
||||
}
|
||||
# If used, remember to uncomment 'lifecycle' block in the
|
||||
# modules/vpc-sc/google_access_context_manager_service_perimeter resource.
|
||||
service_perimeter_standard = var.service_perimeter_standard
|
||||
}
|
||||
|
||||
module "project-services" {
|
||||
|
@ -102,6 +111,9 @@ module "project-services" {
|
|||
service_encryption_key_ids = {
|
||||
storage = [var.service_encryption_key_ids.multiregional]
|
||||
}
|
||||
# If used, remember to uncomment 'lifecycle' block in the
|
||||
# modules/vpc-sc/google_access_context_manager_service_perimeter resource.
|
||||
service_perimeter_standard = var.service_perimeter_standard
|
||||
}
|
||||
|
||||
module "project-transformation" {
|
||||
|
@ -126,6 +138,9 @@ module "project-transformation" {
|
|||
storage = [var.service_encryption_key_ids.multiregional]
|
||||
dataflow = [var.service_encryption_key_ids.global]
|
||||
}
|
||||
# If used, remember to uncomment 'lifecycle' block in the
|
||||
# modules/vpc-sc/google_access_context_manager_service_perimeter resource.
|
||||
service_perimeter_standard = var.service_perimeter_standard
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
|
|
|
@ -67,3 +67,10 @@ variable "service_encryption_key_ids" {
|
|||
global = null
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
variable "service_perimeter_standard" {
|
||||
description = "VPC Service control standard perimeter name in the form of 'accessPolicies/ACCESS_POLICY_NAME/servicePerimeters/PERIMETER_NAME'. All projects will be added to the perimeter in enforced mode."
|
||||
type = string
|
||||
default = null
|
||||
}
|
||||
|
|
|
@ -49,6 +49,9 @@ project_ids = {
|
|||
|
||||
Once done testing, you can clean up resources by running `terraform destroy`.
|
||||
|
||||
### CMEK configuration
|
||||
You can configure GCP resources to use existing CMEK keys configuring the 'service_encryption_key_ids' variable. You need to specify a 'global' and a 'multiregional' key.
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
## Variables
|
||||
|
||||
|
|
|
@ -51,8 +51,6 @@ The environment setup is designed to manage a single environment. Various strate
|
|||
| Description | Priority (1:High - 5:Low ) | Status | Remarks |
|
||||
|-------------|----------|:------:|---------|
|
||||
| DLP best practices in the pipeline | 2 | Not Started | |
|
||||
| KMS support (CMEK) | 2 | Not Started | |
|
||||
| VPC-SC | 3 | Not Started | |
|
||||
| Add Composer with a static DAG running the example | 3 | Not Started | |
|
||||
| Integrate [CI/CD composer data processing workflow framework](https://github.com/jaketf/ci-cd-for-data-processing-workflow) | 3 | Not Started | |
|
||||
| Schema changes, how to handle | 4 | Not Started | |
|
||||
|
|
Loading…
Reference in New Issue