Merge branch 'master' into ludo/gke-autopilot-datapath
This commit is contained in:
Julio Castillo
GitHub
commit
b1dfbe29c2
|
|
@ -33,7 +33,7 @@ env:
|
|||
TF_VERSION: 1.3.2
|
||||
|
||||
jobs:
|
||||
doc-examples:
|
||||
examples:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
|
|
@ -68,7 +68,7 @@ jobs:
|
|||
pip install -r tests/requirements.txt
|
||||
pytest -vv tests/examples
|
||||
|
||||
examples:
|
||||
blueprints:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
|
|
|
|||
|
|
@ -36,3 +36,4 @@ examples/cloud-operations/adfs/ansible/vars/vars.yaml
|
|||
examples/cloud-operations/adfs/ansible/gssh.sh
|
||||
examples/cloud-operations/multi-cluster-mesh-gke-fleet-api/ansible/vars.yaml
|
||||
examples/cloud-operations/multi-cluster-mesh-gke-fleet-api/ansible/gssh.sh
|
||||
blueprints/cloud-operations/network-dashboard/cloud-function.zip
|
||||
|
|
|
|||
|
|
@ -8,6 +8,9 @@ All notable changes to this project will be documented in this file.
|
|||
|
||||
### BLUEPRINTS
|
||||
|
||||
- [[#899](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/899)] Static routes monitoring metrics added to network dashboard BP ([maunope](https://github.com/maunope)) <!-- 2022-10-25 11:36:39+00:00 -->
|
||||
- [[#909](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/909)] GCS2BQ: Move images and templates in sub-folders ([lcaggio](https://github.com/lcaggio)) <!-- 2022-10-25 08:31:25+00:00 -->
|
||||
- [[#907](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/907)] Fix CloudSQL blueprint ([lcaggio](https://github.com/lcaggio)) <!-- 2022-10-25 07:08:08+00:00 -->
|
||||
- [[#897](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/897)] Project-factory: allow folder_id to be defined in defaults_file ([Malet](https://github.com/Malet)) <!-- 2022-10-21 08:20:06+00:00 -->
|
||||
- [[#900](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/900)] Improve net dashboard variables ([juliocc](https://github.com/juliocc)) <!-- 2022-10-20 20:59:31+00:00 -->
|
||||
- [[#896](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/896)] Network Dashboard: CFv2 and performance improvements ([aurelienlegrand](https://github.com/aurelienlegrand)) <!-- 2022-10-19 16:59:29+00:00 -->
|
||||
|
|
@ -44,6 +47,7 @@ All notable changes to this project will be documented in this file.
|
|||
|
||||
### FAST
|
||||
|
||||
- [[#911](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/911)] FAST: Additional PGA DNS records ([sruffilli](https://github.com/sruffilli)) <!-- 2022-10-25 12:28:29+00:00 -->
|
||||
- [[#903](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/903)] Initial replacement for CI/CD stage ([ludoo](https://github.com/ludoo)) <!-- 2022-10-23 17:52:46+00:00 -->
|
||||
- [[#898](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/898)] Update FAST bootstrap README.md ([juliocc](https://github.com/juliocc)) <!-- 2022-10-19 15:15:36+00:00 -->
|
||||
- [[#880](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/880)] **incompatible change:** Refactor net-vpc module for Terraform 1.3 ([ludoo](https://github.com/ludoo)) <!-- 2022-10-14 09:02:34+00:00 -->
|
||||
|
|
@ -63,6 +67,9 @@ All notable changes to this project will be documented in this file.
|
|||
|
||||
### MODULES
|
||||
|
||||
- [[#916](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/916)] Add support for DNS routing policies ([juliocc](https://github.com/juliocc)) <!-- 2022-10-25 14:20:53+00:00 -->
|
||||
- [[#918](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/918)] Fix race condition in SimpleNVA ([sruffilli](https://github.com/sruffilli)) <!-- 2022-10-25 13:04:38+00:00 -->
|
||||
- [[#914](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/914)] **incompatible change:** Update DNS module ([juliocc](https://github.com/juliocc)) <!-- 2022-10-25 10:31:11+00:00 -->
|
||||
- [[#904](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/904)] Add missing description field ([dsbutler101](https://github.com/dsbutler101)) <!-- 2022-10-21 15:05:11+00:00 -->
|
||||
- [[#891](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/891)] Add internal_ips output to compute-vm module ([LucaPrete](https://github.com/LucaPrete)) <!-- 2022-10-21 08:38:27+00:00 -->
|
||||
- [[#890](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/890)] Add auto_delete and instance_redistribution_type to compute-vm and compute-mig modules. ([giovannibaratta](https://github.com/giovannibaratta)) <!-- 2022-10-16 19:19:46+00:00 -->
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ This section **[networking blueprints](./networking/)** that implement core patt
|
|||
Currently available blueprints:
|
||||
|
||||
- **cloud operations** - [Resource tracking and remediation via Cloud Asset feeds](./cloud-operations/asset-inventory-feed-remediation), [Granular Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam), [Granular Cloud DNS IAM for Shared VPC](./cloud-operations/dns-shared-vpc), [Compute Engine quota monitoring](./cloud-operations/quota-monitoring), [Scheduled Cloud Asset Inventory Export to Bigquery](./cloud-operations/scheduled-asset-inventory-export-bq), [Packer image builder](./cloud-operations/packer-image-builder), [On-prem SA key management](./cloud-operations/onprem-sa-key-management), [TCP healthcheck for unmanaged GCE instances](./cloud-operations/unmanaged-instances-healthcheck), [HTTP Load Balancer with Cloud Armor](./cloud-operations/glb_and_armor)
|
||||
- **data solutions** - [GCE/GCS CMEK via centralized Cloud KMS](./data-solutions/gcs-to-bq-with-least-privileges/), [Cloud Storage to Bigquery with Cloud Dataflow with least privileges](./data-solutions/gcs-to-bq-with-least-privileges/), [Data Platform Foundations](./data-solutions/data-platform-foundations/), [SQL Server AlwaysOn availability groups blueprint](./data-solutions/sqlserver-alwayson), [Cloud SQL instance with multi-region read replicas](./data-solutions/cloudsql-multiregion/), [Cloud Composer version 2 private instance, supporting Shared VPC and external CMEK key](./data-solutions/composer-2/)
|
||||
- **data solutions** - [GCE/GCS CMEK via centralized Cloud KMS](./data-solutions/cmek-via-centralized-kms/), [Cloud Storage to Bigquery with Cloud Dataflow with least privileges](./data-solutions/gcs-to-bq-with-least-privileges/), [Data Platform Foundations](./data-solutions/data-platform-foundations/), [SQL Server AlwaysOn availability groups blueprint](./data-solutions/sqlserver-alwayson), [Cloud SQL instance with multi-region read replicas](./data-solutions/cloudsql-multiregion/), [Cloud Composer version 2 private instance, supporting Shared VPC and external CMEK key](./data-solutions/composer-2/)
|
||||
- **factories** - [The why and the how of resource factories](./factories/README.md)
|
||||
- **GKE** - [GKE multitenant fleet](./gke/multitenant-fleet/), [Shared VPC with GKE support](./networking/shared-vpc-gke/), [Binary Authorization Pipeline](./gke/binauthz/), [Multi-cluster mesh on GKE (fleet API)](./gke/multi-cluster-mesh-gke-fleet-api/)
|
||||
- **networking** - [hub and spoke via peering](./networking/hub-and-spoke-peering/), [hub and spoke via VPN](./networking/hub-and-spoke-vpn/), [DNS and Google Private Access for on-premises](./networking/onprem-google-access-dns/), [Shared VPC with GKE support](./networking/shared-vpc-gke/), [ILB as next hop](./networking/ilb-next-hop), [Connecting to on-premise services leveraging PSC and hybrid NEGs](./networking/psc-hybrid/), [decentralized firewall](./networking/decentralized-firewall)
|
||||
|
|
|
|||
|
|
@ -62,3 +62,8 @@ This [blueprint](./onprem-sa-key-management) shows how to manage IAM Service Acc
|
|||
<a href="./unmanaged-instances-healthcheck" title="Unmanaged GCE Instance healthchecker"><img src="./unmanaged-instances-healthcheck/diagram.png" align="left" width="280px"></a> This [blueprint](./unmanaged-instances-healthcheck) shows how to leverage [Serverless VPC Access](https://cloud.google.com/vpc/docs/configure-serverless-vpc-access) and Cloud Functions to organize a highly performant TCP healtheck for unmanaged GCE instances.
|
||||
|
||||
<br clear="left">
|
||||
|
||||
## Workload identity federation for Terraform Enterprise workflow
|
||||
<a href="./terraform-enterprise-wif" title="Workload identity federation for Terraform Cloud/Enterprise workflow"><img src="./terraform-enterprise-wif/diagram.png" align="left" width="280px"></a> This [blueprint](./terraform-enterprise-wif) shows how to configure [Wokload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) between [Terraform Cloud/Enterprise](https://developer.hashicorp.com/terraform/enterprise) instance and Google Cloud.
|
||||
|
||||
<br clear="left">
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -46,22 +46,32 @@ The Cloud Function currently tracks usage, limit and utilization of:
|
|||
- internal forwarding rules for internal L7 load balancers per VPC
|
||||
- internal forwarding rules for internal L4 load balancers per VPC peering group
|
||||
- internal forwarding rules for internal L7 load balancers per VPC peering group
|
||||
- Dynamic routes per VPC
|
||||
- Dynamic routes per VPC peering group
|
||||
- Dynamic routes per VPC
|
||||
- Dynamic routes per VPC peering group
|
||||
- Static routes per project (VPC drill down is available for usage)
|
||||
- Static routes per VPC peering group
|
||||
- IP utilization per subnet (% of IP addresses used in a subnet)
|
||||
- VPC firewall rules per project (VPC drill down is available for usage)
|
||||
- Tuples per Firewall Policy
|
||||
|
||||
It writes this values to custom metrics in Cloud Monitoring and creates a dashboard to visualize the current utilization of these metrics in Cloud Monitoring.
|
||||
|
||||
Note that metrics are created in the cloud-function/metrics.yaml file.
|
||||
Note that metrics are created in the cloud-function/metrics.yaml file. You can also edit default limits for a specific network in that file. See the example for `vpc_peering_per_network`.
|
||||
|
||||
## Assumptions and limitations
|
||||
- The CF assumes that all VPCs in peering groups are within the same organization, except for PSA peerings
|
||||
- The CF will only fetch subnet utilization data from the PSA peerings (not the VMs, ILB or routes usage)
|
||||
- The CF assumes global routing is ON, this impacts dynamic routes usage calculation
|
||||
- The CF assumes custom routes importing/exporting is ON, this impacts static and dynamic routes usage calculation
|
||||
- The CF assumes all networks in peering groups have the same global routing and custom routes sharing configuration
|
||||
|
||||
You can also edit default limits for a specific network in that file. See the example for `vpc_peering_per_network`.
|
||||
|
||||
## Next steps and ideas
|
||||
In a future release, we could support:
|
||||
- Static routes per VPC / per VPC peering group
|
||||
- Google managed VPCs that are peered with PSA (such as Cloud SQL or Memorystore)
|
||||
- Dynamic routes calculation for VPCs/PPGs with "global routing" set to OFF
|
||||
- Static routes calculation for projects/PPGs with "custom routes importing/exporting" set to OFF
|
||||
- Calculations for cross Organization peering groups
|
||||
|
||||
If you are interested in this and/or would like to contribute, please contact legranda@google.com.
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
|
|
|||
|
|
@ -163,6 +163,9 @@ def main(event, context=None):
|
|||
l4_forwarding_rules_dict = ilb_fwrules.get_forwarding_rules_dict(config, "L4")
|
||||
l7_forwarding_rules_dict = ilb_fwrules.get_forwarding_rules_dict(config, "L7")
|
||||
subnet_range_dict = networks.get_subnet_ranges_dict(config)
|
||||
static_routes_dict = routes.get_static_routes_dict(config)
|
||||
dynamic_routes_dict = routes.get_dynamic_routes(
|
||||
config, metrics_dict, limits_dict['dynamic_routes_per_network_limit'])
|
||||
|
||||
try:
|
||||
|
||||
|
|
@ -181,10 +184,12 @@ def main(event, context=None):
|
|||
ilb_fwrules.get_forwarding_rules_data(
|
||||
config, metrics_dict, l7_forwarding_rules_dict,
|
||||
limits_dict['internal_forwarding_rules_l7_limit'], "L7")
|
||||
|
||||
routes.get_static_routes_data(config, metrics_dict, static_routes_dict,
|
||||
project_quotas_dict)
|
||||
|
||||
peerings.get_vpc_peering_data(config, metrics_dict,
|
||||
limits_dict['number_of_vpc_peerings_limit'])
|
||||
dynamic_routes_dict = routes.get_dynamic_routes(
|
||||
config, metrics_dict, limits_dict['dynamic_routes_per_network_limit'])
|
||||
|
||||
# Per VPC peering group metrics
|
||||
metrics.get_pgg_data(
|
||||
|
|
@ -207,7 +212,13 @@ def main(event, context=None):
|
|||
["subnet_ranges_per_peering_group"], subnet_range_dict,
|
||||
config["limit_names"]["SUBNET_RANGES"],
|
||||
limits_dict['number_of_subnet_IP_ranges_ppg_limit'])
|
||||
routes.get_dynamic_routes_ppg(
|
||||
#static
|
||||
routes.get_routes_ppg(
|
||||
config, metrics_dict["metrics_per_peering_group"]
|
||||
["static_routes_per_peering_group"], static_routes_dict,
|
||||
limits_dict['static_routes_per_peering_group_limit'])
|
||||
#dynamic
|
||||
routes.get_routes_ppg(
|
||||
config, metrics_dict["metrics_per_peering_group"]
|
||||
["dynamic_routes_per_peering_group"], dynamic_routes_dict,
|
||||
limits_dict['dynamic_routes_per_peering_group_limit'])
|
||||
|
|
|
|||
|
|
@ -99,6 +99,19 @@ metrics_per_network:
|
|||
utilization:
|
||||
name: dynamic_routes_per_network_utilization
|
||||
description: Number of Dynamic routes per network - utilization.
|
||||
#static routes limit is per project, but usage is per network
|
||||
static_routes_per_project:
|
||||
usage:
|
||||
name: static_routes_per_project_vpc_usage
|
||||
description: Number of Static routes per project and network - usage.
|
||||
limit:
|
||||
name: static_routes_per_project_limit
|
||||
description: Number of Static routes per project - limit.
|
||||
values:
|
||||
default_value: 250
|
||||
utilization:
|
||||
name: static_routes_per_project_utilization
|
||||
description: Number of Static routes per project - utilization.
|
||||
metrics_per_peering_group:
|
||||
l4_forwarding_rules_per_peering_group:
|
||||
usage:
|
||||
|
|
@ -160,6 +173,18 @@ metrics_per_peering_group:
|
|||
utilization:
|
||||
name: dynamic_routes_per_peering_group_utilization
|
||||
description: Number of Dynamic routes per peering group - utilization.
|
||||
static_routes_per_peering_group:
|
||||
usage:
|
||||
name: static_routes_per_peering_group_usage
|
||||
description: Number of Static routes per peering group - usage.
|
||||
limit:
|
||||
name: static_routes_per_peering_group_limit
|
||||
description: Number of Static routes per peering group - limit.
|
||||
values:
|
||||
default_value: 300
|
||||
utilization:
|
||||
name: static_routes_per_peering_group_utilization
|
||||
description: Number of Static routes per peering group - utilization.
|
||||
metrics_per_project:
|
||||
firewalls:
|
||||
usage:
|
||||
|
|
|
|||
|
|
@ -26,8 +26,8 @@ from . import metrics, networks, limits
|
|||
|
||||
def get_firewall_policies_dict(config: dict):
|
||||
'''
|
||||
Calls the Asset Inventory API to get all Firewall Policies under the GCP organization
|
||||
|
||||
Calls the Asset Inventory API to get all Firewall Policies under the GCP organization, including children
|
||||
Ignores monitored projects list: returns all policies regardless of their parent resource
|
||||
Parameters:
|
||||
config (dict): The dict containing config like clients and limits
|
||||
Returns:
|
||||
|
|
@ -55,8 +55,8 @@ def get_firewall_policies_dict(config: dict):
|
|||
|
||||
def get_firewal_policies_data(config, metrics_dict, firewall_policies_dict):
|
||||
'''
|
||||
Gets the data for VPC Firewall lorem ipsum
|
||||
|
||||
Gets the data for VPC Firewall Policies in an organization, including children. All folders are considered,
|
||||
only projects in the monitored projects list are considered.
|
||||
Parameters:
|
||||
config (dict): The dict containing config like clients and limits
|
||||
metrics_dict (dictionary of dictionary of string: string): metrics names and descriptions.
|
||||
|
|
@ -91,6 +91,9 @@ def get_firewal_policies_data(config, metrics_dict, firewall_policies_dict):
|
|||
parent_type = re.search("(^\w+)", firewall_policy["parent"]).group(
|
||||
1) if "parent" in firewall_policy else "projects"
|
||||
|
||||
if parent_type == "projects" and parent not in config["monitored_projects"]:
|
||||
continue
|
||||
|
||||
metric_labels = {'parent': parent, 'parent_type': parent_type}
|
||||
|
||||
metric_labels["name"] = firewall_policy[
|
||||
|
|
|
|||
|
|
@ -42,7 +42,7 @@ def get_quotas_dict(quotas_list):
|
|||
|
||||
def get_quota_project_limit(config, regions=["global"]):
|
||||
'''
|
||||
Retrieves limit for a specific project quota
|
||||
Retrieves quotas for all monitored project in selected regions, default 'global'
|
||||
Parameters:
|
||||
project_link (string): Project link.
|
||||
Returns:
|
||||
|
|
@ -158,7 +158,7 @@ def get_quota_current_limit(config, project_link, metric_name):
|
|||
|
||||
def count_effective_limit(config, project_id, network_dict, usage_metric_name,
|
||||
limit_metric_name, utilization_metric_name,
|
||||
limit_dict):
|
||||
limit_dict, timestamp=None):
|
||||
'''
|
||||
Calculates the effective limits (using algorithm in the link below) for peering groups and writes data (usage, limit, utilization) to the custom metrics.
|
||||
Source: https://cloud.google.com/vpc/docs/quota#vpc-peering-effective-limit
|
||||
|
|
@ -171,11 +171,13 @@ def count_effective_limit(config, project_id, network_dict, usage_metric_name,
|
|||
limit_metric_name (string): Name of the custom metric to be populated for limit per VPC peering group.
|
||||
utilization_metric_name (string): Name of the custom metric to be populated for utilization per VPC peering group.
|
||||
limit_dict (dictionary of string:int): Dictionary containing the limit per peering group (either VPC specific or default limit).
|
||||
timestamp (time): timestamp to be recorded for all points
|
||||
Returns:
|
||||
None
|
||||
'''
|
||||
|
||||
timestamp = time.time()
|
||||
if timestamp == None:
|
||||
timestamp = time.time()
|
||||
|
||||
if network_dict['peerings'] == []:
|
||||
return
|
||||
|
|
|
|||
|
|
@ -91,7 +91,8 @@ def create_metric(metric_name, description, monitoring_project, config):
|
|||
def append_data_to_series_buffer(config, metric_name, metric_value,
|
||||
metric_labels, timestamp=None):
|
||||
'''
|
||||
Writes data to Cloud Monitoring custom metrics.
|
||||
Appends data to Cloud Monitoring custom metrics, using a buffer. buffer is flushed every BUFFER_LEN elements,
|
||||
any unflushed series is discarded upon function closure
|
||||
Parameters:
|
||||
config (dict): The dict containing config like clients and limits
|
||||
metric_name (string): Name of the metric
|
||||
|
|
@ -139,7 +140,7 @@ def append_data_to_series_buffer(config, metric_name, metric_value,
|
|||
|
||||
def flush_series_buffer(config):
|
||||
'''
|
||||
writes buffered metrics to Google Cloud Monitoring, empties buffer upon failure
|
||||
writes buffered metrics to Google Cloud Monitoring, empties buffer upon both failure/success
|
||||
config (dict): The dict containing config like clients and limits
|
||||
'''
|
||||
try:
|
||||
|
|
@ -188,6 +189,7 @@ def get_pgg_data(config, metric_dict, usage_dict, limit_metric, limit_dict):
|
|||
|
||||
current_quota_limit_view = customize_quota_view(current_quota_limit)
|
||||
|
||||
timestamp = time.time()
|
||||
# For each network in this GCP project
|
||||
for network_dict in network_dict_list:
|
||||
if network_dict['network_id'] == 0:
|
||||
|
|
@ -238,7 +240,7 @@ def get_pgg_data(config, metric_dict, usage_dict, limit_metric, limit_dict):
|
|||
metric_dict["usage"]["name"],
|
||||
metric_dict["limit"]["name"],
|
||||
metric_dict["utilization"]["name"],
|
||||
limit_dict)
|
||||
limit_dict, timestamp)
|
||||
print(
|
||||
f"Buffered {metric_dict['usage']['name']} for peering group {network_dict['network_name']} in {project_id}"
|
||||
)
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@
|
|||
import time
|
||||
|
||||
from collections import defaultdict
|
||||
from google.protobuf import field_mask_pb2
|
||||
from . import metrics, networks, limits, peerings, routers
|
||||
|
||||
|
||||
|
|
@ -78,8 +79,8 @@ def get_routes_for_network(config, network_link, project_id, routers_dict):
|
|||
|
||||
def get_dynamic_routes(config, metrics_dict, limits_dict):
|
||||
'''
|
||||
Writes all dynamic routes per VPC to custom metrics.
|
||||
|
||||
This function gets the usage, limit and utilization for the dynamic routes per VPC
|
||||
note: assumes global routing is ON for all VPCs
|
||||
Parameters:
|
||||
config (dict): The dict containing config like clients and limits
|
||||
metrics_dict (dictionary of dictionary of string: string): metrics names and descriptions.
|
||||
|
|
@ -128,10 +129,10 @@ def get_dynamic_routes(config, metrics_dict, limits_dict):
|
|||
return dynamic_routes_dict
|
||||
|
||||
|
||||
def get_dynamic_routes_ppg(config, metric_dict, usage_dict, limit_dict):
|
||||
def get_routes_ppg(config, metric_dict, usage_dict, limit_dict):
|
||||
'''
|
||||
This function gets the usage, limit and utilization for the dynamic routes per VPC peering group.
|
||||
|
||||
This function gets the usage, limit and utilization for the static or dynamic routes per VPC peering group.
|
||||
note: assumes global routing is ON for all VPCs for dynamic routes, assumes share custom routes is on for all peered networks
|
||||
Parameters:
|
||||
config (dict): The dict containing config like clients and limits
|
||||
metric_dict (dictionary of string: string): Dictionary with the metric names and description, that will be used to populate the metrics
|
||||
|
|
@ -140,11 +141,12 @@ def get_dynamic_routes_ppg(config, metric_dict, usage_dict, limit_dict):
|
|||
Returns:
|
||||
None
|
||||
'''
|
||||
for project in config["monitored_projects"]:
|
||||
network_dict_list = peerings.gather_peering_data(config, project)
|
||||
timestamp = time.time()
|
||||
for project_id in config["monitored_projects"]:
|
||||
network_dict_list = peerings.gather_peering_data(config, project_id)
|
||||
|
||||
for network_dict in network_dict_list:
|
||||
network_link = f"https://www.googleapis.com/compute/v1/projects/{project}/global/networks/{network_dict['network_name']}"
|
||||
network_link = f"https://www.googleapis.com/compute/v1/projects/{project_id}/global/networks/{network_dict['network_name']}"
|
||||
|
||||
limit = limits.get_ppg(network_link, limit_dict)
|
||||
|
||||
|
|
@ -169,11 +171,119 @@ def get_dynamic_routes_ppg(config, metric_dict, usage_dict, limit_dict):
|
|||
peered_network_dict["usage"] = peered_usage
|
||||
peered_network_dict["limit"] = peered_limit
|
||||
|
||||
limits.count_effective_limit(config, project, network_dict,
|
||||
limits.count_effective_limit(config, project_id, network_dict,
|
||||
metric_dict["usage"]["name"],
|
||||
metric_dict["limit"]["name"],
|
||||
metric_dict["utilization"]["name"],
|
||||
limit_dict)
|
||||
limit_dict, timestamp)
|
||||
print(
|
||||
f"Wrote {metric_dict['usage']['name']} for peering group {network_dict['network_name']} in {project}"
|
||||
f"Buffered {metric_dict['usage']['name']} for peering group {network_dict['network_name']} in {project_id}"
|
||||
)
|
||||
|
||||
|
||||
def get_static_routes_dict(config):
|
||||
'''
|
||||
Calls the Asset Inventory API to get all static custom routes under the GCP organization.
|
||||
Parameters:
|
||||
config (dict): The dict containing config like clients and limits
|
||||
Returns:
|
||||
routes_per_vpc_dict (dictionary of string: int): Keys are the network links and values are the number of custom static routes per network.
|
||||
'''
|
||||
routes_per_vpc_dict = defaultdict()
|
||||
usage_dict = defaultdict()
|
||||
|
||||
read_mask = field_mask_pb2.FieldMask()
|
||||
read_mask.FromJsonString('name,versionedResources')
|
||||
|
||||
response = config["clients"]["asset_client"].search_all_resources(
|
||||
request={
|
||||
"scope": f"organizations/{config['organization']}",
|
||||
"asset_types": ["compute.googleapis.com/Route"],
|
||||
"read_mask": read_mask
|
||||
})
|
||||
|
||||
for resource in response:
|
||||
for versioned in resource.versioned_resources:
|
||||
static_route = dict()
|
||||
for field_name, field_value in versioned.resource.items():
|
||||
static_route[field_name] = field_value
|
||||
static_route["project_id"] = static_route["network"].split('/')[6]
|
||||
static_route["network_name"] = static_route["network"].split('/')[-1]
|
||||
network_link = f"https://www.googleapis.com/compute/v1/projects/{static_route['project_id']}/global/networks/{static_route['network_name']}"
|
||||
#exclude default vpc and peering routes, dynamic routes are not in Cloud Asset Inventory
|
||||
if "nextHopPeering" not in static_route and "nextHopNetwork" not in static_route:
|
||||
if network_link not in routes_per_vpc_dict:
|
||||
routes_per_vpc_dict[network_link] = dict()
|
||||
routes_per_vpc_dict[network_link]["project_id"] = static_route[
|
||||
"project_id"]
|
||||
routes_per_vpc_dict[network_link]["network_name"] = static_route[
|
||||
"network_name"]
|
||||
if static_route["destRange"] not in routes_per_vpc_dict[network_link]:
|
||||
routes_per_vpc_dict[network_link][static_route["destRange"]] = {}
|
||||
if "usage" not in routes_per_vpc_dict[network_link]:
|
||||
routes_per_vpc_dict[network_link]["usage"] = 0
|
||||
routes_per_vpc_dict[network_link][
|
||||
"usage"] = routes_per_vpc_dict[network_link]["usage"] + 1
|
||||
|
||||
#output a dict with network links and usage only
|
||||
return {
|
||||
network_link_out: routes_per_vpc_dict[network_link_out]["usage"]
|
||||
for network_link_out in routes_per_vpc_dict
|
||||
}
|
||||
|
||||
|
||||
def get_static_routes_data(config, metrics_dict, static_routes_dict,
|
||||
project_quotas_dict):
|
||||
'''
|
||||
Determines and writes the number of static routes for each VPC in monitored projects, the per project limit and the per project utilization
|
||||
note: assumes custom routes sharing is ON for all VPCs
|
||||
Parameters:
|
||||
config (dict): The dict containing config like clients and limits
|
||||
metric_dict (dictionary of string: string): Dictionary with the metric names and description, that will be used to populate the metrics
|
||||
static_routes_dict (dictionary of dictionary: int): Keys are the network links and values are the number of custom static routes per network.
|
||||
project_quotas_dict (dictionary of string:int): Dictionary with the network link as key and the limit as value.
|
||||
Returns:
|
||||
None
|
||||
'''
|
||||
timestamp = time.time()
|
||||
project_usage = {project: 0 for project in config["monitored_projects"]}
|
||||
|
||||
#usage is drilled down by network
|
||||
for network_link in static_routes_dict:
|
||||
|
||||
project_id = network_link.split('/')[6]
|
||||
if (project_id not in config["monitored_projects"]):
|
||||
continue
|
||||
network_name = network_link.split('/')[-1]
|
||||
|
||||
project_usage[project_id] = project_usage[project_id] + static_routes_dict[
|
||||
network_link]
|
||||
|
||||
metric_labels = {"project": project_id, "network_name": network_name}
|
||||
metrics.append_data_to_series_buffer(
|
||||
config, metrics_dict["metrics_per_network"]["static_routes_per_project"]
|
||||
["usage"]["name"], static_routes_dict[network_link], metric_labels,
|
||||
timestamp=timestamp)
|
||||
|
||||
#limit and utilization are calculated by project
|
||||
for project_id in project_usage:
|
||||
current_quota_limit = project_quotas_dict[project_id]['global']["routes"][
|
||||
"limit"]
|
||||
if current_quota_limit is None:
|
||||
print(
|
||||
f"Could not determine static routes metric for projects/{project_id} due to missing quotas"
|
||||
)
|
||||
continue
|
||||
# limit and utilization are calculted by project
|
||||
metric_labels = {"project": project_id}
|
||||
metrics.append_data_to_series_buffer(
|
||||
config, metrics_dict["metrics_per_network"]["static_routes_per_project"]
|
||||
["limit"]["name"], current_quota_limit, metric_labels,
|
||||
timestamp=timestamp)
|
||||
metrics.append_data_to_series_buffer(
|
||||
config, metrics_dict["metrics_per_network"]["static_routes_per_project"]
|
||||
["utilization"]["name"],
|
||||
project_usage[project_id] / current_quota_limit, metric_labels,
|
||||
timestamp=timestamp)
|
||||
|
||||
return
|
||||
|
|
|
|||
|
|
@ -84,7 +84,7 @@ def get_firewalls_data(config, metrics_dict, project_quotas_dict,
|
|||
current_quota_limit = project_quotas_dict[project_id]['global']["firewalls"]
|
||||
if current_quota_limit is None:
|
||||
print(
|
||||
f"Could not determine VPC firewal rules to metric for projects/{project_id} due to missing quotas"
|
||||
f"Could not determine VPC firewal rules metric for projects/{project_id} due to missing quotas"
|
||||
)
|
||||
continue
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"category": "CUSTOM",
|
||||
"displayName": "quotas_utilization_updated",
|
||||
"displayName": "quotas_utilization",
|
||||
"mosaicLayout": {
|
||||
"columns": 12,
|
||||
"tiles": [
|
||||
|
|
@ -22,13 +22,11 @@
|
|||
"timeSeriesFilter": {
|
||||
"aggregation": {
|
||||
"alignmentPeriod": "3600s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_NEXT_OLDER"
|
||||
},
|
||||
"filter": "metric.type=\"custom.googleapis.com/internal_forwarding_rules_l4_utilization\" resource.type=\"global\"",
|
||||
"secondaryAggregation": {
|
||||
"alignmentPeriod": "1800s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_MEAN"
|
||||
}
|
||||
}
|
||||
|
|
@ -64,13 +62,11 @@
|
|||
"timeSeriesFilter": {
|
||||
"aggregation": {
|
||||
"alignmentPeriod": "3600s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_NEXT_OLDER"
|
||||
},
|
||||
"filter": "metric.type=\"custom.googleapis.com/internal_forwarding_rules_l7_utilization\" resource.type=\"global\"",
|
||||
"secondaryAggregation": {
|
||||
"alignmentPeriod": "60s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_MEAN"
|
||||
}
|
||||
}
|
||||
|
|
@ -106,13 +102,11 @@
|
|||
"timeSeriesFilter": {
|
||||
"aggregation": {
|
||||
"alignmentPeriod": "3600s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_NEXT_OLDER"
|
||||
},
|
||||
"filter": "metric.type=\"custom.googleapis.com/number_of_instances_utilization\" resource.type=\"global\"",
|
||||
"secondaryAggregation": {
|
||||
"alignmentPeriod": "60s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_MEAN"
|
||||
}
|
||||
}
|
||||
|
|
@ -148,13 +142,11 @@
|
|||
"timeSeriesFilter": {
|
||||
"aggregation": {
|
||||
"alignmentPeriod": "3600s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_NEXT_OLDER"
|
||||
},
|
||||
"filter": "metric.type=\"custom.googleapis.com/number_of_vpc_peerings_utilization\" resource.type=\"global\"",
|
||||
"secondaryAggregation": {
|
||||
"alignmentPeriod": "60s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_MEAN"
|
||||
}
|
||||
}
|
||||
|
|
@ -190,13 +182,11 @@
|
|||
"timeSeriesFilter": {
|
||||
"aggregation": {
|
||||
"alignmentPeriod": "3600s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_NEXT_OLDER"
|
||||
},
|
||||
"filter": "metric.type=\"custom.googleapis.com/number_of_active_vpc_peerings_utilization\" resource.type=\"global\"",
|
||||
"secondaryAggregation": {
|
||||
"alignmentPeriod": "60s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_INTERPOLATE"
|
||||
}
|
||||
}
|
||||
|
|
@ -232,13 +222,11 @@
|
|||
"timeSeriesFilter": {
|
||||
"aggregation": {
|
||||
"alignmentPeriod": "3600s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_NEXT_OLDER"
|
||||
},
|
||||
"filter": "metric.type=\"custom.googleapis.com/number_of_subnet_IP_ranges_ppg_utilization\" resource.type=\"global\"",
|
||||
"secondaryAggregation": {
|
||||
"alignmentPeriod": "3600s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_MEAN"
|
||||
}
|
||||
}
|
||||
|
|
@ -274,13 +262,11 @@
|
|||
"timeSeriesFilter": {
|
||||
"aggregation": {
|
||||
"alignmentPeriod": "3600s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_NEXT_OLDER"
|
||||
},
|
||||
"filter": "metric.type=\"custom.googleapis.com/internal_forwarding_rules_l4_ppg_utilization\" resource.type=\"global\"",
|
||||
"secondaryAggregation": {
|
||||
"alignmentPeriod": "3600s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_MEAN"
|
||||
}
|
||||
}
|
||||
|
|
@ -316,13 +302,11 @@
|
|||
"timeSeriesFilter": {
|
||||
"aggregation": {
|
||||
"alignmentPeriod": "3600s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_NEXT_OLDER"
|
||||
},
|
||||
"filter": "metric.type=\"custom.googleapis.com/internal_forwarding_rules_l7_ppg_utilization\" resource.type=\"global\"",
|
||||
"secondaryAggregation": {
|
||||
"alignmentPeriod": "60s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_MEAN"
|
||||
}
|
||||
}
|
||||
|
|
@ -358,7 +342,6 @@
|
|||
"timeSeriesFilter": {
|
||||
"aggregation": {
|
||||
"alignmentPeriod": "3600s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_NEXT_OLDER"
|
||||
},
|
||||
"filter": "metric.type=\"custom.googleapis.com/number_of_instances_ppg_utilization\" resource.type=\"global\""
|
||||
|
|
@ -395,7 +378,6 @@
|
|||
"timeSeriesFilter": {
|
||||
"aggregation": {
|
||||
"alignmentPeriod": "60s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_MEAN"
|
||||
},
|
||||
"filter": "metric.type=\"custom.googleapis.com/dynamic_routes_per_network_utilization\" resource.type=\"global\""
|
||||
|
|
@ -452,7 +434,7 @@
|
|||
},
|
||||
"width": 6,
|
||||
"xPos": 0,
|
||||
"yPos": 24
|
||||
"yPos": 32
|
||||
},
|
||||
{
|
||||
"height": 4,
|
||||
|
|
@ -492,7 +474,7 @@
|
|||
},
|
||||
"width": 6,
|
||||
"xPos": 6,
|
||||
"yPos": 24
|
||||
"yPos": 32
|
||||
},
|
||||
{
|
||||
"height": 4,
|
||||
|
|
@ -512,7 +494,6 @@
|
|||
"timeSeriesFilter": {
|
||||
"aggregation": {
|
||||
"alignmentPeriod": "60s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_MEAN"
|
||||
},
|
||||
"filter": "metric.type=\"custom.googleapis.com/firewall_policy_tuples_per_policy_utilization\" resource.type=\"global\""
|
||||
|
|
@ -528,7 +509,7 @@
|
|||
}
|
||||
},
|
||||
"width": 6,
|
||||
"xPos": 0,
|
||||
"xPos": 6,
|
||||
"yPos": 28
|
||||
},
|
||||
{
|
||||
|
|
@ -549,7 +530,6 @@
|
|||
"timeSeriesFilter": {
|
||||
"aggregation": {
|
||||
"alignmentPeriod": "60s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_MEAN"
|
||||
},
|
||||
"filter": "metric.type=\"custom.googleapis.com/ip_addresses_per_subnet_utilization\" resource.type=\"global\""
|
||||
|
|
@ -586,7 +566,6 @@
|
|||
"timeSeriesFilter": {
|
||||
"aggregation": {
|
||||
"alignmentPeriod": "60s",
|
||||
"crossSeriesReducer": "REDUCE_NONE",
|
||||
"perSeriesAligner": "ALIGN_MEAN"
|
||||
},
|
||||
"filter": "metric.type=\"custom.googleapis.com/dynamic_routes_per_peering_group_utilization\" resource.type=\"global\""
|
||||
|
|
@ -604,6 +583,124 @@
|
|||
"width": 6,
|
||||
"xPos": 6,
|
||||
"yPos": 20
|
||||
},
|
||||
{
|
||||
"height": 4,
|
||||
"widget": {
|
||||
"title": "static_routes_per_project_vpc_usage",
|
||||
"xyChart": {
|
||||
"chartOptions": {
|
||||
"mode": "COLOR"
|
||||
},
|
||||
"dataSets": [
|
||||
{
|
||||
"minAlignmentPeriod": "60s",
|
||||
"plotType": "LINE",
|
||||
"targetAxis": "Y1",
|
||||
"timeSeriesQuery": {
|
||||
"apiSource": "DEFAULT_CLOUD",
|
||||
"timeSeriesFilter": {
|
||||
"aggregation": {
|
||||
"alignmentPeriod": "60s",
|
||||
"crossSeriesReducer": "REDUCE_SUM",
|
||||
"groupByFields": [
|
||||
"metric.label.\"project\""
|
||||
],
|
||||
"perSeriesAligner": "ALIGN_MEAN"
|
||||
},
|
||||
"filter": "metric.type=\"custom.googleapis.com/static_routes_per_project_vpc_usage\" resource.type=\"global\"",
|
||||
"secondaryAggregation": {
|
||||
"alignmentPeriod": "60s",
|
||||
"perSeriesAligner": "ALIGN_NONE"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"thresholds": [],
|
||||
"timeshiftDuration": "0s",
|
||||
"yAxis": {
|
||||
"label": "y1Axis",
|
||||
"scale": "LINEAR"
|
||||
}
|
||||
}
|
||||
},
|
||||
"width": 6,
|
||||
"xPos": 0,
|
||||
"yPos": 24
|
||||
},
|
||||
{
|
||||
"height": 4,
|
||||
"widget": {
|
||||
"title": "static_routes_per_ppg_utilization",
|
||||
"xyChart": {
|
||||
"chartOptions": {
|
||||
"mode": "COLOR"
|
||||
},
|
||||
"dataSets": [
|
||||
{
|
||||
"minAlignmentPeriod": "60s",
|
||||
"plotType": "LINE",
|
||||
"targetAxis": "Y1",
|
||||
"timeSeriesQuery": {
|
||||
"apiSource": "DEFAULT_CLOUD",
|
||||
"timeSeriesFilter": {
|
||||
"aggregation": {
|
||||
"alignmentPeriod": "60s",
|
||||
"perSeriesAligner": "ALIGN_MEAN"
|
||||
},
|
||||
"filter": "metric.type=\"custom.googleapis.com/static_routes_per_peering_group_utilization\" resource.type=\"global\""
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"thresholds": [],
|
||||
"timeshiftDuration": "0s",
|
||||
"yAxis": {
|
||||
"label": "y1Axis",
|
||||
"scale": "LINEAR"
|
||||
}
|
||||
}
|
||||
},
|
||||
"width": 6,
|
||||
"xPos": 0,
|
||||
"yPos": 28
|
||||
},
|
||||
{
|
||||
"height": 4,
|
||||
"widget": {
|
||||
"title": "static_routes_per_project_utilization",
|
||||
"xyChart": {
|
||||
"chartOptions": {
|
||||
"mode": "COLOR"
|
||||
},
|
||||
"dataSets": [
|
||||
{
|
||||
"minAlignmentPeriod": "60s",
|
||||
"plotType": "LINE",
|
||||
"targetAxis": "Y1",
|
||||
"timeSeriesQuery": {
|
||||
"apiSource": "DEFAULT_CLOUD",
|
||||
"timeSeriesFilter": {
|
||||
"aggregation": {
|
||||
"alignmentPeriod": "60s",
|
||||
"perSeriesAligner": "ALIGN_MEAN"
|
||||
},
|
||||
"filter": "metric.type=\"custom.googleapis.com/static_routes_per_project_utilization\" resource.type=\"global\""
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"timeshiftDuration": "0s",
|
||||
"yAxis": {
|
||||
"label": "y1Axis",
|
||||
"scale": "LINEAR"
|
||||
}
|
||||
}
|
||||
},
|
||||
"width": 6,
|
||||
"xPos": 6,
|
||||
"yPos": 24
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,115 @@
|
|||
# Configuring workload identity federation for Terraform Cloud/Enterprise workflow
|
||||
|
||||
The most common way to use Terraform Cloud for GCP deployments is to store a GCP Service Account Key as a part of TFE Workflow configuration, as we all know there are security risks due to the fact that keys are long term credentials that could be compromised.
|
||||
|
||||
Workload identity federation enables applications running outside of Google Cloud to replace long-lived service account keys with short-lived access tokens. This is achieved by configuring Google Cloud to trust an external identity provider, so applications can use the credentials issued by the external identity provider to impersonate a service account.
|
||||
|
||||
This blueprint shows how to set up [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) between [Terraform Cloud/Enterprise](https://developer.hashicorp.com/terraform/enterprise) instance and Google Cloud. This will be possible by configuring workload identity federation to trust oidc tokens generated for a specific workflow in a Terraform Enterprise organization.
|
||||
|
||||
The following diagram illustrates how the VM will get a short-lived access token and use it to access a resource:
|
||||
|
||||
|
||||
|
||||
## Running the blueprint
|
||||
|
||||
### Create Terraform Enterprise Workflow
|
||||
If you don't have an existing Terraform Enterprise organization you can sign up for a [free trial](https://app.terraform.io/public/signup/account) account.
|
||||
|
||||
Create a new Workspace for a `CLI-driven workflow` (Identity Federation will work for any workflow type, but for simplicity of the blueprint we use CLI driven workflow).
|
||||
|
||||
Note workspace name and id (id starts with `ws-`), we will use them on a later stage.
|
||||
|
||||
Go to the organization settings and note the org name and id (id starts with `org-`).
|
||||
|
||||
### Deploy GCP Workload Identity Pool Provider for Terraform Enterprise
|
||||
|
||||
> **_NOTE:_** This is a preparation part and should be executed on behalf of a user with enough permissions.
|
||||
|
||||
Required permissions when new project is created:
|
||||
- Project Creator on the parent folder/org.
|
||||
|
||||
Required permissions when an existing project is used:
|
||||
- Workload Identity Admin on the project level
|
||||
- Project IAM Admin on the project level
|
||||
|
||||
Fill out required variables, use TFE Org and Workspace IDs from the previous steps (IDs are not the names).
|
||||
```bash
|
||||
cd gcp-workload-identity-provider
|
||||
|
||||
mv terraform.auto.tfvars.template terraform.auto.tfvars
|
||||
|
||||
vi terraform.auto.tfvars
|
||||
```
|
||||
|
||||
Authenticate using application default credentials, execute terraform code and deploy resources
|
||||
```
|
||||
gcloud auth application-default login
|
||||
|
||||
terraform init
|
||||
|
||||
terraform apply
|
||||
```
|
||||
|
||||
As a result a set of outputs will be provided (your values will be different), note the output since we will use it on the next steps.
|
||||
|
||||
```
|
||||
impersonate_service_account_email = "sa-tfe@fe-test-oidc.iam.gserviceaccount.com"
|
||||
project_id = "tfe-test-oidc"
|
||||
workload_identity_audience = "//iam.googleapis.com/projects/476538149566/locations/global/workloadIdentityPools/tfe-pool/providers/tfe-provider"
|
||||
workload_identity_pool_provider_id = "projects/476538149566/locations/global/workloadIdentityPools/tfe-pool/providers/tfe-provider"
|
||||
```
|
||||
|
||||
### Configure OIDC provider for your TFE Workflow
|
||||
|
||||
To enable OIDC for a TFE workflow it's enough to setup an environment variable `TFC_WORKLOAD_IDENTITY_AUDIENCE`.
|
||||
|
||||
Go the the Workflow -> Variables and add a new variable `TFC_WORKLOAD_IDENTITY_AUDIENCE` equal to the value of `workload_identity_audience` output, in our example it's:
|
||||
|
||||
```
|
||||
TFC_WORKLOAD_IDENTITY_AUDIENCE = "//iam.googleapis.com/projects/476538149566/locations/global/workloadIdentityPools/tfe-pool/providers/tfe-provider"
|
||||
```
|
||||
|
||||
At that point we setup GCP Identity Federation to trust TFE generated OIDC tokens, so the TFE workflow can use the token to impersonate a GCP Service Account.
|
||||
|
||||
## Testing the blueprint
|
||||
|
||||
In order to test the setup we will deploy a GCS bucket from TFE Workflow using OIDC token for Service Account Impersonation.
|
||||
|
||||
### Configure backend and variables
|
||||
|
||||
First, we need to configure TFE Remote backend for our testing terraform code, use TFE Organization name and workspace name (names are not the same as ids)
|
||||
|
||||
```
|
||||
cd ../tfc-workflow-using-wif
|
||||
|
||||
mv backend.tf.template backend.tf
|
||||
|
||||
|
||||
vi backend.tf
|
||||
|
||||
```
|
||||
|
||||
Fill out variables based on the output from the preparation steps:
|
||||
|
||||
```
|
||||
mv terraform.auto.tfvars.template terraform.auto.tfvars
|
||||
|
||||
vi terraform.auto.tfvars
|
||||
|
||||
```
|
||||
|
||||
### Authenticate terraform for triggering CLI-driven workflow
|
||||
|
||||
Follow this [documentation](https://learn.hashicorp.com/tutorials/terraform/cloud-login) to login ti terraform cloud from the CLI.
|
||||
|
||||
### Trigger the workflow
|
||||
|
||||
```
|
||||
terraform init
|
||||
|
||||
terraform apply
|
||||
```
|
||||
|
||||
As a result we have a successfully deployed GCS bucket from Terraform Enterprise workflow using Workload Identity Federation.
|
||||
|
||||
Once done testing, you can clean up resources by running `terraform destroy` first in the `tfc-workflow-using-wif` and then `gcp-workload-identity-provider` folders.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 28 KiB |
|
|
@ -0,0 +1,33 @@
|
|||
# GCP Workload Identity Provider for Terraform Enterprise
|
||||
|
||||
This terraform code is a part of [GCP Workload Identity Federation for Terraform Enterprise](../) blueprint.
|
||||
|
||||
The codebase provisions the following list of resources:
|
||||
|
||||
- GCS Bucket
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| [billing_account](variables.tf#L16) | Billing account id used as default for new projects. | <code>string</code> | ✓ | |
|
||||
| [project_id](variables.tf#L38) | Existing project id. | <code>string</code> | ✓ | |
|
||||
| [tfe_organization_id](variables.tf#L43) | | <code></code> | ✓ | |
|
||||
| [tfe_workspace_id](variables.tf#L48) | | <code></code> | ✓ | |
|
||||
| [issuer_uri](variables.tf#L65) | Terraform Enterprise uri. Replace the uri if a self hosted instance is used. | <code>string</code> | | <code>"https://app.terraform.io/"</code> |
|
||||
| [parent](variables.tf#L27) | Parent folder or organization in 'folders/folder_id' or 'organizations/org_id' format. | <code>string</code> | | <code>null</code> |
|
||||
| [project_create](variables.tf#L21) | Create project instead of using an existing one. | <code>bool</code> | | <code>true</code> |
|
||||
| [workload_identity_pool_id](variables.tf#L53) | Workload identity pool id. | <code>string</code> | | <code>"tfe-pool"</code> |
|
||||
| [workload_identity_pool_provider_id](variables.tf#L59) | Workload identity pool provider id. | <code>string</code> | | <code>"tfe-provider"</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
| name | description | sensitive |
|
||||
|---|---|:---:|
|
||||
| [impersonate_service_account_email](outputs.tf#L31) | | |
|
||||
| [project_id](outputs.tf#L16) | | |
|
||||
| [workload_identity_audience](outputs.tf#L26) | | |
|
||||
| [workload_identity_pool_provider_id](outputs.tf#L21) | GCP workload identity pool provider ID. | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
|
@ -0,0 +1,91 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
###############################################################################
|
||||
# GCP PROJECT #
|
||||
###############################################################################
|
||||
|
||||
module "project" {
|
||||
source = "../../../../modules/project"
|
||||
name = var.project_id
|
||||
project_create = var.project_create
|
||||
parent = var.parent
|
||||
billing_account = var.billing_account
|
||||
services = [
|
||||
"iam.googleapis.com",
|
||||
"cloudresourcemanager.googleapis.com",
|
||||
"iamcredentials.googleapis.com",
|
||||
"sts.googleapis.com",
|
||||
"storage.googleapis.com"
|
||||
]
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Workload Identity Pool and Provider #
|
||||
###############################################################################
|
||||
|
||||
resource "google_iam_workload_identity_pool" "tfe-pool" {
|
||||
project = module.project.project_id
|
||||
workload_identity_pool_id = var.workload_identity_pool_id
|
||||
display_name = "TFE Pool"
|
||||
description = "Identity pool for Terraform Enterprise OIDC integration"
|
||||
}
|
||||
|
||||
resource "google_iam_workload_identity_pool_provider" "tfe-pool-provider" {
|
||||
project = module.project.project_id
|
||||
workload_identity_pool_id = google_iam_workload_identity_pool.tfe-pool.workload_identity_pool_id
|
||||
workload_identity_pool_provider_id = var.workload_identity_pool_provider_id
|
||||
display_name = "TFE Pool Provider"
|
||||
description = "OIDC identity pool provider for TFE Integration"
|
||||
# Use condition to make sure only token generated for a specific TFE Org can be used across org workspaces
|
||||
attribute_condition = "attribute.terraform_organization_id == \"${var.tfe_organization_id}\""
|
||||
attribute_mapping = {
|
||||
"google.subject" = "assertion.sub"
|
||||
"attribute.aud" = "assertion.aud"
|
||||
"attribute.terraform_run_phase" = "assertion.terraform_run_phase"
|
||||
"attribute.terraform_workspace_id" = "assertion.terraform_workspace_id"
|
||||
"attribute.terraform_workspace_name" = "assertion.terraform_workspace_name"
|
||||
"attribute.terraform_organization_id" = "assertion.terraform_organization_id"
|
||||
"attribute.terraform_organization_name" = "assertion.terraform_organization_name"
|
||||
"attribute.terraform_run_id" = "assertion.terraform_run_id"
|
||||
"attribute.terraform_full_workspace" = "assertion.terraform_full_workspace"
|
||||
}
|
||||
oidc {
|
||||
# Should be different if self hosted TFE instance is used
|
||||
issuer_uri = var.issuer_uri
|
||||
}
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Service Account and IAM bindings #
|
||||
###############################################################################
|
||||
|
||||
module "sa-tfe" {
|
||||
source = "../../../../modules/iam-service-account"
|
||||
project_id = module.project.project_id
|
||||
name = "sa-tfe"
|
||||
|
||||
iam = {
|
||||
# We allow only tokens generated by a specific TFE workspace impersonation of the service account,
|
||||
# that way one identity pool can be used for a TFE Organization, but every workspace will be able to impersonate only a specifc SA
|
||||
"roles/iam.workloadIdentityUser" = ["principalSet://iam.googleapis.com/${google_iam_workload_identity_pool.tfe-pool.name}/attribute.terraform_workspace_id/${var.tfe_workspace_id}"]
|
||||
}
|
||||
|
||||
iam_project_roles = {
|
||||
"${module.project.project_id}" = [
|
||||
"roles/storage.admin"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
output "project_id" {
|
||||
description = "GCP Project ID."
|
||||
value = module.project.project_id
|
||||
}
|
||||
|
||||
output "workload_identity_pool_provider_id" {
|
||||
description = "GCP workload identity pool provider ID."
|
||||
value = google_iam_workload_identity_pool_provider.tfe-pool-provider.name
|
||||
}
|
||||
|
||||
output "workload_identity_audience" {
|
||||
description = "TFC Workload Identity Audience."
|
||||
value = "//iam.googleapis.com/${google_iam_workload_identity_pool_provider.tfe-pool-provider.name}"
|
||||
}
|
||||
|
||||
output "impersonate_service_account_email" {
|
||||
description = "Service account to be impersonated by workload identity."
|
||||
value = module.sa-tfe.email
|
||||
}
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
parent = "folders/437102807785"
|
||||
project_id = "my-project-id"
|
||||
tfe_organization_id = "org-W3bz9neazHrZz99U"
|
||||
tfe_workspace_id = "ws-DFxEE3NmeMdaAvoK"
|
||||
billing_account = "015617-1B8CBC-AF10D9"
|
||||
|
|
@ -0,0 +1,69 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
variable "billing_account" {
|
||||
description = "Billing account id used as default for new projects."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "project_create" {
|
||||
description = "Create project instead of using an existing one."
|
||||
type = bool
|
||||
default = true
|
||||
}
|
||||
|
||||
variable "parent" {
|
||||
description = "Parent folder or organization in 'folders/folder_id' or 'organizations/org_id' format."
|
||||
type = string
|
||||
default = null
|
||||
validation {
|
||||
condition = var.parent == null || can(regex("(organizations|folders)/[0-9]+", var.parent))
|
||||
error_message = "Parent must be of the form folders/folder_id or organizations/organization_id."
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
variable "project_id" {
|
||||
description = "Existing project id."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "tfe_organization_id" {
|
||||
description = "TFE organization id."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "tfe_workspace_id" {
|
||||
description = "TFE workspace id."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "workload_identity_pool_id" {
|
||||
description = "Workload identity pool id."
|
||||
type = string
|
||||
default = "tfe-pool"
|
||||
}
|
||||
|
||||
variable "workload_identity_pool_provider_id" {
|
||||
description = "Workload identity pool provider id."
|
||||
type = string
|
||||
default = "tfe-provider"
|
||||
}
|
||||
|
||||
variable "issuer_uri" {
|
||||
description = "Terraform Enterprise uri. Replace the uri if a self hosted instance is used."
|
||||
type = string
|
||||
default = "https://app.terraform.io/"
|
||||
}
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
# GCP Workload Identity Provider for Terraform Enterprise
|
||||
|
||||
This terraform code is a part of [GCP Workload Identity Federation for Terraform Enterprise](../) blueprint. For instructions please refer to the blueprint [readme](../README.md).
|
||||
|
||||
The codebase provisions the following list of resources:
|
||||
|
||||
- GCS Bucket
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| [impersonate_service_account_email](variables.tf#L26) | | <code></code> | ✓ | |
|
||||
| [project_id](variables.tf#L16) | | <code></code> | ✓ | |
|
||||
| [workload_identity_pool_provider_id](variables.tf#L21) | GCP workload identity pool provider ID. | <code>string</code> | ✓ | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
# The block below configures Terraform to use the 'remote' backend with Terraform Cloud.
|
||||
# For more information, see https://www.terraform.io/docs/backends/types/remote.html
|
||||
|
||||
terraform {
|
||||
backend "remote" {
|
||||
organization = "<TFE-ORG-NAME>"
|
||||
|
||||
workspaces {
|
||||
name = "<TFE-WORKSPACE-NAME>"
|
||||
}
|
||||
}
|
||||
|
||||
required_version = ">= 0.14.0"
|
||||
}
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
###############################################################################
|
||||
# TEST RESOURCE TO VALIDATE WIF #
|
||||
###############################################################################
|
||||
|
||||
resource "google_storage_bucket" "test-bucket" {
|
||||
project = var.project_id
|
||||
name = "${var.project_id}-tfe-oidc-test-bucket"
|
||||
location = "US"
|
||||
force_destroy = true
|
||||
}
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
module "tfe_oidc" {
|
||||
source = "./tfc-oidc"
|
||||
|
||||
workload_identity_pool_provider_id = var.workload_identity_pool_provider_id
|
||||
impersonate_service_account_email = var.impersonate_service_account_email
|
||||
}
|
||||
|
||||
provider "google" {
|
||||
credentials = module.tfe_oidc.credentials
|
||||
}
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
project_id = "tfe-oidc-workflow"
|
||||
workload_identity_pool_provider_id = "projects/683987109094/locations/global/workloadIdentityPools/tfe-pool/providers/tfe-provider"
|
||||
impersonate_service_account_email = "sa-tfe@tfe-oidc-workflow2.iam.gserviceaccount.com"
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
# Terraform Enterprise OIDC Credential for GCP Workload Identity Federation
|
||||
|
||||
This is a helper module to prepare GCP Credentials from Terraform Enterprise workload identity token. For more information see [Terraform Enterprise Workload Identity Federation](../) blueprint.
|
||||
|
||||
## Example
|
||||
```hcl
|
||||
module "tfe_oidc" {
|
||||
source = "./tfe_oidc"
|
||||
|
||||
workload_identity_pool_provider_id = "projects/683987109094/locations/global/workloadIdentityPools/tfe-pool/providers/tfe-provider"
|
||||
impersonate_service_account_email = "tfe-test@tfe-test-wif.iam.gserviceaccount.com"
|
||||
}
|
||||
|
||||
provider "google" {
|
||||
credentials = module.tfe_oidc.credentials
|
||||
}
|
||||
|
||||
provider "google-beta" {
|
||||
credentials = module.tfe_oidc.credentials
|
||||
}
|
||||
|
||||
# tftest skip
|
||||
```
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| [impersonate_service_account_email](variables.tf#L22) | Service account to be impersonated by workload identity federation. | <code>string</code> | ✓ | |
|
||||
| [workload_identity_pool_provider_id](variables.tf#L17) | GCP workload identity pool provider ID. | <code>string</code> | ✓ | |
|
||||
| [tmp_oidc_token_path](variables.tf#L27) | Name of the temporary file where TFC OIDC token will be stored to authentificate terraform provider google. | <code>string</code> | | <code>".oidc_token"</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
| name | description | sensitive |
|
||||
|---|---|:---:|
|
||||
| [credentials](outputs.tf#L17) | | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
locals {
|
||||
audience = "//iam.googleapis.com/${var.workload_identity_pool_provider_id}"
|
||||
}
|
||||
|
||||
data "external" "oidc_token_file" {
|
||||
program = ["bash", "${path.module}/write_token.sh", "${var.tmp_oidc_token_path}"]
|
||||
}
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
output "credentials" {
|
||||
value = jsonencode({
|
||||
"type" : "external_account",
|
||||
"audience" : "${local.audience}",
|
||||
"subject_token_type" : "urn:ietf:params:oauth:token-type:jwt",
|
||||
"token_url" : "https://sts.googleapis.com/v1/token",
|
||||
"credential_source" : data.external.oidc_token_file.result
|
||||
"service_account_impersonation_url" : "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/${var.impersonate_service_account_email}:generateAccessToken"
|
||||
})
|
||||
}
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
variable "workload_identity_pool_provider_id" {
|
||||
description = "GCP workload identity pool provider ID."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "impersonate_service_account_email" {
|
||||
description = "Service account to be impersonated by workload identity federation."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "tmp_oidc_token_path" {
|
||||
description = "Name of the temporary file where TFC OIDC token will be stored to authentificate terraform provider google."
|
||||
type = string
|
||||
default = ".oidc_token"
|
||||
}
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
terraform {
|
||||
required_version = ">= 1.3.1"
|
||||
}
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Exit if any of the intermediate steps fail
|
||||
set -e
|
||||
|
||||
FILENAME=$@
|
||||
|
||||
echo $TFC_WORKLOAD_IDENTITY_TOKEN > $FILENAME
|
||||
|
||||
echo -n "{\"file\":\"${FILENAME}\"}"
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
variable "project_id" {
|
||||
description = "GCP project ID."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "workload_identity_pool_provider_id" {
|
||||
description = "GCP workload identity pool provider ID."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "impersonate_service_account_email" {
|
||||
description = "Service account to be impersonated by workload identity."
|
||||
type = string
|
||||
}
|
||||
|
|
@ -13,7 +13,7 @@ They are meant to be used as minimal but complete starting points to create actu
|
|||
|
||||
### Cloud Storage to Bigquery with Cloud Dataflow with least privileges
|
||||
|
||||
<a href="./gcs-to-bq-with-least-privileges/" title="Cloud Storage to Bigquery with Cloud Dataflow with least privileges"><img src="./gcs-to-bq-with-least-privileges/diagram.png" align="left" width="280px"></a> This [blueprint](./gcs-to-bq-with-least-privileges/) implements resources required to run GCS to BigQuery Dataflow pipelines. The solution rely on a set of Services account created with the least privileges principle.
|
||||
<a href="./gcs-to-bq-with-least-privileges/" title="Cloud Storage to Bigquery with Cloud Dataflow with least privileges"><img src="./gcs-to-bq-with-least-privileges/images/diagram.png" align="left" width="280px"></a> This [blueprint](./gcs-to-bq-with-least-privileges/) implements resources required to run GCS to BigQuery Dataflow pipelines. The solution rely on a set of Services account created with the least privileges principle.
|
||||
<br clear="left">
|
||||
|
||||
### Data Platform Foundations
|
||||
|
|
|
|||
|
|
@ -50,7 +50,7 @@ resource "google_sql_user" "service-account" {
|
|||
for_each = toset(var.data_eng_principals)
|
||||
project = module.project.project_id
|
||||
# Omit the .gserviceaccount.com suffix in the email
|
||||
name = regex("(.+)(gserviceaccount)", module.service-account-sql.email)[0]
|
||||
name = regex("(.+)(.gserviceaccount)", module.service-account-sql.email)[0]
|
||||
instance = module.db.name
|
||||
type = "CLOUD_IAM_SERVICE_ACCOUNT"
|
||||
}
|
||||
|
|
|
|||
|
|
@ -38,8 +38,8 @@ output "demo_commands" {
|
|||
description = "Demo commands."
|
||||
value = {
|
||||
"01_ssh" = "gcloud compute ssh ${module.test-vm.instance.name} --project ${module.project.name} --zone ${var.regions.primary}-b"
|
||||
"02_cloud_sql_proxy" = "cloud_sql_proxy -instances=${module.db.connection_name}=tcp:5432 &"
|
||||
"03_psql" = "psql 'host=127.0.0.1 port=5432 sslmode=disable dbname=${var.postgres_database} user=postgres'"
|
||||
"02_cloud_sql_proxy" = "cloud_sql_proxy -enable_iam_login -instances=${module.db.connection_name}=tcp:5432 &"
|
||||
"03_psql" = "psql 'host=127.0.0.1 port=5432 sslmode=disable dbname=${var.postgres_database} user=postgres password=PASSWORD'"
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@ Whether you’re transferring from another Cloud Service Provider or you’re ta
|
|||
|
||||
## Architecture
|
||||
|
||||
|
||||
|
||||
|
||||
The main components that we would be setting up are (to learn more about these products, click on the hyperlinks):
|
||||
|
||||
|
|
@ -61,11 +61,11 @@ __Note__: To grant a user a role, take a look at the [Granting and Revoking Acce
|
|||
Click on the button below, sign in if required and when the prompt appears, click on “confirm”.
|
||||
|
||||
|
||||
[](https://goo.gle/GoDataPipe)
|
||||
[](https://goo.gle/GoDataPipe)
|
||||
|
||||
This will clone the repository to your cloud shell and a screen like this one will appear:
|
||||
|
||||
|
||||
|
||||
|
||||
Before you deploy the architecture, make sure you run the following command to move your cloudshell session into your service project:
|
||||
|
||||
|
|
@ -87,7 +87,7 @@ Before we deploy the architecture, you will need the following information:
|
|||
|
||||
2. In the editor, edit the terraform.tfvars.sample file with the variables you gathered in the step above.
|
||||
|
||||
|
||||
|
||||
|
||||
* a. Fill in __data_eng_principals__ with the list of Users or Groups to impersonate service accounts.
|
||||
|
||||
|
|
@ -105,7 +105,7 @@ Before we deploy the architecture, you will need the following information:
|
|||
|
||||
The resource creation will take a few minutes, at the end this is the output you should expect for successful completion along with a list of the created resources:
|
||||
|
||||
|
||||
|
||||
|
||||
__Congratulations!__ You have successfully deployed the foundation for running your first ETL pipeline on Google Cloud.
|
||||
|
||||
|
|
@ -168,16 +168,16 @@ This command will start a dataflow job called test_batch_01 that uses a Dataflow
|
|||
|
||||
The expected output is the following:
|
||||
|
||||
|
||||
|
||||
|
||||
Then, if you navigate to Dataflow on the console, you will see the following:
|
||||
|
||||
|
||||
|
||||
|
||||
This shows the job you started from the cloudshell is currently running in Dataflow.
|
||||
If you click on the job name, you can see the job graph created and how every step of the Dataflow pipeline is moving along:
|
||||
|
||||
|
||||
|
||||
|
||||
Once the job completes, you can navigate to BigQuery in the console and under __SERVICE_PROJECT_ID__ → datalake → person, you can see the data that was successfully imported into BigQuery through the Dataflow job.
|
||||
|
||||
|
|
|
|||
|
Before Width: | Height: | Size: 144 KiB After Width: | Height: | Size: 144 KiB |
|
Before Width: | Height: | Size: 32 KiB After Width: | Height: | Size: 32 KiB |
|
Before Width: | Height: | Size: 72 KiB After Width: | Height: | Size: 72 KiB |
|
Before Width: | Height: | Size: 39 KiB After Width: | Height: | Size: 39 KiB |
|
Before Width: | Height: | Size: 68 KiB After Width: | Height: | Size: 68 KiB |
|
Before Width: | Height: | Size: 19 KiB After Width: | Height: | Size: 19 KiB |
|
Before Width: | Height: | Size: 67 KiB After Width: | Height: | Size: 67 KiB |
|
Before Width: | Height: | Size: 10 KiB After Width: | Height: | Size: 10 KiB |
|
|
@ -47,7 +47,7 @@ output "command_01_gcs" {
|
|||
|
||||
output "command_02_dataflow" {
|
||||
description = "Command to run Dataflow template impersonating the service account."
|
||||
value = templatefile("${path.module}/dataflow.tftpl", {
|
||||
value = templatefile("${path.module}/templates/dataflow.tftpl", {
|
||||
sa_orch_email = module.service-account-orch.email
|
||||
project_id = module.project.project_id
|
||||
region = var.region
|
||||
|
|
@ -68,7 +68,7 @@ output "command_02_dataflow" {
|
|||
|
||||
output "command_03_bq" {
|
||||
description = "BigQuery command to query imported data."
|
||||
value = templatefile("${path.module}/bigquery.tftpl", {
|
||||
value = templatefile("${path.module}/templates/bigquery.tftpl", {
|
||||
project_id = module.project.project_id
|
||||
bigquery_dataset = module.bigquery-dataset.dataset_id
|
||||
bigquery_table = module.bigquery-dataset.tables["person"].table_id
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -84,7 +84,7 @@ module "dns-api-prod" {
|
|||
domain = "googleapis.com."
|
||||
client_networks = [module.vpc-prod.self_link]
|
||||
recordsets = {
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
"CNAME *" = { records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -96,7 +96,7 @@ module "dns-api-dev" {
|
|||
domain = "googleapis.com."
|
||||
client_networks = [module.vpc-dev.self_link]
|
||||
recordsets = {
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
"CNAME *" = { records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -63,7 +63,7 @@ module "dev-dns-zone" {
|
|||
domain = "dev.example.com."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A localhost" = { ttl = 300, records = ["127.0.0.1"] }
|
||||
"A test-r2" = { ttl = 300, records = [module.dev-r2-vm.internal_ip] }
|
||||
"A localhost" = { records = ["127.0.0.1"] }
|
||||
"A test-r2" = { records = [module.dev-r2-vm.internal_ip] }
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -53,7 +53,7 @@ module "landing-dns-zone" {
|
|||
domain = "example.com."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A localhost" = { ttl = 300, records = ["127.0.0.1"] }
|
||||
"A test-r1" = { ttl = 300, records = [module.landing-r1-vm.internal_ip] }
|
||||
"A localhost" = { records = ["127.0.0.1"] }
|
||||
"A test-r1" = { records = [module.landing-r1-vm.internal_ip] }
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -63,7 +63,7 @@ module "prod-dns-zone" {
|
|||
domain = "prd.example.com."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A localhost" = { ttl = 300, records = ["127.0.0.1"] }
|
||||
"A test-r1" = { ttl = 300, records = [module.prod-r1-vm.internal_ip] }
|
||||
"A localhost" = { records = ["127.0.0.1"] }
|
||||
"A test-r1" = { records = [module.prod-r1-vm.internal_ip] }
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -169,9 +169,9 @@ module "dns-gcp" {
|
|||
domain = "gcp.example.org."
|
||||
client_networks = [module.vpc.self_link]
|
||||
recordsets = {
|
||||
"A localhost" = { ttl = 300, records = ["127.0.0.1"] }
|
||||
"A test-1" = { ttl = 300, records = [module.vm-test1.internal_ip] }
|
||||
"A test-2" = { ttl = 300, records = [module.vm-test2.internal_ip] }
|
||||
"A localhost" = { records = ["127.0.0.1"] }
|
||||
"A test-1" = { records = [module.vm-test1.internal_ip] }
|
||||
"A test-2" = { records = [module.vm-test2.internal_ip] }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -183,9 +183,9 @@ module "dns-api" {
|
|||
domain = "googleapis.com."
|
||||
client_networks = [module.vpc.self_link]
|
||||
recordsets = {
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
"A private" = { ttl = 300, records = local.vips.private }
|
||||
"A restricted" = { ttl = 300, records = local.vips.restricted }
|
||||
"CNAME *" = { records = ["private.googleapis.com."] }
|
||||
"A private" = { records = local.vips.private }
|
||||
"A restricted" = { records = local.vips.restricted }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -218,7 +218,7 @@ module "private-dns-onprem" {
|
|||
domain = "${var.region}-${module.project.project_id}.cloudfunctions.net."
|
||||
client_networks = [module.vpc-onprem.self_link]
|
||||
recordsets = {
|
||||
"A " = { ttl = 300, records = [module.addresses.psc_addresses[local.psc_name].address] }
|
||||
"A " = { records = [module.addresses.psc_addresses[local.psc_name].address] }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -157,8 +157,8 @@ module "host-dns" {
|
|||
domain = "example.com."
|
||||
client_networks = [module.vpc-shared.self_link]
|
||||
recordsets = {
|
||||
"A localhost" = { ttl = 300, records = ["127.0.0.1"] }
|
||||
"A bastion" = { ttl = 300, records = [module.vm-bastion.internal_ip] }
|
||||
"A localhost" = { records = ["127.0.0.1"] }
|
||||
"A bastion" = { records = [module.vm-bastion.internal_ip] }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0"
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0"
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -172,6 +172,13 @@ DNS configuration is further centralized by leveraging peering zones, so that
|
|||
|
||||
- the hub/landing Cloud DNS hosts configurations for on-prem forwarding, Google API domains, and the top-level private zone/s (e.g. gcp.example.com)
|
||||
- the spokes Cloud DNS host configurations for the environment-specific domains (e.g. prod.gcp.example.com), which are bound to the hub/landing leveraging [cross-project binding](https://cloud.google.com/dns/docs/zones/zones-overview#cross-project_binding); a peering zone for the `.` (root) zone is then created on each spoke, delegating all DNS resolution to hub/landing.
|
||||
- Private Google Access is enabled for a selection of the [supported domains](https://cloud.google.com/vpc/docs/configure-private-google-access#domain-options), namely
|
||||
- `private.googleapis.com`
|
||||
- `restricted.googleapis.com`
|
||||
- `gcr.io`
|
||||
- `packages.cloud.google.com`
|
||||
- `pkg.dev`
|
||||
- `pki.goog`
|
||||
|
||||
To complete the configuration, the 35.199.192.0/19 range should be routed to the VPN tunnels from on-premises, and the following names should be configured for DNS forwarding to cloud:
|
||||
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ module "dev-dns-private-zone" {
|
|||
domain = "dev.gcp.example.com."
|
||||
client_networks = [module.landing-trusted-vpc.self_link, module.landing-untrusted-vpc.self_link]
|
||||
recordsets = {
|
||||
"A localhost" = { type = "A", ttl = 300, records = ["127.0.0.1"] }
|
||||
"A localhost" = { records = ["127.0.0.1"] }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -55,11 +55,11 @@ module "gcp-example-dns-private-zone" {
|
|||
module.landing-trusted-vpc.self_link
|
||||
]
|
||||
recordsets = {
|
||||
"A localhost" = { type = "A", ttl = 300, records = ["127.0.0.1"] }
|
||||
"A localhost" = { records = ["127.0.0.1"] }
|
||||
}
|
||||
}
|
||||
|
||||
# Google API zone to trigger Private Access
|
||||
# Google APIs
|
||||
|
||||
module "googleapis-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
|
|
@ -72,12 +72,84 @@ module "googleapis-private-zone" {
|
|||
module.landing-trusted-vpc.self_link
|
||||
]
|
||||
recordsets = {
|
||||
"A private" = { type = "A", ttl = 300, records = [
|
||||
"A private" = { records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"A restricted" = { type = "A", ttl = 300, records = [
|
||||
"A restricted" = { records = [
|
||||
"199.36.153.4", "199.36.153.5", "199.36.153.6", "199.36.153.7"
|
||||
] }
|
||||
"CNAME *" = { type = "CNAME", ttl = 300, records = ["private.googleapis.com."] }
|
||||
"CNAME *" = { records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "gcrio-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.landing-project.project_id
|
||||
type = "private"
|
||||
name = "gcr-io"
|
||||
domain = "gcr.io."
|
||||
client_networks = [
|
||||
module.landing-untrusted-vpc.self_link,
|
||||
module.landing-trusted-vpc.self_link
|
||||
]
|
||||
recordsets = {
|
||||
"A gcr.io." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "packages-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.landing-project.project_id
|
||||
type = "private"
|
||||
name = "packages-cloud"
|
||||
domain = "packages.cloud.google.com."
|
||||
client_networks = [
|
||||
module.landing-untrusted-vpc.self_link,
|
||||
module.landing-trusted-vpc.self_link
|
||||
]
|
||||
recordsets = {
|
||||
"A packages.cloud.google.com." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "pkgdev-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.landing-project.project_id
|
||||
type = "private"
|
||||
name = "pkg-dev"
|
||||
domain = "pkg.dev."
|
||||
client_networks = [
|
||||
module.landing-untrusted-vpc.self_link,
|
||||
module.landing-trusted-vpc.self_link
|
||||
]
|
||||
recordsets = {
|
||||
"A pkg.dev." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "pkigoog-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.landing-project.project_id
|
||||
type = "private"
|
||||
name = "pki-goog"
|
||||
domain = "pki.goog."
|
||||
client_networks = [
|
||||
module.landing-untrusted-vpc.self_link,
|
||||
module.landing-trusted-vpc.self_link
|
||||
]
|
||||
recordsets = {
|
||||
"A pki.goog." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ module "prod-dns-private-zone" {
|
|||
domain = "prod.gcp.example.com."
|
||||
client_networks = [module.landing-trusted-vpc.self_link, module.landing-untrusted-vpc.self_link]
|
||||
recordsets = {
|
||||
"A localhost" = { type = "A", ttl = 300, records = ["127.0.0.1"] }
|
||||
"A localhost" = { records = ["127.0.0.1"] }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -102,6 +102,13 @@ DNS configuration is further centralized by leveraging peering zones, so that
|
|||
|
||||
- the hub/landing Cloud DNS hosts configurations for on-prem forwarding, Google API domains, and the top-level private zone/s (e.g. gcp.example.com)
|
||||
- the spokes Cloud DNS host configurations for the environment-specific domains (e.g. prod.gcp.example.com), which are bound to the hub/landing leveraging [cross-project binding](https://cloud.google.com/dns/docs/zones/zones-overview#cross-project_binding); a peering zone for the `.` (root) zone is then created on each spoke, delegating all DNS resolution to hub/landing.
|
||||
- Private Google Access is enabled for a selection of the [supported domains](https://cloud.google.com/vpc/docs/configure-private-google-access#domain-options), namely
|
||||
- `private.googleapis.com`
|
||||
- `restricted.googleapis.com`
|
||||
- `gcr.io`
|
||||
- `packages.cloud.google.com`
|
||||
- `pkg.dev`
|
||||
- `pki.goog`
|
||||
|
||||
To complete the configuration, the 35.199.192.0/19 range should be routed on the VPN tunnels from on-prem, and the following names configured for DNS forwarding to cloud:
|
||||
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ module "dev-dns-private-zone" {
|
|||
domain = "dev.gcp.example.com."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A localhost" = { type = "A", ttl = 300, records = ["127.0.0.1"] }
|
||||
"A localhost" = { records = ["127.0.0.1"] }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -46,11 +46,11 @@ module "gcp-example-dns-private-zone" {
|
|||
domain = "gcp.example.com."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A localhost" = { type = "A", ttl = 300, records = ["127.0.0.1"] }
|
||||
"A localhost" = { records = ["127.0.0.1"] }
|
||||
}
|
||||
}
|
||||
|
||||
# Google API zone to trigger Private Access
|
||||
# Google APIs
|
||||
|
||||
module "googleapis-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
|
|
@ -60,12 +60,72 @@ module "googleapis-private-zone" {
|
|||
domain = "googleapis.com."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A private" = { type = "A", ttl = 300, records = [
|
||||
"A private" = { records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"A restricted" = { type = "A", ttl = 300, records = [
|
||||
"A restricted" = { records = [
|
||||
"199.36.153.4", "199.36.153.5", "199.36.153.6", "199.36.153.7"
|
||||
] }
|
||||
"CNAME *" = { type = "CNAME", ttl = 300, records = ["private.googleapis.com."] }
|
||||
"CNAME *" = { records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "gcrio-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.landing-project.project_id
|
||||
type = "private"
|
||||
name = "gcr-io"
|
||||
domain = "gcr.io."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A gcr.io." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "packages-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.landing-project.project_id
|
||||
type = "private"
|
||||
name = "packages-cloud"
|
||||
domain = "packages.cloud.google.com."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A packages.cloud.google.com." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "pkgdev-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.landing-project.project_id
|
||||
type = "private"
|
||||
name = "pkg-dev"
|
||||
domain = "pkg.dev."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A pkg.dev." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "pkigoog-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.landing-project.project_id
|
||||
type = "private"
|
||||
name = "pki-goog"
|
||||
domain = "pki.goog."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A pki.goog." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ module "prod-dns-private-zone" {
|
|||
domain = "prod.gcp.example.com."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A localhost" = { type = "A", ttl = 300, records = ["127.0.0.1"] }
|
||||
"A localhost" = { records = ["127.0.0.1"] }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -69,6 +69,13 @@ DNS often goes hand in hand with networking, especially on GCP where Cloud DNS z
|
|||
|
||||
- on-prem to cloud via private zones for cloud-managed domains, and an [inbound policy](https://cloud.google.com/dns/docs/server-policies-overview#dns-server-policy-in) used as forwarding target or via delegation (requires some extra configuration) from on-prem DNS resolvers
|
||||
- cloud to on-prem via forwarding zones for the on-prem managed domains
|
||||
- Private Google Access is enabled for a selection of the [supported domains](https://cloud.google.com/vpc/docs/configure-private-google-access#domain-options), namely
|
||||
- `private.googleapis.com`
|
||||
- `restricted.googleapis.com`
|
||||
- `gcr.io`
|
||||
- `packages.cloud.google.com`
|
||||
- `pkg.dev`
|
||||
- `pki.goog`
|
||||
|
||||
To complete the configuration, the 35.199.192.0/19 range should be routed on the VPN tunnels from on-prem, and the following names configured for DNS forwarding to cloud:
|
||||
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ module "dev-dns-private-zone" {
|
|||
domain = "dev.gcp.example.com."
|
||||
client_networks = [module.dev-spoke-vpc.self_link]
|
||||
recordsets = {
|
||||
"A localhost" = { type = "A", ttl = 300, records = ["127.0.0.1"] }
|
||||
"A localhost" = { records = ["127.0.0.1"] }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -50,6 +50,8 @@ module "dev-reverse-10-dns-forwarding" {
|
|||
forwarders = { for ip in var.dns.dev : ip => null }
|
||||
}
|
||||
|
||||
# Google APIs
|
||||
|
||||
module "dev-googleapis-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.dev-spoke-project.project_id
|
||||
|
|
@ -58,12 +60,72 @@ module "dev-googleapis-private-zone" {
|
|||
domain = "googleapis.com."
|
||||
client_networks = [module.dev-spoke-vpc.self_link]
|
||||
recordsets = {
|
||||
"A private" = { type = "A", ttl = 300, records = [
|
||||
"A private" = { records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"A restricted" = { type = "A", ttl = 300, records = [
|
||||
"A restricted" = { records = [
|
||||
"199.36.153.4", "199.36.153.5", "199.36.153.6", "199.36.153.7"
|
||||
] }
|
||||
"CNAME *" = { type = "CNAME", ttl = 300, records = ["private.googleapis.com."] }
|
||||
"CNAME *" = { records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "dev-gcrio-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.dev-spoke-project.project_id
|
||||
type = "private"
|
||||
name = "gcr-io"
|
||||
domain = "gcr.io."
|
||||
client_networks = [module.dev-spoke-vpc.self_link]
|
||||
recordsets = {
|
||||
"A gcr.io." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "dev-packages-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.dev-spoke-project.project_id
|
||||
type = "private"
|
||||
name = "packages-cloud"
|
||||
domain = "packages.cloud.google.com."
|
||||
client_networks = [module.dev-spoke-vpc.self_link]
|
||||
recordsets = {
|
||||
"A packages.cloud.google.com." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "dev-pkgdev-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.dev-spoke-project.project_id
|
||||
type = "private"
|
||||
name = "pkg-dev"
|
||||
domain = "pkg.dev."
|
||||
client_networks = [module.dev-spoke-vpc.self_link]
|
||||
recordsets = {
|
||||
"A pkg.dev." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "dev-pkigoog-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.dev-spoke-project.project_id
|
||||
type = "private"
|
||||
name = "pki-goog"
|
||||
domain = "pki.goog."
|
||||
client_networks = [module.dev-spoke-vpc.self_link]
|
||||
recordsets = {
|
||||
"A pki.goog." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ module "prod-dns-private-zone" {
|
|||
domain = "prod.gcp.example.com."
|
||||
client_networks = [module.prod-spoke-vpc.self_link]
|
||||
recordsets = {
|
||||
"A localhost" = { type = "A", ttl = 300, records = ["127.0.0.1"] }
|
||||
"A localhost" = { records = ["127.0.0.1"] }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -50,6 +50,7 @@ module "prod-reverse-10-dns-forwarding" {
|
|||
forwarders = { for ip in var.dns.prod : ip => null }
|
||||
}
|
||||
|
||||
# Google APIs
|
||||
|
||||
module "prod-googleapis-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
|
|
@ -59,12 +60,72 @@ module "prod-googleapis-private-zone" {
|
|||
domain = "googleapis.com."
|
||||
client_networks = [module.prod-spoke-vpc.self_link]
|
||||
recordsets = {
|
||||
"A private" = { type = "A", ttl = 300, records = [
|
||||
"A private" = { records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"A restricted" = { type = "A", ttl = 300, records = [
|
||||
"A restricted" = { records = [
|
||||
"199.36.153.4", "199.36.153.5", "199.36.153.6", "199.36.153.7"
|
||||
] }
|
||||
"CNAME *" = { type = "CNAME", ttl = 300, records = ["private.googleapis.com."] }
|
||||
"CNAME *" = { records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "prod-gcrio-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.prod-spoke-project.project_id
|
||||
type = "private"
|
||||
name = "gcr-io"
|
||||
domain = "gcr.io."
|
||||
client_networks = [module.prod-spoke-vpc.self_link]
|
||||
recordsets = {
|
||||
"A gcr.io." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "prod-packages-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.prod-spoke-project.project_id
|
||||
type = "private"
|
||||
name = "packages-cloud"
|
||||
domain = "packages.cloud.google.com."
|
||||
client_networks = [module.prod-spoke-vpc.self_link]
|
||||
recordsets = {
|
||||
"A packages.cloud.google.com." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "prod-pkgdev-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.prod-spoke-project.project_id
|
||||
type = "private"
|
||||
name = "pkg-dev"
|
||||
domain = "pkg.dev."
|
||||
client_networks = [module.prod-spoke-vpc.self_link]
|
||||
recordsets = {
|
||||
"A pkg.dev." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "prod-pkigoog-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.prod-spoke-project.project_id
|
||||
type = "private"
|
||||
name = "pki-goog"
|
||||
domain = "pki.goog."
|
||||
client_networks = [module.prod-spoke-vpc.self_link]
|
||||
recordsets = {
|
||||
"A pki.goog." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -108,6 +108,13 @@ DNS configuration is further centralized by leveraging peering zones, so that
|
|||
|
||||
- the hub/landing Cloud DNS hosts configurations for on-prem forwarding, Google API domains, and the top-level private zone/s (e.g. gcp.example.com)
|
||||
- the spokes Cloud DNS host configurations for the environment-specific domains (e.g. prod.gcp.example.com), which are bound to the hub/landing leveraging [cross-project binding](https://cloud.google.com/dns/docs/zones/zones-overview#cross-project_binding); a peering zone for the `.` (root) zone is then created on each spoke, delegating all DNS resolution to hub/landing.
|
||||
- Private Google Access is enabled for a selection of the [supported domains](https://cloud.google.com/vpc/docs/configure-private-google-access#domain-options), namely
|
||||
- `private.googleapis.com`
|
||||
- `restricted.googleapis.com`
|
||||
- `gcr.io`
|
||||
- `packages.cloud.google.com`
|
||||
- `pkg.dev`
|
||||
- `pki.goog`
|
||||
|
||||
To complete the configuration, the 35.199.192.0/19 range should be routed on the VPN tunnels from on-prem, and the following names configured for DNS forwarding to cloud:
|
||||
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ module "dev-dns-private-zone" {
|
|||
domain = "dev.gcp.example.com."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A localhost" = { type = "A", ttl = 300, records = ["127.0.0.1"] }
|
||||
"A localhost" = { records = ["127.0.0.1"] }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -46,11 +46,11 @@ module "gcp-example-dns-private-zone" {
|
|||
domain = "gcp.example.com."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A localhost" = { type = "A", ttl = 300, records = ["127.0.0.1"] }
|
||||
"A localhost" = { records = ["127.0.0.1"] }
|
||||
}
|
||||
}
|
||||
|
||||
# Google API zone to trigger Private Access
|
||||
# Google APIs
|
||||
|
||||
module "googleapis-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
|
|
@ -60,12 +60,72 @@ module "googleapis-private-zone" {
|
|||
domain = "googleapis.com."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A private" = { type = "A", ttl = 300, records = [
|
||||
"A private" = { records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"A restricted" = { type = "A", ttl = 300, records = [
|
||||
"A restricted" = { records = [
|
||||
"199.36.153.4", "199.36.153.5", "199.36.153.6", "199.36.153.7"
|
||||
] }
|
||||
"CNAME *" = { type = "CNAME", ttl = 300, records = ["private.googleapis.com."] }
|
||||
"CNAME *" = { records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "gcrio-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.landing-project.project_id
|
||||
type = "private"
|
||||
name = "gcr-io"
|
||||
domain = "gcr.io."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A gcr.io." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "packages-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.landing-project.project_id
|
||||
type = "private"
|
||||
name = "packages-cloud"
|
||||
domain = "packages.cloud.google.com."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A packages.cloud.google.com." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "pkgdev-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.landing-project.project_id
|
||||
type = "private"
|
||||
name = "pkg-dev"
|
||||
domain = "pkg.dev."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A pkg.dev." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
||||
module "pkigoog-private-zone" {
|
||||
source = "../../../modules/dns"
|
||||
project_id = module.landing-project.project_id
|
||||
type = "private"
|
||||
name = "pki-goog"
|
||||
domain = "pki.goog."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A pki.goog." = { ttl = 300, records = [
|
||||
"199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"
|
||||
] }
|
||||
"CNAME *" = { ttl = 300, records = ["private.googleapis.com."] }
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ module "prod-dns-private-zone" {
|
|||
domain = "prod.gcp.example.com."
|
||||
client_networks = [module.landing-vpc.self_link]
|
||||
recordsets = {
|
||||
"A localhost" = { type = "A", ttl = 300, records = ["127.0.0.1"] }
|
||||
"A localhost" = { records = ["127.0.0.1"] }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,11 @@ terraform {
|
|||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
version = ">= 4.40.0" # tftest
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue