diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c238e84..acd5836f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,9 +5,31 @@ All notable changes to this project will be documented in this file. ## [Unreleased] - end to end example: `Scheduled Cloud Asset Inventory Export to Bigquery` +- add support for logging and better type for the `retention_policies` variable in `gcs` module +- **incompatible change** deprecate bucket_policy_only in favor of uniform_bucket_level_access in `gcs` module + +## [3.3.0] - 2020-09-01 + +- remove extra readers in `gcs-to-bq-with-dataflow` example (issue: 128) +- make VPC creation optional in `net-vpc` module to allow managing a pre-existing VPC +- make HA VPN gateway creation optional in `net-vpn-ha` module +- add retention_policy in `gcs` module +- refactor `net-address` module variables, and add support for internal address `purpose` + +## [3.2.0] - 2020-08-29 + +- **incompatible change** add alias IP support in `cloud-vm` module +- add tests for `data-solutions` examples +- fix apply errors on dynamic resources in dataflow example +- make zone creation optional in `dns` module +- new `quota-monitoring` end-to-end example in `cloud-operations` + ## [3.1.1] - 2020-08-26 + - fix error in `project` module +- **incompatible change** make HA VPN Gateway creation optional for `net-vpn-ha` module. Now an existing HA VPN Gateway can be used. Updating to the new version of the module will cause VPN Gateway recreation which can be handled by `terraform state rm/terraform import` operations. + ## [3.1.0] - 2020-08-16 - **incompatible change** add support for specifying a different project id in the GKE cluster module; if using the `peering_config` variable, `peering_config.project_id` now needs to be explicitly set, a `null` value will reuse the `project_id` variable for the peering @@ -171,7 +193,9 @@ All notable changes to this project will be documented in this file. - merge development branch with suite of new modules and end-to-end examples -[Unreleased]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v3.1.1...HEAD +[Unreleased]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v3.3.0...HEAD +[3.3.0]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v3.2.0...v3.3.0 +[3.2.0]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v3.1.1...v3.2.0 [3.1.1]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v3.1.0...v3.1.1 [3.1.0]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v3.0.0...v3.1.0 [3.0.0]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v2.8.0...v3.0.0 diff --git a/README.md b/README.md index ed81ca43..d204b6c9 100644 --- a/README.md +++ b/README.md @@ -19,8 +19,7 @@ Currently available examples: - **foundations** - [single level hierarchy](./foundations/environments/) (environments), [multiple level hierarchy](./foundations/business-units/) (business units + environments) - **networking** - [hub and spoke via peering](./networking/hub-and-spoke-peering/), [hub and spoke via VPN](./networking/hub-and-spoke-vpn/), [DNS and Google Private Access for on-premises](./networking/onprem-google-access-dns/), [Shared VPC with GKE support](./networking/shared-vpc-gke/), [ILB as next hop](./networking/ilb-next-hop) - **data solutions** - [GCE/GCS CMEK via centralized Cloud KMS](./data-solutions/cmek-via-centralized-kms/), [Cloud Storage to Bigquery with Cloud Dataflow](./data-solutions/gcs-to-bq-with-dataflow/) -- **cloud operations** - [Resource tracking and remediation via Cloud Asset feeds](.//cloud-operations/asset-inventory-feed-remediation), [Scheduled Cloud Asset Inventory Export to Bigquery](./cloud-operations/scheduled-asset-inventory-export-bq), [Granular Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam) - +- **cloud operations** - [Resource tracking and remediation via Cloud Asset feeds](.//cloud-operations/asset-inventory-feed-remediation), [Granular Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam), [Compute Engine quota monitoring](./cloud-operations/quota-monitoring), [Scheduled Cloud Asset Inventory Export to Bigquery](./cloud-operations/scheduled-asset-inventory-export-bq) For more information see the README files in the [foundations](./foundations/), [networking](./networking/), [data solutions](./data-solutions/) and [cloud operations](./cloud-operations/) folders. ## Modules diff --git a/cloud-operations/README.md b/cloud-operations/README.md index a17e6de4..afe4f15c 100644 --- a/cloud-operations/README.md +++ b/cloud-operations/README.md @@ -12,13 +12,18 @@ The example's feed tracks changes to Google Compute instances, and the Cloud Fun ## Scheduled Cloud Asset Inventory Export to Bigquery - This [example](./scheduled-asset-inventory-export-bq) shows how to leverage [Cloud Asset Inventory Exporting to Bigquery](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery) feature to keep track of your organization wide assets over time storing information in Bigquery. Data stored in Bigquery can then be used for different purposes, for example: dashboarding, analysis. + This [example](./scheduled-asset-inventory-export-bq) shows how to leverage the [Cloud Asset Inventory Exporting to Bigquery](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery) feature, to keep track of your organization's assets over time storing information in Bigquery. Data stored in Bigquery can then be used for different purposes like dashboarding or analysis.
## Granular Cloud DNS IAM via Service Directory - This [example](./dns-fine-grained-iam) shows how to leverage Service Directory](https://cloud.google.com/blog/products/networking/introducing-service-directory) and Cloud DNS Service Directory private zones, to implement fine-grained IAM controls on DNS. The example creates a Service Directory namespace, a Cloud DNS private zone that uses it as its authoritative source, service accounts with different levels of permissions, and VMs to test them. + This [example](./dns-fine-grained-iam) shows how to leverage [Service Directory](https://cloud.google.com/blog/products/networking/introducing-service-directory) and Cloud DNS Service Directory private zones, to implement fine-grained IAM controls on DNS. The example creates a Service Directory namespace, a Cloud DNS private zone that uses it as its authoritative source, service accounts with different levels of permissions, and VMs to test them.
+## Compute Engine quota monitoring + + This [example](./quota-monitoring) shows a practical way of collecting and monitoring [Compute Engine resource quotas](https://cloud.google.com/compute/quotas) via Cloud Monitoring metrics as an alternative to the recently released [built-in quota metrics](https://cloud.google.com/monitoring/alerts/using-quota-metrics). A simple alert on quota thresholds is also part of the example. + +
diff --git a/cloud-operations/asset-inventory-feed-remediation/main.tf b/cloud-operations/asset-inventory-feed-remediation/main.tf index 5c845086..0d6b29ef 100644 --- a/cloud-operations/asset-inventory-feed-remediation/main.tf +++ b/cloud-operations/asset-inventory-feed-remediation/main.tf @@ -108,10 +108,11 @@ module "simple-vm-example" { region = var.region name = var.name network_interfaces = [{ - network = module.vpc.self_link, - subnetwork = try(module.vpc.subnet_self_links["${var.region}/${var.name}-default"], ""), - nat = false, + network = module.vpc.self_link + subnetwork = try(module.vpc.subnet_self_links["${var.region}/${var.name}-default"], "") + nat = false addresses = null + alias_ips = null }] tags = ["${var.project_id}-test-feed", "shared-test-feed"] instance_count = 1 diff --git a/cloud-operations/dns-fine-grained-iam/main.tf b/cloud-operations/dns-fine-grained-iam/main.tf index 2bc0bbdf..35e59dc0 100644 --- a/cloud-operations/dns-fine-grained-iam/main.tf +++ b/cloud-operations/dns-fine-grained-iam/main.tf @@ -111,10 +111,11 @@ module "vm-ns-editor" { region = var.region name = "${var.name}-ns" network_interfaces = [{ - network = module.vpc.self_link, - subnetwork = module.vpc.subnet_self_links["${var.region}/${var.name}-default"], - nat = false, + network = module.vpc.self_link + subnetwork = module.vpc.subnet_self_links["${var.region}/${var.name}-default"] + nat = false addresses = null + alias_ips = null }] metadata = { startup-script = local.startup-script } service_account_create = true @@ -128,10 +129,11 @@ module "vm-svc-editor" { region = var.region name = "${var.name}-svc" network_interfaces = [{ - network = module.vpc.self_link, - subnetwork = module.vpc.subnet_self_links["${var.region}/${var.name}-default"], - nat = false, + network = module.vpc.self_link + subnetwork = module.vpc.subnet_self_links["${var.region}/${var.name}-default"] + nat = false addresses = null + alias_ips = null }] metadata = { startup-script = local.startup-script } service_account_create = true diff --git a/cloud-operations/quota-monitoring/README.md b/cloud-operations/quota-monitoring/README.md new file mode 100644 index 00000000..a92abf68 --- /dev/null +++ b/cloud-operations/quota-monitoring/README.md @@ -0,0 +1,42 @@ +# Compute Engine quota monitoring + +This example improves on the [GCE quota exporter tool](https://github.com/GoogleCloudPlatform/professional-services/tree/master/tools/gce-quota-sync) (by the same author of this example), and shows a practical way of collecting and monitoring [Compute Engine resource quotas](https://cloud.google.com/compute/quotas) via Cloud Monitoring metrics as an alternative to the recently released [built-in quota metrics](https://cloud.google.com/monitoring/alerts/using-quota-metrics). + +Compared to the built-in metrics, it offers a simpler representation of quotas and quota ratios which is especially useful in charts, it allows filtering or combining quotas between different projects regardless of their monitoring workspace, and it creates a default alerting policy without the need to interact directly with the monitoring API. + +Regardless of its specific purpose, this example is also useful in showing how to manipulate and write time series to cloud monitoring. The resources it creates are shown in the high level diagram below: + +GCP resource diagram + +The solution is designed so that the Cloud Function arguments that control function execution (eg to set which project quotas to monitor) are defined in the Cloud Scheduler payload set in the PubSub message, so that a single function can be used for different configurations by creating more schedules. + +Quota time series are stored using a [custom metric](https://cloud.google.com/monitoring/custom-metrics) with the `custom.googleapis.com/quota/gce` type and [gauge kind](https://cloud.google.com/monitoring/api/v3/kinds-and-types#metric-kinds), tracking the ratio between quota and limit as double to aid in visualization and alerting. Labels are set with the quota name, project id (which may differ from the monitoring workspace projects), value, and limit. This is how they look like in the metrics explorer. + +GCP resource diagram + +The solution also creates a basic monitoring alert policy, to demonstrate how to raise alerts when any of the tracked quota ratios go over a predefined threshold. + +## Running the example + +Clone this repository or [open it in cloud shell](https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2Fterraform-google-modules%2Fcloud-foundation-fabric&cloudshell_print=cloud-shell-readme.txt&cloudshell_working_dir=cloud-operations%2Fquota-monitoring), then go through the following steps to create resources: + +- `terraform init` +- `terraform apply -var project_id=my-project-id` + + +## Variables + +| name | description | type | required | default | +|---|---|:---: |:---:|:---:| +| project_id | Project id that references existing project. | string | ✓ | | +| *bundle_path* | Path used to write the intermediate Cloud Function code bundle. | string | | ./bundle.zip | +| *name* | Arbitrary string used to name created resources. | string | | quota-monitor | +| *project_create* | Create project instead ofusing an existing one. | bool | | false | +| *quota_config* | Cloud function configuration. | object({...}) | | ... | +| *region* | Compute region used in the example. | string | | europe-west1 | +| *schedule_config* | Schedule timer configuration in crontab format | string | | 0 * * * * | + +## Outputs + + + diff --git a/cloud-operations/quota-monitoring/backend.tf.sample b/cloud-operations/quota-monitoring/backend.tf.sample new file mode 100644 index 00000000..528c4eb2 --- /dev/null +++ b/cloud-operations/quota-monitoring/backend.tf.sample @@ -0,0 +1,23 @@ +# Copyright 2019 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# set a valid bucket below and rename this file to backend.tf + +terraform { + backend "gcs" { + bucket = "" + prefix = "fabric/operations/quota-monitoring" + } +} + diff --git a/cloud-operations/quota-monitoring/cf/main.py b/cloud-operations/quota-monitoring/cf/main.py new file mode 100755 index 00000000..a3ff4a09 --- /dev/null +++ b/cloud-operations/quota-monitoring/cf/main.py @@ -0,0 +1,201 @@ +#! /usr/bin/env python3 +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Sync GCE quota usage to Stackdriver for multiple projects. + +This tool fetches global and/or regional quotas from the GCE API for +multiple projects, and sends them to Stackdriver as custom metrics, where they +can be used to set alert policies or create charts. +""" + +import base64 +import datetime +import json +import logging +import os +import warnings + +import click + +from google.api_core.exceptions import GoogleAPIError +from google.cloud import monitoring_v3 + +import googleapiclient.discovery +import googleapiclient.errors + + +_BATCH_SIZE = 5 +_METRIC_KIND = monitoring_v3.enums.MetricDescriptor.MetricKind.GAUGE +_METRIC_TYPE = 'custom.googleapis.com/quota/gce' + + +def _add_series(project_id, series, client=None): + """Write metrics series to Stackdriver. + + Args: + project_id: series will be written to this project id's account + series: the time series to be written, as a list of + monitoring_v3.types.TimeSeries instances + client: optional monitoring_v3.MetricServiceClient will be used + instead of obtaining a new one + """ + client = client or monitoring_v3.MetricServiceClient() + project_name = client.project_path(project_id) + if isinstance(series, monitoring_v3.types.TimeSeries): + series = [series] + try: + client.create_time_series(project_name, series) + except GoogleAPIError as e: + raise RuntimeError('Error from monitoring API: %s' % e) + + +def _configure_logging(verbose=True): + """Basic logging configuration. + + Args: + verbose: enable verbose logging + """ + level = logging.DEBUG if verbose else logging.INFO + logging.basicConfig(level=level) + warnings.filterwarnings('ignore', r'.*end user credentials.*', UserWarning) + + +def _fetch_quotas(project, region='global', compute=None): + """Fetch GCE per - project or per - region quotas from the API. + + Args: + project: fetch global or regional quotas for this project id + region: which quotas to fetch, 'global' or region name + compute: optional instance of googleapiclient.discovery.build will be used + instead of obtaining a new one + """ + compute = compute or googleapiclient.discovery.build('compute', 'v1') + try: + if region != 'global': + req = compute.regions().get(project=project, region=region) + else: + req = compute.projects().get(project=project) + resp = req.execute() + return resp['quotas'] + except (GoogleAPIError, googleapiclient.errors.HttpError) as e: + logging.debug('API Error: %s', e, exc_info=True) + raise RuntimeError('Error fetching quota (project: %s, region: %s)' % + (project, region)) + + +def _get_series(metric_labels, value, metric_type=_METRIC_TYPE, dt=None): + """Create a Stackdriver monitoring time series from value and labels. + + Args: + metric_labels: dict with labels that will be used in the time series + value: time series value + metric_type: which metric is this series for + dt: datetime.datetime instance used for the series end time + """ + series = monitoring_v3.types.TimeSeries() + series.metric.type = metric_type + series.resource.type = 'global' + for label in metric_labels: + series.metric.labels[label] = metric_labels[label] + point = series.points.add() + point.value.double_value = value + point.interval.end_time.FromDatetime(dt or datetime.datetime.utcnow()) + return series + + +def _quota_to_series(project, region, quota): + """Convert API quota objects to Stackdriver monitoring time series. + + Args: + project: set in converted time series labels + region: set in converted time series labels + quota: quota object received from the GCE API + """ + labels = dict((k, str(v)) for k, v in quota.items()) + labels['project'] = project + labels['region'] = region + try: + value = quota['usage'] / float(quota['limit']) + except ZeroDivisionError: + value = 0 + return _get_series(labels, value) + + +@click.command() +@click.option('--monitoring-project', required=True, + help='monitoring project id') +@click.option('--gce-project', multiple=True, + help='project ids (multiple), defaults to monitoring project') +@click.option('--gce-region', multiple=True, + help='regions (multiple), defaults to "global"') +@click.option('--verbose', is_flag=True, help='Verbose output') +@click.argument('keywords', nargs=-1) +def main_cli(monitoring_project=None, gce_project=None, gce_region=None, + verbose=False, keywords=None): + """Fetch GCE quotas and writes them as custom metrics to Stackdriver. + + If KEYWORDS are specified as arguments, only quotas matching one of the + keywords will be stored in Stackdriver. + """ + try: + _main(monitoring_project, gce_project, gce_region, verbose, keywords) + except RuntimeError: + logging.exception('exception raised') + + +def main(event, context): + """Cloud Function entry point.""" + try: + data = json.loads(base64.b64decode(event['data']).decode('utf-8')) + _main(os.environ.get('GCP_PROJECT'), **data) + # uncomment once https://issuetracker.google.com/issues/155215191 is fixed + # except RuntimeError: + # raise + except Exception: + logging.exception('exception in cloud function entry point') + + +def _main(monitoring_project, gce_project=None, gce_region=None, verbose=False, + keywords=None): + """Module entry point used by cli and cloud function wrappers.""" + _configure_logging(verbose=verbose) + gce_projects = gce_project or [monitoring_project] + gce_regions = gce_region or ['global'] + keywords = set(keywords or []) + logging.debug('monitoring project %s', monitoring_project) + logging.debug('projects %s regions %s', gce_projects, gce_regions) + logging.debug('keywords %s', keywords) + quotas = [] + compute = googleapiclient.discovery.build( + 'compute', 'v1', cache_discovery=False) + for project in gce_projects: + logging.debug('project %s', project) + for region in gce_regions: + logging.debug('region %s', region) + for quota in _fetch_quotas(project, region, compute=compute): + if keywords and not any(k in quota['metric'] for k in keywords): + # logging.debug('skipping %s', quota) + continue + logging.debug('quota %s', quota) + quotas.append((project, region, quota)) + client, i = monitoring_v3.MetricServiceClient(), 0 + while i < len(quotas): + series = [_quota_to_series(*q) for q in quotas[i:i + _BATCH_SIZE]] + _add_series(monitoring_project, series, client) + i += _BATCH_SIZE + + +if __name__ == '__main__': + main_cli() diff --git a/cloud-operations/quota-monitoring/cf/requirements.txt b/cloud-operations/quota-monitoring/cf/requirements.txt new file mode 100644 index 00000000..b9c9e011 --- /dev/null +++ b/cloud-operations/quota-monitoring/cf/requirements.txt @@ -0,0 +1,3 @@ +Click>=7.0 +google-api-python-client>=1.10.1 +google-cloud-monitoring>=1.1.0 diff --git a/cloud-operations/quota-monitoring/cloud-shell-readme.txt b/cloud-operations/quota-monitoring/cloud-shell-readme.txt new file mode 100644 index 00000000..62e65af9 --- /dev/null +++ b/cloud-operations/quota-monitoring/cloud-shell-readme.txt @@ -0,0 +1,9 @@ + + +################################# Quickstart ################################# + +- terraform init +- terraform apply -var project_id=$GOOGLE_CLOUD_PROJECT + +Refer to the README.md file for more info and testing flow. + diff --git a/cloud-operations/quota-monitoring/diagram.png b/cloud-operations/quota-monitoring/diagram.png new file mode 100644 index 00000000..c68131f2 Binary files /dev/null and b/cloud-operations/quota-monitoring/diagram.png differ diff --git a/cloud-operations/quota-monitoring/explorer.png b/cloud-operations/quota-monitoring/explorer.png new file mode 100644 index 00000000..80f50254 Binary files /dev/null and b/cloud-operations/quota-monitoring/explorer.png differ diff --git a/cloud-operations/quota-monitoring/main.tf b/cloud-operations/quota-monitoring/main.tf new file mode 100644 index 00000000..0727ae4b --- /dev/null +++ b/cloud-operations/quota-monitoring/main.tf @@ -0,0 +1,148 @@ +/** + * Copyright 2020 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + projects = ( + var.quota_config.projects == null + ? [var.project_id] + : var.quota_config.projects + ) +} + +module "project" { + source = "../../modules/project" + name = var.project_id + project_create = var.project_create + services = [ + "compute.googleapis.com", + "cloudfunctions.googleapis.com" + ] + service_config = { + disable_on_destroy = false, + disable_dependent_services = false + } + iam_roles = [ + "roles/monitoring.metricWriter", + ] + iam_members = { + "roles/monitoring.metricWriter" = [module.cf.service_account_iam_email] + } +} + +module "pubsub" { + source = "../../modules/pubsub" + project_id = module.project.project_id + name = var.name + subscriptions = { + "${var.name}-default" = null + } + # the Cloud Scheduler robot service account already has pubsub.topics.publish + # at the project level via roles/cloudscheduler.serviceAgent +} + +module "cf" { + source = "../../modules/cloud-function" + project_id = module.project.project_id + name = var.name + bucket_name = "${var.name}-${random_pet.random.id}" + bucket_config = { + location = var.region + lifecycle_delete_age = null + } + bundle_config = { + source_dir = "cf" + output_path = var.bundle_path + } + # https://github.com/hashicorp/terraform-provider-archive/issues/40 + # https://issuetracker.google.com/issues/155215191 + environment_variables = { + USE_WORKER_V2 = "true" + PYTHON37_DRAIN_LOGS_ON_CRASH_WAIT_SEC = "5" + } + service_account_create = true + trigger_config = { + event = "google.pubsub.topic.publish" + resource = module.pubsub.topic.id + retry = null + } +} + +resource "google_cloud_scheduler_job" "job" { + project = var.project_id + region = var.region + name = var.name + schedule = var.schedule_config + time_zone = "UTC" + + pubsub_target { + attributes = {} + topic_name = module.pubsub.topic.id + data = base64encode(jsonencode({ + gce_project = var.quota_config.projects + gce_region = var.quota_config.regions + keywords = var.quota_config.filters + })) + } +} + +resource "google_project_iam_member" "network_viewer" { + for_each = toset(local.projects) + project = each.key + role = "roles/compute.networkViewer" + member = module.cf.service_account_iam_email +} + +resource "google_project_iam_member" "quota_viewer" { + for_each = toset(local.projects) + project = each.key + role = "roles/servicemanagement.quotaViewer" + member = module.cf.service_account_iam_email +} + +resource "google_monitoring_alert_policy" "alert_policy" { + project = module.project.project_id + display_name = "Quota monitor" + combiner = "OR" + conditions { + display_name = "simple quota threshold" + condition_threshold { + filter = "metric.type=\"custom.googleapis.com/quota/gce\" resource.type=\"global\"" + threshold_value = 0.75 + comparison = "COMPARISON_GT" + duration = "0s" + aggregations { + alignment_period = "60s" + group_by_fields = [] + per_series_aligner = "ALIGN_MEAN" + } + trigger { + count = 1 + percent = 0 + } + } + } + enabled = false + user_labels = { + name = var.name + } + documentation { + content = "GCE quota over threshold." + } +} + +resource "random_pet" "random" { + length = 1 +} diff --git a/cloud-operations/quota-monitoring/outputs.tf b/cloud-operations/quota-monitoring/outputs.tf new file mode 100644 index 00000000..264a34bf --- /dev/null +++ b/cloud-operations/quota-monitoring/outputs.tf @@ -0,0 +1,16 @@ +/** + * Copyright 2020 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + diff --git a/cloud-operations/quota-monitoring/variables.tf b/cloud-operations/quota-monitoring/variables.tf new file mode 100644 index 00000000..c54a44bb --- /dev/null +++ b/cloud-operations/quota-monitoring/variables.tf @@ -0,0 +1,64 @@ +/** + * Copyright 2020 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "bundle_path" { + description = "Path used to write the intermediate Cloud Function code bundle." + type = string + default = "./bundle.zip" +} + +variable "name" { + description = "Arbitrary string used to name created resources." + type = string + default = "quota-monitor" +} + +variable "project_create" { + description = "Create project instead ofusing an existing one." + type = bool + default = false +} + +variable "project_id" { + description = "Project id that references existing project." + type = string +} + +variable "quota_config" { + description = "Cloud function configuration." + type = object({ + filters = list(string) + projects = list(string) + regions = list(string) + }) + default = { + filters = null + projects = null + regions = null + } +} + +variable "region" { + description = "Compute region used in the example." + type = string + default = "europe-west1" +} + +variable "schedule_config" { + description = "Schedule timer configuration in crontab format" + type = string + default = "0 * * * *" +} diff --git a/cloud-operations/scheduled-asset-inventory-export-bq/README.md b/cloud-operations/scheduled-asset-inventory-export-bq/README.md index be27eb71..9c63f2b4 100644 --- a/cloud-operations/scheduled-asset-inventory-export-bq/README.md +++ b/cloud-operations/scheduled-asset-inventory-export-bq/README.md @@ -1,5 +1,6 @@ # Scheduled Cloud Asset Inventory Export to Bigquery +<<<<<<< HEAD This example shows how to leverage [Cloud Asset Inventory Exporting to Bigquery](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery) feature to keep track of your project wide assets over time storing information in Bigquery. The data stored in Bigquery can then be used for different purposes: @@ -13,15 +14,23 @@ The example uses export resources at project level for ease of testing, in actua - the `roles/cloudasset.viewer` on the service account should be set at folder or organization level The resources created in this example are shown in the high level diagram below: +======= +This example shows how to leverage the [Cloud Asset Inventory Exporting to Bigquery](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery) feature, to keep track of your organization's assets over time storing information in Bigquery. The data stored in Bigquery can then be used for different purposes like dashboarding or analysis. + +This example shows an export to Bigquery scheduled on a daily basis. The resources created in this example are shown in the high level diagram below: +>>>>>>> a2392aeda1c0a3da04b80666d3eb4b09061eeb04 ## Prerequisites -Ensure that you grant your account one of the following roles on your project, folder, or organization. - - Cloud Asset Viewer role (roles/cloudasset.viewer) - - Owner primitive role (roles/owner) + +Ensure that you grant your account one of the following roles on your project, folder, or organization: + +- Cloud Asset Viewer role (`roles/cloudasset.viewer`) +- Owner primitive role (`roles/owner`) ## Running the example + Clone this repository, specify your variables in a `terraform.tvars` and then go through the following steps to create resources: - `terraform init` @@ -31,9 +40,10 @@ Once done testing, you can clean up resources by running `terraform destroy`. To ## Testing the example -You can now run queries on the data you exported on Bigquery. [Here](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery#querying_an_asset_snapshot) you can find some example of queries you can run. +Once resources are created, you can run queries on the data you exported on Bigquery. [Here](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery#querying_an_asset_snapshot) you can find some example of queries you can run. + +You can also create a dashboard connecting [Datalab](https://datastudio.google.com/) or any other BI tools of your choice to your Bigquery datase. -You can also create a dashborad connecting [Datalab](https://datastudio.google.com/) or any other BI tools of your choice to your Bigquery datase.. ## Variables diff --git a/data-solutions/cmek-via-centralized-kms/README.md b/data-solutions/cmek-via-centralized-kms/README.md index 62b42210..03c32f27 100644 --- a/data-solutions/cmek-via-centralized-kms/README.md +++ b/data-solutions/cmek-via-centralized-kms/README.md @@ -44,7 +44,6 @@ This sample creates several distinct groups of resources: | *vpc_ip_cidr_range* | Ip range used in the subnet deployef in the Service Project. | string | | 10.0.0.0/20 | | *vpc_name* | Name of the VPC created in the Service Project. | string | | local | | *vpc_subnet_name* | Name of the subnet created in the Service Project. | string | | subnet | -| *zone* | The zone where resources will be deployed. | string | | europe-west1-b | ## Outputs diff --git a/data-solutions/cmek-via-centralized-kms/main.tf b/data-solutions/cmek-via-centralized-kms/main.tf index 464f208f..08b7b7de 100644 --- a/data-solutions/cmek-via-centralized-kms/main.tf +++ b/data-solutions/cmek-via-centralized-kms/main.tf @@ -104,13 +104,13 @@ module "kms_vm_example" { source = "../../modules/compute-vm" project_id = module.project-service.project_id region = var.region - zone = var.zone name = "kms-vm" network_interfaces = [{ network = module.vpc.self_link, subnetwork = module.vpc.subnet_self_links["${var.region}/subnet"], nat = false, addresses = null + alias_ips = null }] attached_disks = [ { diff --git a/data-solutions/cmek-via-centralized-kms/variables.tf b/data-solutions/cmek-via-centralized-kms/variables.tf index fdc0ac94..04b70784 100644 --- a/data-solutions/cmek-via-centralized-kms/variables.tf +++ b/data-solutions/cmek-via-centralized-kms/variables.tf @@ -64,9 +64,3 @@ variable "vpc_ip_cidr_range" { type = string default = "10.0.0.0/20" } - -variable "zone" { - description = "The zone where resources will be deployed." - type = string - default = "europe-west1-b" -} diff --git a/data-solutions/gcs-to-bq-with-dataflow/README.md b/data-solutions/gcs-to-bq-with-dataflow/README.md index 23a4470d..da3c35c5 100644 --- a/data-solutions/gcs-to-bq-with-dataflow/README.md +++ b/data-solutions/gcs-to-bq-with-dataflow/README.md @@ -125,7 +125,6 @@ You can check data imported into Google BigQuery from the Google Cloud Console U | *vpc_ip_cidr_range* | Ip range used in the subnet deployef in the Service Project. | string | | 10.0.0.0/20 | | *vpc_name* | Name of the VPC created in the Service Project. | string | | local | | *vpc_subnet_name* | Name of the subnet created in the Service Project. | string | | subnet | -| *zone* | The zone where resources will be deployed. | string | | europe-west1-b | ## Outputs diff --git a/data-solutions/gcs-to-bq-with-dataflow/main.tf b/data-solutions/gcs-to-bq-with-dataflow/main.tf index 84521c03..39a9ffd3 100644 --- a/data-solutions/gcs-to-bq-with-dataflow/main.tf +++ b/data-solutions/gcs-to-bq-with-dataflow/main.tf @@ -61,9 +61,9 @@ module "service-account-bq" { project_id = module.project-service.project_id names = ["bq-test"] iam_project_roles = { - (module.project-service.project_id) = [ + (var.project_service_name) = [ "roles/logging.logWriter", - "roles/monitoring.metricWriter", + "roles/monitoring.metricWriter", "roles/bigquery.admin" ] } @@ -74,12 +74,12 @@ module "service-account-gce" { project_id = module.project-service.project_id names = ["gce-test"] iam_project_roles = { - (module.project-service.project_id) = [ + (var.project_service_name) = [ "roles/logging.logWriter", "roles/monitoring.metricWriter", "roles/dataflow.admin", "roles/iam.serviceAccountUser", - "roles/bigquery.dataOwner", + "roles/bigquery.dataOwner", "roles/bigquery.jobUser" # Needed to import data using 'bq' command ] } @@ -90,7 +90,7 @@ module "service-account-df" { project_id = module.project-service.project_id names = ["df-test"] iam_project_roles = { - (module.project-service.project_id) = [ + (var.project_service_name) = [ "roles/dataflow.worker", "roles/bigquery.dataOwner", "roles/bigquery.metadataViewer", @@ -143,7 +143,7 @@ module "kms" { #"serviceAccount:${module.project-service.service_accounts.default.bq}", "serviceAccount:${data.google_bigquery_default_service_account.bq_sa.email}", ] - }, + }, } } @@ -156,7 +156,7 @@ module "kms-regional" { } keys = { key-df = null } key_iam_roles = { - key-df = ["roles/cloudkms.cryptoKeyEncrypterDecrypter"] + key-df = ["roles/cloudkms.cryptoKeyEncrypterDecrypter"] } key_iam_members = { key-df = { @@ -164,7 +164,7 @@ module "kms-regional" { "serviceAccount:${module.project-service.service_accounts.robots.dataflow}", "serviceAccount:${module.project-service.service_accounts.robots.compute}", ] - } + } } } @@ -210,13 +210,13 @@ module "vm_example" { source = "../../modules/compute-vm" project_id = module.project-service.project_id region = var.region - zone = var.zone name = "vm-example" network_interfaces = [{ network = module.vpc.self_link, subnetwork = module.vpc.subnet_self_links["${var.region}/${var.vpc_subnet_name}"], nat = false, addresses = null + alias_ips = null }] attached_disks = [ { @@ -259,9 +259,9 @@ module "kms-gcs" { prefix = module.project-service.project_id names = ["data", "df-tmplocation"] iam_roles = { - data = ["roles/storage.admin","roles/storage.objectViewer"], + data = ["roles/storage.admin", "roles/storage.objectViewer"], df-tmplocation = ["roles/storage.admin"] - } + } iam_members = { data = { "roles/storage.admin" = [ @@ -269,22 +269,22 @@ module "kms-gcs" { ], "roles/storage.viewer" = [ "serviceAccount:${module.service-account-df.email}", - ], + ], }, df-tmplocation = { "roles/storage.admin" = [ "serviceAccount:${module.service-account-gce.email}", "serviceAccount:${module.service-account-df.email}", ] - } - } + } + } encryption_keys = { data = module.kms.keys.key-gcs.self_link, df-tmplocation = module.kms.keys.key-gcs.self_link, } force_destroy = { data = true, - df-tmplocation = true, + df-tmplocation = true, } } @@ -293,15 +293,14 @@ module "kms-gcs" { ############################################################################### module "bigquery-dataset" { - source = "../../modules/bigquery-dataset" - project_id = module.project-service.project_id - id = "bq_dataset" + source = "../../modules/bigquery-dataset" + project_id = module.project-service.project_id + id = "bq_dataset" access_roles = { reader-group = { role = "READER", type = "domain" } owner = { role = "OWNER", type = "user_by_email" } } access_identities = { - reader-group = "caggioland.com" owner = module.service-account-bq.email } encryption_key = module.kms.keys.key-bq.self_link @@ -315,11 +314,11 @@ module "bigquery-dataset" { range = null # use start/end/interval for range time = null } - schema = file("schema_bq_import.json") + schema = file("${path.module}/schema_bq_import.json") options = { - clustering = null - expiration_time = null - encryption_key = module.kms.keys.key-bq.self_link + clustering = null + expiration_time = null + encryption_key = module.kms.keys.key-bq.self_link } }, df_import = { @@ -331,11 +330,11 @@ module "bigquery-dataset" { range = null # use start/end/interval for range time = null } - schema = file("schema_df_import.json") + schema = file("${path.module}/schema_df_import.json") options = { - clustering = null + clustering = null expiration_time = null - encryption_key = module.kms.keys.key-bq.self_link + encryption_key = module.kms.keys.key-bq.self_link } } } diff --git a/data-solutions/gcs-to-bq-with-dataflow/variables.tf b/data-solutions/gcs-to-bq-with-dataflow/variables.tf index a44f874a..be76a0e5 100644 --- a/data-solutions/gcs-to-bq-with-dataflow/variables.tf +++ b/data-solutions/gcs-to-bq-with-dataflow/variables.tf @@ -63,12 +63,6 @@ variable "vpc_ip_cidr_range" { default = "10.0.0.0/20" } -variable "zone" { - description = "The zone where resources will be deployed." - type = string - default = "europe-west1-b" -} - variable "ssh_source_ranges" { description = "IP CIDR ranges that will be allowed to connect via SSH to the onprem instance." type = list(string) diff --git a/modules/compute-vm/README.md b/modules/compute-vm/README.md index 128087c0..ed6513d9 100644 --- a/modules/compute-vm/README.md +++ b/modules/compute-vm/README.md @@ -20,10 +20,11 @@ module "simple-vm-example" { region = "europe-west1" name = "test" network_interfaces = [{ - network = local.network_self_link, - subnetwork = local.subnet_self_link, - nat = false, + network = local.network_self_link + subnetwork = local.subnet_self_link + nat = false addresses = null + alias_ips = null }] service_account_create = true instance_count = 1 @@ -41,10 +42,11 @@ module "kms-vm-example" { region = local.region name = "kms-test" network_interfaces = [{ - network = local.network_self_link, - subnetwork = local.subnet_self_link, - nat = false, + network = local.network_self_link + subnetwork = local.subnet_self_link + nat = false addresses = null + alias_ips = null }] attached_disks = [ { @@ -85,10 +87,11 @@ module "cos-test" { region = "europe-west1" name = "test" network_interfaces = [{ - network = local.network_self_link, - subnetwork = local.subnet_self_link, - nat = false, + network = local.network_self_link + subnetwork = local.subnet_self_link + nat = false addresses = null + alias_ips = null }] instance_count = 1 boot_disk = { @@ -115,10 +118,11 @@ module "instance-group" { region = "europe-west1" name = "ilb-test" network_interfaces = [{ - network = local.network_self_link, - subnetwork = local.subnetwork_self_link, - nat = false, + network = local.network_self_link + subnetwork = local.subnetwork_self_link + nat = false addresses = null + alias_ips = null }] boot_disk = { image = "projects/cos-cloud/global/images/family/cos-stable" @@ -141,7 +145,7 @@ module "instance-group" { | name | description | type | required | default | |---|---|:---: |:---:|:---:| | name | Instances base name. | string | ✓ | | -| network_interfaces | Network interfaces configuration. Use self links for Shared VPC, set addresses to null if not needed. | list(object({...})) | ✓ | | +| network_interfaces | Network interfaces configuration. Use self links for Shared VPC, set addresses and alias_ips to null if not needed. | list(object({...})) | ✓ | | | project_id | Project id. | string | ✓ | | | region | Compute region. | string | ✓ | | | *attached_disk_defaults* | Defaults for attached disks options. | object({...}) | | ... | diff --git a/modules/compute-vm/main.tf b/modules/compute-vm/main.tf index 834a1a60..90b88c0c 100644 --- a/modules/compute-vm/main.tf +++ b/modules/compute-vm/main.tf @@ -145,6 +145,14 @@ resource "google_compute_instance" "default" { ) } } + dynamic alias_ip_range { + for_each = config.value.alias_ips != null ? config.value.alias_ips : [] + iterator = alias_ips + content { + ip_cidr_range = alias_ips.value.ip_cidr_range + subnetwork_range_name = alias_ips.value.subnetwork_range_name + } + } } } diff --git a/modules/compute-vm/variables.tf b/modules/compute-vm/variables.tf index 61b3508d..709a7548 100644 --- a/modules/compute-vm/variables.tf +++ b/modules/compute-vm/variables.tf @@ -144,7 +144,7 @@ variable "name" { } variable "network_interfaces" { - description = "Network interfaces configuration. Use self links for Shared VPC, set addresses to null if not needed." + description = "Network interfaces configuration. Use self links for Shared VPC, set addresses and alias_ips to null if not needed." type = list(object({ nat = bool network = string @@ -153,6 +153,10 @@ variable "network_interfaces" { internal = list(string) external = list(string) }) + alias_ips = list(object({ + ip_cidr_range = string + subnetwork_range_name = string + })) })) } diff --git a/modules/dns/README.md b/modules/dns/README.md index d391c4cf..2c00ebb1 100644 --- a/modules/dns/README.md +++ b/modules/dns/README.md @@ -38,6 +38,7 @@ module "private-dns" { | *recordsets* | List of DNS record objects to manage. | list(object({...})) | | [] | | *service_directory_namespace* | Service directory namespace id (URL), only valid for 'service-directory' zone types. | string | | null | | *type* | Type of zone to create, valid values are 'public', 'private', 'forwarding', 'peering', 'service-directory'. | string | | private | +| *zone_create* | Create zone. When set to false, uses a data source to reference existing zone. | bool | | true | ## Outputs diff --git a/modules/dns/main.tf b/modules/dns/main.tf index f29d2768..d7e98b52 100644 --- a/modules/dns/main.tf +++ b/modules/dns/main.tf @@ -19,10 +19,14 @@ locals { for record in var.recordsets : join("/", [record.name, record.type]) => record } - zone = try( - google_dns_managed_zone.non-public.0, try( - google_dns_managed_zone.public.0, null - ) + zone = ( + var.zone_create + ? try( + google_dns_managed_zone.non-public.0, try( + google_dns_managed_zone.public.0, null + ) + ) + : try(data.google_dns_managed_zone.public.0, null) ) dns_keys = try( data.google_dns_keys.dns_keys.0, null @@ -30,7 +34,7 @@ locals { } resource "google_dns_managed_zone" "non-public" { - count = var.type != "public" ? 1 : 0 + count = (var.zone_create && var.type != "public" ) ? 1 : 0 provider = google-beta project = var.project_id name = var.name @@ -89,8 +93,13 @@ resource "google_dns_managed_zone" "non-public" { } +data "google_dns_managed_zone" "public" { + count = var.zone_create ? 0 : 1 + name = var.name +} + resource "google_dns_managed_zone" "public" { - count = var.type == "public" ? 1 : 0 + count = (var.zone_create && var.type == "public" ) ? 1 : 0 project = var.project_id name = var.name dns_name = var.domain @@ -123,8 +132,8 @@ resource "google_dns_managed_zone" "public" { } data "google_dns_keys" "dns_keys" { - count = var.dnssec_config == {} || var.type != "public" ? 0 : 1 - managed_zone = google_dns_managed_zone.public.0.id + count = var.zone_create && ( var.dnssec_config == {} || var.type != "public" ) ? 0 : 1 + managed_zone = local.zone.id } resource "google_dns_record_set" "cloud-static-records" { diff --git a/modules/dns/variables.tf b/modules/dns/variables.tf index f38fb36a..43aad86b 100644 --- a/modules/dns/variables.tf +++ b/modules/dns/variables.tf @@ -98,3 +98,11 @@ variable "type" { type = string default = "private" } + +variable "zone_create" { + description = "Create zone. When set to false, uses a data source to reference existing zone." + type = bool + default = true +} + + diff --git a/modules/folders-unit/main.tf b/modules/folders-unit/main.tf index ac012ab3..3034d20f 100644 --- a/modules/folders-unit/main.tf +++ b/modules/folders-unit/main.tf @@ -95,7 +95,7 @@ resource "google_storage_bucket" "tfstate" { location = var.gcs_defaults.location storage_class = var.gcs_defaults.storage_class force_destroy = false - bucket_policy_only = true + uniform_bucket_level_access = true versioning { enabled = true } diff --git a/modules/gcs/README.md b/modules/gcs/README.md index 839faadd..672ec12e 100644 --- a/modules/gcs/README.md +++ b/modules/gcs/README.md @@ -45,12 +45,44 @@ module "buckets" { iam_roles = { bucket-two = ["roles/storage.admin"] } - kms_keys = { + encryption_keys = { bucket-two = local.kms_key.self_link, } } ``` +### Example with retention policy + +```hcl +module "buckets" { + source = "./modules/gcs" + project_id = "myproject" + prefix = "test" + names = ["bucket-one", "bucket-two"] + bucket_policy_only = { + bucket-one = false + } + iam_members = { + bucket-two = { + "roles/storage.admin" = ["group:storage@example.com"] + } + } + iam_roles = { + bucket-two = ["roles/storage.admin"] + } + + retention_policies = { + bucket-one = { retention_period = 100 , is_locked = true} + bucket-two = { retention_period = 900 , is_locked = false} + } + + logging_config = { + bucket-one = { log_bucket = bucket_name_for_logging , log_object_prefix = null} + bucket-two = { log_bucket = bucket_name_for_logging , log_object_prefix = "logs_for_bucket_two"} + } +} +``` + ## Variables @@ -58,14 +90,16 @@ module "buckets" { |---|---|:---: |:---:|:---:| | names | Bucket name suffixes. | list(string) | ✓ | | | project_id | Bucket project id. | string | ✓ | | -| *bucket_policy_only* | Optional map to disable object ACLS keyed by name, defaults to true. | map(bool) | | {} | +| *uniform_bucket_level_access* | Optional map to enable object ACLs keyed by name, defaults to true. | map(bool) | | {} | | *encryption_keys* | Per-bucket KMS keys that will be used for encryption. | map(string) | | {} | | *force_destroy* | Optional map to set force destroy keyed by name, defaults to false. | map(bool) | | {} | | *iam_members* | IAM members keyed by bucket name and role. | map(map(list(string))) | | {} | | *iam_roles* | IAM roles keyed by bucket name. | map(list(string)) | | {} | | *labels* | Labels to be attached to all buckets. | map(string) | | {} | | *location* | Bucket location. | string | | EU | +| *logging_config* | Per-bucket logging. | map(object({...})) | | {} | | *prefix* | Prefix used to generate the bucket name. | string | | null | +| *retention_policies* | Per-bucket retention policy. | map(object({...})) | | {} | | *storage_class* | Bucket storage class. | string | | MULTI_REGIONAL | | *versioning* | Optional map to set versioning keyed by name, defaults to false. | map(bool) | | {} | diff --git a/modules/gcs/main.tf b/modules/gcs/main.tf index f345ed3b..c93734a6 100644 --- a/modules/gcs/main.tf +++ b/modules/gcs/main.tf @@ -36,7 +36,15 @@ locals { ? "" : join("-", [var.prefix, lower(var.location), ""]) ) - kms_keys = { for name in var.names : name => lookup(var.encryption_keys, name, null) } + kms_keys = { + for name in var.names : name => lookup(var.encryption_keys, name, null) + } + retention_policy = { + for name in var.names : name => lookup(var.retention_policies, name, null) + } + logging_config = { + for name in var.names : name => lookup(var.logging_config, name, null) + } } resource "google_storage_bucket" "buckets" { @@ -46,7 +54,7 @@ resource "google_storage_bucket" "buckets" { location = var.location storage_class = var.storage_class force_destroy = lookup(var.force_destroy, each.key, false) - bucket_policy_only = lookup(var.bucket_policy_only, each.key, true) + uniform_bucket_level_access = lookup(var.uniform_bucket_level_access, each.key, true) versioning { enabled = lookup(var.versioning, each.key, false) } @@ -63,6 +71,22 @@ resource "google_storage_bucket" "buckets" { default_kms_key_name = local.kms_keys[each.key] } } + + dynamic retention_policy { + for_each = local.retention_policy[each.key] == null ? [] : [""] + content { + retention_period = local.retention_policy[each.key]["retention_period"] + is_locked = local.retention_policy[each.key]["is_locked"] + } + } + + dynamic logging { + for_each = local.logging_config[each.key] == null ? [] : [""] + content { + log_bucket = local.logging_config[each.key]["log_bucket"] + log_object_prefix = local.logging_config[each.key]["log_object_prefix"] + } + } } resource "google_storage_bucket_iam_binding" "bindings" { diff --git a/modules/gcs/variables.tf b/modules/gcs/variables.tf index cdc63ecc..19ada263 100644 --- a/modules/gcs/variables.tf +++ b/modules/gcs/variables.tf @@ -14,8 +14,8 @@ * limitations under the License. */ -variable "bucket_policy_only" { - description = "Optional map to disable object ACLS keyed by name, defaults to true." +variable "uniform_bucket_level_access" { + description = "Optional map to allow using object ACLs (false) or not (true, this is the recommended behavior) , defaults to true (which is the recommended practice, but not the behavior of storage API)." type = map(bool) default = {} } @@ -56,6 +56,15 @@ variable "location" { default = "EU" } +variable "logging_config" { + description = "Per-bucket logging." + type = map(object({ + log_bucket = string + log_object_prefix = string + })) + default = {} +} + variable "names" { description = "Bucket name suffixes." type = list(string) @@ -72,6 +81,15 @@ variable "project_id" { type = string } +variable "retention_policies" { + description = "Per-bucket retention policy." + type = map(object({ + retention_period = number + is_locked = bool + })) + default = {} +} + variable "storage_class" { description = "Bucket storage class." type = string diff --git a/modules/net-address/README.md b/modules/net-address/README.md index 9c1169b5..eb3e1168 100644 --- a/modules/net-address/README.md +++ b/modules/net-address/README.md @@ -1,14 +1,46 @@ # Net Address Reservation Module -## Example +This module allows reserving Compute Engine external, global, and internal addresses. + +## Examples + +### External and global addresses ```hcl module "addresses" { source = "./modules/net-address" project_id = local.projects.host external_addresses = { - nat-1 = module.vpc.subnet_regions["default"], - vpn-remote = module.vpc.subnet_regions["default"], + nat-1 = var.region + vpn-remote = var.region + } + global_addresses = ["app-1", "app-2"] +} +``` + +### Internal addresses + +```hcl +module "addresses" { + source = "./modules/net-address" + project_id = local.projects.host + internal_addresses = { + ilb-1 = { + region = var.region + subnetwork = module.vpc.subnet_self_links["${var.region}-test"] + } + ilb-2 = { + region = var.region + subnetwork = module.vpc.subnet_self_links["${var.region}-test"] + } + } + # optional configuration + internal_addresses_config = { + ilb-1 = { + address = null + purpose = "SHARED_LOADBALANCER_VIP" + tier = null + } } } ``` @@ -21,9 +53,8 @@ module "addresses" { | project_id | Project where the addresses will be created. | string | ✓ | | | *external_addresses* | Map of external address regions, keyed by name. | map(string) | | {} | | *global_addresses* | List of global addresses to create. | list(string) | | [] | -| *internal_address_addresses* | Optional explicit addresses for internal addresses, keyed by name. | map(string) | | {} | -| *internal_address_tiers* | Optional network tiers for internal addresses, keyed by name. | map(string) | | {} | | *internal_addresses* | Map of internal addresses to create, keyed by name. | map(object({...})) | | {} | +| *internal_addresses_config* | Optional configuration for internal addresses, keyed by name. Unused options can be set to null. | map(object({...})) | | {} | ## Outputs diff --git a/modules/net-address/main.tf b/modules/net-address/main.tf index b752f2aa..ae43174e 100644 --- a/modules/net-address/main.tf +++ b/modules/net-address/main.tf @@ -31,6 +31,7 @@ resource "google_compute_address" "external" { } resource "google_compute_address" "internal" { + provider = google-beta for_each = var.internal_addresses project = var.project_id name = each.key @@ -38,7 +39,8 @@ resource "google_compute_address" "internal" { address_type = "INTERNAL" region = each.value.region subnetwork = each.value.subnetwork - address = lookup(var.internal_address_addresses, each.key, null) - network_tier = lookup(var.internal_address_tiers, each.key, null) + address = try(var.internal_addresses_config[each.key].address, null) + network_tier = try(var.internal_addresses_config[each.key].tier, null) + purpose = try(var.internal_addresses_config[each.key].purpose, null) # labels = lookup(var.internal_address_labels, each.key, {}) } diff --git a/modules/net-address/outputs.tf b/modules/net-address/outputs.tf index 7d26158a..188e88c1 100644 --- a/modules/net-address/outputs.tf +++ b/modules/net-address/outputs.tf @@ -31,7 +31,6 @@ output "global_addresses" { address.name => { address = address.address self_link = address.self_link - status = address.status } } } diff --git a/modules/net-address/variables.tf b/modules/net-address/variables.tf index 02b85f68..e5eda945 100644 --- a/modules/net-address/variables.tf +++ b/modules/net-address/variables.tf @@ -41,16 +41,14 @@ variable "internal_addresses" { default = {} } -variable "internal_address_addresses" { - description = "Optional explicit addresses for internal addresses, keyed by name." - type = map(string) - default = {} -} - -variable "internal_address_tiers" { - description = "Optional network tiers for internal addresses, keyed by name." - type = map(string) - default = {} +variable "internal_addresses_config" { + description = "Optional configuration for internal addresses, keyed by name. Unused options can be set to null." + type = map(object({ + address = string + purpose = string + tier = string + })) + default = {} } # variable "internal_address_labels" { diff --git a/modules/net-address/versions.tf b/modules/net-address/versions.tf index ce6918e0..ef2d3464 100644 --- a/modules/net-address/versions.tf +++ b/modules/net-address/versions.tf @@ -16,4 +16,7 @@ terraform { required_version = ">= 0.12.6" + required_providers { + google-beta = "~> 3.28.0" + } } diff --git a/modules/net-vpc/README.md b/modules/net-vpc/README.md index 334c7fbb..29085729 100644 --- a/modules/net-vpc/README.md +++ b/modules/net-vpc/README.md @@ -127,6 +127,7 @@ module "vpc-host" { | *subnet_flow_logs* | Optional map of boolean to control flow logs (default is disabled), keyed by subnet 'region/name'. | map(bool) | | {} | | *subnet_private_access* | Optional map of boolean to control private Google access (default is enabled), keyed by subnet 'region/name'. | map(bool) | | {} | | *subnets* | The list of subnets being created | list(object({...})) | | [] | +| *vpc_create* | Create VPC. When set to false, uses a data source to reference existing VPC. | bool | | true | ## Outputs diff --git a/modules/net-vpc/main.tf b/modules/net-vpc/main.tf index 3c21f467..67bd102d 100644 --- a/modules/net-vpc/main.tf +++ b/modules/net-vpc/main.tf @@ -66,9 +66,21 @@ locals { for subnet in var.subnets : "${subnet.region}/${subnet.name}" => subnet } + network = ( + var.vpc_create + ? try(google_compute_network.network.0, null) + : try(data.google_compute_network.network.0, null) + ) +} + +data "google_compute_network" "network" { + count = var.vpc_create ? 0 : 1 + project = var.project_id + name = var.name } resource "google_compute_network" "network" { + count = var.vpc_create ? 1 : 0 project = var.project_id name = var.name description = var.description @@ -80,8 +92,8 @@ resource "google_compute_network" "network" { resource "google_compute_network_peering" "local" { provider = google-beta count = var.peering_config == null ? 0 : 1 - name = "${google_compute_network.network.name}-${local.peer_network}" - network = google_compute_network.network.self_link + name = "${var.name}-${local.peer_network}" + network = local.network.self_link peer_network = var.peering_config.peer_vpc_self_link export_custom_routes = var.peering_config.export_routes import_custom_routes = var.peering_config.import_routes @@ -90,9 +102,9 @@ resource "google_compute_network_peering" "local" { resource "google_compute_network_peering" "remote" { provider = google-beta count = var.peering_config == null ? 0 : 1 - name = "${local.peer_network}-${google_compute_network.network.name}" + name = "${local.peer_network}-${var.name}" network = var.peering_config.peer_vpc_self_link - peer_network = google_compute_network.network.self_link + peer_network = local.network.self_link export_custom_routes = var.peering_config.import_routes import_custom_routes = var.peering_config.export_routes depends_on = [google_compute_network_peering.local] @@ -101,7 +113,7 @@ resource "google_compute_network_peering" "remote" { resource "google_compute_shared_vpc_host_project" "shared_vpc_host" { count = var.shared_vpc_host ? 1 : 0 project = var.project_id - depends_on = [google_compute_network.network] + depends_on = [local.network] } resource "google_compute_shared_vpc_service_project" "service_projects" { @@ -118,7 +130,7 @@ resource "google_compute_shared_vpc_service_project" "service_projects" { resource "google_compute_subnetwork" "subnetwork" { for_each = local.subnets project = var.project_id - network = google_compute_network.network.name + network = local.network.name region = each.value.region name = each.value.name ip_cidr_range = each.value.ip_cidr_range @@ -153,7 +165,7 @@ resource "google_compute_subnetwork_iam_binding" "binding" { resource "google_compute_route" "gateway" { for_each = local.routes_gateway project = var.project_id - network = google_compute_network.network.name + network = local.network.name name = "${var.name}-${each.key}" description = "Terraform-managed." dest_range = each.value.dest_range @@ -165,7 +177,7 @@ resource "google_compute_route" "gateway" { resource "google_compute_route" "ilb" { for_each = local.routes_ilb project = var.project_id - network = google_compute_network.network.name + network = local.network.name name = "${var.name}-${each.key}" description = "Terraform-managed." dest_range = each.value.dest_range @@ -177,7 +189,7 @@ resource "google_compute_route" "ilb" { resource "google_compute_route" "instance" { for_each = local.routes_instance project = var.project_id - network = google_compute_network.network.name + network = local.network.name name = "${var.name}-${each.key}" description = "Terraform-managed." dest_range = each.value.dest_range @@ -191,7 +203,7 @@ resource "google_compute_route" "instance" { resource "google_compute_route" "ip" { for_each = local.routes_ip project = var.project_id - network = google_compute_network.network.name + network = local.network.name name = "${var.name}-${each.key}" description = "Terraform-managed." dest_range = each.value.dest_range @@ -203,7 +215,7 @@ resource "google_compute_route" "ip" { resource "google_compute_route" "vpn_tunnel" { for_each = local.routes_vpn_tunnel project = var.project_id - network = google_compute_network.network.name + network = local.network.name name = "${var.name}-${each.key}" description = "Terraform-managed." dest_range = each.value.dest_range diff --git a/modules/net-vpc/outputs.tf b/modules/net-vpc/outputs.tf index 64649135..5dfe4066 100644 --- a/modules/net-vpc/outputs.tf +++ b/modules/net-vpc/outputs.tf @@ -16,17 +16,17 @@ output "network" { description = "Network resource." - value = google_compute_network.network + value = local.network } output "name" { description = "The name of the VPC being created." - value = google_compute_network.network.name + value = local.network.name } output "self_link" { description = "The URI of the VPC being created." - value = google_compute_network.network.self_link + value = local.network.self_link } output "project_id" { diff --git a/modules/net-vpc/variables.tf b/modules/net-vpc/variables.tf index d06eb401..d9ea1ee2 100644 --- a/modules/net-vpc/variables.tf +++ b/modules/net-vpc/variables.tf @@ -143,3 +143,9 @@ variable "subnet_private_access" { type = map(bool) default = {} } + +variable "vpc_create" { + description = "Create VPC. When set to false, uses a data source to reference existing VPC." + type = bool + default = true +} diff --git a/modules/net-vpn-ha/README.md b/modules/net-vpn-ha/README.md index 2357fb6b..959f7fa7 100644 --- a/modules/net-vpn-ha/README.md +++ b/modules/net-vpn-ha/README.md @@ -136,7 +136,7 @@ module "vpn_ha" { | name | description | type | required | default | |---|---|:---: |:---:|:---:| -| name | VPN gateway name, and prefix used for dependent resources. | string | ✓ | | +| name | VPN Gateway name (if an existing VPN Gateway is not used), and prefix used for dependent resources. | string | ✓ | | | network | VPC used for the gateway and routes. | string | ✓ | | | project_id | Project where resources will be created. | string | ✓ | | | region | Region used for resources. | string | ✓ | | @@ -146,16 +146,18 @@ module "vpn_ha" { | *router_advertise_config* | Router custom advertisement configuration, ip_ranges is a map of address ranges and descriptions. | object({...}) | | null | | *router_asn* | Router ASN used for auto-created router. | number | | 64514 | | *router_create* | Create router. | bool | | true | -| *router_name* | Router name used for auto created router, or to specify existing router to use. Leave blank to use VPN name for auto created router. | string | | | +| *router_name* | Router name used for auto created router, or to specify an existing router to use if `router_create` is set to `true`. Leave blank to use VPN name for auto created router. | string | | | | *tunnels* | VPN tunnel configurations, bgp_peer_options is usually null. | map(object({...})) | | {} | +| *vpn_gateway* | HA VPN Gateway Self Link for using an existing HA VPN Gateway, leave empty if `vpn_gateway_create` is set to `true`. | string | | null | +| *vpn_gateway_create* | Create HA VPN Gateway. | bool | | true | ## Outputs | name | description | sensitive | |---|---|:---:| | external_gateway | External VPN gateway resource. | | -| gateway | HA VPN gateway resource. | | -| name | VPN gateway name. | | +| gateway | VPN gateway resource (only if auto-created). | | +| name | VPN gateway name (only if auto-created). | | | random_secret | Generated secret. | ✓ | | router | Router resource (only if auto-created). | | | router_name | Router name. | | diff --git a/modules/net-vpn-ha/main.tf b/modules/net-vpn-ha/main.tf index 141bffc7..7f6e4fa0 100644 --- a/modules/net-vpn-ha/main.tf +++ b/modules/net-vpn-ha/main.tf @@ -27,11 +27,17 @@ locals { ? try(google_compute_router.router[0].name, null) : var.router_name ) + vpn_gateway = ( + var.vpn_gateway_create + ? try(google_compute_ha_vpn_gateway.ha_gateway[0].self_link, null) + : var.vpn_gateway + ) secret = random_id.secret.b64_url } resource "google_compute_ha_vpn_gateway" "ha_gateway" { provider = google-beta + count = var.vpn_gateway_create ? 1 : 0 name = var.name project = var.project_id region = var.region @@ -55,12 +61,11 @@ resource "google_compute_external_vpn_gateway" "external_gateway" { } resource "google_compute_router" "router" { - provider = google-beta - count = var.router_create ? 1 : 0 - name = var.router_name == "" ? "vpn-${var.name}" : var.router_name - project = var.project_id - region = var.region - network = var.network + count = var.router_create ? 1 : 0 + name = var.router_name == "" ? "vpn-${var.name}" : var.router_name + project = var.project_id + region = var.region + network = var.network bgp { advertise_mode = ( var.router_advertise_config == null @@ -135,7 +140,6 @@ resource "google_compute_router_peer" "bgp_peer" { } resource "google_compute_router_interface" "router_interface" { - provider = google-beta for_each = var.tunnels project = var.project_id region = var.region @@ -162,7 +166,7 @@ resource "google_compute_vpn_tunnel" "tunnels" { ? local.secret : each.value.shared_secret ) - vpn_gateway = google_compute_ha_vpn_gateway.ha_gateway.self_link + vpn_gateway = local.vpn_gateway } resource "random_id" "secret" { diff --git a/modules/net-vpn-ha/outputs.tf b/modules/net-vpn-ha/outputs.tf index 94269377..7227e8f3 100644 --- a/modules/net-vpn-ha/outputs.tf +++ b/modules/net-vpn-ha/outputs.tf @@ -1,4 +1,3 @@ - /** * Copyright 2019 Google LLC * @@ -16,8 +15,12 @@ */ output "gateway" { - description = "HA VPN gateway resource." - value = google_compute_ha_vpn_gateway.ha_gateway + description = "VPN gateway resource (only if auto-created)." + value = ( + var.vpn_gateway_create + ? google_compute_ha_vpn_gateway.ha_gateway[0] + : null + ) } output "external_gateway" { @@ -30,13 +33,21 @@ output "external_gateway" { } output "name" { - description = "VPN gateway name." - value = google_compute_ha_vpn_gateway.ha_gateway.name + description = "VPN gateway name (only if auto-created). " + value = ( + var.vpn_gateway_create + ? google_compute_ha_vpn_gateway.ha_gateway[0].name + : null + ) } output "router" { description = "Router resource (only if auto-created)." - value = var.router_name == "" ? google_compute_router.router[0] : null + value = ( + var.router_name == "" + ? google_compute_router.router[0] + : null + ) } output "router_name" { @@ -46,7 +57,7 @@ output "router_name" { output "self_link" { description = "HA VPN gateway self link." - value = google_compute_ha_vpn_gateway.ha_gateway.self_link + value = local.vpn_gateway } output "tunnels" { diff --git a/modules/net-vpn-ha/variables.tf b/modules/net-vpn-ha/variables.tf index 55f4ec89..81016add 100644 --- a/modules/net-vpn-ha/variables.tf +++ b/modules/net-vpn-ha/variables.tf @@ -15,10 +15,22 @@ */ variable "name" { - description = "VPN gateway name, and prefix used for dependent resources." + description = "VPN Gateway name (if an existing VPN Gateway is not used), and prefix used for dependent resources." type = string } +variable "vpn_gateway_create" { + description = "Create HA VPN Gateway." + type = bool + default = true +} + +variable "vpn_gateway" { + description = "HA VPN Gateway Self Link for using an existing HA VPN Gateway, leave empty if `vpn_gateway_create` is set to `true`." + type = string + default = null +} + variable "network" { description = "VPC used for the gateway and routes." type = string @@ -81,7 +93,7 @@ variable "router_create" { } variable "router_name" { - description = "Router name used for auto created router, or to specify existing router to use. Leave blank to use VPN name for auto created router." + description = "Router name used for auto created router, or to specify an existing router to use if `router_create` is set to `true`. Leave blank to use VPN name for auto created router." type = string default = "" } diff --git a/modules/organization/outputs.tf b/modules/organization/outputs.tf index 2a829c4d..f33d54f1 100644 --- a/modules/organization/outputs.tf +++ b/modules/organization/outputs.tf @@ -18,7 +18,7 @@ output "org_id" { description = "Organization id dependent on module resources." value = var.org_id depends_on = [ - google_organization_iam_audit_config, + google_organization_iam_audit_config.config, google_organization_iam_binding.authoritative, google_organization_iam_custom_role.roles, google_organization_iam_member.additive, diff --git a/networking/hub-and-spoke-peering/main.tf b/networking/hub-and-spoke-peering/main.tf index 686ef547..f77d49f6 100644 --- a/networking/hub-and-spoke-peering/main.tf +++ b/networking/hub-and-spoke-peering/main.tf @@ -149,10 +149,11 @@ module "vm-spoke-1" { region = var.region name = "spoke-1-test" network_interfaces = [{ - network = module.vpc-spoke-1.self_link, + network = module.vpc-spoke-1.self_link subnetwork = module.vpc-spoke-1.subnet_self_links["${var.region}/spoke-1-default"] - nat = false, + nat = false addresses = null + alias_ips = null }] metadata = { startup-script = local.vm-startup-script } service_account = module.service-account-gce.email @@ -166,10 +167,11 @@ module "vm-spoke-2" { region = var.region name = "spoke-2-test" network_interfaces = [{ - network = module.vpc-spoke-2.self_link, + network = module.vpc-spoke-2.self_link subnetwork = module.vpc-spoke-2.subnet_self_links["${var.region}/spoke-2-default"] - nat = false, + nat = false addresses = null + alias_ips = null }] metadata = { startup-script = local.vm-startup-script } service_account = module.service-account-gce.email diff --git a/networking/hub-and-spoke-vpn/main.tf b/networking/hub-and-spoke-vpn/main.tf index 1cad68be..78eea3aa 100644 --- a/networking/hub-and-spoke-vpn/main.tf +++ b/networking/hub-and-spoke-vpn/main.tf @@ -249,10 +249,11 @@ module "vm-spoke-1" { region = var.regions.b name = "spoke-1-test" network_interfaces = [{ - network = module.vpc-spoke-1.self_link, + network = module.vpc-spoke-1.self_link subnetwork = module.vpc-spoke-1.subnet_self_links["${var.regions.b}/spoke-1-b"] - nat = false, + nat = false addresses = null + alias_ips = null }] tags = ["ssh"] metadata = { startup-script = local.vm-startup-script } @@ -264,10 +265,11 @@ module "vm-spoke-2" { region = var.regions.b name = "spoke-2-test" network_interfaces = [{ - network = module.vpc-spoke-2.self_link, - subnetwork = module.vpc-spoke-2.subnet_self_links["${var.regions.b}/spoke-2-b"], - nat = false, + network = module.vpc-spoke-2.self_link + subnetwork = module.vpc-spoke-2.subnet_self_links["${var.regions.b}/spoke-2-b"] + nat = false addresses = null + alias_ips = null }] tags = ["ssh"] metadata = { startup-script = local.vm-startup-script } diff --git a/networking/ilb-next-hop/gateways.tf b/networking/ilb-next-hop/gateways.tf index 7684e130..9a0da360 100644 --- a/networking/ilb-next-hop/gateways.tf +++ b/networking/ilb-next-hop/gateways.tf @@ -32,13 +32,15 @@ module "gw" { network = module.vpc-left.self_link subnetwork = values(module.vpc-left.subnet_self_links)[0], nat = false, - addresses = null + addresses = null, + alias_ips = null }, { network = module.vpc-right.self_link subnetwork = values(module.vpc-right.subnet_self_links)[0], nat = false, - addresses = null + addresses = null, + alias_ips = null } ] tags = ["ssh"] diff --git a/networking/ilb-next-hop/vms.tf b/networking/ilb-next-hop/vms.tf index 6a8f22a0..d3fbc0f8 100644 --- a/networking/ilb-next-hop/vms.tf +++ b/networking/ilb-next-hop/vms.tf @@ -31,9 +31,10 @@ module "vm-left" { network_interfaces = [ { network = module.vpc-left.self_link - subnetwork = values(module.vpc-left.subnet_self_links)[0], - nat = false, + subnetwork = values(module.vpc-left.subnet_self_links)[0] + nat = false addresses = null + alias_ips = null } ] tags = ["ssh"] @@ -56,9 +57,10 @@ module "vm-right" { network_interfaces = [ { network = module.vpc-right.self_link - subnetwork = values(module.vpc-right.subnet_self_links)[0], - nat = false, + subnetwork = values(module.vpc-right.subnet_self_links)[0] + nat = false addresses = null + alias_ips = null } ] tags = ["ssh"] diff --git a/networking/onprem-google-access-dns/main.tf b/networking/onprem-google-access-dns/main.tf index 1e29defa..2d803673 100644 --- a/networking/onprem-google-access-dns/main.tf +++ b/networking/onprem-google-access-dns/main.tf @@ -187,10 +187,11 @@ module "vm-test" { region = var.region name = "test" network_interfaces = [{ - network = module.vpc.self_link, + network = module.vpc.self_link subnetwork = module.vpc.subnet_self_links["${var.region}/subnet"] - nat = false, + nat = false addresses = null + alias_ips = null }] metadata = { startup-script = local.vm-startup-script } service_account = module.service-account-gce.email @@ -250,8 +251,9 @@ module "vm-onprem" { network_interfaces = [{ network = module.vpc.name subnetwork = module.vpc.subnet_self_links["${var.region}/subnet"] - nat = true, + nat = true addresses = null + alias_ips = null }] service_account = module.service-account-onprem.email service_account_scopes = ["https://www.googleapis.com/auth/cloud-platform"] diff --git a/networking/shared-vpc-gke/main.tf b/networking/shared-vpc-gke/main.tf index 00bd89cc..9beef9aa 100644 --- a/networking/shared-vpc-gke/main.tf +++ b/networking/shared-vpc-gke/main.tf @@ -182,10 +182,11 @@ module "vm-bastion" { region = var.region name = "bastion" network_interfaces = [{ - network = module.vpc-shared.self_link, - subnetwork = lookup(module.vpc-shared.subnet_self_links, "${var.region}/gce", null), - nat = false, + network = module.vpc-shared.self_link + subnetwork = lookup(module.vpc-shared.subnet_self_links, "${var.region}/gce", null) + nat = false addresses = null + alias_ips = null }] instance_count = 1 tags = ["ssh"] diff --git a/tests/cloud_operations/quota_monitoring/__init__.py b/tests/cloud_operations/quota_monitoring/__init__.py new file mode 100644 index 00000000..6913f02e --- /dev/null +++ b/tests/cloud_operations/quota_monitoring/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tests/cloud_operations/quota_monitoring/fixture/cf/README b/tests/cloud_operations/quota_monitoring/fixture/cf/README new file mode 100644 index 00000000..e69de29b diff --git a/tests/cloud_operations/quota_monitoring/fixture/main.tf b/tests/cloud_operations/quota_monitoring/fixture/main.tf new file mode 100644 index 00000000..3f2810ae --- /dev/null +++ b/tests/cloud_operations/quota_monitoring/fixture/main.tf @@ -0,0 +1,22 @@ +/** + * Copyright 2020 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +module "test" { + source = "../../../../cloud-operations/quota-monitoring" + name = var.name + project_create = var.project_create + project_id = var.project_id +} diff --git a/tests/cloud_operations/quota_monitoring/fixture/variables.tf b/tests/cloud_operations/quota_monitoring/fixture/variables.tf new file mode 100644 index 00000000..ce52e598 --- /dev/null +++ b/tests/cloud_operations/quota_monitoring/fixture/variables.tf @@ -0,0 +1,38 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +variable "name" { + type = string + default = "dns-sd-test" +} + +variable "project_create" { + type = bool + default = true +} + +variable "project_id" { + type = string + default = "test" +} + +variable "region" { + type = string + default = "europe-west1" +} + +variable "zone_domain" { + type = string + default = "svc.example.org." +} diff --git a/tests/cloud_operations/quota_monitoring/test_plan.py b/tests/cloud_operations/quota_monitoring/test_plan.py new file mode 100644 index 00000000..7b195b5c --- /dev/null +++ b/tests/cloud_operations/quota_monitoring/test_plan.py @@ -0,0 +1,27 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import os +import pytest + + +FIXTURES_DIR = os.path.join(os.path.dirname(__file__), 'fixture') + + +def test_resources(e2e_plan_runner): + "Test that plan works and the numbers of resources is as expected." + modules, resources = e2e_plan_runner(FIXTURES_DIR) + assert len(modules) == 3 + assert len(resources) == 10 diff --git a/tests/data_solutions/cmek_via_centralized_kms/__init__.py b/tests/data_solutions/cmek_via_centralized_kms/__init__.py new file mode 100644 index 00000000..6913f02e --- /dev/null +++ b/tests/data_solutions/cmek_via_centralized_kms/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tests/data_solutions/cmek_via_centralized_kms/fixture/main.tf b/tests/data_solutions/cmek_via_centralized_kms/fixture/main.tf new file mode 100644 index 00000000..374460dc --- /dev/null +++ b/tests/data_solutions/cmek_via_centralized_kms/fixture/main.tf @@ -0,0 +1,21 @@ +/** + * Copyright 2020 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +module "test" { + source = "../../../../data-solutions/cmek-via-centralized-kms/" + billing_account = var.billing_account + root_node = var.root_node +} diff --git a/tests/data_solutions/cmek_via_centralized_kms/fixture/variables.tf b/tests/data_solutions/cmek_via_centralized_kms/fixture/variables.tf new file mode 100644 index 00000000..764f047d --- /dev/null +++ b/tests/data_solutions/cmek_via_centralized_kms/fixture/variables.tf @@ -0,0 +1,26 @@ +/** + * Copyright 2020 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "billing_account" { + type = string + default = "123456-123456-123456" +} + +variable "root_node" { + description = "The resource name of the parent Folder or Organization. Must be of the form folders/folder_id or organizations/org_id." + type = string + default = "folders/12345678" +} diff --git a/tests/data_solutions/cmek_via_centralized_kms/test_plan.py b/tests/data_solutions/cmek_via_centralized_kms/test_plan.py new file mode 100644 index 00000000..21514522 --- /dev/null +++ b/tests/data_solutions/cmek_via_centralized_kms/test_plan.py @@ -0,0 +1,27 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import os +import pytest + + +FIXTURES_DIR = os.path.join(os.path.dirname(__file__), 'fixture') + + +def test_resources(e2e_plan_runner): + "Test that plan works and the numbers of resources is as expected." + modules, resources = e2e_plan_runner(FIXTURES_DIR) + assert len(modules) == 7 + assert len(resources) == 22 diff --git a/tests/data_solutions/gc_to_bq_with_dataflow/__init__.py b/tests/data_solutions/gc_to_bq_with_dataflow/__init__.py new file mode 100644 index 00000000..6913f02e --- /dev/null +++ b/tests/data_solutions/gc_to_bq_with_dataflow/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tests/data_solutions/gc_to_bq_with_dataflow/fixture/main.tf b/tests/data_solutions/gc_to_bq_with_dataflow/fixture/main.tf new file mode 100644 index 00000000..cf6ce48f --- /dev/null +++ b/tests/data_solutions/gc_to_bq_with_dataflow/fixture/main.tf @@ -0,0 +1,23 @@ +/** + * Copyright 2020 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +module "test" { + source = "../../../../data-solutions/gcs-to-bq-with-dataflow/" + billing_account = var.billing_account + project_kms_name = var.project_kms_name + project_service_name = var.project_service_name + root_node = var.root_node +} diff --git a/tests/data_solutions/gc_to_bq_with_dataflow/fixture/variables.tf b/tests/data_solutions/gc_to_bq_with_dataflow/fixture/variables.tf new file mode 100644 index 00000000..0a2696ba --- /dev/null +++ b/tests/data_solutions/gc_to_bq_with_dataflow/fixture/variables.tf @@ -0,0 +1,36 @@ +/** + * Copyright 2020 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "billing_account" { + type = string + default = "123456-123456-123456" +} + +variable "root_node" { + description = "The resource name of the parent Folder or Organization. Must be of the form folders/folder_id or organizations/org_id." + type = string + default = "folders/12345678" +} + +variable "project_service_name" { + type = string + default = "project-srv" +} + +variable "project_kms_name" { + type = string + default = "project-kms" +} diff --git a/tests/data_solutions/gc_to_bq_with_dataflow/test_plan.py b/tests/data_solutions/gc_to_bq_with_dataflow/test_plan.py new file mode 100644 index 00000000..1828f7f4 --- /dev/null +++ b/tests/data_solutions/gc_to_bq_with_dataflow/test_plan.py @@ -0,0 +1,27 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import os +import pytest + + +FIXTURES_DIR = os.path.join(os.path.dirname(__file__), 'fixture') + + +def test_resources(e2e_plan_runner): + "Test that plan works and the numbers of resources is as expected." + modules, resources = e2e_plan_runner(FIXTURES_DIR) + assert len(modules) == 13 + assert len(resources) == 61 diff --git a/tests/modules/compute_vm/fixture/variables.tf b/tests/modules/compute_vm/fixture/variables.tf index b2a5c6ed..21603452 100644 --- a/tests/modules/compute_vm/fixture/variables.tf +++ b/tests/modules/compute_vm/fixture/variables.tf @@ -53,12 +53,17 @@ variable "network_interfaces" { internal = list(string) external = list(string) }) + alias_ips = list(object({ + ip_cidr_range = string + subnetwork_range_name = string + })) })) default = [{ network = "https://www.googleapis.com/compute/v1/projects/my-project/global/networks/default", subnetwork = "https://www.googleapis.com/compute/v1/projects/my-project/regions/europe-west1/subnetworks/default-default", nat = false, addresses = null + alias_ips = null }] } diff --git a/tests/modules/compute_vm/test_plan_interfaces.py b/tests/modules/compute_vm/test_plan_interfaces.py index 560be3cb..3711b481 100644 --- a/tests/modules/compute_vm/test_plan_interfaces.py +++ b/tests/modules/compute_vm/test_plan_interfaces.py @@ -26,6 +26,7 @@ def test_no_addresses(plan_runner): subnetwork = "https://www.googleapis.com/compute/v1/projects/my-project/regions/europe-west1/subnetworks/default-default", nat = false, addresses = {external=[], internal=[]} + alias_ips = null }] ''' _, resources = plan_runner( @@ -39,6 +40,7 @@ def test_internal_addresses(plan_runner): subnetwork = "https://www.googleapis.com/compute/v1/projects/my-project/regions/europe-west1/subnetworks/default-default", nat = false, addresses = {external=[], internal=["1.1.1.2", "1.1.1.3"]} + alias_ips = null }] ''' _, resources = plan_runner( @@ -53,6 +55,7 @@ def test_internal_addresses_nat(plan_runner): subnetwork = "https://www.googleapis.com/compute/v1/projects/my-project/regions/europe-west1/subnetworks/default-default", nat = true, addresses = {external=[], internal=["1.1.1.2", "1.1.1.3"]} + alias_ips = null }] ''' _, resources = plan_runner( @@ -67,6 +70,7 @@ def test_all_addresses(plan_runner): subnetwork = "https://www.googleapis.com/compute/v1/projects/my-project/regions/europe-west1/subnetworks/default-default", nat = true, addresses = {external=["2.2.2.2", "2.2.2.3"], internal=["1.1.1.2", "1.1.1.3"]} + alias_ips = null }] ''' _, resources = plan_runner( diff --git a/tests/modules/gcs/fixture/main.tf b/tests/modules/gcs/fixture/main.tf index c0f4b4cb..6275c616 100644 --- a/tests/modules/gcs/fixture/main.tf +++ b/tests/modules/gcs/fixture/main.tf @@ -17,12 +17,14 @@ module "test" { source = "../../../../modules/gcs" project_id = "my-project" - names = ["bucket-a", "bucket-b"] - prefix = var.prefix + uniform_bucket_level_access = var.uniform_bucket_level_access + force_destroy = var.force_destroy iam_members = var.iam_members iam_roles = var.iam_roles labels = var.labels - bucket_policy_only = var.bucket_policy_only - force_destroy = var.force_destroy + logging_config = var.logging_config + names = ["bucket-a", "bucket-b"] + prefix = var.prefix + retention_policies = var.retention_policies versioning = var.versioning } diff --git a/tests/modules/gcs/fixture/variables.tf b/tests/modules/gcs/fixture/variables.tf index 08e95396..f79485f8 100644 --- a/tests/modules/gcs/fixture/variables.tf +++ b/tests/modules/gcs/fixture/variables.tf @@ -14,7 +14,7 @@ * limitations under the License. */ -variable "bucket_policy_only" { +variable "uniform_bucket_level_access" { type = map(bool) default = { bucket-a = false } } @@ -39,11 +39,36 @@ variable "labels" { default = { environment = "test" } } +variable "logging_config" { + type = map(object({ + log_bucket = string + log_object_prefix = string + })) + default = { + bucket-a = { log_bucket = "foo", log_object_prefix = null } + } +} + variable "prefix" { type = string default = null } +variable "project_id" { + type = string + default = "my-project" +} + +variable "retention_policies" { + type = map(object({ + retention_period = number + is_locked = bool + })) + default = { + bucket-b = { retention_period = 5, is_locked = false } + } +} + variable "storage_class" { type = string default = "MULTI_REGIONAL" diff --git a/tests/modules/gcs/test_plan.py b/tests/modules/gcs/test_plan.py index c2a59343..051eb042 100644 --- a/tests/modules/gcs/test_plan.py +++ b/tests/modules/gcs/test_plan.py @@ -44,7 +44,7 @@ def test_prefix(plan_runner): def test_map_values(plan_runner): "Test that map values set the correct attributes on buckets." _, resources = plan_runner(FIXTURES_DIR) - bpo = dict((r['values']['name'], r['values']['bucket_policy_only']) + bpo = dict((r['values']['name'], r['values']['uniform_bucket_level_access']) for r in resources) assert bpo == {'bucket-a': False, 'bucket-b': True} force_destroy = dict((r['values']['name'], r['values']['force_destroy']) @@ -55,6 +55,18 @@ def test_map_values(plan_runner): assert versioning == { 'bucket-a': [{'enabled': True}], 'bucket-b': [{'enabled': False}] } + logging_config = dict((r['values']['name'], r['values']['logging']) + for r in resources) + assert logging_config == { + 'bucket-a': [{'log_bucket': 'foo'}], + 'bucket-b': [] + } + retention_policies = dict((r['values']['name'], r['values']['retention_policy']) + for r in resources) + assert retention_policies == { + 'bucket-a': [], + 'bucket-b': [{'is_locked': False, 'retention_period': 5}] + } for r in resources: assert r['values']['labels'] == { 'environment': 'test', 'location': 'eu', diff --git a/tests/modules/net_address/__init__.py b/tests/modules/net_address/__init__.py new file mode 100644 index 00000000..6913f02e --- /dev/null +++ b/tests/modules/net_address/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tests/modules/net_address/fixture/main.tf b/tests/modules/net_address/fixture/main.tf new file mode 100644 index 00000000..e10bf7d2 --- /dev/null +++ b/tests/modules/net_address/fixture/main.tf @@ -0,0 +1,24 @@ +/** + * Copyright 2020 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +module "test" { + source = "../../../../modules/net-address" + external_addresses = var.external_addresses + global_addresses = var.global_addresses + internal_addresses = var.internal_addresses + internal_addresses_config = var.internal_addresses_config + project_id = var.project_id +} diff --git a/tests/modules/net_address/fixture/outputs.tf b/tests/modules/net_address/fixture/outputs.tf new file mode 100644 index 00000000..77b8211f --- /dev/null +++ b/tests/modules/net_address/fixture/outputs.tf @@ -0,0 +1,19 @@ +/** + * Copyright 2020 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +output "module" { + value = module.test +} diff --git a/tests/modules/net_address/fixture/variables.tf b/tests/modules/net_address/fixture/variables.tf new file mode 100644 index 00000000..9d350819 --- /dev/null +++ b/tests/modules/net_address/fixture/variables.tf @@ -0,0 +1,47 @@ +/** + * Copyright 2020 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "external_addresses" { + type = map(string) + default = {} +} + +variable "global_addresses" { + type = list(string) + default = [] +} + +variable "internal_addresses" { + type = map(object({ + region = string + subnetwork = string + })) + default = {} +} + +variable "internal_addresses_config" { + type = map(object({ + address = string + purpose = string + tier = string + })) + default = {} +} + +variable "project_id" { + type = string + default = "my-project" +} diff --git a/tests/modules/net_address/test_plan.py b/tests/modules/net_address/test_plan.py new file mode 100644 index 00000000..968f05dc --- /dev/null +++ b/tests/modules/net_address/test_plan.py @@ -0,0 +1,70 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import os +import pytest + + +FIXTURES_DIR = os.path.join(os.path.dirname(__file__), 'fixture') + + +def test_external_addresses(plan_runner): + addresses = '{one = "europe-west1", two = "europe-west2"}' + _, resources = plan_runner(FIXTURES_DIR, external_addresses=addresses) + assert [r['values']['name'] for r in resources] == ['one', 'two'] + assert set(r['values']['address_type'] + for r in resources) == set(['EXTERNAL']) + assert [r['values']['region'] + for r in resources] == ['europe-west1', 'europe-west2'] + + +def test_global_addresses(plan_runner): + _, resources = plan_runner(FIXTURES_DIR, global_addresses='["one", "two"]') + assert [r['values']['name'] for r in resources] == ['one', 'two'] + assert set(r['values']['address_type'] for r in resources) == set([None]) + + +def test_internal_addresses(plan_runner): + addresses = ( + '{one = {region = "europe-west1", subnetwork = "foobar"}, ' + 'two = {region = "europe-west2", subnetwork = "foobarz"}}' + ) + _, resources = plan_runner(FIXTURES_DIR, internal_addresses=addresses) + assert [r['values']['name'] for r in resources] == ['one', 'two'] + assert set(r['values']['address_type'] + for r in resources) == set(['INTERNAL']) + assert [r['values']['region'] + for r in resources] == ['europe-west1', 'europe-west2'] + + +def test_internal_addresses_config(plan_runner): + addresses = ( + '{one = {region = "europe-west1", subnetwork = "foobar"}, ' + 'two = {region = "europe-west2", subnetwork = "foobarz"}}' + ) + config = ( + '{one = {address = "10.0.0.2", purpose = "SHARED_LOADBALANCER_VIP", ' + 'tier=null}}' + ) + _, resources = plan_runner(FIXTURES_DIR, + internal_addresses=addresses, + internal_addresses_config=config) + assert [r['values']['name'] for r in resources] == ['one', 'two'] + assert set(r['values']['address_type'] + for r in resources) == set(['INTERNAL']) + assert [r['values'].get('address') + for r in resources] == ['10.0.0.2', None] + assert [r['values'].get('purpose') + for r in resources] == ['SHARED_LOADBALANCER_VIP', None] diff --git a/tests/modules/net_vpc/test_plan_subnets.py b/tests/modules/net_vpc/test_plan_subnets.py index 473a1c34..2d14f6f5 100644 --- a/tests/modules/net_vpc/test_plan_subnets.py +++ b/tests/modules/net_vpc/test_plan_subnets.py @@ -58,7 +58,9 @@ def test_subnet_log_configs(plan_runner): for r in resources: if r['type'] != 'google_compute_subnetwork': continue - flow_logs[r['values']['name']] = r['values']['log_config'] + flow_logs[r['values']['name']] = [{key: config[key] for key in config.keys() + & {'aggregation_interval', 'flow_sampling', 'metadata'}} + for config in r['values']['log_config']] assert flow_logs == { # enable, override one default option 'a': [{