Merge pull request #137 from terraform-google-modules/asset_inventory

e2e example: Scheduled Cloud Asset Inventory Export to Bigquery

CHANGELOG.md

@@ -3,6 +3,7 @@
 All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
+ - end to end example: `Scheduled Cloud Asset Inventory Export to Bigquery`
 
 ## [3.4.0] - 2020-09-24
 

README.md

@@ -19,8 +19,7 @@ Currently available examples:
 - **foundations** - [single level hierarchy](./foundations/environments/) (environments), [multiple level hierarchy](./foundations/business-units/) (business units + environments)
 - **networking** - [hub and spoke via peering](./networking/hub-and-spoke-peering/), [hub and spoke via VPN](./networking/hub-and-spoke-vpn/), [DNS and Google Private Access for on-premises](./networking/onprem-google-access-dns/), [Shared VPC with GKE support](./networking/shared-vpc-gke/), [ILB as next hop](./networking/ilb-next-hop)
 - **data solutions** - [GCE/GCS CMEK via centralized Cloud KMS](./data-solutions/cmek-via-centralized-kms/), [Cloud Storage to Bigquery with Cloud Dataflow](./data-solutions/gcs-to-bq-with-dataflow/)
-- **cloud operations** - [Resource tracking and remediation via Cloud Asset feeds](.//cloud-operations/asset-inventory-feed-remediation), [Granular Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam), [Compute Engine quota monitoring](./cloud-operations/quota-monitoring)
-
+- **cloud operations** - [Resource tracking and remediation via Cloud Asset feeds](.//cloud-operations/asset-inventory-feed-remediation), [Granular Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam), [Compute Engine quota monitoring](./cloud-operations/quota-monitoring), [Scheduled Cloud Asset Inventory Export to Bigquery](./cloud-operations/scheduled-asset-inventory-export-bq)
 For more information see the README files in the [foundations](./foundations/), [networking](./networking/), [data solutions](./data-solutions/) and [cloud operations](./cloud-operations/) folders.
 
 ## Modules

cloud-operations/README.md

@@ -10,6 +10,12 @@ The example's feed tracks changes to Google Compute instances, and the Cloud Fun
 
 <br clear="left">
 
+## Scheduled Cloud Asset Inventory Export to Bigquery
+
+<a href="./scheduled-asset-inventory-export-bq" title="Scheduled Cloud Asset Inventory Export to Bigquery"><img src="./scheduled-asset-inventory-export-bq/diagram.png" align="left" width="280px"></a> This [example](./scheduled-asset-inventory-export-bq) shows how to leverage the [Cloud Asset Inventory Exporting to Bigquery](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery) feature, to keep track of your organization's assets over time storing information in Bigquery. Data stored in Bigquery can then be used for different purposes like dashboarding or analysis.
+
+<br clear="left">
+
 ## Granular Cloud DNS IAM via Service Directory
 
 <a href="./dns-fine-grained-iam" title="Fine-grained Cloud DNS IAM with Service Directory"><img src="./dns-fine-grained-iam/diagram.png" align="left" width="280px"></a> This [example](./dns-fine-grained-iam) shows how to leverage [Service Directory](https://cloud.google.com/blog/products/networking/introducing-service-directory) and Cloud DNS Service Directory private zones, to implement fine-grained IAM controls on DNS. The example creates a Service Directory namespace, a Cloud DNS private zone that uses it as its authoritative source, service accounts with different levels of permissions, and VMs to test them.

cloud-operations/scheduled-asset-inventory-export-bq/README.md

@@ -0,0 +1,62 @@
+# Scheduled Cloud Asset Inventory Export to Bigquery
+
+This example shows how to leverage [Cloud Asset Inventory Exporting to Bigquery](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery) feature to keep track of your project wide assets over time storing information in Bigquery.
+
+The data stored in Bigquery can then be used for different purposes:
+
+- dashboarding
+- analysis
+
+The example uses export resources at the project level for ease of testing, in actual use a few changes are needed to operate at the resource hierarchy level:
+
+- the export should be set at the folder or organization level
+- the `roles/cloudasset.viewer` on the service account should be set at the folder or organization level
+
+The resources created in this example are shown in the high level diagram below:
+
+<img src="diagram.png" width="640px">
+
+## Prerequisites
+
+Ensure that you grant your account one of the following roles on your project, folder, or organization:
+
+- Cloud Asset Viewer role (`roles/cloudasset.viewer`)
+- Owner primitive role (`roles/owner`)
+
+## Running the example
+
+Clone this repository, specify your variables in a `terraform.tvars` and then go through the following steps to create resources:
+
+- `terraform init`
+- `terraform apply`
+
+Once done testing, you can clean up resources by running `terraform destroy`. To persist state, check out the `backend.tf.sample` file.
+
+## Testing the example
+
+Once resources are created, you can run queries on the data you exported on Bigquery. [Here](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery#querying_an_asset_snapshot) you can find some example of queries you can run.
+
+You can also create a dashboard connecting [Datalab](https://datastudio.google.com/) or any other BI tools of your choice to your Bigquery datase.
+
+<!-- BEGIN TFDOC -->
+## Variables
+
+| name | description | type | required | default |
+|---|---|:---: |:---:|:---:|
+| billing_account | Billing account id used as default for new projects. | <code title="">string</code> | ✓ |  |
+| cai_config | Cloud Asset inventory export config. | <code title="object&#40;&#123;&#10;bq_dataset   &#61; string&#10;bq_table     &#61; string&#10;&#125;&#41;">object({...})</code> | ✓ |  |
+| project_id | Project id that references existing project. | <code title="">string</code> | ✓ |  |
+| root_node | The resource name of the parent Folder or Organization. Must be of the form folders/folder_id or organizations/org_id. | <code title="">string</code> | ✓ |  |
+| *bundle_path* | Path used to write the intermediate Cloud Function code bundle. | <code title="">string</code> |  | <code title="">./bundle.zip</code> |
+| *location* | Appe Engine location used in the example. | <code title="">string</code> |  | <code title="">europe-west</code> |
+| *name* | Arbitrary string used to name created resources. | <code title="">string</code> |  | <code title="">asset-inventory</code> |
+| *project_create* | Create project instead ofusing an existing one. | <code title="">bool</code> |  | <code title="">true</code> |
+| *region* | Compute region used in the example. | <code title="">string</code> |  | <code title="">europe-west1</code> |
+
+## Outputs
+
+| name | description | sensitive |
+|---|---|:---:|
+| bq-dataset | Bigquery instance details. |  |
+| cloud-function | Cloud Function instance details. |  |
+<!-- END TFDOC -->

cloud-operations/scheduled-asset-inventory-export-bq/backend.tf.sample

@@ -0,0 +1,23 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# set a valid bucket below and rename this file to backend.tf
+
+terraform {
+  backend "gcs" {
+    bucket = ""
+    prefix = "fabric/operations/inventory"
+  }
+}
+

cloud-operations/scheduled-asset-inventory-export-bq/cf/main.py

@@ -0,0 +1,112 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+'''Cloud Function module to export data for a given day.
+
+This module is designed to be plugged in a Cloud Function, attached to Cloud
+Scheduler trigger to create a Cloud Asset Inventory Export to BigQuery.
+
+'''
+
+import base64
+import datetime
+import json
+import logging
+import os
+import warnings
+
+import click
+
+from google.api_core.exceptions import GoogleAPIError
+from google.cloud import asset_v1
+
+import googleapiclient.discovery
+import googleapiclient.errors
+
+
+def _configure_logging(verbose=True):
+  '''Basic logging configuration.
+  Args:
+    verbose: enable verbose logging
+  '''
+  level = logging.DEBUG if verbose else logging.INFO
+  logging.basicConfig(level=level)
+  warnings.filterwarnings('ignore', r'.*end user credentials.*', UserWarning)
+
+
+@click.command()
+@click.option('--project', required=True, help='Project ID')
+@click.option('--bq-project', required=True, help='Bigquery project to use.')
+@click.option('--bq-dataset', required=True, help='Bigquery dataset to use.')
+@click.option('--bq-table', required=True, help='Bigquery table name to use.')
+@click.option('--read-time', required=False, help=(
+    'Day to take an asset snapshot in \'YYYYMMDD\' format, uses current day '
+    ' as default. Export will run at midnight of the specified day.'))
+@click.option('--verbose', is_flag=True, help='Verbose output')
+def main_cli(project=None, bq_project=None, bq_dataset=None, bq_table=None,
+             read_time=None, verbose=False):
+  '''Trigger Cloud Asset inventory export to Bigquery. Data will be stored in
+  the dataset specified on a dated table with the name specified.
+  '''
+  try:
+    _main(project, bq_project, bq_dataset, bq_table, read_time, verbose)
+  except RuntimeError:
+    logging.exception('exception raised')
+
+
+def main(event, context):
+  'Cloud Function entry point.'
+  try:
+    data = json.loads(base64.b64decode(event['data']).decode('utf-8'))
+    print(data)
+    _main(**data)
+  # uncomment once https://issuetracker.google.com/issues/155215191 is fixed
+  # except RuntimeError:
+  #  raise
+  except Exception:
+    logging.exception('exception in cloud function entry point')
+
+
+def _main(project=None, bq_project=None, bq_dataset=None, bq_table=None, read_time=None, verbose=False):
+  'Module entry point used by cli and cloud function wrappers.'
+
+  _configure_logging(verbose)
+  if not read_time:
+    read_time = datetime.datetime.now()
+  client = asset_v1.AssetServiceClient()
+  parent = 'projects/%s' % project
+  content_type = asset_v1.ContentType.RESOURCE
+  output_config = asset_v1.OutputConfig()
+  output_config.bigquery_destination.dataset = 'projects/%s/datasets/%s' % (
+      bq_project, bq_dataset)
+  output_config.bigquery_destination.table = '%s_%s' % (
+      bq_table, read_time.strftime('%Y%m%d'))
+  output_config.bigquery_destination.force = True
+  try:
+    response = client.export_assets(
+        request={
+            'parent': parent,
+            'read_time': read_time,
+            'content_type': content_type,
+            'output_config': output_config
+        }
+    )
+  except (GoogleAPIError, googleapiclient.errors.HttpError) as e:
+    logging.debug('API Error: %s', e, exc_info=True)
+    raise RuntimeError(
+        'Error fetching Asset Inventory entries (project: %s)' % parent, e)
+
+
+if __name__ == '__main__':
+  main_cli()

cloud-operations/scheduled-asset-inventory-export-bq/cf/requirements.txt

@@ -0,0 +1,4 @@
+Click>=7.0
+google-api-python-client>=1.10.1
+google-cloud-monitoring>=1.1.0
+google-cloud-asset

cloud-operations/scheduled-asset-inventory-export-bq/main.tf

@@ -0,0 +1,134 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+###############################################################################
+#                                Projects                                     #
+###############################################################################
+module "project" {
+  source          = "../../modules/project"
+  name            = var.project_id
+  parent          = var.root_node
+  billing_account = var.billing_account
+  project_create  = var.project_create
+  services = [
+    "bigquery.googleapis.com",
+    "cloudasset.googleapis.com",
+    "compute.googleapis.com",
+    "cloudfunctions.googleapis.com",
+    "cloudbuild.googleapis.com",
+    "cloudscheduler.googleapis.com",
+    "pubsub.googleapis.com"
+  ]
+}
+
+module "service-account" {
+  source     = "../../modules/iam-service-accounts"
+  project_id = module.project.project_id
+  names      = ["${var.name}-cf"]
+  iam_project_roles = {
+    (var.project_id) = ["roles/cloudasset.viewer"]
+  }
+}
+
+###############################################################################
+#                                Pub/Sub                                      #
+###############################################################################
+module "pubsub" {
+  source     = "../../modules/pubsub"
+  project_id = module.project.project_id
+  name       = var.name
+  subscriptions = {
+    "${var.name}-default" = null
+  }
+  # the Cloud Scheduler robot service account already has pubsub.topics.publish
+  # at the project level via roles/cloudscheduler.serviceAgent
+}
+
+###############################################################################
+#                             Cloud Function                                  #
+###############################################################################
+module "cf" {
+  source      = "../../modules/cloud-function"
+  project_id  = module.project.project_id
+  name        = var.name
+  bucket_name = "${var.name}-${random_pet.random.id}"
+  bucket_config = {
+    location             = var.region
+    lifecycle_delete_age = null
+  }
+  bundle_config = {
+    source_dir  = "cf"
+    output_path = var.bundle_path
+  }
+  service_account = module.service-account.email
+  trigger_config = {
+    event    = "google.pubsub.topic.publish"
+    resource = module.pubsub.topic.id
+    retry    = null
+  }
+}
+
+resource "random_pet" "random" {
+  length = 1
+}
+
+###############################################################################
+#                            Cloud Scheduler                                  #
+###############################################################################
+resource "google_app_engine_application" "app" {
+  project     = module.project.project_id
+  location_id = var.location
+}
+
+resource "google_cloud_scheduler_job" "job" {
+  project     = google_app_engine_application.app.project
+  region      = var.region
+  name        = "test-job"
+  description = "test http job"
+  schedule    = "* 9 * * 1"
+  time_zone   = "Etc/UTC"
+
+  pubsub_target {
+    attributes = {}
+    topic_name = module.pubsub.topic.id
+    data = base64encode(jsonencode({
+      project    = module.project.project_id
+      bq_project = module.project.project_id
+      bq_dataset = var.cai_config.bq_dataset
+      bq_table   = var.cai_config.bq_table
+    }))
+  }
+}
+
+###############################################################################
+#                                Bigquery                                     #
+###############################################################################
+module "bq" {
+  source     = "../../modules/bigquery-dataset"
+  project_id = module.project.project_id
+  id         = var.cai_config.bq_dataset
+  access_roles = {
+    owner = { role = "OWNER", type = "user_by_email" }
+  }
+  access_identities = {
+    owner = module.service-account.email
+  }
+  options = {
+    default_table_expiration_ms     = null
+    default_partition_expiration_ms = null
+    delete_contents_on_destroy      = true
+  }
+}

cloud-operations/scheduled-asset-inventory-export-bq/outputs.tf

@@ -0,0 +1,25 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+output "bq-dataset" {
+  description = "Bigquery instance details."
+  value       = module.bq.dataset
+}
+
+output "cloud-function" {
+  description = "Cloud Function instance details."
+  value       = module.cf.function
+}

cloud-operations/scheduled-asset-inventory-export-bq/variables.tf

@@ -0,0 +1,69 @@
+/**
+ * Copyright 2020 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+variable "billing_account" {
+  description = "Billing account id used as default for new projects."
+  type        = string
+}
+
+variable "bundle_path" {
+  description = "Path used to write the intermediate Cloud Function code bundle."
+  type        = string
+  default     = "./bundle.zip"
+}
+
+variable "cai_config" {
+  description = "Cloud Asset inventory export config."
+  type = object({
+    bq_dataset   = string
+    bq_table     = string
+  })
+}
+
+variable "location" {
+  description = "Appe Engine location used in the example."
+  type        = string
+  default     = "europe-west"
+}
+
+
+variable "name" {
+  description = "Arbitrary string used to name created resources."
+  type        = string
+  default     = "asset-inventory"
+}
+
+variable "project_create" {
+  description = "Create project instead ofusing an existing one."
+  type        = bool
+  default     = true
+}
+
+variable "project_id" {
+  description = "Project id that references existing project."
+  type        = string
+}
+
+variable "region" {
+  description = "Compute region used in the example."
+  type        = string
+  default     = "europe-west1"
+}
+
+variable "root_node" {
+  description = "The resource name of the parent Folder or Organization. Must be of the form folders/folder_id or organizations/org_id."
+  type        = string
+}

cloud-operations/scheduled-asset-inventory-export-bq/versions.tf

@@ -0,0 +1,17 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+terraform {
+  required_version = ">= 0.12.6"
+}