Merge pull request #137 from terraform-google-modules/asset_inventory
e2e example: Scheduled Cloud Asset Inventory Export to Bigquery
This commit is contained in:
commit
7ef84a5cd5
|
@ -3,6 +3,7 @@
|
|||
All notable changes to this project will be documented in this file.
|
||||
|
||||
## [Unreleased]
|
||||
- end to end example: `Scheduled Cloud Asset Inventory Export to Bigquery`
|
||||
|
||||
## [3.4.0] - 2020-09-24
|
||||
|
||||
|
|
|
@ -19,8 +19,7 @@ Currently available examples:
|
|||
- **foundations** - [single level hierarchy](./foundations/environments/) (environments), [multiple level hierarchy](./foundations/business-units/) (business units + environments)
|
||||
- **networking** - [hub and spoke via peering](./networking/hub-and-spoke-peering/), [hub and spoke via VPN](./networking/hub-and-spoke-vpn/), [DNS and Google Private Access for on-premises](./networking/onprem-google-access-dns/), [Shared VPC with GKE support](./networking/shared-vpc-gke/), [ILB as next hop](./networking/ilb-next-hop)
|
||||
- **data solutions** - [GCE/GCS CMEK via centralized Cloud KMS](./data-solutions/cmek-via-centralized-kms/), [Cloud Storage to Bigquery with Cloud Dataflow](./data-solutions/gcs-to-bq-with-dataflow/)
|
||||
- **cloud operations** - [Resource tracking and remediation via Cloud Asset feeds](.//cloud-operations/asset-inventory-feed-remediation), [Granular Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam), [Compute Engine quota monitoring](./cloud-operations/quota-monitoring)
|
||||
|
||||
- **cloud operations** - [Resource tracking and remediation via Cloud Asset feeds](.//cloud-operations/asset-inventory-feed-remediation), [Granular Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam), [Compute Engine quota monitoring](./cloud-operations/quota-monitoring), [Scheduled Cloud Asset Inventory Export to Bigquery](./cloud-operations/scheduled-asset-inventory-export-bq)
|
||||
For more information see the README files in the [foundations](./foundations/), [networking](./networking/), [data solutions](./data-solutions/) and [cloud operations](./cloud-operations/) folders.
|
||||
|
||||
## Modules
|
||||
|
|
|
@ -10,6 +10,12 @@ The example's feed tracks changes to Google Compute instances, and the Cloud Fun
|
|||
|
||||
<br clear="left">
|
||||
|
||||
## Scheduled Cloud Asset Inventory Export to Bigquery
|
||||
|
||||
<a href="./scheduled-asset-inventory-export-bq" title="Scheduled Cloud Asset Inventory Export to Bigquery"><img src="./scheduled-asset-inventory-export-bq/diagram.png" align="left" width="280px"></a> This [example](./scheduled-asset-inventory-export-bq) shows how to leverage the [Cloud Asset Inventory Exporting to Bigquery](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery) feature, to keep track of your organization's assets over time storing information in Bigquery. Data stored in Bigquery can then be used for different purposes like dashboarding or analysis.
|
||||
|
||||
<br clear="left">
|
||||
|
||||
## Granular Cloud DNS IAM via Service Directory
|
||||
|
||||
<a href="./dns-fine-grained-iam" title="Fine-grained Cloud DNS IAM with Service Directory"><img src="./dns-fine-grained-iam/diagram.png" align="left" width="280px"></a> This [example](./dns-fine-grained-iam) shows how to leverage [Service Directory](https://cloud.google.com/blog/products/networking/introducing-service-directory) and Cloud DNS Service Directory private zones, to implement fine-grained IAM controls on DNS. The example creates a Service Directory namespace, a Cloud DNS private zone that uses it as its authoritative source, service accounts with different levels of permissions, and VMs to test them.
|
||||
|
|
|
@ -0,0 +1,62 @@
|
|||
# Scheduled Cloud Asset Inventory Export to Bigquery
|
||||
|
||||
This example shows how to leverage [Cloud Asset Inventory Exporting to Bigquery](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery) feature to keep track of your project wide assets over time storing information in Bigquery.
|
||||
|
||||
The data stored in Bigquery can then be used for different purposes:
|
||||
|
||||
- dashboarding
|
||||
- analysis
|
||||
|
||||
The example uses export resources at the project level for ease of testing, in actual use a few changes are needed to operate at the resource hierarchy level:
|
||||
|
||||
- the export should be set at the folder or organization level
|
||||
- the `roles/cloudasset.viewer` on the service account should be set at the folder or organization level
|
||||
|
||||
The resources created in this example are shown in the high level diagram below:
|
||||
|
||||
<img src="diagram.png" width="640px">
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Ensure that you grant your account one of the following roles on your project, folder, or organization:
|
||||
|
||||
- Cloud Asset Viewer role (`roles/cloudasset.viewer`)
|
||||
- Owner primitive role (`roles/owner`)
|
||||
|
||||
## Running the example
|
||||
|
||||
Clone this repository, specify your variables in a `terraform.tvars` and then go through the following steps to create resources:
|
||||
|
||||
- `terraform init`
|
||||
- `terraform apply`
|
||||
|
||||
Once done testing, you can clean up resources by running `terraform destroy`. To persist state, check out the `backend.tf.sample` file.
|
||||
|
||||
## Testing the example
|
||||
|
||||
Once resources are created, you can run queries on the data you exported on Bigquery. [Here](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery#querying_an_asset_snapshot) you can find some example of queries you can run.
|
||||
|
||||
You can also create a dashboard connecting [Datalab](https://datastudio.google.com/) or any other BI tools of your choice to your Bigquery datase.
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---: |:---:|:---:|
|
||||
| billing_account | Billing account id used as default for new projects. | <code title="">string</code> | ✓ | |
|
||||
| cai_config | Cloud Asset inventory export config. | <code title="object({ bq_dataset = string bq_table = string })">object({...})</code> | ✓ | |
|
||||
| project_id | Project id that references existing project. | <code title="">string</code> | ✓ | |
|
||||
| root_node | The resource name of the parent Folder or Organization. Must be of the form folders/folder_id or organizations/org_id. | <code title="">string</code> | ✓ | |
|
||||
| *bundle_path* | Path used to write the intermediate Cloud Function code bundle. | <code title="">string</code> | | <code title="">./bundle.zip</code> |
|
||||
| *location* | Appe Engine location used in the example. | <code title="">string</code> | | <code title="">europe-west</code> |
|
||||
| *name* | Arbitrary string used to name created resources. | <code title="">string</code> | | <code title="">asset-inventory</code> |
|
||||
| *project_create* | Create project instead ofusing an existing one. | <code title="">bool</code> | | <code title="">true</code> |
|
||||
| *region* | Compute region used in the example. | <code title="">string</code> | | <code title="">europe-west1</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
| name | description | sensitive |
|
||||
|---|---|:---:|
|
||||
| bq-dataset | Bigquery instance details. | |
|
||||
| cloud-function | Cloud Function instance details. | |
|
||||
<!-- END TFDOC -->
|
|
@ -0,0 +1,23 @@
|
|||
# Copyright 2019 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# set a valid bucket below and rename this file to backend.tf
|
||||
|
||||
terraform {
|
||||
backend "gcs" {
|
||||
bucket = ""
|
||||
prefix = "fabric/operations/inventory"
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,112 @@
|
|||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
'''Cloud Function module to export data for a given day.
|
||||
|
||||
This module is designed to be plugged in a Cloud Function, attached to Cloud
|
||||
Scheduler trigger to create a Cloud Asset Inventory Export to BigQuery.
|
||||
|
||||
'''
|
||||
|
||||
import base64
|
||||
import datetime
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import warnings
|
||||
|
||||
import click
|
||||
|
||||
from google.api_core.exceptions import GoogleAPIError
|
||||
from google.cloud import asset_v1
|
||||
|
||||
import googleapiclient.discovery
|
||||
import googleapiclient.errors
|
||||
|
||||
|
||||
def _configure_logging(verbose=True):
|
||||
'''Basic logging configuration.
|
||||
Args:
|
||||
verbose: enable verbose logging
|
||||
'''
|
||||
level = logging.DEBUG if verbose else logging.INFO
|
||||
logging.basicConfig(level=level)
|
||||
warnings.filterwarnings('ignore', r'.*end user credentials.*', UserWarning)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option('--project', required=True, help='Project ID')
|
||||
@click.option('--bq-project', required=True, help='Bigquery project to use.')
|
||||
@click.option('--bq-dataset', required=True, help='Bigquery dataset to use.')
|
||||
@click.option('--bq-table', required=True, help='Bigquery table name to use.')
|
||||
@click.option('--read-time', required=False, help=(
|
||||
'Day to take an asset snapshot in \'YYYYMMDD\' format, uses current day '
|
||||
' as default. Export will run at midnight of the specified day.'))
|
||||
@click.option('--verbose', is_flag=True, help='Verbose output')
|
||||
def main_cli(project=None, bq_project=None, bq_dataset=None, bq_table=None,
|
||||
read_time=None, verbose=False):
|
||||
'''Trigger Cloud Asset inventory export to Bigquery. Data will be stored in
|
||||
the dataset specified on a dated table with the name specified.
|
||||
'''
|
||||
try:
|
||||
_main(project, bq_project, bq_dataset, bq_table, read_time, verbose)
|
||||
except RuntimeError:
|
||||
logging.exception('exception raised')
|
||||
|
||||
|
||||
def main(event, context):
|
||||
'Cloud Function entry point.'
|
||||
try:
|
||||
data = json.loads(base64.b64decode(event['data']).decode('utf-8'))
|
||||
print(data)
|
||||
_main(**data)
|
||||
# uncomment once https://issuetracker.google.com/issues/155215191 is fixed
|
||||
# except RuntimeError:
|
||||
# raise
|
||||
except Exception:
|
||||
logging.exception('exception in cloud function entry point')
|
||||
|
||||
|
||||
def _main(project=None, bq_project=None, bq_dataset=None, bq_table=None, read_time=None, verbose=False):
|
||||
'Module entry point used by cli and cloud function wrappers.'
|
||||
|
||||
_configure_logging(verbose)
|
||||
if not read_time:
|
||||
read_time = datetime.datetime.now()
|
||||
client = asset_v1.AssetServiceClient()
|
||||
parent = 'projects/%s' % project
|
||||
content_type = asset_v1.ContentType.RESOURCE
|
||||
output_config = asset_v1.OutputConfig()
|
||||
output_config.bigquery_destination.dataset = 'projects/%s/datasets/%s' % (
|
||||
bq_project, bq_dataset)
|
||||
output_config.bigquery_destination.table = '%s_%s' % (
|
||||
bq_table, read_time.strftime('%Y%m%d'))
|
||||
output_config.bigquery_destination.force = True
|
||||
try:
|
||||
response = client.export_assets(
|
||||
request={
|
||||
'parent': parent,
|
||||
'read_time': read_time,
|
||||
'content_type': content_type,
|
||||
'output_config': output_config
|
||||
}
|
||||
)
|
||||
except (GoogleAPIError, googleapiclient.errors.HttpError) as e:
|
||||
logging.debug('API Error: %s', e, exc_info=True)
|
||||
raise RuntimeError(
|
||||
'Error fetching Asset Inventory entries (project: %s)' % parent, e)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main_cli()
|
|
@ -0,0 +1,4 @@
|
|||
Click>=7.0
|
||||
google-api-python-client>=1.10.1
|
||||
google-cloud-monitoring>=1.1.0
|
||||
google-cloud-asset
|
Binary file not shown.
After Width: | Height: | Size: 27 KiB |
|
@ -0,0 +1,134 @@
|
|||
/**
|
||||
* Copyright 2020 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
###############################################################################
|
||||
# Projects #
|
||||
###############################################################################
|
||||
module "project" {
|
||||
source = "../../modules/project"
|
||||
name = var.project_id
|
||||
parent = var.root_node
|
||||
billing_account = var.billing_account
|
||||
project_create = var.project_create
|
||||
services = [
|
||||
"bigquery.googleapis.com",
|
||||
"cloudasset.googleapis.com",
|
||||
"compute.googleapis.com",
|
||||
"cloudfunctions.googleapis.com",
|
||||
"cloudbuild.googleapis.com",
|
||||
"cloudscheduler.googleapis.com",
|
||||
"pubsub.googleapis.com"
|
||||
]
|
||||
}
|
||||
|
||||
module "service-account" {
|
||||
source = "../../modules/iam-service-accounts"
|
||||
project_id = module.project.project_id
|
||||
names = ["${var.name}-cf"]
|
||||
iam_project_roles = {
|
||||
(var.project_id) = ["roles/cloudasset.viewer"]
|
||||
}
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Pub/Sub #
|
||||
###############################################################################
|
||||
module "pubsub" {
|
||||
source = "../../modules/pubsub"
|
||||
project_id = module.project.project_id
|
||||
name = var.name
|
||||
subscriptions = {
|
||||
"${var.name}-default" = null
|
||||
}
|
||||
# the Cloud Scheduler robot service account already has pubsub.topics.publish
|
||||
# at the project level via roles/cloudscheduler.serviceAgent
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Cloud Function #
|
||||
###############################################################################
|
||||
module "cf" {
|
||||
source = "../../modules/cloud-function"
|
||||
project_id = module.project.project_id
|
||||
name = var.name
|
||||
bucket_name = "${var.name}-${random_pet.random.id}"
|
||||
bucket_config = {
|
||||
location = var.region
|
||||
lifecycle_delete_age = null
|
||||
}
|
||||
bundle_config = {
|
||||
source_dir = "cf"
|
||||
output_path = var.bundle_path
|
||||
}
|
||||
service_account = module.service-account.email
|
||||
trigger_config = {
|
||||
event = "google.pubsub.topic.publish"
|
||||
resource = module.pubsub.topic.id
|
||||
retry = null
|
||||
}
|
||||
}
|
||||
|
||||
resource "random_pet" "random" {
|
||||
length = 1
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Cloud Scheduler #
|
||||
###############################################################################
|
||||
resource "google_app_engine_application" "app" {
|
||||
project = module.project.project_id
|
||||
location_id = var.location
|
||||
}
|
||||
|
||||
resource "google_cloud_scheduler_job" "job" {
|
||||
project = google_app_engine_application.app.project
|
||||
region = var.region
|
||||
name = "test-job"
|
||||
description = "test http job"
|
||||
schedule = "* 9 * * 1"
|
||||
time_zone = "Etc/UTC"
|
||||
|
||||
pubsub_target {
|
||||
attributes = {}
|
||||
topic_name = module.pubsub.topic.id
|
||||
data = base64encode(jsonencode({
|
||||
project = module.project.project_id
|
||||
bq_project = module.project.project_id
|
||||
bq_dataset = var.cai_config.bq_dataset
|
||||
bq_table = var.cai_config.bq_table
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Bigquery #
|
||||
###############################################################################
|
||||
module "bq" {
|
||||
source = "../../modules/bigquery-dataset"
|
||||
project_id = module.project.project_id
|
||||
id = var.cai_config.bq_dataset
|
||||
access_roles = {
|
||||
owner = { role = "OWNER", type = "user_by_email" }
|
||||
}
|
||||
access_identities = {
|
||||
owner = module.service-account.email
|
||||
}
|
||||
options = {
|
||||
default_table_expiration_ms = null
|
||||
default_partition_expiration_ms = null
|
||||
delete_contents_on_destroy = true
|
||||
}
|
||||
}
|
|
@ -0,0 +1,25 @@
|
|||
/**
|
||||
* Copyright 2020 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
output "bq-dataset" {
|
||||
description = "Bigquery instance details."
|
||||
value = module.bq.dataset
|
||||
}
|
||||
|
||||
output "cloud-function" {
|
||||
description = "Cloud Function instance details."
|
||||
value = module.cf.function
|
||||
}
|
|
@ -0,0 +1,69 @@
|
|||
/**
|
||||
* Copyright 2020 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
variable "billing_account" {
|
||||
description = "Billing account id used as default for new projects."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "bundle_path" {
|
||||
description = "Path used to write the intermediate Cloud Function code bundle."
|
||||
type = string
|
||||
default = "./bundle.zip"
|
||||
}
|
||||
|
||||
variable "cai_config" {
|
||||
description = "Cloud Asset inventory export config."
|
||||
type = object({
|
||||
bq_dataset = string
|
||||
bq_table = string
|
||||
})
|
||||
}
|
||||
|
||||
variable "location" {
|
||||
description = "Appe Engine location used in the example."
|
||||
type = string
|
||||
default = "europe-west"
|
||||
}
|
||||
|
||||
|
||||
variable "name" {
|
||||
description = "Arbitrary string used to name created resources."
|
||||
type = string
|
||||
default = "asset-inventory"
|
||||
}
|
||||
|
||||
variable "project_create" {
|
||||
description = "Create project instead ofusing an existing one."
|
||||
type = bool
|
||||
default = true
|
||||
}
|
||||
|
||||
variable "project_id" {
|
||||
description = "Project id that references existing project."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
description = "Compute region used in the example."
|
||||
type = string
|
||||
default = "europe-west1"
|
||||
}
|
||||
|
||||
variable "root_node" {
|
||||
description = "The resource name of the parent Folder or Organization. Must be of the form folders/folder_id or organizations/org_id."
|
||||
type = string
|
||||
}
|
|
@ -0,0 +1,17 @@
|
|||
# Copyright 2020 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
terraform {
|
||||
required_version = ">= 0.12.6"
|
||||
}
|
Loading…
Reference in New Issue