diff --git a/cloud-operations/scheduled-asset-inventory-export-bq/README.md b/cloud-operations/scheduled-asset-inventory-export-bq/README.md index ef8bca88..69c6e589 100644 --- a/cloud-operations/scheduled-asset-inventory-export-bq/README.md +++ b/cloud-operations/scheduled-asset-inventory-export-bq/README.md @@ -38,17 +38,29 @@ Once resources are created, you can run queries on the data you exported on Bigq You can also create a dashboard connecting [Datalab](https://datastudio.google.com/) or any other BI tools of your choice to your Bigquery datase. +## File exporter for JSON, CSV. + +Regular file-based exports of Cloud Asset Inventory may be useful for scale-out network dependency discovery like [Planet Exporter](https://github.com/williamchanrico/planet-exporter) or to update asset tracking or configuration management workflows. Bigquery supports multiple [export formats](https://cloud.google.com/bigquery/docs/exporting-data#export_formats_and_compression_types) and one may upload objects to Storage Bucket using provided Cloud Function. Specify `job.DestinationFormat` as defined in [documentation](https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.job.DestinationFormat.html), e.g. `NEWLINE_DELIMITED_JSON`. + +It helps to create custom [scheduled query](https://cloud.google.com/bigquery/docs/scheduling-queries#console) for to comply with downstream systems' fields, and to time it with CAI export into BQ for freshness. See [sample queries](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery-sample-queries). + +Note: Cloud Function's Service Account needs write-capable IAM role on `bucket`. + + ## Variables | name | description | type | required | default | |---|---|:---: |:---:|:---:| -| cai_config | Cloud Asset inventory export config. | object({...}) | ✓ | | +| cai_config | Cloud Asset Inventory export config. | object({...}) | ✓ | | | project_id | Project id that references existing project. | string | ✓ | | | *billing_account* | Billing account id used as default for new projects. | string | | null | | *bundle_path* | Path used to write the intermediate Cloud Function code bundle. | string | | ./bundle.zip | +| *bundle_path_cffile* | Path used to write the intermediate Cloud Function code bundle. | string | | ./bundle_cffile.zip | +| *file_config* | Optional BQ table as a file export function config. | object({...}) | | ... | | *location* | Appe Engine location used in the example. | string | | europe-west | | *name* | Arbitrary string used to name created resources. | string | | asset-inventory | +| *name_cffile* | Arbitrary string used to name created resources. | string | | cffile-exporter | | *project_create* | Create project instead ofusing an existing one. | bool | | true | | *region* | Compute region used in the example. | string | | europe-west1 | | *root_node* | The resource name of the parent folder or organization for project creation, in 'folders/folder_id' or 'organizations/org_id' format. | string | | null | diff --git a/cloud-operations/scheduled-asset-inventory-export-bq/variables.tf b/cloud-operations/scheduled-asset-inventory-export-bq/variables.tf index 83823b90..c2137d89 100644 --- a/cloud-operations/scheduled-asset-inventory-export-bq/variables.tf +++ b/cloud-operations/scheduled-asset-inventory-export-bq/variables.tf @@ -51,6 +51,13 @@ variable "file_config" { bq_dataset = string bq_table = string }) + default = { + bucket = null + filename = null + format = null + bq_dataset = null + bq_table = null + } } variable "location" { diff --git a/tests/cloud_operations/scheduled_asset_inventory_export_bq/test_plan.py b/tests/cloud_operations/scheduled_asset_inventory_export_bq/test_plan.py index f3d07509..484496a5 100644 --- a/tests/cloud_operations/scheduled_asset_inventory_export_bq/test_plan.py +++ b/tests/cloud_operations/scheduled_asset_inventory_export_bq/test_plan.py @@ -24,4 +24,4 @@ def test_resources(e2e_plan_runner): "Test that plan works and the numbers of resources is as expected." modules, resources = e2e_plan_runner(FIXTURES_DIR) assert len(modules) == 7 - assert len(resources) == 28 + assert len(resources) == 29