Merge branch 'master' into yuryninog-patch-1
This commit is contained in:
yuryninog
GitHub
commit
a3da60bc71
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
@ -50,7 +50,7 @@ steps:
|
|||
"factories/firewall-vpc-rules",
|
||||
"foundations",
|
||||
"modules",
|
||||
"networking"
|
||||
"networking",
|
||||
]
|
||||
|
||||
substitutions:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,4 @@
|
|||
|
||||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
@ -19,7 +18,7 @@ name: |
|
|||
on:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
- cron: '0 2 * * *'
|
||||
- cron: "0 2 * * *"
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
|
@ -44,4 +43,3 @@ jobs:
|
|||
else
|
||||
echo "No changes in last 24 hours"
|
||||
fi
|
||||
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
@ -12,7 +12,7 @@
|
|||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
name: 'Linting'
|
||||
name: "Linting"
|
||||
on:
|
||||
pull_request:
|
||||
branches:
|
||||
|
|
@ -30,7 +30,7 @@ jobs:
|
|||
- name: Set up Python
|
||||
uses: actions/setup-python@v2
|
||||
with:
|
||||
python-version: '3.9'
|
||||
python-version: "3.9"
|
||||
|
||||
- name: Set up Terraform
|
||||
uses: hashicorp/setup-terraform@v1
|
||||
|
|
@ -54,4 +54,10 @@ jobs:
|
|||
- name: Check documentation
|
||||
id: documentation
|
||||
run: |
|
||||
python3 tools/check_documentation.py cloud-operations/ data-solutions/ data-solutions/data-platform-foundations/ factories/ factories/firewall-vpc-rules/ foundations/ modules/ networking/
|
||||
python3 tools/check_documentation.py \
|
||||
cloud-operations \
|
||||
data-solutions \
|
||||
factories \
|
||||
foundations \
|
||||
modules \
|
||||
networking
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
@ -12,10 +12,10 @@
|
|||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
name: 'Tests'
|
||||
name: "Tests"
|
||||
on:
|
||||
schedule:
|
||||
- cron: '45 2 * * *'
|
||||
- cron: "45 2 * * *"
|
||||
pull_request:
|
||||
branches:
|
||||
- master
|
||||
|
|
@ -35,7 +35,7 @@ jobs:
|
|||
- name: Set up Python
|
||||
uses: actions/setup-python@v2
|
||||
with:
|
||||
python-version: '3.9'
|
||||
python-version: "3.9"
|
||||
|
||||
- name: Set up Terraform
|
||||
uses: hashicorp/setup-terraform@v1
|
||||
|
|
@ -64,7 +64,7 @@ jobs:
|
|||
- name: Set up Python
|
||||
uses: actions/setup-python@v2
|
||||
with:
|
||||
python-version: '3.9'
|
||||
python-version: "3.9"
|
||||
|
||||
- name: Set up Terraform
|
||||
uses: hashicorp/setup-terraform@v1
|
||||
|
|
@ -93,7 +93,7 @@ jobs:
|
|||
- name: Set up Python
|
||||
uses: actions/setup-python@v2
|
||||
with:
|
||||
python-version: '3.9'
|
||||
python-version: "3.9"
|
||||
|
||||
- name: Set up Terraform
|
||||
uses: hashicorp/setup-terraform@v1
|
||||
|
|
|
|||
36
CHANGELOG.md
36
CHANGELOG.md
|
|
@ -3,13 +3,40 @@
|
|||
All notable changes to this project will be documented in this file.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [1.0.1] - 2022-01-03
|
||||
|
||||
- remove lifecycle block from vpc sc perimeter resources
|
||||
|
||||
## [10.0.0] - 2021-12-31
|
||||
|
||||
- fix cases where bridge perimeter status resources are `null` in `vpc-sc` module
|
||||
- re-release 9.0.3 as a major release as it contains breaking changes
|
||||
- update hierarchical firewall resources to use the newer `google_compute_firewall_*` resources
|
||||
- **incompatible change** rename `firewall_policy_attachments` to `firewall_policy_association` in the `organization` and `folder` modules
|
||||
- **incompatible change** updated API for the `net-vpc-sc` module
|
||||
|
||||
## [9.0.3] - 2021-12-31
|
||||
|
||||
- update hierarchical firewall resources to use the newer `google_compute_firewall_*` resources
|
||||
- **incompatible change** rename `firewall_policy_attachments` to `firewall_policy_association` in the `organization` and `folder` modules
|
||||
- **incompatible change** updated API for the `net-vpc-sc` module
|
||||
|
||||
## [9.0.2] - 2021-12-22
|
||||
|
||||
- ignore description changes in firewall policy rule to avoid permadiff, add factory example to `folder` module documentation
|
||||
|
||||
## [9.0.0] - 2021-12-22
|
||||
|
||||
- new `cloud-run` module
|
||||
- added gVNIC support to `compute-vm` module
|
||||
- added a rule factory to `net-vpc-firewall` module
|
||||
- added a subnet factory to `net-vpc` module
|
||||
- **incompatible change** added support for partitioned tables to `organization` module sinks
|
||||
- **incompatible change** renamed `private_service_networking_range` variable to `psc_ranges` in `net-vpc`module, and changed its type to `list(string)`
|
||||
- added a firewall policy factory to `organization` module
|
||||
- added a firewall policy factory to `organization` and `firewall` module
|
||||
- refactored `tfdoc`
|
||||
- added support for metric scopes to the `project` module
|
||||
|
||||
## [8.0.0] - 2021-10-21
|
||||
|
||||
|
|
@ -381,7 +408,12 @@ All notable changes to this project will be documented in this file.
|
|||
|
||||
- merge development branch with suite of new modules and end-to-end examples
|
||||
|
||||
[Unreleased]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v8.0.0...HEAD
|
||||
[Unreleased]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v10.0.1...HEAD
|
||||
[10.0.1]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v10.0.0...v10.0.1
|
||||
[10.0.0]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v9.0.3...v10.0.0
|
||||
[9.0.3]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v9.0.2...v9.0.3
|
||||
[9.0.2]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v9.0.0...v9.0.2
|
||||
[9.0.0]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v8.0.0...v9.0.0
|
||||
[8.0.0]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v7.0.0...v8.0.0
|
||||
[7.0.0]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v6.0.0...v7.0.0
|
||||
[6.0.0]: https://github.com/terraform-google-modules/cloud-foundation-fabric/compare/v5.1.0...v6.0.0
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@ Currently available examples:
|
|||
- **foundations** - [single level hierarchy](./foundations/environments/) (environments), [multiple level hierarchy](./foundations/business-units/) (business units + environments)
|
||||
- **networking** - [hub and spoke via peering](./networking/hub-and-spoke-peering/), [hub and spoke via VPN](./networking/hub-and-spoke-vpn/), [DNS and Google Private Access for on-premises](./networking/onprem-google-access-dns/), [Shared VPC with GKE support](./networking/shared-vpc-gke/), [ILB as next hop](./networking/ilb-next-hop), [PSC for on-premises Cloud Function invocation](./networking/private-cloud-function-from-onprem/), [decentralized firewall](./networking/decentralized-firewall)
|
||||
- **data solutions** - [GCE/GCS CMEK via centralized Cloud KMS](./data-solutions/cmek-via-centralized-kms/), [Cloud Storage to Bigquery with Cloud Dataflow](./data-solutions/gcs-to-bq-with-dataflow/)
|
||||
- **cloud operations** - [Resource tracking and remediation via Cloud Asset feeds](.//cloud-operations/asset-inventory-feed-remediation), [Granular Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam), [Granular Cloud DNS IAM for Shared VPC](./cloud-operations/dns-shared-vpc), [Compute Engine quota monitoring](./cloud-operations/quota-monitoring), [Scheduled Cloud Asset Inventory Export to Bigquery](./cloud-operations/scheduled-asset-inventory-export-bq), [Packer image builder](./cloud-operations/packer-image-builder)
|
||||
- **cloud operations** - [Resource tracking and remediation via Cloud Asset feeds](.//cloud-operations/asset-inventory-feed-remediation), [Granular Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam), [Granular Cloud DNS IAM for Shared VPC](./cloud-operations/dns-shared-vpc), [Compute Engine quota monitoring](./cloud-operations/quota-monitoring), [Scheduled Cloud Asset Inventory Export to Bigquery](./cloud-operations/scheduled-asset-inventory-export-bq), [Packer image builder](./cloud-operations/packer-image-builder), [On-prem SA key management](./cloud-operations/onprem-sa-key-management)
|
||||
- **third party solutions** - [OpenShift cluster on Shared VPC](./third-party-solutions/openshift)
|
||||
- **factories** - [Example environments](./factories/example-environments), [Hierarchical Firewall Policies](./factories/firewall-hierarchical-policies), [VPC Firewall Rules](./factories/firewall-vpc-rules), [Subnets](./factories/subnets)
|
||||
|
||||
|
|
|
|||
|
|
@ -45,3 +45,10 @@ The example's feed tracks changes to Google Compute instances, and the Cloud Fun
|
|||
<a href="./packer-image-builder" title="Packer image builder"><img src="./packer-image-builder/diagram.png" align="left" width="280px"></a> This [example](./packer-image-builder) shows how to deploy infrastructure for a Compute Engine image builder based on [Hashicorp's Packer tool](https://www.packer.io).
|
||||
|
||||
<br clear="left">
|
||||
|
||||
## On-prem Service Account key management
|
||||
|
||||
|
||||
This [example](./onprem-sa-key-management) shows how to manage IAM Service Account Keys by manually generating a key pair and uploading the public part of the key to GCP.
|
||||
s
|
||||
<br clear="left">
|
||||
|
|
@ -52,16 +52,18 @@ Run the `subscription_pull` command until it returns nothing, then run the follo
|
|||
- the `tag_show` command to verify that the function output matches the resource state
|
||||
|
||||
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---: |:---:|:---:|
|
||||
| project_id | Project id that references existing project. | <code title="">string</code> | ✓ | |
|
||||
| *bundle_path* | Path used to write the intermediate Cloud Function code bundle. | <code title="">string</code> | | <code title="">./bundle.zip</code> |
|
||||
| *name* | Arbitrary string used to name created resources. | <code title="">string</code> | | <code title="">asset-feed</code> |
|
||||
| *project_create* | Create project instead of using an existing one. | <code title="">bool</code> | | <code title="">false</code> |
|
||||
| *region* | Compute region used in the example. | <code title="">string</code> | | <code title="">europe-west1</code> |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| project_id | Project id that references existing project. | <code>string</code> | ✓ | |
|
||||
| bundle_path | Path used to write the intermediate Cloud Function code bundle. | <code>string</code> | | <code>"./bundle.zip"</code> |
|
||||
| name | Arbitrary string used to name created resources. | <code>string</code> | | <code>"asset-feed"</code> |
|
||||
| project_create | Create project instead of using an existing one. | <code>bool</code> | | <code>false</code> |
|
||||
| region | Compute region used in the example. | <code>string</code> | | <code>"europe-west1"</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
|
|
@ -71,5 +73,7 @@ Run the `subscription_pull` command until it returns nothing, then run the follo
|
|||
| subscription_pull | Subscription pull command. | |
|
||||
| tag_add | Instance add tag command. | |
|
||||
| tag_show | Instance add tag command. | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -99,16 +99,18 @@ dig app1.svc.example.org +short
|
|||
# 127.0.0.7
|
||||
```
|
||||
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---: |:---:|:---:|
|
||||
| project_id | Existing project id. | <code title="">string</code> | ✓ | |
|
||||
| *name* | Arbitrary string used to name created resources. | <code title="">string</code> | | <code title="">dns-sd-test</code> |
|
||||
| *project_create* | Create project instead ofusing an existing one. | <code title="">bool</code> | | <code title="">false</code> |
|
||||
| *region* | Compute region used in the example. | <code title="">string</code> | | <code title="">europe-west1</code> |
|
||||
| *zone_domain* | Domain name used for the DNS zone. | <code title="">string</code> | | <code title="">svc.example.org.</code> |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| project_id | Existing project id. | <code>string</code> | ✓ | |
|
||||
| name | Arbitrary string used to name created resources. | <code>string</code> | | <code>"dns-sd-test"</code> |
|
||||
| project_create | Create project instead ofusing an existing one. | <code>bool</code> | | <code>false</code> |
|
||||
| region | Compute region used in the example. | <code>string</code> | | <code>"europe-west1"</code> |
|
||||
| zone_domain | Domain name used for the DNS zone. | <code>string</code> | | <code>"svc.example.org."</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
|
|
@ -116,4 +118,6 @@ dig app1.svc.example.org +short
|
|||
|---|---|:---:|
|
||||
| gcloud_commands | Commands used to SSH to the VMs. | |
|
||||
| vms | VM names. | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -19,22 +19,26 @@ The resources created in this example are shown in the high level diagram below:
|
|||
|
||||
Note that Terraform 0.13 at least is required due to the use of `for_each` with modules.
|
||||
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---: |:---:|:---:|
|
||||
| billing_account_id | Billing account associated with the GCP Projects that will be created for each team. | <code title="">string</code> | ✓ | |
|
||||
| folder_id | Folder ID in which DNS projects will be created. | <code title="">string</code> | ✓ | |
|
||||
| shared_vpc_link | Shared VPC self link, used for DNS peering. | <code title="">string</code> | ✓ | |
|
||||
| *dns_domain* | DNS domain under which each application team DNS domain will be created. | <code title="">string</code> | | <code title="">example.org</code> |
|
||||
| *prefix* | Customer name to use as prefix for resources' naming. | <code title="">string</code> | | <code title="">test-dns</code> |
|
||||
| *project_services* | Service APIs enabled by default. | <code title="list(string)">list(string)</code> | | <code title="[ "compute.googleapis.com", "dns.googleapis.com", ]">...</code> |
|
||||
| *teams* | List of application teams requiring their own Cloud DNS instance. | <code title="list(string)">list(string)</code> | | <code title="[ "team1", "team2", ]">...</code> |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| billing_account_id | Billing account associated with the GCP Projects that will be created for each team. | <code>string</code> | ✓ | |
|
||||
| folder_id | Folder ID in which DNS projects will be created. | <code>string</code> | ✓ | |
|
||||
| shared_vpc_link | Shared VPC self link, used for DNS peering. | <code>string</code> | ✓ | |
|
||||
| dns_domain | DNS domain under which each application team DNS domain will be created. | <code>string</code> | | <code>"example.org"</code> |
|
||||
| prefix | Customer name to use as prefix for resources' naming. | <code>string</code> | | <code>"test-dns"</code> |
|
||||
| project_services | Service APIs enabled by default. | <code>list(string)</code> | | <code title="[ "compute.googleapis.com", "dns.googleapis.com", ]">[…]</code> |
|
||||
| teams | List of application teams requiring their own Cloud DNS instance. | <code>list(string)</code> | | <code title="[ "team1", "team2", ]">[…]</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
| name | description | sensitive |
|
||||
|---|---|:---:|
|
||||
| teams | Team resources | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -64,18 +64,19 @@ If you get any warnings, check the roles and remove any of them granting any of
|
|||
- `resourcemanager.organizations.setIamPolicy`
|
||||
|
||||
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---: |:---:|:---:|
|
||||
| project_administrators | List identities granted administrator permissions. | <code title="list(string)">list(string)</code> | ✓ | |
|
||||
| project_id | GCP project id where to grant direct and delegated roles to the users listed in project_administrators. | <code title="">string</code> | ✓ | |
|
||||
| *delegated_role_grants* | List of roles that project administrators will be allowed to grant/revoke. | <code title="list(string)">list(string)</code> | | <code title="[ "roles/storage.admin", "roles/storage.hmacKeyAdmin", "roles/storage.legacyBucketOwner", "roles/storage.objectAdmin", "roles/storage.objectCreator", "roles/storage.objectViewer", "roles/compute.admin", "roles/compute.imageUser", "roles/compute.instanceAdmin", "roles/compute.instanceAdmin.v1", "roles/compute.networkAdmin", "roles/compute.networkUser", "roles/compute.networkViewer", "roles/compute.orgFirewallPolicyAdmin", "roles/compute.orgFirewallPolicyUser", "roles/compute.orgSecurityPolicyAdmin", "roles/compute.orgSecurityPolicyUser", "roles/compute.orgSecurityResourceAdmin", "roles/compute.osAdminLogin", "roles/compute.osLogin", "roles/compute.osLoginExternalUser", "roles/compute.packetMirroringAdmin", "roles/compute.packetMirroringUser", "roles/compute.publicIpAdmin", "roles/compute.securityAdmin", "roles/compute.serviceAgent", "roles/compute.storageAdmin", "roles/compute.viewer", "roles/viewer" ]">...</code> |
|
||||
| *direct_role_grants* | List of roles granted directly to project administrators. | <code title="list(string)">list(string)</code> | | <code title="[ "roles/compute.admin", "roles/storage.admin", ]">...</code> |
|
||||
| *project_create* | Create project instead of using an existing one. | <code title="">bool</code> | | <code title="">false</code> |
|
||||
| *restricted_role_grant* | Role grant to which the restrictions will apply. | <code title="">string</code> | | <code title="">roles/resourcemanager.projectIamAdmin</code> |
|
||||
|
||||
## Outputs
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| project_administrators | List identities granted administrator permissions. | <code>list(string)</code> | ✓ | |
|
||||
| project_id | GCP project id where to grant direct and delegated roles to the users listed in project_administrators. | <code>string</code> | ✓ | |
|
||||
| delegated_role_grants | List of roles that project administrators will be allowed to grant/revoke. | <code>list(string)</code> | | <code title="[ "roles/storage.admin", "roles/storage.hmacKeyAdmin", "roles/storage.legacyBucketOwner", "roles/storage.objectAdmin", "roles/storage.objectCreator", "roles/storage.objectViewer", "roles/compute.admin", "roles/compute.imageUser", "roles/compute.instanceAdmin", "roles/compute.instanceAdmin.v1", "roles/compute.networkAdmin", "roles/compute.networkUser", "roles/compute.networkViewer", "roles/compute.orgFirewallPolicyAdmin", "roles/compute.orgFirewallPolicyUser", "roles/compute.orgSecurityPolicyAdmin", "roles/compute.orgSecurityPolicyUser", "roles/compute.orgSecurityResourceAdmin", "roles/compute.osAdminLogin", "roles/compute.osLogin", "roles/compute.osLoginExternalUser", "roles/compute.packetMirroringAdmin", "roles/compute.packetMirroringUser", "roles/compute.publicIpAdmin", "roles/compute.securityAdmin", "roles/compute.serviceAgent", "roles/compute.storageAdmin", "roles/compute.viewer", "roles/viewer" ]">[…]</code> |
|
||||
| direct_role_grants | List of roles granted directly to project administrators. | <code>list(string)</code> | | <code title="[ "roles/compute.admin", "roles/storage.admin", ]">[…]</code> |
|
||||
| project_create | Create project instead of using an existing one. | <code>bool</code> | | <code>false</code> |
|
||||
| restricted_role_grant | Role grant to which the restrictions will apply. | <code>string</code> | | <code>"roles/resourcemanager.projectIamAdmin"</code> |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
@ -28,50 +28,50 @@ SENSITIVE_PERMISSIONS = {
|
|||
|
||||
|
||||
def get_role_permissions(role):
|
||||
if role.startswith("roles/"):
|
||||
endpoint = iam_service.roles()
|
||||
elif role.startswith("projects/"):
|
||||
endpoint = iam_service.projects().roles()
|
||||
elif role.startswith("organizations/"):
|
||||
endpoint = iam_service.organizations().roles()
|
||||
else:
|
||||
raise Exception(f"Invalid role {role}")
|
||||
if role.startswith("roles/"):
|
||||
endpoint = iam_service.roles()
|
||||
elif role.startswith("projects/"):
|
||||
endpoint = iam_service.projects().roles()
|
||||
elif role.startswith("organizations/"):
|
||||
endpoint = iam_service.organizations().roles()
|
||||
else:
|
||||
raise Exception(f"Invalid role {role}")
|
||||
|
||||
response = endpoint.get(name=role).execute()
|
||||
permissions = response.get("includedPermissions")
|
||||
return permissions
|
||||
response = endpoint.get(name=role).execute()
|
||||
permissions = response.get("includedPermissions")
|
||||
return permissions
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.argument("file", type=click.File("r"))
|
||||
def main(file):
|
||||
"""Verify that the set of GCP roles in FILE does not include the
|
||||
permission setIamPolicy at project, folder or organization level
|
||||
"""Verify that the set of GCP roles in FILE does not include the
|
||||
permission setIamPolicy at project, folder or organization level
|
||||
|
||||
This program authenticates against GCP using default application
|
||||
credentials to query project and organization level roles.
|
||||
This program authenticates against GCP using default application
|
||||
credentials to query project and organization level roles.
|
||||
|
||||
"""
|
||||
clean_roles = [x.rstrip(" \n") for x in file]
|
||||
roles = (x for x in clean_roles if x)
|
||||
"""
|
||||
clean_roles = [x.rstrip(" \n") for x in file]
|
||||
roles = (x for x in clean_roles if x)
|
||||
|
||||
allok = True
|
||||
for role in roles:
|
||||
try:
|
||||
permissions = set(get_role_permissions(role))
|
||||
except Error as e:
|
||||
print(f"WARNING: can't read {role}: {e}")
|
||||
allok = False
|
||||
else:
|
||||
matched_sensitive_permissions = SENSITIVE_PERMISSIONS & permissions
|
||||
if matched_sensitive_permissions:
|
||||
print(f"WARNING: {role} contains {matched_sensitive_permissions}")
|
||||
allok = False
|
||||
else:
|
||||
print(f"{role} ok")
|
||||
allok = True
|
||||
for role in roles:
|
||||
try:
|
||||
permissions = set(get_role_permissions(role))
|
||||
except Error as e:
|
||||
print(f"WARNING: can't read {role}: {e}")
|
||||
allok = False
|
||||
else:
|
||||
matched_sensitive_permissions = SENSITIVE_PERMISSIONS & permissions
|
||||
if matched_sensitive_permissions:
|
||||
print(f"WARNING: {role} contains {matched_sensitive_permissions}")
|
||||
allok = False
|
||||
else:
|
||||
print(f"{role} ok")
|
||||
|
||||
exit(0 if allok else 1)
|
||||
exit(0 if allok else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
main()
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,83 @@
|
|||
# Managing on-prem service account keys by uploading public keys
|
||||
|
||||
When managing GCP Service Accounts with terraform, it's often a question on **how to avoid Service Account Key in the terraform state?**
|
||||
|
||||
This example shows how to manage IAM Service Account Keys by manually generating a key pair and uploading the public part of the key to GCP. It has the following benefits:
|
||||
|
||||
- no [passing keys between users](https://cloud.google.com/iam/docs/best-practices-for-managing-service-account-keys#pass-between-users) or systems
|
||||
- no private keys stored in the terraform state (only public part of the key is in the state)
|
||||
- let keys [expire automatically](https://cloud.google.com/iam/docs/best-practices-for-managing-service-account-keys#key-expiryhaving)
|
||||
|
||||
|
||||
## Running the example
|
||||
|
||||
Clone this repository or [open it in cloud shell](https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2Fterraform-google-modules%2Fcloud-foundation-fabric&cloudshell_print=cloud-shell-readme.txt&cloudshell_working_dir=cloud-operations%2Fonprem-sa-key-management&cloudshell_open_in_editor=cloudshell_open%2Fcloud-foundation-fabric%2Fcloud-operations%2Fonprem-sa-key-management%2Fvariables.tf), then go through the following steps to create resources:
|
||||
|
||||
Cleaning up example keys
|
||||
```bash
|
||||
rm -f /public-keys/data-uploader/
|
||||
rm -f /public-keys/prisma-security/
|
||||
```
|
||||
|
||||
Generate keys for service accounts
|
||||
```bash
|
||||
mkdir keys && cd keys
|
||||
openssl req -x509 -nodes -newkey rsa:2048 -days 30 \
|
||||
-keyout data_uploader_private_key.pem \
|
||||
-out ../public-keys/data-uploader/public_key.pem \
|
||||
-subj "/CN=unused"
|
||||
openssl req -x509 -nodes -newkey rsa:2048 -days 30 \
|
||||
-keyout prisma_security_private_key.pem \
|
||||
-out ../public-keys/prisma-security/public_key.pem \
|
||||
-subj "/CN=unused"
|
||||
```
|
||||
|
||||
Deploy service accounts and keys
|
||||
```bash
|
||||
cd ..
|
||||
terraform init
|
||||
terraform apply -var project_id=$GOOGLE_CLOUD_PROJECT
|
||||
|
||||
```
|
||||
|
||||
Extract JSON credentials templates from terraform output and put the private part of the keys into templates
|
||||
```bash
|
||||
terraform show -json | jq '.values.outputs."sa-credentials".value."data-uploader"."public_key.pem" | fromjson' > data-uploader.json
|
||||
terraform show -json | jq '.values.outputs."sa-credentials".value."prisma-security"."public_key.pem" | fromjson' > prisma-security.json
|
||||
|
||||
contents=$(jq --arg key "$(cat keys/data_uploader_private_key.pem)" '.private_key=$key' data-uploader.json) && echo "$contents" > data-uploader.json
|
||||
contents=$(jq --arg key "$(cat keys/prisma_security_private_key.pem)" '.private_key=$key' prisma-security.json) && echo "$contents" > prisma-security.json
|
||||
```
|
||||
|
||||
## Testing the example
|
||||
Validate that service accounts json credentials are valid
|
||||
```bash
|
||||
gcloud auth activate-service-account --key-file prisma-security.json
|
||||
gcloud auth activate-service-account --key-file data-uploader.json
|
||||
```
|
||||
|
||||
## Cleaning up
|
||||
```bash
|
||||
terraform destroy -var project_id=$GOOGLE_CLOUD_PROJECT
|
||||
```
|
||||
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| project_id | Project id. | <code>string</code> | ✓ | |
|
||||
| project_create | Create project instead of using an existing one. | <code>bool</code> | | <code>false</code> |
|
||||
| service_accounts | List of service accounts. | <code title="list(object({ name = string iam_project_roles = list(string) public_keys_path = string }))">list(object({…}))</code> | | <code title="[ { name = "data-uploader" iam_project_roles = [ "roles/bigquery.dataOwner", "roles/bigquery.jobUser", "roles/storage.objectAdmin" ] public_keys_path = "public-keys/data-uploader/" }, { name = "prisma-security" iam_project_roles = [ "roles/iam.securityReviewer" ] public_keys_path = "public-keys/prisma-security/" }, ]">[…]</code> |
|
||||
| services | Service APIs to enable. | <code>list(string)</code> | | <code>[]</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
| name | description | sensitive |
|
||||
|---|---|:---:|
|
||||
| sa-credentials | SA json key templates. | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# set a valid bucket below and rename this file to backend.tf
|
||||
|
||||
terraform {
|
||||
backend "gcs" {
|
||||
bucket = ""
|
||||
prefix = "fabric/operations/onprem-sa-key-management"
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
|
||||
|
||||
################################# Quickstart #################################
|
||||
|
||||
# cleaning up example keys
|
||||
|
||||
- rm -f /public-keys/data-uploader/
|
||||
- rm -f /public-keys/prisma-security/
|
||||
|
||||
# generate keys for service accounts
|
||||
|
||||
- mkdir keys && cd keys
|
||||
- openssl req -x509 -nodes -newkey rsa:2048 -days 30 \
|
||||
-keyout data_uploader_private_key.pem \
|
||||
-out ../public-keys/data-uploader/public_key.pem \
|
||||
-subj "/CN=unused"
|
||||
- openssl req -x509 -nodes -newkey rsa:2048 -days 30 \
|
||||
-keyout prisma_security_private_key.pem \
|
||||
-out ../public-keys/prisma-security/public_key.pem \
|
||||
-subj "/CN=unused"
|
||||
|
||||
# deploy service accounts and keys
|
||||
|
||||
- cd ..
|
||||
- terraform init
|
||||
- terraform apply -var project_id=$GOOGLE_CLOUD_PROJECT
|
||||
|
||||
|
||||
# extract JSON credentials templates from terraform output and put the private part of the keys into templates
|
||||
|
||||
- terraform show -json | jq '.values.outputs."sa-credentials".value."data-uploader"."public_key.pem" | fromjson' > data-uploader.json
|
||||
- terraform show -json | jq '.values.outputs."sa-credentials".value."prisma-security"."public_key.pem" | fromjson' > prisma-security.json
|
||||
|
||||
- contents=$(jq --arg key "$(cat keys/data_uploader_private_key.pem)" '.private_key=$key' data-uploader.json) && echo "$contents" > data-uploader.json
|
||||
- contents=$(jq --arg key "$(cat keys/prisma_security_private_key.pem)" '.private_key=$key' prisma-security.json) && echo "$contents" > prisma-security.json
|
||||
|
||||
|
||||
|
||||
# validate that service accounts json credentials are valid
|
||||
|
||||
- gcloud auth activate-service-account --key-file prisma-security.json
|
||||
- gcloud auth activate-service-account --key-file data-uploader.json
|
||||
|
||||
|
||||
# cleaning up
|
||||
- terraform destroy -var project_id=$GOOGLE_CLOUD_PROJECT
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
locals {
|
||||
service_accounts = { for sa in var.service_accounts : sa.name => sa }
|
||||
}
|
||||
|
||||
module "project" {
|
||||
source = "../../modules/project"
|
||||
name = var.project_id
|
||||
project_create = var.project_create
|
||||
services = var.services
|
||||
}
|
||||
|
||||
module "integration-sa" {
|
||||
source = "../../modules/iam-service-account"
|
||||
for_each = local.service_accounts
|
||||
project_id = module.project.project_id
|
||||
name = each.value.name
|
||||
iam_project_roles = {
|
||||
(module.project.project_id) = each.value.iam_project_roles
|
||||
}
|
||||
public_keys_directory = each.value.public_keys_path
|
||||
}
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
output "sa-credentials" {
|
||||
description = "SA json key templates."
|
||||
value = { for key, value in module.integration-sa : key => value.service_account_credentials }
|
||||
}
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
-----BEGIN CERTIFICATE-----
|
||||
MIICnjCCAYYCCQDhgw8htVCGmTANBgkqhkiG9w0BAQsFADARMQ8wDQYDVQQDDAZ1
|
||||
bnVzZWQwHhcNMjExMjA2MDgwNTAyWhcNMzExMjA0MDgwNTAyWjARMQ8wDQYDVQQD
|
||||
DAZ1bnVzZWQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC0xlwdjkBS
|
||||
1ovANJ1RXKpFdbPQWYlqKUUo+/KLClNYC9KxRqrc+u5FtPIdCPv5WRH5sz+z8gcf
|
||||
3zJMht0dO7fOwJ9wSDKzvHkMUdXTGBPbm2i9PNA6f+YEwJQjWJlAHFH4Lp3x6ddT
|
||||
4KO4FRQEkN/5V1+sfmyGGFaSXaoi+PcDcQHvfUUlp5iyX4I+8tqwh1kdg1M5orkE
|
||||
7iBG0wHWzfOSmZq5in6t9+lWzOZeYapi8bVBm7Vz+dmHZPKS6EGmAXS1wpLCSKHB
|
||||
uv23KXY4gAXOPHiDI70JpeNiSJBE9WgXs+nL78vNjLTvDhpC10b9nOxLjRc6wA5b
|
||||
3q2Am0dW1DPRAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAJqyTIibNZM/q30Fn+vR
|
||||
V9q++19CIervZig1uCarH1M86cpPYRfKcYHOi6tnoCTL9VG8Ky8pbmkZNkET7vnN
|
||||
OQirpsPmqu3d+FBoqXUt8w1mT1JVr0YiTo3i07zTH8rvQKHjEfPxR73IAyYNvJ3D
|
||||
k3SdUvU3xXOa+otOQcBKIxX6mJPLhzXgZd144KCfD95qOvpoQOsNW4UWXZ3sPC0k
|
||||
VcMlN5O8/+D65y63nNtyECXvLicLdn/cdpA2H7Tqhz2ZZR+6tLcDW1kSsA8b6+rQ
|
||||
1IaKpF+TYo0jMD+WLatRrOHXOWije8871zooAXq9MLVJrT889TdsmEIYT7YPWIeJ
|
||||
Jcg=
|
||||
-----END CERTIFICATE-----
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
-----BEGIN CERTIFICATE-----
|
||||
MIICnjCCAYYCCQDXMv59IiZqfTANBgkqhkiG9w0BAQsFADARMQ8wDQYDVQQDDAZ1
|
||||
bnVzZWQwHhcNMjExMjA2MDgwNjE3WhcNMzExMjA0MDgwNjE3WjARMQ8wDQYDVQQD
|
||||
DAZ1bnVzZWQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDOK6XwgTzL
|
||||
icSITBrQBmhnYNOuggDhQr40j8/pIuTOiFZbd+ne3MhcFxpE58T9cOXgR0i/S4ok
|
||||
+kcGE74H2U7RsRpNi7fJhi62T9e2CXpibURQNJD6y0lXBQkfx6kCrhyvXqHbTxm5
|
||||
J0f5mpLlze+w7ATikmYI0mrU9XjtnRJOdxtGfiIaQ/suGTaZ0z4tZgAXy9RnwUAb
|
||||
LPXn0BD1+GYpCs82+1q7HpMIf343VRH0AdsQJteQSj5LKfaZZTNUF9NIgKtMylck
|
||||
z0Pt8TmBU0GtJX/XkSWCwMUdqdedXkvhY1XoAZPjaEBSSwq6P15PmdpDq9q2TYjD
|
||||
8U3kuCX0AlWjAgMBAAEwDQYJKoZIhvcNAQELBQADggEBADAmjI1sg150MK97DCSl
|
||||
d5OpEShCypaEZSLb/mFONW6mTX2OSdF9ipd9B07BQ2DrL8Xou2/V1aDtQZOWPIGu
|
||||
Hlm1LKw8sZY2rWX0Rq/v/NxY5iGRlwPMh7Rn9fnpHgaC1PktoDJEcvNMpzBjtfKn
|
||||
beKP9MNSChAFTTbJVWO5xT/ljE/yoPL3jyJKzKHH7y7AfbonrbQjAENbX/WCRYh3
|
||||
zOEWZG/fusRcKkZ/cO7wFFP1gzJFE9wFRu7LOA/FntCixtVSnclsOnunQfqQEVmp
|
||||
Y0IjfceIerJysCTo0I5HfRw0DOFfZimallOa4Mv5BDmzMWWyX9TvppHCnmqvM2El
|
||||
ISY=
|
||||
-----END CERTIFICATE-----
|
||||
|
|
@ -0,0 +1,60 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
variable "project_create" {
|
||||
description = "Create project instead of using an existing one."
|
||||
type = bool
|
||||
default = false
|
||||
}
|
||||
|
||||
variable "project_id" {
|
||||
description = "Project id."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "service_accounts" {
|
||||
description = "List of service accounts."
|
||||
type = list(object({
|
||||
name = string
|
||||
iam_project_roles = list(string)
|
||||
public_keys_path = string
|
||||
}))
|
||||
default = [
|
||||
{
|
||||
name = "data-uploader"
|
||||
iam_project_roles = [
|
||||
"roles/bigquery.dataOwner",
|
||||
"roles/bigquery.jobUser",
|
||||
"roles/storage.objectAdmin"
|
||||
]
|
||||
public_keys_path = "public-keys/data-uploader/"
|
||||
},
|
||||
{
|
||||
name = "prisma-security"
|
||||
iam_project_roles = [
|
||||
"roles/iam.securityReviewer"
|
||||
]
|
||||
public_keys_path = "public-keys/prisma-security/"
|
||||
},
|
||||
]
|
||||
|
||||
}
|
||||
|
||||
variable "services" {
|
||||
description = "Service APIs to enable."
|
||||
type = list(string)
|
||||
default = []
|
||||
}
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
@ -67,21 +67,23 @@ The following example assumes that provisioning of a Compute Engine VM requires
|
|||
the resources over the Internet (i.e. to install OS packages). Since Compute VM has no public IP
|
||||
address for security reasons, Internet connectivity is done with [Cloud NAT](https://cloud.google.com/nat/docs/overview).
|
||||
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---: |:---:|:---:|
|
||||
| project_id | Project id that references existing project. | <code title="">string</code> | ✓ | |
|
||||
| *billing_account* | Billing account id used as default for new projects. | <code title="">string</code> | | <code title="">null</code> |
|
||||
| *cidrs* | CIDR ranges for subnets | <code title="map(string)">map(string)</code> | | <code title="{ image-builder = "10.0.0.0/24" }">...</code> |
|
||||
| *create_packer_vars* | Create packer variables file using template file and terraform output. | <code title="">bool</code> | | <code title="">false</code> |
|
||||
| *packer_account_users* | List of members that will be allowed to impersonate Packer image builder service account in IAM format, i.e. 'user:{emailid}'. | <code title="list(string)">list(string)</code> | | <code title="">[]</code> |
|
||||
| *packer_source_cidrs* | List of CIDR ranges allowed to connect to the temporary VM for provisioning. | <code title="list(string)">list(string)</code> | | <code title="">["0.0.0.0/0"]</code> |
|
||||
| *project_create* | Create project instead of using an existing one. | <code title="">bool</code> | | <code title="">true</code> |
|
||||
| *region* | Default region for resources | <code title="">string</code> | | <code title="">europe-west1</code> |
|
||||
| *root_node* | The resource name of the parent folder or organization for project creation, in 'folders/folder_id' or 'organizations/org_id' format. | <code title="">string</code> | | <code title="">null</code> |
|
||||
| *use_iap* | Use IAP tunnel to connect to Compute Engine instance for provisioning. | <code title="">bool</code> | | <code title="">true</code> |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| project_id | Project id that references existing project. | <code>string</code> | ✓ | |
|
||||
| billing_account | Billing account id used as default for new projects. | <code>string</code> | | <code>null</code> |
|
||||
| cidrs | CIDR ranges for subnets | <code>map(string)</code> | | <code title="{ image-builder = "10.0.0.0/24" }">{…}</code> |
|
||||
| create_packer_vars | Create packer variables file using template file and terraform output. | <code>bool</code> | | <code>false</code> |
|
||||
| packer_account_users | List of members that will be allowed to impersonate Packer image builder service account in IAM format, i.e. 'user:{emailid}'. | <code>list(string)</code> | | <code>[]</code> |
|
||||
| packer_source_cidrs | List of CIDR ranges allowed to connect to the temporary VM for provisioning. | <code>list(string)</code> | | <code>["0.0.0.0/0"]</code> |
|
||||
| project_create | Create project instead of using an existing one. | <code>bool</code> | | <code>true</code> |
|
||||
| region | Default region for resources | <code>string</code> | | <code>"europe-west1"</code> |
|
||||
| root_node | The resource name of the parent folder or organization for project creation, in 'folders/folder_id' or 'organizations/org_id' format. | <code>string</code> | | <code>null</code> |
|
||||
| use_iap | Use IAP tunnel to connect to Compute Engine instance for provisioning. | <code>bool</code> | | <code>true</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
|
|
@ -91,4 +93,6 @@ address for security reasons, Internet connectivity is done with [Cloud NAT](htt
|
|||
| compute_sa | Packer's temporary VM service account email. | |
|
||||
| compute_subnetwork | Name of a subnetwork for Packer's temporary VM. | |
|
||||
| compute_zone | Name of a compute engine zone for Packer's temporary VM. | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
#!/bin/sh
|
||||
|
||||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -23,20 +23,21 @@ Clone this repository or [open it in cloud shell](https://ssh.cloud.google.com/c
|
|||
- `terraform init`
|
||||
- `terraform apply -var project_id=my-project-id`
|
||||
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---: |:---:|:---:|
|
||||
| project_id | Project id that references existing project. | <code title="">string</code> | ✓ | |
|
||||
| *bundle_path* | Path used to write the intermediate Cloud Function code bundle. | <code title="">string</code> | | <code title="">./bundle.zip</code> |
|
||||
| *name* | Arbitrary string used to name created resources. | <code title="">string</code> | | <code title="">quota-monitor</code> |
|
||||
| *project_create* | Create project instead ofusing an existing one. | <code title="">bool</code> | | <code title="">false</code> |
|
||||
| *quota_config* | Cloud function configuration. | <code title="object({ filters = list(string) projects = list(string) regions = list(string) })">object({...})</code> | | <code title="{ filters = null projects = null regions = null }">...</code> |
|
||||
| *region* | Compute region used in the example. | <code title="">string</code> | | <code title="">europe-west1</code> |
|
||||
| *schedule_config* | Schedule timer configuration in crontab format | <code title="">string</code> | | <code title="">0 * * * *</code> |
|
||||
|
||||
## Outputs
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| project_id | Project id that references existing project. | <code>string</code> | ✓ | |
|
||||
| bundle_path | Path used to write the intermediate Cloud Function code bundle. | <code>string</code> | | <code>"./bundle.zip"</code> |
|
||||
| name | Arbitrary string used to name created resources. | <code>string</code> | | <code>"quota-monitor"</code> |
|
||||
| project_create | Create project instead ofusing an existing one. | <code>bool</code> | | <code>false</code> |
|
||||
| quota_config | Cloud function configuration. | <code title="object({ filters = list(string) projects = list(string) regions = list(string) })">object({…})</code> | | <code title="{ filters = null projects = null regions = null }">{…}</code> |
|
||||
| region | Compute region used in the example. | <code>string</code> | | <code>"europe-west1"</code> |
|
||||
| schedule_config | Schedule timer configuration in crontab format | <code>string</code> | | <code>"0 * * * *"</code> |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
#! /usr/bin/env python3
|
||||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -36,22 +36,42 @@ Once done testing, you can clean up resources by running `terraform destroy`. To
|
|||
|
||||
Once resources are created, you can run queries on the data you exported on Bigquery. [Here](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery#querying_an_asset_snapshot) you can find some example of queries you can run.
|
||||
|
||||
You can also create a dashboard connecting [Datalab](https://datastudio.google.com/) or any other BI tools of your choice to your Bigquery datase.
|
||||
You can also create a dashboard connecting [Datalab](https://datastudio.google.com/) or any other BI tools of your choice to your Bigquery dataset.
|
||||
|
||||
## File exporter for JSON, CSV (optional).
|
||||
|
||||
This is an optional part.
|
||||
|
||||
Regular file-based exports of data from Cloud Asset Inventory may be useful for e.g. scale-out network dependencies discovery tools like [Planet Exporter](https://github.com/williamchanrico/planet-exporter), or to update legacy workloads tracking or configuration management systems. Bigquery supports multiple [export formats](https://cloud.google.com/bigquery/docs/exporting-data#export_formats_and_compression_types) and one may upload objects to Storage Bucket using provided Cloud Function. Specify `job.DestinationFormat` as defined in [documentation](https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.job.DestinationFormat.html), e.g. `NEWLINE_DELIMITED_JSON`.
|
||||
|
||||
It helps to create custom [scheduled query](https://cloud.google.com/bigquery/docs/scheduling-queries#console) from CAI export tables, and to write out results in to dedicated table (with overwrites). Define such query's output columns to comply with downstream systems' fields requirements, and time query execution after CAI export into BQ for freshness. See [sample queries](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery-sample-queries).
|
||||
|
||||
This is an optional part, created if `cai_gcs_export` is set to `true`. The high level diagram extends to the following:
|
||||
|
||||
<img src="diagram_optional.png" width="640px">
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---: |:---:|:---:|
|
||||
| cai_config | Cloud Asset inventory export config. | <code title="object({ bq_dataset = string bq_table = string target_node = string })">object({...})</code> | ✓ | |
|
||||
| project_id | Project id that references existing project. | <code title="">string</code> | ✓ | |
|
||||
| *billing_account* | Billing account id used as default for new projects. | <code title="">string</code> | | <code title="">null</code> |
|
||||
| *bundle_path* | Path used to write the intermediate Cloud Function code bundle. | <code title="">string</code> | | <code title="">./bundle.zip</code> |
|
||||
| *location* | Appe Engine location used in the example. | <code title="">string</code> | | <code title="">europe-west</code> |
|
||||
| *name* | Arbitrary string used to name created resources. | <code title="">string</code> | | <code title="">asset-inventory</code> |
|
||||
| *project_create* | Create project instead ofusing an existing one. | <code title="">bool</code> | | <code title="">true</code> |
|
||||
| *region* | Compute region used in the example. | <code title="">string</code> | | <code title="">europe-west1</code> |
|
||||
| *root_node* | The resource name of the parent folder or organization for project creation, in 'folders/folder_id' or 'organizations/org_id' format. | <code title="">string</code> | | <code title="">null</code> |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| cai_config | Cloud Asset Inventory export config. | <code title="object({ bq_dataset = string bq_table = string bq_table_overwrite = bool target_node = string })">object({…})</code> | ✓ | |
|
||||
| project_id | Project id that references existing project. | <code>string</code> | ✓ | |
|
||||
| billing_account | Billing account id used as default for new projects. | <code>string</code> | | <code>null</code> |
|
||||
| bundle_path | Path used to write the intermediate Cloud Function code bundle. | <code>string</code> | | <code>"./bundle.zip"</code> |
|
||||
| bundle_path_cffile | Path used to write the intermediate Cloud Function code bundle. | <code>string</code> | | <code>"./bundle_cffile.zip"</code> |
|
||||
| cai_gcs_export | Enable optional part to export tables to GCS | <code>bool</code> | | <code>false</code> |
|
||||
| file_config | Optional BQ table as a file export function config. | <code title="object({ bucket = string filename = string format = string bq_dataset = string bq_table = string })">object({…})</code> | | <code title="{ bucket = null filename = null format = null bq_dataset = null bq_table = null }">{…}</code> |
|
||||
| location | Appe Engine location used in the example. | <code>string</code> | | <code>"europe-west"</code> |
|
||||
| name | Arbitrary string used to name created resources. | <code>string</code> | | <code>"asset-inventory"</code> |
|
||||
| name_cffile | Arbitrary string used to name created resources. | <code>string</code> | | <code>"cffile-exporter"</code> |
|
||||
| project_create | Create project instead ofusing an existing one. | <code>bool</code> | | <code>true</code> |
|
||||
| region | Compute region used in the example. | <code>string</code> | | <code>"europe-west1"</code> |
|
||||
| root_node | The resource name of the parent folder or organization for project creation, in 'folders/folder_id' or 'organizations/org_id' format. | <code>string</code> | | <code>null</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
|
|
@ -59,4 +79,7 @@ You can also create a dashboard connecting [Datalab](https://datastudio.google.c
|
|||
|---|---|:---:|
|
||||
| bq-dataset | Bigquery instance details. | |
|
||||
| cloud-function | Cloud Function instance details. | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
@ -50,18 +50,20 @@ def _configure_logging(verbose=True):
|
|||
@click.option('--bq-project', required=True, help='Bigquery project to use.')
|
||||
@click.option('--bq-dataset', required=True, help='Bigquery dataset to use.')
|
||||
@click.option('--bq-table', required=True, help='Bigquery table name to use.')
|
||||
@click.option('--bq-table-overwrite', required=True, help='Overwrite existing BQ table or create new datetime() one.')
|
||||
@click.option('--target-node', required=True, help='Node in Google Cloud resource hierarchy.')
|
||||
@click.option('--read-time', required=False, help=(
|
||||
'Day to take an asset snapshot in \'YYYYMMDD\' format, uses current day '
|
||||
' as default. Export will run at midnight of the specified day.'))
|
||||
@click.option('--verbose', is_flag=True, help='Verbose output')
|
||||
def main_cli(project=None, bq_project=None, bq_dataset=None, bq_table=None, target_node=None,
|
||||
def main_cli(project=None, bq_project=None, bq_dataset=None, bq_table=None, bq_table_overwrite=None, target_node=None,
|
||||
read_time=None, verbose=False):
|
||||
'''Trigger Cloud Asset inventory export to Bigquery. Data will be stored in
|
||||
the dataset specified on a dated table with the name specified.
|
||||
'''
|
||||
try:
|
||||
_main(project, bq_project, bq_dataset, bq_table, target_node, read_time, verbose)
|
||||
_main(project, bq_project, bq_dataset, bq_table,
|
||||
bq_table_overwrite, target_node, read_time, verbose)
|
||||
except RuntimeError:
|
||||
logging.exception('exception raised')
|
||||
|
||||
|
|
@ -79,19 +81,22 @@ def main(event, context):
|
|||
logging.exception('exception in cloud function entry point')
|
||||
|
||||
|
||||
def _main(project=None, bq_project=None, bq_dataset=None, bq_table=None, target_node=None, read_time=None, verbose=False):
|
||||
def _main(project=None, bq_project=None, bq_dataset=None, bq_table=None, bq_table_overwrite=None, target_node=None, read_time=None, verbose=False):
|
||||
'Module entry point used by cli and cloud function wrappers.'
|
||||
|
||||
_configure_logging(verbose)
|
||||
if not read_time:
|
||||
read_time = datetime.datetime.now()
|
||||
client = asset_v1.AssetServiceClient()
|
||||
content_type = asset_v1.ContentType.RESOURCE
|
||||
output_config = asset_v1.OutputConfig()
|
||||
client = asset_v1.AssetServiceClient()
|
||||
if bq_table_overwrite == False:
|
||||
read_time = datetime.datetime.now()
|
||||
output_config.bigquery_destination.table = '%s_%s' % (
|
||||
bq_table, read_time.strftime('%Y%m%d'))
|
||||
else:
|
||||
output_config.bigquery_destination.table = '%s_latest' % (
|
||||
bq_table)
|
||||
content_type = asset_v1.ContentType.RESOURCE
|
||||
output_config.bigquery_destination.dataset = 'projects/%s/datasets/%s' % (
|
||||
bq_project, bq_dataset)
|
||||
output_config.bigquery_destination.table = '%s_%s' % (
|
||||
bq_table, read_time.strftime('%Y%m%d'))
|
||||
output_config.bigquery_destination.separate_tables_per_asset_type = True
|
||||
output_config.bigquery_destination.force = True
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -0,0 +1,101 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
'''Cloud Function module to export BQ table as JSON.
|
||||
|
||||
This module is designed to be plugged in a Cloud Function, attached to Cloud
|
||||
Scheduler trigger to create a JSON of IP to hostname mappings from BigQuery.
|
||||
|
||||
'''
|
||||
|
||||
import base64
|
||||
import datetime
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import warnings
|
||||
|
||||
from google.api_core.exceptions import GoogleAPIError
|
||||
from google.cloud import bigquery
|
||||
|
||||
import click
|
||||
|
||||
import googleapiclient.discovery
|
||||
import googleapiclient.errors
|
||||
|
||||
|
||||
def _configure_logging(verbose=True):
|
||||
'''Basic logging configuration.
|
||||
Args:
|
||||
verbose: enable verbose logging
|
||||
'''
|
||||
level = logging.DEBUG if verbose else logging.INFO
|
||||
logging.basicConfig(level=level)
|
||||
warnings.filterwarnings('ignore', r'.*end user credentials.*', UserWarning)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option('--bucket', required=True, help='GCS bucket for export')
|
||||
@click.option('--filename', required=True, help='Path and filename with extension to export e.g. folder/export.json .')
|
||||
@click.option('--format', required=True, help='The exported file format, e.g. NEWLINE_DELIMITED_JSON or CSV.')
|
||||
@click.option('--bq-dataset', required=True, help='Bigquery dataset where table for export is located.')
|
||||
@click.option('--bq-table', required=True, help='Bigquery table to export.')
|
||||
@click.option('--verbose', is_flag=True, help='Verbose output')
|
||||
def main_cli(bucket=None, filename=None, format=None, bq_dataset=None, bq_table=None, verbose=False):
|
||||
'''Trigger Cloud Asset inventory export from Bigquery to file. Data will be stored in
|
||||
the dataset specified on a dated table with the name specified.
|
||||
'''
|
||||
try:
|
||||
_main(bucket, filename, format, bq_dataset, bq_table, verbose)
|
||||
except RuntimeError:
|
||||
logging.exception('exception raised')
|
||||
|
||||
|
||||
def main(event, context):
|
||||
'Cloud Function entry point.'
|
||||
try:
|
||||
data = json.loads(base64.b64decode(event['data']).decode('utf-8'))
|
||||
print(data)
|
||||
_main(**data)
|
||||
# uncomment once https://issuetracker.google.com/issues/155215191 is fixed
|
||||
# except RuntimeError:
|
||||
# raise
|
||||
except Exception:
|
||||
logging.exception('exception in cloud function entry point')
|
||||
|
||||
|
||||
def _main(bucket=None, filename=None, format=None, bq_dataset=None, bq_table=None, verbose=False):
|
||||
'Module entry point used by cli and cloud function wrappers.'
|
||||
|
||||
_configure_logging(verbose)
|
||||
client = bigquery.Client()
|
||||
destination_uri = 'gs://{}/{}'.format(bucket, filename)
|
||||
dataset_ref = client.dataset(bq_dataset)
|
||||
table_ref = dataset_ref.table(bq_table)
|
||||
job_config = bigquery.job.ExtractJobConfig()
|
||||
job_config.destination_format = (
|
||||
getattr(bigquery.DestinationFormat, format))
|
||||
extract_job = client.extract_table(
|
||||
table_ref, destination_uri, job_config=job_config
|
||||
)
|
||||
try:
|
||||
extract_job.result()
|
||||
except (GoogleAPIError, googleapiclient.errors.HttpError) as e:
|
||||
logging.debug('API Error: %s', e, exc_info=True)
|
||||
raise RuntimeError(
|
||||
'Error exporting BQ table %s as a file' % bq_table, e)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main_cli()
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
google-api-python-client>=1.10.1
|
||||
google-cloud-monitoring>=1.1.0
|
||||
google-cloud-bigquery
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 54 KiB |
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
@ -14,6 +14,8 @@
|
|||
* limitations under the License.
|
||||
*/
|
||||
|
||||
|
||||
|
||||
###############################################################################
|
||||
# Projects #
|
||||
###############################################################################
|
||||
|
|
@ -47,6 +49,7 @@ module "service-account" {
|
|||
iam_project_roles = {
|
||||
(var.project_id) = [
|
||||
"roles/cloudasset.owner",
|
||||
"roles/bigquery.jobUser"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
|
@ -66,6 +69,17 @@ module "pubsub" {
|
|||
# at the project level via roles/cloudscheduler.serviceAgent
|
||||
}
|
||||
|
||||
module "pubsub_file" {
|
||||
source = "../../modules/pubsub"
|
||||
project_id = module.project.project_id
|
||||
name = var.name_cffile
|
||||
subscriptions = {
|
||||
"${var.name_cffile}-default" = null
|
||||
}
|
||||
# the Cloud Scheduler robot service account already has pubsub.topics.publish
|
||||
# at the project level via roles/cloudscheduler.serviceAgent
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Cloud Function #
|
||||
###############################################################################
|
||||
|
|
@ -93,6 +107,30 @@ module "cf" {
|
|||
}
|
||||
}
|
||||
|
||||
module "cffile" {
|
||||
count = var.cai_gcs_export ? 1 : 0
|
||||
source = "../../modules/cloud-function"
|
||||
project_id = module.project.project_id
|
||||
region = var.region
|
||||
name = var.name_cffile
|
||||
bucket_name = "${var.name_cffile}-${random_pet.random.id}"
|
||||
bucket_config = {
|
||||
location = var.region
|
||||
lifecycle_delete_age = null
|
||||
}
|
||||
bundle_config = {
|
||||
source_dir = "cffile"
|
||||
output_path = var.bundle_path_cffile
|
||||
excludes = null
|
||||
}
|
||||
service_account = module.service-account.email
|
||||
trigger_config = {
|
||||
event = "google.pubsub.topic.publish"
|
||||
resource = module.pubsub_file.topic.id
|
||||
retry = null
|
||||
}
|
||||
}
|
||||
|
||||
resource "random_pet" "random" {
|
||||
length = 1
|
||||
}
|
||||
|
|
@ -118,11 +156,34 @@ resource "google_cloud_scheduler_job" "job" {
|
|||
attributes = {}
|
||||
topic_name = module.pubsub.topic.id
|
||||
data = base64encode(jsonencode({
|
||||
project = module.project.project_id
|
||||
bq_project = module.project.project_id
|
||||
bq_dataset = var.cai_config.bq_dataset
|
||||
bq_table = var.cai_config.bq_table
|
||||
target_node = var.cai_config.target_node
|
||||
project = module.project.project_id
|
||||
bq_project = module.project.project_id
|
||||
bq_dataset = var.cai_config.bq_dataset
|
||||
bq_table = var.cai_config.bq_table
|
||||
bq_table_overwrite = var.cai_config.bq_table_overwrite
|
||||
target_node = var.cai_config.target_node
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_cloud_scheduler_job" "job_file" {
|
||||
count = var.cai_gcs_export ? 1 : 0
|
||||
project = google_app_engine_application.app.project
|
||||
region = var.region
|
||||
name = "file-export-job"
|
||||
description = "File export from BQ Job"
|
||||
schedule = "* 9 * * 1"
|
||||
time_zone = "Etc/UTC"
|
||||
|
||||
pubsub_target {
|
||||
attributes = {}
|
||||
topic_name = module.pubsub_file.topic.id
|
||||
data = base64encode(jsonencode({
|
||||
bucket = var.file_config.bucket
|
||||
filename = var.file_config.filename
|
||||
format = var.file_config.format
|
||||
bq_dataset = var.file_config.bq_dataset
|
||||
bq_table = var.file_config.bq_table
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
@ -26,15 +26,50 @@ variable "bundle_path" {
|
|||
default = "./bundle.zip"
|
||||
}
|
||||
|
||||
|
||||
variable "bundle_path_cffile" {
|
||||
description = "Path used to write the intermediate Cloud Function code bundle."
|
||||
type = string
|
||||
default = "./bundle_cffile.zip"
|
||||
}
|
||||
|
||||
variable "cai_config" {
|
||||
description = "Cloud Asset inventory export config."
|
||||
description = "Cloud Asset Inventory export config."
|
||||
type = object({
|
||||
bq_dataset = string
|
||||
bq_table = string
|
||||
target_node = string
|
||||
bq_dataset = string
|
||||
bq_table = string
|
||||
bq_table_overwrite = bool
|
||||
target_node = string
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
variable "cai_gcs_export" {
|
||||
description = "Enable optional part to export tables to GCS"
|
||||
type = bool
|
||||
default = false
|
||||
}
|
||||
|
||||
|
||||
variable "file_config" {
|
||||
description = "Optional BQ table as a file export function config."
|
||||
type = object({
|
||||
bucket = string
|
||||
filename = string
|
||||
format = string
|
||||
bq_dataset = string
|
||||
bq_table = string
|
||||
})
|
||||
default = {
|
||||
bucket = null
|
||||
filename = null
|
||||
format = null
|
||||
bq_dataset = null
|
||||
bq_table = null
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
variable "location" {
|
||||
description = "Appe Engine location used in the example."
|
||||
type = string
|
||||
|
|
@ -48,6 +83,15 @@ variable "name" {
|
|||
default = "asset-inventory"
|
||||
}
|
||||
|
||||
|
||||
|
||||
variable "name_cffile" {
|
||||
description = "Arbitrary string used to name created resources."
|
||||
type = string
|
||||
default = "cffile-exporter"
|
||||
}
|
||||
|
||||
|
||||
variable "project_create" {
|
||||
description = "Create project instead ofusing an existing one."
|
||||
type = bool
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -30,20 +30,22 @@ This sample creates several distinct groups of resources:
|
|||
- GCS
|
||||
- One bucket encrypted with a CMEK Cryptokey hosted in Cloud KMS
|
||||
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---: |:---:|:---:|
|
||||
| billing_account | Billing account id used as default for new projects. | <code title="">string</code> | ✓ | |
|
||||
| root_node | The resource name of the parent Folder or Organization. Must be of the form folders/folder_id or organizations/org_id. | <code title="">string</code> | ✓ | |
|
||||
| *location* | The location where resources will be deployed. | <code title="">string</code> | | <code title="">europe</code> |
|
||||
| *project_kms_name* | Name for the new KMS Project. | <code title="">string</code> | | <code title="">my-project-kms-001</code> |
|
||||
| *project_service_name* | Name for the new Service Project. | <code title="">string</code> | | <code title="">my-project-service-001</code> |
|
||||
| *region* | The region where resources will be deployed. | <code title="">string</code> | | <code title="">europe-west1</code> |
|
||||
| *vpc_ip_cidr_range* | Ip range used in the subnet deployef in the Service Project. | <code title="">string</code> | | <code title="">10.0.0.0/20</code> |
|
||||
| *vpc_name* | Name of the VPC created in the Service Project. | <code title="">string</code> | | <code title="">local</code> |
|
||||
| *vpc_subnet_name* | Name of the subnet created in the Service Project. | <code title="">string</code> | | <code title="">subnet</code> |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| billing_account | Billing account id used as default for new projects. | <code>string</code> | ✓ | |
|
||||
| root_node | The resource name of the parent Folder or Organization. Must be of the form folders/folder_id or organizations/org_id. | <code>string</code> | ✓ | |
|
||||
| location | The location where resources will be deployed. | <code>string</code> | | <code>"europe"</code> |
|
||||
| project_kms_name | Name for the new KMS Project. | <code>string</code> | | <code>"my-project-kms-001"</code> |
|
||||
| project_service_name | Name for the new Service Project. | <code>string</code> | | <code>"my-project-service-001"</code> |
|
||||
| region | The region where resources will be deployed. | <code>string</code> | | <code>"europe-west1"</code> |
|
||||
| vpc_ip_cidr_range | Ip range used in the subnet deployef in the Service Project. | <code>string</code> | | <code>"10.0.0.0/20"</code> |
|
||||
| vpc_name | Name of the VPC created in the Service Project. | <code>string</code> | | <code>"local"</code> |
|
||||
| vpc_subnet_name | Name of the subnet created in the Service Project. | <code>string</code> | | <code>"subnet"</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
|
|
@ -54,4 +56,6 @@ This sample creates several distinct groups of resources:
|
|||
| projects | Project ids. | |
|
||||
| vm | GCE VM. | |
|
||||
| vm_keys | GCE VM Cloud KMS crypto keys. | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -47,19 +47,21 @@ gcloud access-context-manager perimeters list --format="json" | grep name
|
|||
|
||||
The script use 'google_access_context_manager_service_perimeter_resource' terraform resource. If this resource is used alongside the 'vpc-sc' module, remember to uncomment the lifecycle block in the 'vpc-sc' module so they don't fight over which resources should be in the perimeter.
|
||||
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---: |:---:|:---:|
|
||||
| billing_account_id | Billing account id. | <code title="">string</code> | ✓ | |
|
||||
| root_node | Parent folder or organization in 'folders/folder_id' or 'organizations/org_id' format. | <code title="">string</code> | ✓ | |
|
||||
| *admins* | List of users allowed to impersonate the service account | <code title="list(string)">list(string)</code> | | <code title="">null</code> |
|
||||
| *prefix* | Prefix used to generate project id and name. | <code title="">string</code> | | <code title="">null</code> |
|
||||
| *project_names* | Override this variable if you need non-standard names. | <code title="object({ datamart = string dwh = string landing = string services = string transformation = string })">object({...})</code> | | <code title="{ datamart = "datamart" dwh = "datawh" landing = "landing" services = "services" transformation = "transformation" }">...</code> |
|
||||
| *service_account_names* | Override this variable if you need non-standard names. | <code title="object({ main = string })">object({...})</code> | | <code title="{ main = "data-platform-main" }">...</code> |
|
||||
| *service_encryption_key_ids* | Cloud KMS encryption key in {LOCATION => [KEY_URL]} format. Keys belong to existing project. | <code title="object({ multiregional = string global = string })">object({...})</code> | | <code title="{ multiregional = null global = null }">...</code> |
|
||||
| *service_perimeter_standard* | VPC Service control standard perimeter name in the form of 'accessPolicies/ACCESS_POLICY_NAME/servicePerimeters/PERIMETER_NAME'. All projects will be added to the perimeter in enforced mode. | <code title="">string</code> | | <code title="">null</code> |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| billing_account_id | Billing account id. | <code>string</code> | ✓ | |
|
||||
| root_node | Parent folder or organization in 'folders/folder_id' or 'organizations/org_id' format. | <code>string</code> | ✓ | |
|
||||
| admins | List of users allowed to impersonate the service account | <code>list(string)</code> | | <code>null</code> |
|
||||
| prefix | Prefix used to generate project id and name. | <code>string</code> | | <code>null</code> |
|
||||
| project_names | Override this variable if you need non-standard names. | <code title="object({ datamart = string dwh = string landing = string services = string transformation = string })">object({…})</code> | | <code title="{ datamart = "datamart" dwh = "datawh" landing = "landing" services = "services" transformation = "transformation" }">{…}</code> |
|
||||
| service_account_names | Override this variable if you need non-standard names. | <code title="object({ main = string })">object({…})</code> | | <code title="{ main = "data-platform-main" }">{…}</code> |
|
||||
| service_encryption_key_ids | Cloud KMS encryption key in {LOCATION => [KEY_URL]} format. Keys belong to existing project. | <code title="object({ multiregional = string global = string })">object({…})</code> | | <code title="{ multiregional = null global = null }">{…}</code> |
|
||||
| service_perimeter_standard | VPC Service control standard perimeter name in the form of 'accessPolicies/ACCESS_POLICY_NAME/servicePerimeters/PERIMETER_NAME'. All projects will be added to the perimeter in enforced mode. | <code>string</code> | | <code>null</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
|
|
@ -68,4 +70,6 @@ The script use 'google_access_context_manager_service_perimeter_resource' terraf
|
|||
| project_ids | Project ids for created projects. | |
|
||||
| service_account | Main service account. | |
|
||||
| service_encryption_key_ids | Cloud KMS encryption keys in {LOCATION => [KEY_URL]} format. | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -51,23 +51,25 @@ Once done testing, you can clean up resources by running `terraform destroy`.
|
|||
### CMEK configuration
|
||||
You can configure GCP resources to use existing CMEK keys configuring the 'service_encryption_key_ids' variable. You need to specify a 'global' and a 'multiregional' key.
|
||||
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---: |:---:|:---:|
|
||||
| project_ids | Project IDs. | <code title="object({ datamart = string dwh = string landing = string services = string transformation = string })">object({...})</code> | ✓ | |
|
||||
| *admins* | List of users allowed to impersonate the service account | <code title="list(string)">list(string)</code> | | <code title="">null</code> |
|
||||
| *datamart_bq_datasets* | Datamart Bigquery datasets | <code title="map(object({ iam = map(list(string)) location = string }))">map(object({...}))</code> | | <code title="{ bq_datamart_dataset = { location = "EU" iam = { } } }">...</code> |
|
||||
| *dwh_bq_datasets* | DWH Bigquery datasets | <code title="map(object({ location = string iam = map(list(string)) }))">map(object({...}))</code> | | <code title="{ bq_raw_dataset = { iam = {} location = "EU" } }">...</code> |
|
||||
| *landing_buckets* | List of landing buckets to create | <code title="map(object({ location = string name = string }))">map(object({...}))</code> | | <code title="{ raw-data = { location = "EU" name = "raw-data" } data-schema = { location = "EU" name = "data-schema" } }">...</code> |
|
||||
| *landing_pubsub* | List of landing pubsub topics and subscriptions to create | <code title="map(map(object({ iam = map(list(string)) labels = map(string) options = object({ ack_deadline_seconds = number message_retention_duration = number retain_acked_messages = bool expiration_policy_ttl = number }) })))">map(map(object({...})))</code> | | <code title="{ landing-1 = { sub1 = { iam = { } labels = {} options = null } sub2 = { iam = {} labels = {}, options = null }, } }">...</code> |
|
||||
| *landing_service_account* | landing service accounts list. | <code title="">string</code> | | <code title="">sa-landing</code> |
|
||||
| *service_account_names* | Project service accounts list. | <code title="object({ datamart = string dwh = string landing = string services = string transformation = string })">object({...})</code> | | <code title="{ datamart = "sa-datamart" dwh = "sa-datawh" landing = "sa-landing" services = "sa-services" transformation = "sa-transformation" }">...</code> |
|
||||
| *service_encryption_key_ids* | Cloud KMS encryption key in {LOCATION => [KEY_URL]} format. Keys belong to existing project. | <code title="object({ multiregional = string global = string })">object({...})</code> | | <code title="{ multiregional = null global = null }">...</code> |
|
||||
| *transformation_buckets* | List of transformation buckets to create | <code title="map(object({ location = string name = string }))">map(object({...}))</code> | | <code title="{ temp = { location = "EU" name = "temp" }, templates = { location = "EU" name = "templates" }, }">...</code> |
|
||||
| *transformation_subnets* | List of subnets to create in the transformation Project. | <code title="list(object({ ip_cidr_range = string name = string region = string secondary_ip_range = map(string) }))">list(object({...}))</code> | | <code title="[ { ip_cidr_range = "10.1.0.0/20" name = "transformation-subnet" region = "europe-west3" secondary_ip_range = {} }, ]">...</code> |
|
||||
| *transformation_vpc_name* | Name of the VPC created in the transformation Project. | <code title="">string</code> | | <code title="">transformation-vpc</code> |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| project_ids | Project IDs. | <code title="object({ datamart = string dwh = string landing = string services = string transformation = string })">object({…})</code> | ✓ | |
|
||||
| admins | List of users allowed to impersonate the service account | <code>list(string)</code> | | <code>null</code> |
|
||||
| datamart_bq_datasets | Datamart Bigquery datasets | <code title="map(object({ iam = map(list(string)) location = string }))">map(object({…}))</code> | | <code title="{ bq_datamart_dataset = { location = "EU" iam = { } } }">{…}</code> |
|
||||
| dwh_bq_datasets | DWH Bigquery datasets | <code title="map(object({ location = string iam = map(list(string)) }))">map(object({…}))</code> | | <code title="{ bq_raw_dataset = { iam = {} location = "EU" } }">{…}</code> |
|
||||
| landing_buckets | List of landing buckets to create | <code title="map(object({ location = string name = string }))">map(object({…}))</code> | | <code title="{ raw-data = { location = "EU" name = "raw-data" } data-schema = { location = "EU" name = "data-schema" } }">{…}</code> |
|
||||
| landing_pubsub | List of landing pubsub topics and subscriptions to create | <code title="map(map(object({ iam = map(list(string)) labels = map(string) options = object({ ack_deadline_seconds = number message_retention_duration = number retain_acked_messages = bool expiration_policy_ttl = number }) })))">map(map(object({…})))</code> | | <code title="{ landing-1 = { sub1 = { iam = { } labels = {} options = null } sub2 = { iam = {} labels = {}, options = null }, } }">{…}</code> |
|
||||
| landing_service_account | landing service accounts list. | <code>string</code> | | <code>"sa-landing"</code> |
|
||||
| service_account_names | Project service accounts list. | <code title="object({ datamart = string dwh = string landing = string services = string transformation = string })">object({…})</code> | | <code title="{ datamart = "sa-datamart" dwh = "sa-datawh" landing = "sa-landing" services = "sa-services" transformation = "sa-transformation" }">{…}</code> |
|
||||
| service_encryption_key_ids | Cloud KMS encryption key in {LOCATION => [KEY_URL]} format. Keys belong to existing project. | <code title="object({ multiregional = string global = string })">object({…})</code> | | <code title="{ multiregional = null global = null }">{…}</code> |
|
||||
| transformation_buckets | List of transformation buckets to create | <code title="map(object({ location = string name = string }))">map(object({…}))</code> | | <code title="{ temp = { location = "EU" name = "temp" }, templates = { location = "EU" name = "templates" }, }">{…}</code> |
|
||||
| transformation_subnets | List of subnets to create in the transformation Project. | <code title="list(object({ ip_cidr_range = string name = string region = string secondary_ip_range = map(string) }))">list(object({…}))</code> | | <code title="[ { ip_cidr_range = "10.1.0.0/20" name = "transformation-subnet" region = "europe-west3" secondary_ip_range = {} }, ]">[…]</code> |
|
||||
| transformation_vpc_name | Name of the VPC created in the transformation Project. | <code>string</code> | | <code>"transformation-vpc"</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
|
|
@ -79,4 +81,6 @@ You can configure GCP resources to use existing CMEK keys configuring the 'servi
|
|||
| landing-pubsub | List of pubsub topics and subscriptions created for the landing project. | |
|
||||
| transformation-buckets | List of buckets created for the transformation project. | |
|
||||
| transformation-vpc | Transformation VPC details | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -110,21 +110,23 @@ schema_bq_import.json
|
|||
|
||||
You can check data imported into Google BigQuery from the Google Cloud Console UI.
|
||||
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---: |:---:|:---:|
|
||||
| billing_account | Billing account id used as default for new projects. | <code title="">string</code> | ✓ | |
|
||||
| project_kms_name | Name for the new KMS Project. | <code title="">string</code> | ✓ | |
|
||||
| project_service_name | Name for the new Service Project. | <code title="">string</code> | ✓ | |
|
||||
| root_node | The resource name of the parent Folder or Organization. Must be of the form folders/folder_id or organizations/org_id. | <code title="">string</code> | ✓ | |
|
||||
| *location* | The location where resources will be deployed. | <code title="">string</code> | | <code title="">europe</code> |
|
||||
| *region* | The region where resources will be deployed. | <code title="">string</code> | | <code title="">europe-west1</code> |
|
||||
| *ssh_source_ranges* | IP CIDR ranges that will be allowed to connect via SSH to the onprem instance. | <code title="list(string)">list(string)</code> | | <code title="">["0.0.0.0/0"]</code> |
|
||||
| *vpc_ip_cidr_range* | Ip range used in the subnet deployef in the Service Project. | <code title="">string</code> | | <code title="">10.0.0.0/20</code> |
|
||||
| *vpc_name* | Name of the VPC created in the Service Project. | <code title="">string</code> | | <code title="">local</code> |
|
||||
| *vpc_subnet_name* | Name of the subnet created in the Service Project. | <code title="">string</code> | | <code title="">subnet</code> |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| billing_account | Billing account id used as default for new projects. | <code>string</code> | ✓ | |
|
||||
| project_kms_name | Name for the new KMS Project. | <code>string</code> | ✓ | |
|
||||
| project_service_name | Name for the new Service Project. | <code>string</code> | ✓ | |
|
||||
| root_node | The resource name of the parent Folder or Organization. Must be of the form folders/folder_id or organizations/org_id. | <code>string</code> | ✓ | |
|
||||
| location | The location where resources will be deployed. | <code>string</code> | | <code>"europe"</code> |
|
||||
| region | The region where resources will be deployed. | <code>string</code> | | <code>"europe-west1"</code> |
|
||||
| ssh_source_ranges | IP CIDR ranges that will be allowed to connect via SSH to the onprem instance. | <code>list(string)</code> | | <code>["0.0.0.0/0"]</code> |
|
||||
| vpc_ip_cidr_range | Ip range used in the subnet deployef in the Service Project. | <code>string</code> | | <code>"10.0.0.0/20"</code> |
|
||||
| vpc_name | Name of the VPC created in the Service Project. | <code>string</code> | | <code>"local"</code> |
|
||||
| vpc_subnet_name | Name of the subnet created in the Service Project. | <code>string</code> | | <code>"subnet"</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
|
|
@ -134,4 +136,6 @@ You can check data imported into Google BigQuery from the Google Cloud Console U
|
|||
| buckets | GCS Bucket Cloud KMS crypto keys. | |
|
||||
| projects | Project ids. | |
|
||||
| vm | GCE VM. | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2021 Google LLC
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -148,17 +148,21 @@ web_frontends:
|
|||
- web-frontends@project-wf2.iam.gserviceaccount.com
|
||||
```
|
||||
|
||||
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---: |:---:|:---:|
|
||||
| config_folder | Relative path of the folder containing the hierarchical firewall configuration | <code title="">string</code> | ✓ | |
|
||||
| templates_folder | Relative path of the folder containing the cidr/service account templates | <code title="">string</code> | ✓ | |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| config_folder | Relative path of the folder containing the hierarchical firewall configuration | <code>string</code> | ✓ | |
|
||||
| templates_folder | Relative path of the folder containing the cidr/service account templates | <code>string</code> | ✓ | |
|
||||
|
||||
## Outputs
|
||||
|
||||
| name | description | sensitive |
|
||||
|---|---|:---:|
|
||||
| hierarchical-firewall-rules | Generated Hierarchical Firewall Rules | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/**
|
||||
* Copyright 2021 Google LLC
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue