Merge pull request #722 from GoogleCloudPlatform/org-policy-rework

OrgPolicy module (factory) using new org-policy API, #698
This commit is contained in:
Aleksandr Averbukh 2022-07-08 15:38:42 +02:00 committed by GitHub
commit e21a0f7541
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
15 changed files with 614 additions and 3 deletions

View File

@ -29,7 +29,7 @@ The current list of modules supports most of the core foundational and networkin
Currently available modules:
- **foundational** - [folder](./modules/folder), [organization](./modules/organization), [project](./modules/project), [service accounts](./modules/iam-service-account), [logging bucket](./modules/logging-bucket), [billing budget](./modules/billing-budget), [naming convention](./modules/naming-convention), [projects-data-source](./modules/projects-data-source)
- **foundational** - [folder](./modules/folder), [organization](./modules/organization), [project](./modules/project), [service accounts](./modules/iam-service-account), [logging bucket](./modules/logging-bucket), [billing budget](./modules/billing-budget), [naming convention](./modules/naming-convention), [projects-data-source](./modules/projects-data-source), [organization-policy](./modules/organization-policy)
- **networking** - [VPC](./modules/net-vpc), [VPC firewall](./modules/net-vpc-firewall), [VPC peering](./modules/net-vpc-peering), [VPN static](./modules/net-vpn-static), [VPN dynamic](./modules/net-vpn-dynamic), [HA VPN](./modules/net-vpn-ha), [NAT](./modules/net-cloudnat), [address reservation](./modules/net-address), [DNS](./modules/dns), [L4 ILB](./modules/net-ilb), [L7 ILB](./modules/net-ilb-l7), [Service Directory](./modules/service-directory), [Cloud Endpoints](./modules/endpoints)
- **compute** - [VM/VM group](./modules/compute-vm), [MIG](./modules/compute-mig), [GKE cluster](./modules/gke-cluster), [GKE nodepool](./modules/gke-nodepool), [GKE hub](./modules/gke-hub), [COS container](./modules/cloud-config-container/cos-generic-metadata/) (coredns, mysql, onprem, squid)
- **data** - [GCS](./modules/gcs), [BigQuery dataset](./modules/bigquery-dataset), [Pub/Sub](./modules/pubsub), [Datafusion](./modules/datafusion), [Bigtable instance](./modules/bigtable-instance), [Cloud SQL instance](./modules/cloudsql-instance), [Data Catalog Policy Tag](./modules/data-catalog-policy-tag)

View File

@ -12,7 +12,7 @@ Nested folder structure for yaml configurations is optionally supported, which a
```hcl
module "prod-firewall" {
source = "./modules/net-vpc-firewall-yaml"
source = "./examples/factories/net-vpc-firewall-yaml"
project_id = "my-prod-project"
network = "my-prod-network"
@ -27,7 +27,7 @@ module "prod-firewall" {
}
module "dev-firewall" {
source = "./modules/net-vpc-firewall-yaml"
source = "./examples/factories/net-vpc-firewall-yaml"
project_id = "my-dev-project"
network = "my-dev-network"

View File

@ -37,6 +37,7 @@ These modules are used in the examples included in this repository. If you are u
- [project](./project)
- [projects-data-source](./projects-data-source)
- [service account](./iam-service-account)
- [organization policy](./organization-policy)
## Networking modules

View File

@ -0,0 +1,167 @@
# Google Cloud Organization Policy
This module allows creation and management of [GCP Organization Policies](https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints) by defining them in a well formatted `yaml` files or with HCL.
Yaml based factory can simplify centralized management of Org Policies for a DevSecOps team by providing a simple way to define/structure policies and exclusions.
> **_NOTE:_** This module uses experimental feature `module_variable_optional_attrs` which will be included into [terraform release 1.3](https://github.com/hashicorp/terraform/releases/tag/v1.3.0-alpha20220706).
## Example
### Terraform code
```hcl
# using configuration provided in a set of yaml files
module "org-policy-factory" {
source = "./modules/organization-policy"
config_directory = "./policies"
}
# using configuration provided in the module variable
module "org-policy" {
source = "./modules/organization-policy"
policies = {
"folders/1234567890" = {
# enforce boolean policy with no conditions
"iam.disableServiceAccountKeyUpload" = {
rules = [
{
enforce = true
}
]
},
# Deny All for compute.vmCanIpForward policy
"compute.vmCanIpForward" = {
inherit_from_parent = false
rules = [
deny = [] # stands for deny_all
]
}
},
"organizations/1234567890" = {
# allow only internal ingress when match condition env=prod
"run.allowedIngress" = {
rules = [
{
allow = ["internal"]
condition = {
description= "allow ingress"
expression = "resource.matchTag('123456789/environment', 'prod')"
title = "allow-for-prod-org"
}
}
]
}
}
}
}
# tftest skip
```
## Org Policy definition format and structure
### Structure of `policies` variable
```hcl
policies = {
"parent_id" = { # parent id in format projects/project-id, folders/1234567890 or organizations/1234567890.
"policy_name" = { # policy constraint id, for example compute.vmExternalIpAccess.
inherit_from_parent = true|false # (Optional) Only for list constraints. Determines the inheritance behavior for this policy.
reset = true|false # (Optional) Ignores policies set above this resource and restores the constraint_default enforcement behavior.
rules = [ # Up to 10 PolicyRules are allowed.
{
allow = ["value1", "value2"] # (Optional) Only for list constraints. Stands for `allow_all` if set to empty list `[]` or to `values.allowed_values` if set to a list of values
denyl = ["value3", "value4"] # (Optional) Only for list constraints. Stands for `deny_all` if set to empty list `[]` or to `values.denied_values` if set to a list of values
enforce = true|false # (Optional) Only for boolean constraints. If true, then the Policy is enforced.
condition = { # (Optional) A condition which determines whether this rule is used in the evaluation of the policy.
description = "Condition description" # (Optional)
expression = "Condition expression" # (Optional) For example "resource.matchTag('123456789/environment', 'prod')".
location = "policy-error.log" # (Optional) String indicating the location of the expression for error reporting.
title = "condition-title" # (Optional)
}
}
]
}
}
}
# tftest skip
```
### Structure of configuration provided in a yaml file/s
Configuration should be placed in a set of yaml files in the config directory. Policy entry structure as follows:
```yaml
parent_id: # parent id in format projects/project-id, folders/1234567890 or organizations/1234567890.
policy_name1: # policy constraint id, for example compute.vmExternalIpAccess.
inherit_from_parent: true|false # (Optional) Only for list constraints. Determines the inheritance behavior for this policy.
reset: true|false # (Optional) Ignores policies set above this resource and restores the constraint_default enforcement behavior.
rules:
- allow: ["value1", "value2"] # (Optional) Only for list constraints. Stands for `allow_all` if set to empty list `[]` or to `values.allowed_values` if set to a list of values
deny: ["value3", "value4"] # (Optional) Only for list constraints. Stands for `deny_all` if set to empty list `[]` or to `values.denied_values` if set to a list of values
enforce: true|false # (Optional) Only for boolean constraints. If true, then the Policy is enforced.
condition: # (Optional) A condition which determines whether this rule is used in the evaluation of the policy.
description: Condition description # (Optional)
expression: Condition expression # (Optional) For example resource.matchTag("123456789/environment", "prod")
location: policy-error.log # (Optional) String indicating the location of the expression for error reporting.
title: condition-title # (Optional)
```
Module allows policies to be distributed into multiple yaml files for a better management and navigation.
```bash
├── org-policies
│ ├── baseline.yaml
│   ├── image-import-projects.yaml
│   └── exclusions.yaml
```
Organization policies example yaml configuration
```bash
cat ./policies/baseline.yaml
organizations/1234567890:
constraints/compute.vmExternalIpAccess:
rules:
- deny_all: true
folders/1234567890:
compute.vmCanIpForward:
inherit_from_parent: false
reset: false
rules:
- allow: [] # Stands for allow_all = true
projects/my-project-id:
run.allowedIngress:
inherit_from_parent: true
rules:
- condition:
description: allow internal ingress
expression: resource.matchTag("123456789/environment", "prod")
location: test.log
title: allow-for-prod
values:
allowed_values: ['internal']
iam.allowServiceAccountCredentialLifetimeExtension:
rules:
- deny: [] # Stands for deny_all = true
compute.disableGlobalLoadBalancing:
reset: true
```
<!-- BEGIN TFDOC -->
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [config_directory](variables.tf#L17) | Paths to a folder where organization policy configs are stored in yaml format. Files suffix must be `.yaml`. | <code>string</code> | | <code>null</code> |
| [policies](variables.tf#L23) | Organization policies keyed by parent in format `projects/project-id`, `folders/1234567890` or `organizations/1234567890`. | <code title="map&#40;map&#40;object&#40;&#123;&#10; inherit_from_parent &#61; optional&#40;bool&#41; &#35; List policy only.&#10; reset &#61; optional&#40;bool&#41;&#10; rules &#61; optional&#40;&#10; list&#40;object&#40;&#123;&#10; allow &#61; optional&#40;list&#40;string&#41;&#41; &#35; List policy only. Stands for &#96;allow_all&#96; if set to empty list &#96;&#91;&#93;&#96; or to &#96;values.allowed_values&#96; if set to a list of values &#10; deny &#61; optional&#40;list&#40;string&#41;&#41; &#35; List policy only. Stands for &#96;deny_all&#96; if set to empty list &#96;&#91;&#93;&#96; or to &#96;values.denied_values&#96; if set to a list of values&#10; enforce &#61; optional&#40;bool&#41; &#35; Boolean policy only. &#10; condition &#61; optional&#40;&#10; object&#40;&#123;&#10; description &#61; optional&#40;string&#41;&#10; expression &#61; optional&#40;string&#41;&#10; location &#61; optional&#40;string&#41;&#10; title &#61; optional&#40;string&#41;&#10; &#125;&#41;&#10; &#41;&#10; &#125;&#41;&#41;&#10; &#41;&#10;&#125;&#41;&#41;&#41;">map&#40;map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
## Outputs
| name | description | sensitive |
|---|---|:---:|
| [policies](outputs.tf#L17) | Organization policies. | |
<!-- END TFDOC -->

View File

@ -0,0 +1,19 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
terraform {
# TODO: Remove once Terraform 1.3 is released https://github.com/hashicorp/terraform/releases/tag/v1.3.0-alpha20220622
experiments = [module_variable_optional_attrs]
}

View File

@ -0,0 +1,102 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
locals {
policy_files = var.config_directory == null ? [] : concat(
[
for config_file in fileset("${path.root}/${var.config_directory}", "**/*.yaml") :
"${path.root}/${var.config_directory}/${config_file}"
]
)
policies_raw = merge(
merge(
[
for config_file in local.policy_files :
try(yamldecode(file(config_file)), {})
]...
), var.policies)
policies_list = flatten([
for parent, policies in local.policies_raw : [
for policy_name, policy in policies : {
parent = parent,
policy_name = policy_name,
inherit_from_parent = try(policy["inherit_from_parent"], null),
reset = try(policy["reset"], null),
rules = [
for rule in try(policy["rules"], []) : {
allow_all = try(length(rule["allow"]), -1) == 0 ? "TRUE" : null
deny_all = try(length(rule["deny"]), -1) == 0 ? "TRUE" : null
enforce = try(rule["enforce"], null) == true ? "TRUE" : try(
rule["enforce"], null) == false ? "FALSE" : null,
condition = try(rule["condition"], null) != null ? {
description = try(rule["condition"]["description"], null),
expression = try(rule["condition"]["expression"], null),
location = try(rule["condition"]["location"], null),
title = try(rule["condition"]["title"], null)
} : null,
values = try(length(rule["allow"]), 0) > 0 || try(length(rule["deny"]), 0) > 0 ? {
allowed_values = try(length(rule["allow"]), 0) > 0 ? rule["allow"] : null
denied_values = try(length(rule["deny"]), 0) > 0 ? rule["deny"] : null
} : null
}
]
}
]
])
policies_map = {
for item in local.policies_list :
format("%s-%s", item["parent"], item["policy_name"]) => item
}
}
resource "google_org_policy_policy" "primary" {
for_each = local.policies_map
name = format("%s/policies/%s", each.value.parent, each.value.policy_name)
parent = each.value.parent
spec {
inherit_from_parent = each.value.inherit_from_parent
reset = each.value.reset
dynamic "rules" {
for_each = each.value.rules
content {
allow_all = rules.value.allow_all
deny_all = rules.value.deny_all
enforce = rules.value.enforce
dynamic "condition" {
for_each = rules.value.condition != null ? [""] : []
content {
description = rules.value.condition.description
expression = rules.value.condition.expression
location = rules.value.condition.location
title = rules.value.condition.title
}
}
dynamic "values" {
for_each = rules.value.values != null ? [""] : []
content {
allowed_values = rules.value.values.allowed_values
denied_values = rules.value.values.denied_values
}
}
}
}
}
}

View File

@ -0,0 +1,20 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
output "policies" {
description = "Organization policies."
value = google_org_policy_policy.primary
}

View File

@ -0,0 +1,45 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
variable "config_directory" {
description = "Paths to a folder where organization policy configs are stored in yaml format. Files suffix must be `.yaml`."
type = string
default = null
}
variable "policies" {
description = "Organization policies keyed by parent in format `projects/project-id`, `folders/1234567890` or `organizations/1234567890`."
type = map(map(object({
inherit_from_parent = optional(bool) # List policy only.
reset = optional(bool)
rules = optional(
list(object({
allow = optional(list(string)) # List policy only. Stands for `allow_all` if set to empty list `[]` or to `values.allowed_values` if set to a list of values
deny = optional(list(string)) # List policy only. Stands for `deny_all` if set to empty list `[]` or to `values.denied_values` if set to a list of values
enforce = optional(bool) # Boolean policy only.
condition = optional(
object({
description = optional(string)
expression = optional(string)
location = optional(string)
title = optional(string)
})
)
}))
)
})))
default = {}
}

View File

@ -0,0 +1,29 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
terraform {
required_version = ">= 1.1.0"
required_providers {
google = {
source = "hashicorp/google"
version = ">= 4.20.0" # tftest
}
google-beta = {
source = "hashicorp/google-beta"
version = ">= 4.20.0" # tftest
}
}
}

View File

@ -0,0 +1,13 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

View File

@ -0,0 +1,18 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
terraform {
# TODO: Remove once Terraform 1.3 is released https://github.com/hashicorp/terraform/releases/tag/v1.3.0-alpha20220622
experiments = [module_variable_optional_attrs]
}

View File

@ -0,0 +1,22 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
module "org-policy" {
source = "../../../../modules/organization-policy"
config_directory = var.config_directory
policies = var.policies
}

View File

@ -0,0 +1,40 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
organizations/1234567890:
constraints/compute.vmExternalIpAccess:
rules:
- deny_all: true
folders/1234567890:
compute.vmCanIpForward:
inherit_from_parent: false
reset: false
rules:
- allow: []
projects/my-project-id:
run.allowedIngress:
inherit_from_parent: true
rules:
- allow: ['internal']
condition:
description: allow internal ingress
expression: resource.matchTag("123456789/environment", "prod")
location: test.log
title: allow-for-prod
iam.allowServiceAccountCredentialLifetimeExtension:
rules:
- deny: []
compute.disableGlobalLoadBalancing:
reset: true

View File

@ -0,0 +1,46 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
variable "config_directory" {
description = "Paths to a folder where organization policy configs are stored in yaml format. Files suffix must be `.yaml`."
type = string
default = null
}
variable "policies" {
description = "Organization policies keyed by parent in format `projects/project-id`, `folders/1234567890` or `organizations/1234567890`."
type = map(map(object({
inherit_from_parent = optional(bool) # List policy only.
reset = optional(bool)
rules = optional(
list(object({
allow = optional(list(string)) # List policy only. Stands for `allow_all` if set to empty list `[]` or to `values.allowed_values` if set to a list of values
deny = optional(list(string)) # List policy only. Stands for `deny_all` if set to empty list `[]` or to `values.denied_values` if set to a list of values
enforce = optional(bool) # Boolean policy only.
condition = optional(
object({
description = optional(string)
expression = optional(string)
location = optional(string)
title = optional(string)
})
)
}))
)
})))
default = {}
}

View File

@ -0,0 +1,89 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
def test_org_policy_simple(plan_runner):
"Test vpc with no extra options."
org_policies = (
'{'
'"folders/1234567890" = {'
' "constraints/iam.disableServiceAccountKeyUpload" = {'
' rules = ['
' {'
' enforce = true,'
' }'
' ]'
' }'
' },'
' "organizations/1234567890" = {'
' "run.allowedIngress" = {'
' rules = ['
' {'
' allow = ["internal"],'
' condition = {'
' description= "allow ingress",'
' expression = "resource.matchTag(\'123456789/environment\', \'prod\')",'
' title = "allow-for-prod-org"'
' }'
' }'
' ]'
' }'
' }'
'}'
)
_, resources = plan_runner(
policies = org_policies
)
assert len(resources) == 2
org_policy = [r for r in resources if r["values"]
["name"].endswith('iam.disableServiceAccountKeyUpload')][0]["values"]
assert org_policy["parent"] == "folders/1234567890"
assert org_policy["spec"][0]["rules"][0]["enforce"] == "TRUE"
def test_org_policy_factory(plan_runner):
"Test yaml based configuration"
_, resources = plan_runner(
config_directory="./policies",
)
assert len(resources) == 5
org_policy = [r for r in resources if r["values"]
["name"].endswith('run.allowedIngress')][0]["values"]["spec"][0]
assert org_policy["inherit_from_parent"] == True
assert org_policy["rules"][0]["condition"][0]["title"] == "allow-for-prod"
assert set(org_policy["rules"][0]["values"][0]["allowed_values"]) == set(["internal"])
def test_combined_org_policy_config(plan_runner):
"Test combined (yaml, hcl) policy configuration"
org_policies = (
'{'
'"folders/3456789012" = {'
' "constraints/iam.disableServiceAccountKeyUpload" = {'
' rules = ['
' {'
' enforce = true'
' }'
' ]'
' }'
' }'
'}'
)
_, resources = plan_runner(
config_directory="./policies",
policies = org_policies
)
assert len(resources) == 6