diff --git a/modules/project-factory/README.md b/modules/project-factory/README.md index e38e411d..e93fc6d0 100644 --- a/modules/project-factory/README.md +++ b/modules/project-factory/README.md @@ -23,10 +23,12 @@ The code is meant to be executed by a high level service accounts with powerful - [Folder hierarchy](#folder-hierarchy) - [Projects](#projects) - - [Leveraging project defaults, merges, optionals](#leveraging-project-defaults-merges-optionals) + - [Factory-wide project defaults, merges, optionals](#factory-wide-project-defaults-merges-optionals) - [Service accounts](#service-accounts) + - [Automation project and resources](#automation-project-and-resources) - [Billing budgets](#billing-budgets) - [Example](#example) +- [Files](#files) - [Variables](#variables) - [Outputs](#outputs) - [Tests](#tests) @@ -51,7 +53,7 @@ Refer to the [example](#example) below for actual examples of the YAML definitio The project factory is configured via the `factories_config.projects_data_path` variable, and project files are also read from the hierarchy describe in the previous section when enabled. The YAML format mirrors the project module, refer to the [example](#example) below for actual examples of the YAML definitions. -### Leveraging project defaults, merges, optionals +### Factory-wide project defaults, merges, optionals In addition to the YAML-based project configurations, the factory accepts three additional sets of inputs via Terraform variables: @@ -81,6 +83,54 @@ service_accounts: Both the `display_name` and `iam_self_roles` attributes are optional. +### Automation project and resources + +Project configurations also support defining service accounts and storage buckets to support automation, created in a separate controlling project so as to be outside of the sphere of control of the managed project. + +Automation resources are defined via the `automation` attribute in project configurations, which supports: + +- a mandatory `project` attribute to define the external controlling project +- an optional `service_accounts` list where each element will define a service account in the controlling project +- an optional `buckets` map where each key will define a bucket in the controlling project, and the map of roles/principals in the corresponding value assigned on the created bucket; principals can refer to the created service accounts by key + +Service accounts and buckets will be prefixed with the project name, and use the key specified in the YAML file as a suffix. + +```yaml +# file name: prod-app-example-0 +# prefix via factory defaults: foo +# project id: foo-prod-app-example-0 +billing_account: 012345-67890A-BCDEF0 +parent: folders/12345678 +services: + - compute.googleapis.com + - stackdriver.googleapis.com +iam: + roles/owner: + - rw + roles/viewer: + - ro +automation: + project: foo-prod-iac-core-0 + service_accounts: + # sa name: foo-prod-app-example-0-rw + rw: + description: Read/write automation sa for app example 0. + # sa name: foo-prod-app-example-0-ro + ro: + description: Read-only automation sa for app example 0. + buckets: + # bucket name: foo-prod-app-example-0-state + state: + description: Terraform state bucket for app example 0. + iam: + roles/storage.objectCreator: + - rw + roles/storage.objectViewer: + - rw + - ro + - group:devops@example.org +``` + ## Billing budgets The billing budgets factory integrates the `[`billing-account`](../billing-account/) module functionality, and adds support for easy referencing budgets in project files. @@ -102,7 +152,7 @@ billing_budgets: - test-100 ``` -The example below shows how to use the billing budgets factory. +A simple billing budget example is show in the [example](#example) below. ## Example @@ -155,7 +205,7 @@ module "project-factory" { projects_data_path = "data/projects" } } -# tftest modules=13 resources=48 files=prj-app-1,prj-app-2,prj-app-3,budget-test-100,h-0-0,h-1-0,h-0-1,h-1-1,h-1-1-p0 inventory=example.yaml +# tftest modules=16 resources=55 files=prj-app-1,prj-app-2,prj-app-3,budget-test-100,h-0-0,h-1-0,h-0-1,h-1-1,h-1-1-p0 inventory=example.yaml ``` A simple hierarchy of folders: @@ -191,7 +241,7 @@ billing_account: 012345-67890A-BCDEF0 services: - container.googleapis.com - storage.googleapis.com -# tftest-file id=h-1-1-p0 path=data/hierarchy/bar/baz/bar-baz-0.yaml +# tftest-file id=h-1-1-p0 path=data/hierarchy/bar/baz/bar-baz-iac-0.yaml ``` More traditional project definitions via the project factory data: @@ -264,12 +314,36 @@ shared_vpc_service_config: # tftest-file id=prj-app-2 path=data/projects/prj-app-2.yaml ``` +This project uses a reference to a hierarchy folder, and defines a controlling project via the `automation` attributes: + ```yaml -# project app-3 -parent: folders/12345678 +parent: bar/baz services: - run.googleapis.com - storage.googleapis.com +iam: + "roles/owner": + - rw + "roles/viewer": + - ro +automation: + project: bar-baz-iac-0 + service_accounts: + rw: + description: Read/write automation sa for app example 0. + ro: + description: Read-only automation sa for app example 0. + buckets: + state: + description: Terraform state bucket for app example 0. + iam: + roles/storage.objectCreator: + - rw + roles/storage.objectViewer: + - rw + - ro + - group:devops@example.org + # tftest-file id=prj-app-3 path=data/projects/prj-app-3.yaml ``` @@ -297,7 +371,21 @@ update_rules: # tftest-file id=budget-test-100 path=data/budgets/test-100.yaml ``` + +## Files + +| name | description | modules | +|---|---|---| +| [automation.tf](./automation.tf) | Automation projects locals and resources. | gcs · iam-service-account | +| [factory-budgets.tf](./factory-budgets.tf) | Billing budget factory locals. | | +| [factory-folders.tf](./factory-folders.tf) | Folder hierarchy factory locals. | | +| [factory-projects.tf](./factory-projects.tf) | Projects factory locals. | | +| [folders.tf](./folders.tf) | Folder hierarchy factory resources. | folder | +| [main.tf](./main.tf) | Projects and billing budgets factory resources. | billing-account · iam-service-account · project | +| [outputs.tf](./outputs.tf) | Module outputs. | | +| [variables.tf](./variables.tf) | Module variables. | | + ## Variables | name | description | type | required | default | diff --git a/modules/project-factory/automation.tf b/modules/project-factory/automation.tf new file mode 100644 index 00000000..104739f7 --- /dev/null +++ b/modules/project-factory/automation.tf @@ -0,0 +1,109 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +# tfdoc:file:description Automation projects locals and resources. + +locals { + automation_buckets = flatten([ + for k, v in local.projects : [ + for ks, kv in try(v.automation.buckets, {}) : merge(kv, { + automation_project = v.automation.project + name = ks + prefix = v.prefix + project = k + }) + ] + ]) + automation_sa = flatten([ + for k, v in local.projects : [ + for ks, kv in try(v.automation.service_accounts, {}) : merge(kv, { + automation_project = v.automation.project + name = ks + prefix = v.prefix + project = k + }) + ] + ]) +} + +module "automation-buckets" { + source = "../gcs" + for_each = { + for k in local.automation_buckets : "${k.project}/${k.name}" => k + } + project_id = each.value.automation_project + prefix = each.value.prefix + name = "${each.value.project}-${each.value.name}" + encryption_key = lookup(each.value, "encryption_key", null) + # try interpolating service accounts by key in principals + iam = { + for k, v in lookup(each.value, "iam", {}) : k => [ + for vv in v : try( + module.automation-service-accounts["${each.value.project}/${vv}"].iam_email, + vv + ) + ] + } + iam_bindings = { + for k, v in lookup(each.value, "iam_bindings", {}) : k => merge(v, { + members = [ + for vv in v.members : try( + module.automation-service-accounts["${each.value.project}/${vv}"].iam_email, + vv + ) + ] + }) + } + iam_bindings_additive = { + for k, v in lookup(each.value, "iam_bindings_additive", {}) : k => merge(v, { + member = try( + module.automation-service-accounts["${each.value.project}/${v.member}"].iam_email, + v.member + ) + }) + } + labels = lookup(each.value, "labels", {}) + location = lookup(each.value, "location", "EU") + storage_class = lookup(each.value, "storage_class", "MULTI_REGIONAL") + uniform_bucket_level_access = lookup(each.value, "uniform_bucket_level_access", true) + versioning = lookup(each.value, "versioning", false) +} + +module "automation-service-accounts" { + source = "../iam-service-account" + for_each = { + for k in local.automation_sa : "${k.project}/${k.name}" => k + } + project_id = each.value.automation_project + prefix = each.value.prefix + name = "${each.value.project}-${each.value.name}" + description = lookup(each.value, "description", null) + display_name = lookup( + each.value, + "display_name", + "Service account ${each.value.name} for ${each.value.project}." + ) + iam = lookup(each.value, "iam", {}) + iam_bindings = lookup(each.value, "iam_bindings", {}) + iam_bindings_additive = lookup(each.value, "iam_bindings_additive", {}) + iam_billing_roles = lookup(each.value, "iam_billing_roles", {}) + iam_folder_roles = lookup(each.value, "iam_folder_roles", {}) + iam_organization_roles = lookup(each.value, "iam_organization_roles", {}) + iam_project_roles = lookup(each.value, "iam_project_roles", {}) + iam_sa_roles = lookup(each.value, "iam_sa_roles", {}) + # we don't interpolate buckets here as we can't use a dynamic key + iam_storage_roles = lookup(each.value, "iam_storage_roles", {}) +} diff --git a/modules/project-factory/factory-budgets.tf b/modules/project-factory/factory-budgets.tf index 228eb2c8..82f4b6bd 100644 --- a/modules/project-factory/factory-budgets.tf +++ b/modules/project-factory/factory-budgets.tf @@ -14,6 +14,8 @@ * limitations under the License. */ +# tfdoc:file:description Billing budget factory locals. + locals { # reimplement the billing account factory here to interpolate projects _budget_path = try(pathexpand(var.factories_config.budgets.budgets_data_path), null) diff --git a/modules/project-factory/factory-folders.tf b/modules/project-factory/factory-folders.tf index 99e91a47..cd9537f4 100644 --- a/modules/project-factory/factory-folders.tf +++ b/modules/project-factory/factory-folders.tf @@ -14,6 +14,8 @@ * limitations under the License. */ +# tfdoc:file:description Folder hierarchy factory locals. + locals { _folders_path = try( pathexpand(var.factories_config.hierarchy.folders_data_path), null @@ -51,53 +53,3 @@ check "hierarchy-data" { error_message = "No default set for hierarchy parent ids." } } - -module "hierarchy-folder-lvl-1" { - source = "../folder" - for_each = { for k, v in local.folders : k => v if v.level == 1 } - parent = try( - # allow the YAML data to set the parent for this level - lookup( - var.factories_config.hierarchy.parent_ids, - each.value.parent, - # use the value as is if it's not in the parents map - each.value.parent - ), - # use the default value in the initial parents map - var.factories_config.hierarchy.parent_ids.default - # fail if we don't have an explicit parent - ) - name = each.value.name - iam = lookup(each.value, "iam", {}) - iam_bindings = lookup(each.value, "iam_bindings", {}) - iam_bindings_additive = lookup(each.value, "iam_bindings_additive", {}) - iam_by_principals = lookup(each.value, "iam_by_principals", {}) - org_policies = lookup(each.value, "org_policies", {}) - tag_bindings = lookup(each.value, "tag_bindings", {}) -} - -module "hierarchy-folder-lvl-2" { - source = "../folder" - for_each = { for k, v in local.folders : k => v if v.level == 2 } - parent = module.hierarchy-folder-lvl-1[each.value.parent_key].id - name = each.value.name - iam = lookup(each.value, "iam", {}) - iam_bindings = lookup(each.value, "iam_bindings", {}) - iam_bindings_additive = lookup(each.value, "iam_bindings_additive", {}) - iam_by_principals = lookup(each.value, "iam_by_principals", {}) - org_policies = lookup(each.value, "org_policies", {}) - tag_bindings = lookup(each.value, "tag_bindings", {}) -} - -module "hierarchy-folder-lvl-3" { - source = "../folder" - for_each = { for k, v in local.folders : k => v if v.level == 3 } - parent = module.hierarchy-folder-lvl-2[each.value.parent_key].id - name = each.value.name - iam = lookup(each.value, "iam", {}) - iam_bindings = lookup(each.value, "iam_bindings", {}) - iam_bindings_additive = lookup(each.value, "iam_bindings_additive", {}) - iam_by_principals = lookup(each.value, "iam_by_principals", {}) - org_policies = lookup(each.value, "org_policies", {}) - tag_bindings = lookup(each.value, "tag_bindings", {}) -} diff --git a/modules/project-factory/factory-projects.tf b/modules/project-factory/factory-projects.tf index 1c5c2f19..aef73ef0 100644 --- a/modules/project-factory/factory-projects.tf +++ b/modules/project-factory/factory-projects.tf @@ -13,6 +13,9 @@ * See the License for the specific language governing permissions and * limitations under the License. */ + +# tfdoc:file:description Projects factory locals. + locals { _hierarchy_projects = ( { diff --git a/modules/project-factory/folders.tf b/modules/project-factory/folders.tf new file mode 100644 index 00000000..a846ce27 --- /dev/null +++ b/modules/project-factory/folders.tf @@ -0,0 +1,67 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +# tfdoc:file:description Folder hierarchy factory resources. + +module "hierarchy-folder-lvl-1" { + source = "../folder" + for_each = { for k, v in local.folders : k => v if v.level == 1 } + parent = try( + # allow the YAML data to set the parent for this level + lookup( + var.factories_config.hierarchy.parent_ids, + each.value.parent, + # use the value as is if it's not in the parents map + each.value.parent + ), + # use the default value in the initial parents map + var.factories_config.hierarchy.parent_ids.default + # fail if we don't have an explicit parent + ) + name = each.value.name + iam = lookup(each.value, "iam", {}) + iam_bindings = lookup(each.value, "iam_bindings", {}) + iam_bindings_additive = lookup(each.value, "iam_bindings_additive", {}) + iam_by_principals = lookup(each.value, "iam_by_principals", {}) + org_policies = lookup(each.value, "org_policies", {}) + tag_bindings = lookup(each.value, "tag_bindings", {}) +} + +module "hierarchy-folder-lvl-2" { + source = "../folder" + for_each = { for k, v in local.folders : k => v if v.level == 2 } + parent = module.hierarchy-folder-lvl-1[each.value.parent_key].id + name = each.value.name + iam = lookup(each.value, "iam", {}) + iam_bindings = lookup(each.value, "iam_bindings", {}) + iam_bindings_additive = lookup(each.value, "iam_bindings_additive", {}) + iam_by_principals = lookup(each.value, "iam_by_principals", {}) + org_policies = lookup(each.value, "org_policies", {}) + tag_bindings = lookup(each.value, "tag_bindings", {}) +} + +module "hierarchy-folder-lvl-3" { + source = "../folder" + for_each = { for k, v in local.folders : k => v if v.level == 3 } + parent = module.hierarchy-folder-lvl-2[each.value.parent_key].id + name = each.value.name + iam = lookup(each.value, "iam", {}) + iam_bindings = lookup(each.value, "iam_bindings", {}) + iam_bindings_additive = lookup(each.value, "iam_bindings_additive", {}) + iam_by_principals = lookup(each.value, "iam_by_principals", {}) + org_policies = lookup(each.value, "org_policies", {}) + tag_bindings = lookup(each.value, "tag_bindings", {}) +} diff --git a/modules/project-factory/main.tf b/modules/project-factory/main.tf index 2f440de6..0df66a7c 100644 --- a/modules/project-factory/main.tf +++ b/modules/project-factory/main.tf @@ -14,6 +14,8 @@ * limitations under the License. */ +# tfdoc:file:description Projects and billing budgets factory resources. + module "projects" { source = "../project" for_each = local.projects @@ -31,10 +33,35 @@ module "projects" { ) default_service_account = try(each.value.default_service_account, "keep") descriptive_name = try(each.value.descriptive_name, null) - iam = try(each.value.iam, {}) - iam_bindings = try(each.value.iam_bindings, {}) - iam_bindings_additive = try(each.value.iam_bindings_additive, {}) - iam_by_principals = try(each.value.iam_by_principals, {}) + # IAM interpolates automation service accounts + iam = { + for k, v in lookup(each.value, "iam", {}) : k => [ + for vv in v : try( + module.automation-service-accounts["${each.key}/${vv}"].iam_email, + vv + ) + ] + } + iam_bindings = { + for k, v in lookup(each.value, "iam_bindings", {}) : k => merge(v, { + members = [ + for vv in v.members : try( + module.automation-service-accounts["${each.key}/${vv}"].iam_email, + vv + ) + ] + }) + } + iam_bindings_additive = { + for k, v in lookup(each.value, "iam_bindings_additive", {}) : k => merge(v, { + member = try( + module.automation-service-accounts["${each.key}/${v.member}"].iam_email, + v.member + ) + }) + } + # IAM principals would trigger dynamic key errors so we don't interpolate + iam_by_principals = try(each.value.iam_by_principals, {}) labels = merge( each.value.labels, var.data_merges.labels ) diff --git a/tests/modules/project_factory/examples/example.yaml b/tests/modules/project_factory/examples/example.yaml index 3e14fc28..592f8e53 100644 --- a/tests/modules/project_factory/examples/example.yaml +++ b/tests/modules/project_factory/examples/example.yaml @@ -12,6 +12,107 @@ # See the License for the specific language governing permissions and # limitations under the License. +values: + module.project-factory.module.automation-buckets["prj-app-3/state"].google_storage_bucket.bucket: + autoclass: + - enabled: false + cors: [] + custom_placement_config: [] + default_event_based_hold: null + enable_object_retention: null + encryption: [] + force_destroy: false + labels: null + lifecycle_rule: [] + location: EU + logging: [] + name: test-pf-prj-app-3-state + project: bar-baz-iac-0 + requester_pays: null + retention_policy: [] + storage_class: MULTI_REGIONAL + timeouts: null + uniform_bucket_level_access: true + versioning: + - enabled: false + ? module.project-factory.module.automation-buckets["prj-app-3/state"].google_storage_bucket_iam_binding.authoritative["roles/storage.objectCreator"] + : bucket: test-pf-prj-app-3-state + condition: [] + members: + - serviceAccount:test-pf-prj-app-3-rw@bar-baz-iac-0.iam.gserviceaccount.com + role: roles/storage.objectCreator + ? module.project-factory.module.automation-buckets["prj-app-3/state"].google_storage_bucket_iam_binding.authoritative["roles/storage.objectViewer"] + : bucket: test-pf-prj-app-3-state + condition: [] + members: + - group:devops@example.org + - serviceAccount:test-pf-prj-app-3-ro@bar-baz-iac-0.iam.gserviceaccount.com + - serviceAccount:test-pf-prj-app-3-rw@bar-baz-iac-0.iam.gserviceaccount.com + role: roles/storage.objectViewer + module.project-factory.module.automation-service-accounts["prj-app-3/ro"].google_service_account.service_account[0]: + account_id: test-pf-prj-app-3-ro + create_ignore_already_exists: null + description: Read-only automation sa for app example 0. + disabled: false + display_name: Service account ro for prj-app-3. + project: bar-baz-iac-0 + timeouts: null + module.project-factory.module.automation-service-accounts["prj-app-3/rw"].google_service_account.service_account[0]: + account_id: test-pf-prj-app-3-rw + create_ignore_already_exists: null + description: Read/write automation sa for app example 0. + disabled: false + display_name: Service account rw for prj-app-3. + project: bar-baz-iac-0 + timeouts: null + module.project-factory.module.hierarchy-folder-lvl-1["bar"].google_folder.folder[0]: + display_name: Bar (level 1) + parent: folders/4567890 + timeouts: null + module.project-factory.module.hierarchy-folder-lvl-1["foo"].google_folder.folder[0]: + display_name: Foo (level 1) + parent: folders/12345678 + timeouts: null + module.project-factory.module.hierarchy-folder-lvl-1["foo"].google_folder_iam_binding.authoritative["roles/viewer"]: + condition: [] + members: + - group:a@example.com + role: roles/viewer + module.project-factory.module.hierarchy-folder-lvl-2["bar/baz"].google_folder.folder[0]: + display_name: Bar Baz (level 2) + timeouts: null + module.project-factory.module.hierarchy-folder-lvl-2["foo/baz"].google_folder.folder[0]: + display_name: Foo Baz (level 2) + timeouts: null + module.project-factory.module.projects["bar-baz-iac-0"].data.google_storage_project_service_account.gcs_sa[0]: + project: test-pf-bar-baz-iac-0 + user_project: null + module.project-factory.module.projects["prj-app-3"].google_project.project[0]: + auto_create_network: false + billing_account: 123456-123456-123456 + effective_labels: + environment: test + labels: + environment: test + name: test-pf-prj-app-3 + project_id: test-pf-prj-app-3 + skip_delete: false + terraform_labels: + environment: test + timeouts: null + module.project-factory.module.projects["prj-app-3"].google_project_iam_binding.authoritative["roles/owner"]: + condition: [] + members: + - serviceAccount:test-pf-prj-app-3-rw@bar-baz-iac-0.iam.gserviceaccount.com + project: test-pf-prj-app-3 + role: roles/owner + module.project-factory.module.projects["prj-app-3"].google_project_iam_binding.authoritative["roles/viewer"]: + condition: [] + members: + - serviceAccount:test-pf-prj-app-3-ro@bar-baz-iac-0.iam.gserviceaccount.com + project: test-pf-prj-app-3 + role: roles/viewer + counts: google_billing_budget: 1 google_compute_shared_vpc_service_project: 1 @@ -23,9 +124,12 @@ counts: google_monitoring_notification_channel: 1 google_org_policy_policy: 1 google_project: 4 + google_project_iam_binding: 2 google_project_iam_member: 6 google_project_service: 14 - google_service_account: 3 + google_service_account: 5 + google_storage_bucket: 1 + google_storage_bucket_iam_binding: 2 google_storage_project_service_account: 4 - modules: 13 - resources: 48 + modules: 16 + resources: 55