2024-03-14 05:03:42 -07:00
# Project and Folder Factory
2022-01-14 01:29:09 -08:00
2024-03-14 05:03:42 -07:00
This module implements end-to-end creation processes for a folder hierarchy, projects and billing budgets via YAML data configurations.
2022-01-14 01:29:09 -08:00
2024-02-26 02:16:52 -08:00
It supports
2022-01-14 01:29:09 -08:00
2024-03-14 05:03:42 -07:00
- filesystem-driven folder hierarchy exposing the full configuration options available in the [folder module ](../folder/ )
2024-02-27 10:13:49 -08:00
- multiple project creation and management exposing the full configuration options available in the [project module ](../project/ ), including KMS key grants and VPC-SC perimeter membership
- optional per-project [service account management ](#service-accounts ) including basic IAM grants
- optional [billing budgets ](#billing-budgets ) factory and budget/project associations
2024-03-14 05:03:42 -07:00
- cross-referencing of hierarchy folders in projects
2024-02-27 10:13:49 -08:00
- optional per-project IaC configuration (TODO)
2024-02-26 02:16:52 -08:00
2024-03-14 05:03:42 -07:00
The factory is implemented as a thin data translation layer for the underlying modules, so that no "magic" or hidden side effects are implemented in code, and debugging or integration of new features are simple.
2022-01-14 01:29:09 -08:00
2023-08-20 00:44:20 -07:00
The code is meant to be executed by a high level service accounts with powerful permissions:
2022-01-14 01:29:09 -08:00
2024-03-14 05:03:42 -07:00
- forlder admin permissions for the hierarchy
2023-08-20 00:44:20 -07:00
- project creation on the nodes (folder or org) where projects will be defined
2024-03-14 05:03:42 -07:00
- Shared VPC connection if service project attachment is desired
- billing cost manager permissions to manage budgets and monitoring permissions if notifications should also be managed here
2022-01-14 01:29:09 -08:00
2024-02-27 10:13:49 -08:00
<!-- BEGIN TOC -->
2024-03-14 05:03:42 -07:00
- [Folder hierarchy ](#folder-hierarchy )
- [Projects ](#projects )
2024-03-19 08:50:06 -07:00
- [Factory-wide project defaults, merges, optionals ](#factory-wide-project-defaults-merges-optionals )
2024-02-27 10:13:49 -08:00
- [Service accounts ](#service-accounts )
2024-03-19 08:50:06 -07:00
- [Automation project and resources ](#automation-project-and-resources )
2024-03-14 05:03:42 -07:00
- [Billing budgets ](#billing-budgets )
2024-02-27 10:13:49 -08:00
- [Example ](#example )
2024-03-19 08:50:06 -07:00
- [Files ](#files )
2024-02-27 10:13:49 -08:00
- [Variables ](#variables )
- [Outputs ](#outputs )
- [Tests ](#tests )
<!-- END TOC -->
2024-03-14 05:03:42 -07:00
## Folder hierarchy
The hierarchy supports up to three levels of folders, which are defined via filesystem directories each including a `_config.yaml` files detailing their attributes.
The hierarchy factory is configured via the `factories_config.hierarchy` variable via one mandatory and one optional argument:
- `factories_config.hierarchy.folders_data_path` is required to enable the hierarchy factory, and must be set to the path containing the YAML definitions
- `factories_config.hierarchy.parent_ids` is an optional map where keys are arbitrary and values are set to resource node ids
Top-level folders in the filesystem hierarchy have no explicit parent, so their parent ids need to be provided in the YAML by either referencing the full id (e.g. `folders/12345678` ) or by referencing a key in the `parent_ids` attribute described above. As a shortcut, a `default` key can be defined whose value is used for any top-level folder which does not directly provide a parent id.
Filesystem directories can also contain project definitions in the same YAML format described below. This approach must be used with caution and is best adopted for stable scenarios, as problems in the filesystem hierarchy definitions might result in the project files not being read and the resources being deleted by Terraform.
Refer to the [example ](#example ) below for actual examples of the YAML definitions.
## Projects
The project factory is configured via the `factories_config.projects_data_path` variable, and project files are also read from the hierarchy describe in the previous section when enabled. The YAML format mirrors the project module, refer to the [example ](#example ) below for actual examples of the YAML definitions.
2024-03-19 08:50:06 -07:00
### Factory-wide project defaults, merges, optionals
2023-10-02 07:13:56 -07:00
2024-02-26 02:16:52 -08:00
In addition to the YAML-based project configurations, the factory accepts three additional sets of inputs via Terraform variables:
2023-10-02 07:13:56 -07:00
2024-02-26 02:16:52 -08:00
- the `data_defaults` variable allows defining defaults for specific project attributes, which are only used if the attributes are not passed in via YAML
- the `data_overrides` variable works similarly to defaults, but the values specified here take precedence over those in YAML files
- the `data_merges` variable allows specifying additional values for map or set based variables, which are merged with the data coming from YAML
2023-10-02 07:13:56 -07:00
2024-02-27 10:13:49 -08:00
Some examples on where to use each of the three sets are [provided below ](#example ).
### Service accounts
Service accounts can be managed as part of each project's YAML configuration. This allows creation of default service accounts used for GCE instances, in firewall rules, or for application-level credentials without resorting to a separate Terraform configuration.
2024-03-05 04:13:02 -08:00
Each service account is represented by one key and a set of optional key/value pairs in the `service_accounts` top-level YAML map, which expose most of the variables available in the `iam-service-account` module:
2024-02-27 10:13:49 -08:00
```yaml
service_accounts:
be-0: {}
fe-1:
display_name: GCE frontend service account.
2024-03-05 04:13:02 -08:00
iam_self_roles:
- roles/storage.objectViewer
2024-02-27 10:13:49 -08:00
iam_project_roles:
2024-03-05 04:13:02 -08:00
my-host-project:
- roles/compute.networkUser
2024-02-27 10:13:49 -08:00
```
2024-03-05 04:13:02 -08:00
Both the `display_name` and `iam_self_roles` attributes are optional.
2024-02-27 10:13:49 -08:00
2024-03-19 08:50:06 -07:00
### Automation project and resources
Project configurations also support defining service accounts and storage buckets to support automation, created in a separate controlling project so as to be outside of the sphere of control of the managed project.
Automation resources are defined via the `automation` attribute in project configurations, which supports:
- a mandatory `project` attribute to define the external controlling project
- an optional `service_accounts` list where each element will define a service account in the controlling project
- an optional `buckets` map where each key will define a bucket in the controlling project, and the map of roles/principals in the corresponding value assigned on the created bucket; principals can refer to the created service accounts by key
Service accounts and buckets will be prefixed with the project name, and use the key specified in the YAML file as a suffix.
```yaml
# file name: prod-app-example-0
# prefix via factory defaults: foo
# project id: foo-prod-app-example-0
billing_account: 012345-67890A-BCDEF0
parent: folders/12345678
services:
- compute.googleapis.com
- stackdriver.googleapis.com
iam:
roles/owner:
- rw
roles/viewer:
- ro
automation:
project: foo-prod-iac-core-0
service_accounts:
# sa name: foo-prod-app-example-0-rw
rw:
description: Read/write automation sa for app example 0.
# sa name: foo-prod-app-example-0-ro
ro:
description: Read-only automation sa for app example 0.
buckets:
# bucket name: foo-prod-app-example-0-state
state:
description: Terraform state bucket for app example 0.
iam:
roles/storage.objectCreator:
- rw
roles/storage.objectViewer:
- rw
- ro
- group:devops@example.org
```
2024-03-14 05:03:42 -07:00
## Billing budgets
2024-02-27 10:13:49 -08:00
2024-03-14 05:03:42 -07:00
The billing budgets factory integrates the `[` billing-account`](../billing-account/) module functionality, and adds support for easy referencing budgets in project files.
2024-02-27 10:13:49 -08:00
To enable support for billing budgets, set the billing account id, optional notification channels, and the data folder for budgets in the `factories_config.budgets` variable, then create billing budgets using YAML definitions following the format described in the `billing-account` module.
Once budgets are defined, they can be referenced in a project file using their file name:
```yaml
billing_account: 012345-67890A-BCDEF0
labels:
app: app-1
team: foo
parent: folders/12345678
services:
- container.googleapis.com
- storage.googleapis.com
billing_budgets:
- test-100
```
2024-03-19 08:50:06 -07:00
A simple billing budget example is show in the [example ](#example ) below.
2022-01-14 01:29:09 -08:00
2023-08-20 00:44:20 -07:00
## Example
2022-01-14 01:29:09 -08:00
2024-03-14 05:03:42 -07:00
The module invocation using all optional features:
2022-01-14 01:29:09 -08:00
```hcl
2023-08-20 00:44:20 -07:00
module "project-factory" {
2024-02-26 02:16:52 -08:00
source = "./fabric/modules/project-factory"
2023-10-02 07:13:56 -07:00
# use a default billing account if none is specified via yaml
2023-08-20 00:44:20 -07:00
data_defaults = {
2024-02-27 10:13:49 -08:00
billing_account = var.billing_account_id
2023-08-20 00:44:20 -07:00
}
2023-10-02 07:13:56 -07:00
# make sure the environment label and stackdriver service are always added
2023-08-20 00:44:20 -07:00
data_merges = {
labels = {
environment = "test"
}
services = [
"stackdriver.googleapis.com"
]
}
2023-10-02 07:13:56 -07:00
# always use this contaxt and prefix, regardless of what is in the yaml file
2023-08-20 00:44:20 -07:00
data_overrides = {
contacts = {
"admin@example.com" = ["ALL"]
}
prefix = "test-pf"
}
2023-10-02 07:13:56 -07:00
# location where the yaml files are read from
2024-02-27 10:13:49 -08:00
factories_config = {
budgets = {
billing_account = var.billing_account_id
budgets_data_path = "data/budgets"
notification_channels = {
billing-default = {
project_id = "foo-billing-audit"
type = "email"
labels = {
email_address = "gcp-billing-admins@example.com"
}
}
}
}
2024-03-14 05:03:42 -07:00
hierarchy = {
folders_data_path = "data/hierarchy"
parent_ids = {
default = "folders/12345678"
}
}
2024-02-27 10:13:49 -08:00
projects_data_path = "data/projects"
}
2022-01-14 01:29:09 -08:00
}
2024-03-19 08:50:06 -07:00
# tftest modules=16 resources=55 files=prj-app-1,prj-app-2,prj-app-3,budget-test-100,h-0-0,h-1-0,h-0-1,h-1-1,h-1-1-p0 inventory=example.yaml
2022-01-14 01:29:09 -08:00
```
2024-03-14 05:03:42 -07:00
A simple hierarchy of folders:
```yaml
name: Foo (level 1)
iam:
roles/viewer:
- group:a@example.com
# tftest-file id=h-0-0 path=data/hierarchy/foo/_config.yaml
```
```yaml
name: Bar (level 1)
parent: folders/4567890
# tftest-file id=h-1-0 path=data/hierarchy/bar/_config.yaml
```
```yaml
name: Foo Baz (level 2)
# tftest-file id=h-0-1 path=data/hierarchy/foo/baz/_config.yaml
```
```yaml
name: Bar Baz (level 2)
# tftest-file id=h-1-1 path=data/hierarchy/bar/baz/_config.yaml
```
One project defined within the folder hierarchy:
```yaml
billing_account: 012345-67890A-BCDEF0
services:
- container.googleapis.com
- storage.googleapis.com
2024-03-19 08:50:06 -07:00
# tftest-file id=h-1-1-p0 path=data/hierarchy/bar/baz/bar-baz-iac-0.yaml
2024-03-14 05:03:42 -07:00
```
More traditional project definitions via the project factory data:
2022-01-14 01:29:09 -08:00
```yaml
2024-02-27 10:13:49 -08:00
# project app-1
2023-08-20 00:44:20 -07:00
billing_account: 012345-67890A-BCDEF0
2022-01-14 01:29:09 -08:00
labels:
2023-08-20 00:44:20 -07:00
app: app-1
team: foo
2023-09-07 05:48:39 -07:00
parent: folders/12345678
2023-08-20 00:44:20 -07:00
service_encryption_key_ids:
compute:
- projects/kms-central-prj/locations/europe-west3/keyRings/my-keyring/cryptoKeys/europe3-gce
services:
2023-11-02 00:24:50 -07:00
- container.googleapis.com
- storage.googleapis.com
2023-08-20 00:44:20 -07:00
service_accounts:
2023-10-26 07:09:03 -07:00
app-1-be:
2024-03-05 04:13:02 -08:00
iam_self_roles:
2023-10-26 07:09:03 -07:00
- roles/logging.logWriter
- roles/monitoring.metricWriter
2024-03-05 04:13:02 -08:00
iam_project_roles:
my-host-project:
- roles/compute.networkUser
2023-10-26 07:09:03 -07:00
app-1-fe:
display_name: "Test app 1 frontend."
2024-03-05 04:13:02 -08:00
iam_project_roles:
my-host-project:
- roles/compute.networkUser
2024-02-27 10:13:49 -08:00
billing_budgets:
- test-100
# tftest-file id=prj-app-1 path=data/projects/prj-app-1.yaml
2022-01-14 01:29:09 -08:00
```
```yaml
2024-02-27 10:13:49 -08:00
# project app-2
2022-11-23 02:09:00 -08:00
labels:
2023-12-15 05:39:21 -08:00
app: app-2
team: foo
2023-09-07 05:48:39 -07:00
parent: folders/12345678
2023-12-15 05:39:21 -08:00
org_policies:
"compute.restrictSharedVpcSubnetworks":
rules:
- allow:
values:
- projects/foo-host/regions/europe-west1/subnetworks/prod-default-ew1
2022-11-23 02:09:00 -08:00
service_accounts:
2023-08-20 00:44:20 -07:00
app-2-be: {}
2023-11-02 00:24:50 -07:00
services:
- compute.googleapis.com
2023-12-07 01:07:48 -08:00
- container.googleapis.com
2023-11-02 00:24:50 -07:00
- run.googleapis.com
- storage.googleapis.com
2023-10-23 06:45:48 -07:00
shared_vpc_service_config:
host_project: foo-host
2023-12-07 01:07:48 -08:00
service_identity_iam:
"roles/vpcaccess.user":
2023-12-15 05:39:21 -08:00
- cloudrun
2023-12-07 01:07:48 -08:00
"roles/container.hostServiceAgentUser":
2023-12-15 05:39:21 -08:00
- container-engine
service_identity_subnet_iam:
europe-west1/prod-default-ew1:
- cloudservices
- container-engine
network_subnet_users:
europe-west1/prod-default-ew1:
- group:team-1@example.com
2022-01-14 01:29:09 -08:00
2024-02-27 10:13:49 -08:00
# tftest-file id=prj-app-2 path=data/projects/prj-app-2.yaml
2022-01-14 01:29:09 -08:00
```
2023-11-02 00:24:50 -07:00
2024-03-19 08:50:06 -07:00
This project uses a reference to a hierarchy folder, and defines a controlling project via the `automation` attributes:
2023-11-02 00:24:50 -07:00
```yaml
2024-03-19 08:50:06 -07:00
parent: bar/baz
2023-11-02 00:24:50 -07:00
services:
- run.googleapis.com
- storage.googleapis.com
2024-03-19 08:50:06 -07:00
iam:
"roles/owner":
- rw
"roles/viewer":
- ro
automation:
project: bar-baz-iac-0
service_accounts:
rw:
description: Read/write automation sa for app example 0.
ro:
description: Read-only automation sa for app example 0.
buckets:
state:
description: Terraform state bucket for app example 0.
iam:
roles/storage.objectCreator:
- rw
roles/storage.objectViewer:
- rw
- ro
- group:devops@example.org
2023-11-02 00:24:50 -07:00
2024-02-27 10:13:49 -08:00
# tftest-file id=prj-app-3 path=data/projects/prj-app-3.yaml
2023-11-02 00:24:50 -07:00
```
2023-12-15 05:39:21 -08:00
2024-03-14 05:03:42 -07:00
And a billing budget:
2024-02-27 10:13:49 -08:00
```yaml
# billing budget test-100
display_name: 100 dollars in current spend
amount:
units: 100
filter:
period:
calendar: MONTH
resource_ancestors:
- folders/1234567890
threshold_rules:
- percent: 0.5
- percent: 0.75
update_rules:
default:
disable_default_iam_recipients: true
monitoring_notification_channels:
- billing-default
# tftest-file id=budget-test-100 path=data/budgets/test-100.yaml
```
2024-03-14 05:03:42 -07:00
2024-03-19 08:50:06 -07:00
<!-- TFDOC OPTS files:1 -->
2022-01-14 01:29:09 -08:00
<!-- BEGIN TFDOC -->
2024-03-19 08:50:06 -07:00
## Files
| name | description | modules |
|---|---|---|
| [automation.tf ](./automation.tf ) | Automation projects locals and resources. | < code > gcs</ code > · < code > iam-service-account</ code > |
| [factory-budgets.tf ](./factory-budgets.tf ) | Billing budget factory locals. | |
| [factory-folders.tf ](./factory-folders.tf ) | Folder hierarchy factory locals. | |
| [factory-projects.tf ](./factory-projects.tf ) | Projects factory locals. | |
| [folders.tf ](./folders.tf ) | Folder hierarchy factory resources. | < code > folder</ code > |
| [main.tf ](./main.tf ) | Projects and billing budgets factory resources. | < code > billing-account</ code > · < code > iam-service-account</ code > · < code > project</ code > |
| [outputs.tf ](./outputs.tf ) | Module outputs. | |
| [variables.tf ](./variables.tf ) | Module variables. | |
2022-01-14 01:29:09 -08:00
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
2024-04-22 00:28:01 -07:00
| [factories_config ](variables.tf#L96 ) | Path to folder with YAML resource description data files. | < code title = "object({ hierarchy = optional(object({ folders_data_path = string parent_ids = optional(map(string), {}) })) projects_data_path = optional(string) budgets = optional(object({ billing_account = string budgets_data_path = string notification_channels = optional(map(any), {}) })) })" > object({…}) </ code > | ✓ | |
| [data_defaults ](variables.tf#L17 ) | Optional default values used when corresponding project data from files are missing. | < code title = "object({ billing_account = optional(string) contacts = optional(map(list(string)), {}) labels = optional(map(string), {}) metric_scopes = optional(list(string), []) parent = optional(string) prefix = optional(string) service_encryption_key_ids = optional(map(list(string)), {}) services = optional(list(string), []) shared_vpc_service_config = optional(object({ host_project = string network_users = optional(list(string), []) service_identity_iam = optional(map(list(string)), {}) service_identity_subnet_iam = optional(map(list(string)), {}) service_iam_grants = optional(list(string), []) network_subnet_users = optional(map(list(string)), {}) }), { host_project = null }) tag_bindings = optional(map(string), {}) service_accounts = optional(map(object({ display_name = optional(string, "Terraform-managed.") iam_self_roles = optional(list(string)) })), {}) vpc_sc = optional(object({ perimeter_name = string perimeter_bridges = optional(list(string), []) is_dry_run = optional(bool, false) })) })" > object({…}) </ code > | | < code > {} </ code > |
| [data_merges ](variables.tf#L52 ) | Optional values that will be merged with corresponding data from files. Combines with `data_defaults` , file data, and `data_overrides` . | < code title = "object({ contacts = optional(map(list(string)), {}) labels = optional(map(string), {}) metric_scopes = optional(list(string), []) service_encryption_key_ids = optional(map(list(string)), {}) services = optional(list(string), []) tag_bindings = optional(map(string), {}) service_accounts = optional(map(object({ display_name = optional(string, "Terraform-managed.") iam_self_roles = optional(list(string)) })), {}) })" > object({…}) </ code > | | < code > {} </ code > |
| [data_overrides ](variables.tf#L71 ) | Optional values that override corresponding data from files. Takes precedence over file data and `data_defaults` . | < code title = "object({ billing_account = optional(string) contacts = optional(map(list(string))) parent = optional(string) prefix = optional(string) service_encryption_key_ids = optional(map(list(string))) tag_bindings = optional(map(string)) services = optional(list(string)) service_accounts = optional(map(object({ display_name = optional(string, "Terraform-managed.") iam_self_roles = optional(list(string)) }))) vpc_sc = optional(object({ perimeter_name = string perimeter_bridges = optional(list(string), []) is_dry_run = optional(bool, false) })) })" > object({…}) </ code > | | < code > {} </ code > |
2022-01-14 01:29:09 -08:00
## Outputs
| name | description | sensitive |
|---|---|:---:|
2024-03-14 05:03:42 -07:00
| [folders ](outputs.tf#L17 ) | Folder ids. | |
| [projects ](outputs.tf#L22 ) | Project module outputs. | |
| [service_accounts ](outputs.tf#L27 ) | Service account emails. | |
2022-01-14 01:29:09 -08:00
<!-- END TFDOC -->
2023-11-02 00:24:50 -07:00
## Tests
These tests validate fixes to the project factory.
```hcl
module "project-factory" {
2024-02-26 02:16:52 -08:00
source = "./fabric/modules/project-factory"
2023-11-02 00:24:50 -07:00
data_defaults = {
billing_account = "012345-67890A-ABCDEF"
}
data_merges = {
labels = {
owner = "foo"
}
services = [
"compute.googleapis.com"
]
}
data_overrides = {
prefix = "foo"
}
2024-02-27 10:13:49 -08:00
factories_config = {
projects_data_path = "data/projects"
}
2023-11-02 00:24:50 -07:00
}
# tftest modules=4 resources=14 files=test-0,test-1,test-2
```
```yaml
parent: folders/1234567890
services:
- iam.googleapis.com
- contactcenteraiplatform.googleapis.com
- container.googleapis.com
2024-02-27 10:13:49 -08:00
# tftest-file id=test-0 path=data/projects/test-0.yaml
2023-11-02 00:24:50 -07:00
```
```yaml
parent: folders/1234567890
services:
- iam.googleapis.com
- contactcenteraiplatform.googleapis.com
2024-02-27 10:13:49 -08:00
# tftest-file id=test-1 path=data/projects/test-1.yaml
2023-11-02 00:24:50 -07:00
```
```yaml
parent: folders/1234567890
services:
- iam.googleapis.com
- storage.googleapis.com
2024-02-27 10:13:49 -08:00
# tftest-file id=test-2 path=data/projects/test-2.yaml
2023-11-02 00:24:50 -07:00
```