From 6700faf662e80491984433a22f13ba2adac03eee Mon Sep 17 00:00:00 2001 From: Julio Castillo Date: Thu, 10 Feb 2022 12:49:48 +0100 Subject: [PATCH 1/7] Document log sinks in stage 0 --- fast/stages/00-bootstrap/README.md | 24 ++++++++++++++++++++---- fast/stages/00-bootstrap/variables.tf | 2 ++ 2 files changed, 22 insertions(+), 4 deletions(-) diff --git a/fast/stages/00-bootstrap/README.md b/fast/stages/00-bootstrap/README.md index 3842aa07..f5bf29d0 100644 --- a/fast/stages/00-bootstrap/README.md +++ b/fast/stages/00-bootstrap/README.md @@ -54,6 +54,11 @@ For same-organization billing, we configure a custom organization role that can For details on configuring the different billing account modes, refer to the [How to run this stage](#how-to-run-this-stage) section below. +### Organization-level logging +We create organization-level log sinks early in the bootstrap process to ensure a proper audit trail is in place from the very beginning. By default, we provide log filters to capture [Cloud Audit Logs](https://cloud.google.com/logging/docs/audit) and [VPC Service Controls violations](https://cloud.google.com/vpc-service-controls/docs/troubleshooting#vpc-sc-errors) into a Bigquery dataset in the top-level audit project. + +The [Customizations](#log-sinks-and-log-destinations) section explains how to change the logs captured and their destination. + ### Naming We are intentionally not supporting random prefix/suffixes for names, as that is an antipattern typically only used in development. It does not map to our customer's actual production usage, where they always adopt a fixed naming convention. @@ -278,6 +283,17 @@ In those cases where roles need to be assigned to end-user service accounts (e.g The one exception to this convention is for roles which are part of the delegated grant condition described above, and which can then be assigned from other stages. In this case, use the `iam_additive` variable as they are implemented with non-authoritative resources. Using non-authoritative bindings ensure that re-executing this stage will not override any bindings set in downstream stages. +### Log sinks and log destinations + +You can customize organization-level logs through the `log_sinks` variable in two ways: + +* creating additional log sinks to capture more logs +* changing the destination of captured logs + +By default, all logs are exported to Bigquery, but FAST can create sinks to Cloud Logging Buckets, GCS, or PubSub. + +If you need to capture additional logs, please refer to GCP's documentation on [scenarios for exporting logging data](https://cloud.google.com/architecture/exporting-stackdriver-logging-for-security-and-access-analytics), where you can find ready-made filter expressions for different use cases. + ### Names and naming convention Configuring the individual tokens for the naming convention described above, has varying degrees of complexity: @@ -311,14 +327,14 @@ Names used in internal references (e.g. `module.foo-prod.id`) are only used by T | name | description | type | required | default | producer | |---|---|:---:|:---:|:---:|:---:| | [billing_account](variables.tf#L17) | Billing account id and organization id ('nnnnnnnn' or null). | object({…}) | ✓ | | | -| [organization](variables.tf#L82) | Organization details. | object({…}) | ✓ | | | -| [prefix](variables.tf#L97) | Prefix used for resources that need unique names. | string | ✓ | | | +| [organization](variables.tf#L84) | Organization details. | object({…}) | ✓ | | | +| [prefix](variables.tf#L99) | Prefix used for resources that need unique names. | string | ✓ | | | | [bootstrap_user](variables.tf#L25) | Email of the nominal user running this stage for the first time. | string | | null | | | [groups](variables.tf#L31) | Group names to grant organization-level permissions. | map(string) | | {…} | | | [iam](variables.tf#L45) | Organization-level custom IAM settings in role => [principal] format. | map(list(string)) | | {} | | | [iam_additive](variables.tf#L51) | Organization-level custom IAM settings in role => [principal] format for non-authoritative bindings. | map(list(string)) | | {} | | -| [log_sinks](variables.tf#L57) | Org-level log sinks, in name => {type, filter} format. | map(object({…})) | | {…} | | -| [outputs_location](variables.tf#L91) | Path where providers and tfvars files for the following stages are written. Leave empty to disable. | string | | null | | +| [log_sinks](variables.tf#L59) | Org-level log sinks, in name => {type, filter} format. | map(object({…})) | | {…} | | +| [outputs_location](variables.tf#L93) | Path where providers and tfvars files for the following stages are written. Leave empty to disable. | string | | null | | ## Outputs diff --git a/fast/stages/00-bootstrap/variables.tf b/fast/stages/00-bootstrap/variables.tf index 9f102c77..1bf0e2de 100644 --- a/fast/stages/00-bootstrap/variables.tf +++ b/fast/stages/00-bootstrap/variables.tf @@ -54,6 +54,8 @@ variable "iam_additive" { default = {} } +# See https://cloud.google.com/architecture/exporting-stackdriver-logging-for-security-and-access-analytics +# for additional logging filter examples variable "log_sinks" { description = "Org-level log sinks, in name => {type, filter} format." type = map(object({ From 2696af1a7ef13282b98946e1769ba21a80592cc1 Mon Sep 17 00:00:00 2001 From: Simone Ruffilli Date: Thu, 10 Feb 2022 16:24:14 +0100 Subject: [PATCH 2/7] Update vpc-sc.tf - manage empty perimeters. (#530) * Update vpc-sc.tf - manage empty perimeters. Co-authored-by: Ludovico Magnocavallo --- fast/stages/02-security/vpc-sc.tf | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/fast/stages/02-security/vpc-sc.tf b/fast/stages/02-security/vpc-sc.tf index b3125541..49611e0f 100644 --- a/fast/stages/02-security/vpc-sc.tf +++ b/fast/stages/02-security/vpc-sc.tf @@ -15,6 +15,7 @@ */ locals { + _perimeter_names = ["dev", "landing", "prod"] # dereference perimeter egress policy names to the actual objects _vpc_sc_perimeter_egress_policies = { for k, v in coalesce(var.vpc_sc_perimeter_egress_policies, {}) : @@ -33,8 +34,8 @@ locals { } # compute the number of projects in each perimeter to detect which to create vpc_sc_counts = { - for k in ["dev", "landing", "prod"] : k => length( - coalesce(try(var.vpc_sc_perimeter_projects[k], null), []) + for k in local._perimeter_names : k => length( + local.vpc_sc_perimeter_projects[k] ) } # define dry run spec at file level for convenience @@ -42,12 +43,12 @@ locals { # compute perimeter bridge resources (projects) vpc_sc_p_bridge_resources = { landing_to_dev = concat( - var.vpc_sc_perimeter_projects.landing, - var.vpc_sc_perimeter_projects.dev + local.vpc_sc_perimeter_projects.landing, + local.vpc_sc_perimeter_projects.dev ) landing_to_prod = concat( - var.vpc_sc_perimeter_projects.landing, - var.vpc_sc_perimeter_projects.prod + local.vpc_sc_perimeter_projects.landing, + local.vpc_sc_perimeter_projects.prod ) } # computer perimeter regular specs / status @@ -56,7 +57,7 @@ locals { access_levels = coalesce( try(var.vpc_sc_perimeter_access_levels.dev, null), [] ) - resources = var.vpc_sc_perimeter_projects.dev + resources = local.vpc_sc_perimeter_projects.dev restricted_services = local.vpc_sc_restricted_services egress_policies = try( local._vpc_sc_perimeter_egress_policies.dev, null @@ -74,7 +75,7 @@ locals { access_levels = coalesce( try(var.vpc_sc_perimeter_access_levels.landing, null), [] ) - resources = var.vpc_sc_perimeter_projects.landing + resources = local.vpc_sc_perimeter_projects.landing restricted_services = local.vpc_sc_restricted_services egress_policies = try( local._vpc_sc_perimeter_egress_policies.landing, null @@ -93,7 +94,7 @@ locals { try(var.vpc_sc_perimeter_access_levels.prod, null), [] ) # combine the security project, and any specified in the variable - resources = var.vpc_sc_perimeter_projects.prod + resources = local.vpc_sc_perimeter_projects.prod restricted_services = local.vpc_sc_restricted_services egress_policies = try( local._vpc_sc_perimeter_egress_policies.prod, null @@ -108,6 +109,17 @@ locals { # } } } + # account for null values in variable + vpc_sc_perimeter_projects = ( + var.vpc_sc_perimeter_projects == null ? + { + for k in local._perimeter_names : k => [] + } + : { + for k, v in local._perimeter_names : + k => v == null ? [] : v + } + ) # get the list of restricted services from the yaml file vpc_sc_restricted_services = yamldecode( file("${path.module}/vpc-sc-restricted-services.yaml") From ae49074921c21f0312f6207fac77639830ad909a Mon Sep 17 00:00:00 2001 From: Ludovico Magnocavallo Date: Thu, 10 Feb 2022 17:25:38 +0100 Subject: [PATCH 3/7] Update terraform.tfvars.sample --- fast/stages/00-bootstrap/terraform.tfvars.sample | 1 + 1 file changed, 1 insertion(+) diff --git a/fast/stages/00-bootstrap/terraform.tfvars.sample b/fast/stages/00-bootstrap/terraform.tfvars.sample index e9b9a90a..7dbe2e6b 100644 --- a/fast/stages/00-bootstrap/terraform.tfvars.sample +++ b/fast/stages/00-bootstrap/terraform.tfvars.sample @@ -1,4 +1,5 @@ # use `gcloud beta billing accounts list` +# if you have too many accounts, check the Cloud Console :) billing_account = { id = "012345-67890A-BCDEF0" organization_id = 1234567890 From c4d36cc66b05de2f848abf9ed055e545bbbc938c Mon Sep 17 00:00:00 2001 From: Ludovico Magnocavallo Date: Thu, 10 Feb 2022 19:12:07 +0100 Subject: [PATCH 4/7] Allow specifying custom role names --- fast/stages/00-bootstrap/organization.tf | 6 +++--- fast/stages/00-bootstrap/outputs.tf | 8 ++++++-- fast/stages/00-bootstrap/variables.tf | 8 ++++++++ fast/stages/02-networking-vpn/vpc-spoke-dev.tf | 2 +- fast/stages/02-networking-vpn/vpc-spoke-prod.tf | 2 +- 5 files changed, 19 insertions(+), 7 deletions(-) diff --git a/fast/stages/00-bootstrap/organization.tf b/fast/stages/00-bootstrap/organization.tf index ea2e4cef..689f378b 100644 --- a/fast/stages/00-bootstrap/organization.tf +++ b/fast/stages/00-bootstrap/organization.tf @@ -147,12 +147,12 @@ module "organization" { iam_additive = local.iam_additive custom_roles = { # this is needed for use in additive IAM bindings, to avoid conflicts - "organizationIamAdmin" = [ + (var.custom_role_names.organization_iam_admin) = [ "resourcemanager.organizations.get", "resourcemanager.organizations.getIamPolicy", "resourcemanager.organizations.setIamPolicy" ] - "serviceProjectNetworkAdmin" = [ + (var.custom_role_names.service_project_network_admin) = [ "compute.globalOperations.get", "compute.organizations.disableXpnResource", "compute.organizations.enableXpnResource", @@ -182,7 +182,7 @@ module "organization" { resource "google_organization_iam_binding" "org_admin_delegated" { org_id = var.organization.id - role = module.organization.custom_role_id.organizationIamAdmin + role = module.organization.custom_role_id[var.custom_role_names.organization_iam_admin] members = [module.automation-tf-resman-sa.iam_email] condition { title = "automation_sa_delegated_grants" diff --git a/fast/stages/00-bootstrap/outputs.tf b/fast/stages/00-bootstrap/outputs.tf index d7924781..d074d91f 100644 --- a/fast/stages/00-bootstrap/outputs.tf +++ b/fast/stages/00-bootstrap/outputs.tf @@ -15,6 +15,10 @@ */ locals { + _custom_roles = { + for k, v in var.custom_role_names : + k => module.organization.custom_role_id[v] + } providers = { "00-bootstrap" = templatefile("${path.module}/../../assets/templates/providers.tpl", { bucket = module.automation-tf-bootstrap-gcs.name @@ -31,14 +35,14 @@ locals { "01-resman" = jsonencode({ automation_project_id = module.automation-project.project_id billing_account = var.billing_account - custom_roles = module.organization.custom_role_id + custom_roles = local._custom_roles groups = var.groups organization = var.organization prefix = var.prefix }) "02-networking" = jsonencode({ billing_account_id = var.billing_account.id - custom_roles = module.organization.custom_role_id + custom_roles = local._custom_roles organization = var.organization prefix = var.prefix }) diff --git a/fast/stages/00-bootstrap/variables.tf b/fast/stages/00-bootstrap/variables.tf index 1bf0e2de..8fe53c7f 100644 --- a/fast/stages/00-bootstrap/variables.tf +++ b/fast/stages/00-bootstrap/variables.tf @@ -28,6 +28,14 @@ variable "bootstrap_user" { default = null } +variable "custom_role_names" { + description = "Names of custom roles defined at the org level." + type = object({ + organization_iam_admin = "organizationIamAdmin" + service_project_network_admin = "serviceProjectNetworkAdmin" + }) +} + variable "groups" { # https://cloud.google.com/docs/enterprise/setup-checklist description = "Group names to grant organization-level permissions." diff --git a/fast/stages/02-networking-vpn/vpc-spoke-dev.tf b/fast/stages/02-networking-vpn/vpc-spoke-dev.tf index 9b3c0f9e..4a3f0f25 100644 --- a/fast/stages/02-networking-vpn/vpc-spoke-dev.tf +++ b/fast/stages/02-networking-vpn/vpc-spoke-dev.tf @@ -40,7 +40,7 @@ module "dev-spoke-project" { metric_scopes = [module.landing-project.project_id] iam = { "roles/dns.admin" = [var.project_factory_sa.dev] - (var.custom_roles.serviceProjectNetworkAdmin) = [ + (var.custom_roles.service_project_network_admin) = [ var.project_factory_sa.prod ] } diff --git a/fast/stages/02-networking-vpn/vpc-spoke-prod.tf b/fast/stages/02-networking-vpn/vpc-spoke-prod.tf index 7f42ab2c..3be90c2e 100644 --- a/fast/stages/02-networking-vpn/vpc-spoke-prod.tf +++ b/fast/stages/02-networking-vpn/vpc-spoke-prod.tf @@ -40,7 +40,7 @@ module "prod-spoke-project" { metric_scopes = [module.landing-project.project_id] iam = { "roles/dns.admin" = [var.project_factory_sa.prod] - (var.custom_roles.serviceProjectNetworkAdmin) = [ + (var.custom_roles.service_project_network_admin) = [ var.project_factory_sa.prod ] } From 3246d1c08d836a1298b331375add796629d0e749 Mon Sep 17 00:00:00 2001 From: Ludovico Magnocavallo Date: Thu, 10 Feb 2022 19:13:55 +0100 Subject: [PATCH 5/7] fix variable --- fast/stages/00-bootstrap/variables.tf | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/fast/stages/00-bootstrap/variables.tf b/fast/stages/00-bootstrap/variables.tf index 8fe53c7f..33709703 100644 --- a/fast/stages/00-bootstrap/variables.tf +++ b/fast/stages/00-bootstrap/variables.tf @@ -31,9 +31,13 @@ variable "bootstrap_user" { variable "custom_role_names" { description = "Names of custom roles defined at the org level." type = object({ + organization_iam_admin = string + service_project_network_admin = string + }) + default = { organization_iam_admin = "organizationIamAdmin" service_project_network_admin = "serviceProjectNetworkAdmin" - }) + } } variable "groups" { From 677f3c8df120905e8868e5ddf419b2d0b53ba87e Mon Sep 17 00:00:00 2001 From: Ludovico Magnocavallo Date: Thu, 10 Feb 2022 19:16:13 +0100 Subject: [PATCH 6/7] use custom role name for billing org too --- fast/stages/00-bootstrap/billing.tf | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/fast/stages/00-bootstrap/billing.tf b/fast/stages/00-bootstrap/billing.tf index e6abc31d..e8227041 100644 --- a/fast/stages/00-bootstrap/billing.tf +++ b/fast/stages/00-bootstrap/billing.tf @@ -73,7 +73,10 @@ resource "google_organization_iam_binding" "billing_org_ext_admin_delegated" { org_id = var.billing_account.organization_id # if the billing org does not have our custom role, user the predefined one # role = "roles/resourcemanager.organizationAdmin" - role = "organizations/${var.billing_account.organization_id}/roles/organizationIamAdmin" + role = join("", [ + "organizations/${var.billing_account.organization_id}/", + "roles/${var.custom_role_names.organization_iam_admin}" + ]) members = [module.automation-tf-resman-sa.iam_email] condition { title = "automation_sa_delegated_grants" From bb974869878d30ad32c321812238eee43d88db8e Mon Sep 17 00:00:00 2001 From: Ludovico Magnocavallo Date: Thu, 10 Feb 2022 19:17:35 +0100 Subject: [PATCH 7/7] tfdoc --- fast/stages/00-bootstrap/README.md | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/fast/stages/00-bootstrap/README.md b/fast/stages/00-bootstrap/README.md index f5bf29d0..a9b804e9 100644 --- a/fast/stages/00-bootstrap/README.md +++ b/fast/stages/00-bootstrap/README.md @@ -327,22 +327,23 @@ Names used in internal references (e.g. `module.foo-prod.id`) are only used by T | name | description | type | required | default | producer | |---|---|:---:|:---:|:---:|:---:| | [billing_account](variables.tf#L17) | Billing account id and organization id ('nnnnnnnn' or null). | object({…}) | ✓ | | | -| [organization](variables.tf#L84) | Organization details. | object({…}) | ✓ | | | -| [prefix](variables.tf#L99) | Prefix used for resources that need unique names. | string | ✓ | | | +| [organization](variables.tf#L96) | Organization details. | object({…}) | ✓ | | | +| [prefix](variables.tf#L111) | Prefix used for resources that need unique names. | string | ✓ | | | | [bootstrap_user](variables.tf#L25) | Email of the nominal user running this stage for the first time. | string | | null | | -| [groups](variables.tf#L31) | Group names to grant organization-level permissions. | map(string) | | {…} | | -| [iam](variables.tf#L45) | Organization-level custom IAM settings in role => [principal] format. | map(list(string)) | | {} | | -| [iam_additive](variables.tf#L51) | Organization-level custom IAM settings in role => [principal] format for non-authoritative bindings. | map(list(string)) | | {} | | -| [log_sinks](variables.tf#L59) | Org-level log sinks, in name => {type, filter} format. | map(object({…})) | | {…} | | -| [outputs_location](variables.tf#L93) | Path where providers and tfvars files for the following stages are written. Leave empty to disable. | string | | null | | +| [custom_role_names](variables.tf#L31) | Names of custom roles defined at the org level. | object({…}) | | {…} | | +| [groups](variables.tf#L43) | Group names to grant organization-level permissions. | map(string) | | {…} | | +| [iam](variables.tf#L57) | Organization-level custom IAM settings in role => [principal] format. | map(list(string)) | | {} | | +| [iam_additive](variables.tf#L63) | Organization-level custom IAM settings in role => [principal] format for non-authoritative bindings. | map(list(string)) | | {} | | +| [log_sinks](variables.tf#L71) | Org-level log sinks, in name => {type, filter} format. | map(object({…})) | | {…} | | +| [outputs_location](variables.tf#L105) | Path where providers and tfvars files for the following stages are written. Leave empty to disable. | string | | null | | ## Outputs | name | description | sensitive | consumers | |---|---|:---:|---| -| [billing_dataset](outputs.tf#L85) | BigQuery dataset prepared for billing export. | | | -| [project_ids](outputs.tf#L90) | Projects created by this stage. | | | -| [providers](outputs.tf#L101) | Terraform provider files for this stage and dependent stages. | ✓ | stage-01 | -| [tfvars](outputs.tf#L110) | Terraform variable files for the following stages. | ✓ | | +| [billing_dataset](outputs.tf#L89) | BigQuery dataset prepared for billing export. | | | +| [project_ids](outputs.tf#L94) | Projects created by this stage. | | | +| [providers](outputs.tf#L105) | Terraform provider files for this stage and dependent stages. | ✓ | stage-01 | +| [tfvars](outputs.tf#L114) | Terraform variable files for the following stages. | ✓ | |