Fix comments

This commit is contained in:
lcaggio 2023-04-19 11:22:50 +02:00
parent 1f4fac2f1d
commit f4490fcaea
7 changed files with 57 additions and 53 deletions

View File

@ -98,5 +98,5 @@ module "test" {
prefix = "prefix" prefix = "prefix"
} }
# tftest modules=9 resources=47 # tftest modules=9 resources=48
``` ```

View File

@ -86,5 +86,5 @@ module "test" {
parent = "folders/467898377" parent = "folders/467898377"
} }
} }
# tftest modules=8 resources=40 # tftest modules=8 resources=41
``` ```

View File

@ -52,19 +52,19 @@ This blueprint can be used as a building block for setting up an end2end ML Ops
| name | description | type | required | default | | name | description | type | required | default |
|---|---|:---:|:---:|:---:| |---|---|:---:|:---:|:---:|
| [notebooks](variables.tf#L73) | Vertex AI workbenchs to be deployed. Service Account runtime/instances deployed. | <code title="map&#40;object&#40;&#123;&#10; type &#61; string&#10; machine_type &#61; optional&#40;string, &#34;n1-standard-4&#34;&#41;&#10; internal_ip_only &#61; optional&#40;bool, true&#41;&#10; idle_shutdown &#61; optional&#40;bool, false&#41;&#10; owner &#61; optional&#40;string, null&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | ✓ | | | [notebooks](variables.tf#L73) | Vertex AI workbenchs to be deployed. Service Account runtime/instances deployed. | <code title="map&#40;object&#40;&#123;&#10; type &#61; string&#10; machine_type &#61; optional&#40;string, &#34;n1-standard-4&#34;&#41;&#10; internal_ip_only &#61; optional&#40;bool, true&#41;&#10; idle_shutdown &#61; optional&#40;bool, false&#41;&#10; owner &#61; optional&#40;string&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | ✓ | |
| [project_config](variables.tf#L100) | Provide 'billing_account_id' value if project creation is needed, uses existing 'project_id' if null. Parent is in 'folders/nnn' or 'organizations/nnn' format. | <code title="object&#40;&#123;&#10; billing_account_id &#61; optional&#40;string, null&#41;&#10; parent &#61; string&#10; project_id &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | | [project_config](variables.tf#L100) | Provide 'billing_account_id' value if project creation is needed, uses existing 'project_id' if null. Parent is in 'folders/nnn' or 'organizations/nnn' format. | <code title="object&#40;&#123;&#10; billing_account_id &#61; optional&#40;string&#41;&#10; parent &#61; optional&#40;string&#41;&#10; project_id &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | |
| [bucket_name](variables.tf#L18) | GCS bucket name to store the Vertex AI artifacts. | <code>string</code> | | <code>null</code> | | [bucket_name](variables.tf#L18) | GCS bucket name to store the Vertex AI artifacts. | <code>string</code> | | <code>null</code> |
| [dataset_name](variables.tf#L24) | BigQuery Dataset to store the training data. | <code>string</code> | | <code>null</code> | | [dataset_name](variables.tf#L24) | BigQuery Dataset to store the training data. | <code>string</code> | | <code>null</code> |
| [groups](variables.tf#L30) | Name of the groups (name@domain.org) to apply opinionated IAM permissions. | <code title="object&#40;&#123;&#10; gcp-ml-ds &#61; optional&#40;string, null&#41;&#10; gcp-ml-eng &#61; optional&#40;string, null&#41;&#10; gcp-ml-viewer &#61; optional&#40;string, null&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; gcp-ml-ds &#61; null&#10; gcp-ml-eng &#61; null&#10; gcp-ml-viewer &#61; null&#10;&#125;">&#123;&#8230;&#125;</code> | | [groups](variables.tf#L30) | Name of the groups (name@domain.org) to apply opinionated IAM permissions. | <code title="object&#40;&#123;&#10; gcp-ml-ds &#61; optional&#40;string&#41;&#10; gcp-ml-eng &#61; optional&#40;string&#41;&#10; gcp-ml-viewer &#61; optional&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; gcp-ml-ds &#61; null&#10; gcp-ml-eng &#61; null&#10; gcp-ml-viewer &#61; null&#10;&#125;">&#123;&#8230;&#125;</code> |
| [identity_pool_claims](variables.tf#L45) | Claims to be used by Workload Identity Federation (i.e.: attribute.repository/ORGANIZATION/REPO). If a not null value is provided, then google_iam_workload_identity_pool resource will be created. | <code>string</code> | | <code>null</code> | | [identity_pool_claims](variables.tf#L45) | Claims to be used by Workload Identity Federation (i.e.: attribute.repository/ORGANIZATION/REPO). If a not null value is provided, then google_iam_workload_identity_pool resource will be created. | <code>string</code> | | <code>null</code> |
| [labels](variables.tf#L51) | Labels to be assigned at project level. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> | | [labels](variables.tf#L51) | Labels to be assigned at project level. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> |
| [location](variables.tf#L57) | Location used for multi-regional resources. | <code>string</code> | | <code>&#34;eu&#34;</code> | | [location](variables.tf#L57) | Location used for multi-regional resources. | <code>string</code> | | <code>&#34;eu&#34;</code> |
| [network_config](variables.tf#L63) | Shared VPC network configurations to use. If null networks will be created in projects with preconfigured values. | <code title="object&#40;&#123;&#10; host_project &#61; string&#10; network_self_link &#61; string&#10; subnet_self_link &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> | | [network_config](variables.tf#L63) | Shared VPC network configurations to use. If null networks will be created in projects with preconfigured values. | <code title="object&#40;&#123;&#10; host_project &#61; string&#10; network_self_link &#61; string&#10; subnet_self_link &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [prefix](variables.tf#L94) | Prefix used for the project id. | <code>string</code> | | <code>null</code> | | [prefix](variables.tf#L94) | Prefix used for the project id. | <code>string</code> | | <code>null</code> |
| [region](variables.tf#L113) | Region used for regional resources. | <code>string</code> | | <code>&#34;europe-west4&#34;</code> | | [region](variables.tf#L114) | Region used for regional resources. | <code>string</code> | | <code>&#34;europe-west4&#34;</code> |
| [repo_name](variables.tf#L119) | Cloud Source Repository name. null to avoid to create it. | <code>string</code> | | <code>null</code> | | [repo_name](variables.tf#L120) | Cloud Source Repository name. null to avoid to create it. | <code>string</code> | | <code>null</code> |
| [service_encryption_keys](variables.tf#L125) | Cloud KMS to use to encrypt different services. Key location should match service region. | <code title="object&#40;&#123;&#10; aiplatform &#61; optional&#40;string, null&#41;&#10; bq &#61; optional&#40;string, null&#41;&#10; notebooks &#61; optional&#40;string, null&#41;&#10; secretmanager &#61; optional&#40;string, null&#41;&#10; storage &#61; optional&#40;string, null&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; aiplatform &#61; null&#10; bq &#61; null&#10; notebooks &#61; null&#10; secretmanager &#61; null&#10; storage &#61; null&#10;&#125;">&#123;&#8230;&#125;</code> | | [service_encryption_keys](variables.tf#L126) | Cloud KMS to use to encrypt different services. Key location should match service region. | <code title="object&#40;&#123;&#10; aiplatform &#61; optional&#40;string&#41;&#10; bq &#61; optional&#40;string&#41;&#10; notebooks &#61; optional&#40;string&#41;&#10; secretmanager &#61; optional&#40;string&#41;&#10; storage &#61; optional&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>&#123;&#125;</code> |
## Outputs ## Outputs

View File

@ -60,9 +60,9 @@ module "secret-manager" {
secrets = { secrets = {
github-key = [var.region] github-key = [var.region]
} }
# encryption_key = { encryption_key = {
# "${var.region}" = try(var.service_encryption_keys["secretmanager"], null) "${var.region}" = var.service_encryption_keys.secretmanager
# } }
iam = { iam = {
github-key = { github-key = {
"roles/secretmanager.secretAccessor" = [ "roles/secretmanager.secretAccessor" = [

View File

@ -64,8 +64,7 @@ locals {
} }
) )
service_encryption_keys = var.service_encryption_keys shared_vpc_project = try(var.network_config.host_project, null)
shared_vpc_project = try(var.network_config.host_project, null)
subnet = ( subnet = (
local.use_shared_vpc local.use_shared_vpc
@ -109,7 +108,7 @@ module "gcs-bucket" {
location = var.region location = var.region
storage_class = "REGIONAL" storage_class = "REGIONAL"
versioning = false versioning = false
encryption_key = try(local.service_encryption_keys.storage, null) encryption_key = var.service_encryption_keys.storage
} }
# Default bucket for Cloud Build to prevent error: "'us' violates constraint gcp.resourceLocations" # Default bucket for Cloud Build to prevent error: "'us' violates constraint gcp.resourceLocations"
@ -122,7 +121,7 @@ module "gcs-bucket-cloudbuild" {
location = var.region location = var.region
storage_class = "REGIONAL" storage_class = "REGIONAL"
versioning = false versioning = false
encryption_key = try(local.service_encryption_keys.storage, null) encryption_key = var.service_encryption_keys.storage
} }
module "bq-dataset" { module "bq-dataset" {
@ -131,7 +130,7 @@ module "bq-dataset" {
project_id = module.project.project_id project_id = module.project.project_id
id = var.dataset_name id = var.dataset_name
location = var.region location = var.region
encryption_key = try(local.service_encryption_keys.bq, null) encryption_key = var.service_encryption_keys.bq
} }
module "vpc-local" { module "vpc-local" {
@ -191,18 +190,27 @@ module "cloudnat" {
module "project" { module "project" {
source = "../../../modules/project" source = "../../../modules/project"
name = var.project_config.project_id name = var.project_config.project_id
parent = try(var.project_config.parent, null) parent = var.project_config.parent
billing_account = try(var.project_config.billing_account_id, null) billing_account = var.project_config.billing_account_id
project_create = var.project_config.billing_account_id != null project_create = var.project_config.billing_account_id != null
prefix = var.prefix prefix = var.prefix
group_iam = local.group_iam group_iam = local.group_iam
iam = { iam = {
"roles/aiplatform.user" = [module.service-account-mlops.iam_email, module.service-account-notebook.iam_email] "roles/aiplatform.user" = [
module.service-account-mlops.iam_email,
module.service-account-notebook.iam_email
]
"roles/artifactregistry.reader" = [module.service-account-mlops.iam_email] "roles/artifactregistry.reader" = [module.service-account-mlops.iam_email]
"roles/artifactregistry.writer" = [module.service-account-github.iam_email] "roles/artifactregistry.writer" = [module.service-account-github.iam_email]
"roles/bigquery.dataEditor" = [module.service-account-mlops.iam_email, module.service-account-notebook.iam_email] "roles/bigquery.dataEditor" = [
"roles/bigquery.jobUser" = [module.service-account-mlops.iam_email, module.service-account-notebook.iam_email] module.service-account-mlops.iam_email,
"roles/bigquery.user" = [module.service-account-mlops.iam_email, module.service-account-notebook.iam_email] module.service-account-notebook.iam_email
]
"roles/bigquery.jobUser" = [
module.service-account-mlops.iam_email,
module.service-account-notebook.iam_email
]
"roles/bigquery.user" = [module.service-account-mlops.iam_email, module.service-account-notebook.iam_email]
"roles/cloudbuild.builds.editor" = [ "roles/cloudbuild.builds.editor" = [
module.service-account-mlops.iam_email, module.service-account-mlops.iam_email,
module.service-account-github.iam_email module.service-account-github.iam_email
@ -232,11 +240,12 @@ module "project" {
labels = var.labels labels = var.labels
service_encryption_key_ids = { service_encryption_key_ids = {
aiplatform = [try(local.service_encryption_keys.aiplatform, null)] aiplatform = [var.service_encryption_keys.aiplatform]
bq = [try(local.service_encryption_keys.bq, null)] bq = [var.service_encryption_keys.bq]
cloudbuild = [try(local.service_encryption_keys.storage, null)] cloudbuild = [var.service_encryption_keys.storage]
notebooks = [try(local.service_encryption_keys.notebooks, null)] notebooks = [var.service_encryption_keys.notebooks]
storage = [try(local.service_encryption_keys.storage, null)] secretmanager = [var.service_encryption_keys.secretmanager]
storage = [var.service_encryption_keys.storage]
} }
services = [ services = [

View File

@ -30,9 +30,9 @@ variable "dataset_name" {
variable "groups" { variable "groups" {
description = "Name of the groups (name@domain.org) to apply opinionated IAM permissions." description = "Name of the groups (name@domain.org) to apply opinionated IAM permissions."
type = object({ type = object({
gcp-ml-ds = optional(string, null) gcp-ml-ds = optional(string)
gcp-ml-eng = optional(string, null) gcp-ml-eng = optional(string)
gcp-ml-viewer = optional(string, null) gcp-ml-viewer = optional(string)
}) })
default = { default = {
gcp-ml-ds = null gcp-ml-ds = null
@ -77,7 +77,7 @@ variable "notebooks" {
machine_type = optional(string, "n1-standard-4") machine_type = optional(string, "n1-standard-4")
internal_ip_only = optional(bool, true) internal_ip_only = optional(bool, true)
idle_shutdown = optional(bool, false) idle_shutdown = optional(bool, false)
owner = optional(string, null) owner = optional(string)
})) }))
validation { validation {
condition = alltrue([ condition = alltrue([
@ -100,14 +100,15 @@ variable "prefix" {
variable "project_config" { variable "project_config" {
description = "Provide 'billing_account_id' value if project creation is needed, uses existing 'project_id' if null. Parent is in 'folders/nnn' or 'organizations/nnn' format." description = "Provide 'billing_account_id' value if project creation is needed, uses existing 'project_id' if null. Parent is in 'folders/nnn' or 'organizations/nnn' format."
type = object({ type = object({
billing_account_id = optional(string, null) billing_account_id = optional(string)
parent = string parent = optional(string)
project_id = string project_id = string
}) })
validation { validation {
condition = var.project_config.project_id != null condition = var.project_config.project_id != null
error_message = "Project id must be set." error_message = "Project id must be set."
} }
nullable = false
} }
variable "region" { variable "region" {
@ -125,17 +126,12 @@ variable "repo_name" {
variable "service_encryption_keys" { variable "service_encryption_keys" {
description = "Cloud KMS to use to encrypt different services. Key location should match service region." description = "Cloud KMS to use to encrypt different services. Key location should match service region."
type = object({ type = object({
aiplatform = optional(string, null) aiplatform = optional(string)
bq = optional(string, null) bq = optional(string)
notebooks = optional(string, null) notebooks = optional(string)
secretmanager = optional(string, null) secretmanager = optional(string)
storage = optional(string, null) storage = optional(string)
}) })
default = { default = {}
aiplatform = null nullable = false
bq = null
notebooks = null
secretmanager = null
storage = null
}
} }

View File

@ -21,10 +21,10 @@ resource "google_vertex_ai_metadata_store" "store" {
description = "Vertex Ai Metadata Store" description = "Vertex Ai Metadata Store"
region = var.region region = var.region
dynamic "encryption_spec" { dynamic "encryption_spec" {
for_each = try(var.service_encryption_keys.aiplatform, null) == null ? [] : [""] for_each = var.service_encryption_keys.aiplatform == null ? [] : [""]
content { content {
kms_key_name = try(var.service_encryption_keys.aiplatform, null) kms_key_name = var.service_encryption_keys.aiplatform
} }
} }
# `state` value will be decided automatically based on the result of the configuration # `state` value will be decided automatically based on the result of the configuration
@ -42,7 +42,6 @@ module "service-account-notebook" {
resource "google_notebooks_runtime" "runtime" { resource "google_notebooks_runtime" "runtime" {
for_each = { for k, v in var.notebooks : k => v if v.type == "MANAGED" } for_each = { for k, v in var.notebooks : k => v if v.type == "MANAGED" }
name = "${var.prefix}-${each.key}" name = "${var.prefix}-${each.key}"
project = module.project.project_id project = module.project.project_id
location = var.region location = var.region
access_config { access_config {
@ -59,9 +58,9 @@ resource "google_notebooks_runtime" "runtime" {
subnet = local.subnet subnet = local.subnet
internal_ip_only = var.notebooks[each.key].internal_ip_only internal_ip_only = var.notebooks[each.key].internal_ip_only
dynamic "encryption_config" { dynamic "encryption_config" {
for_each = try(local.service_encryption_keys.notebooks, null) == null ? [] : [1] for_each = var.service_encryption_keys.notebooks == null ? [] : [1]
content { content {
kms_key = local.service_encryption_keys.notebooks kms_key = var.service_encryption_keys.notebooks
} }
} }
metadata = { metadata = {
@ -83,7 +82,7 @@ resource "google_notebooks_runtime" "runtime" {
resource "google_notebooks_instance" "playground" { resource "google_notebooks_instance" "playground" {
for_each = { for k, v in var.notebooks : k => v if v.type == "USER_MANAGED" } for_each = { for k, v in var.notebooks : k => v if v.type == "USER_MANAGED" }
name = "${var.prefix}-${each.key}" name = "${var.prefix}-${each.key}"
location = format("%s-%s", var.region, "b") location = "${var.region}-b"
machine_type = var.notebooks[each.key].machine_type machine_type = var.notebooks[each.key].machine_type
project = module.project.project_id project = module.project.project_id
@ -95,8 +94,8 @@ resource "google_notebooks_instance" "playground" {
install_gpu_driver = true install_gpu_driver = true
boot_disk_type = "PD_SSD" boot_disk_type = "PD_SSD"
boot_disk_size_gb = 110 boot_disk_size_gb = 110
disk_encryption = try(local.service_encryption_keys.notebooks != null, false) ? "CMEK" : null disk_encryption = var.service_encryption_keys.notebooks != null ? "CMEK" : null
kms_key = try(local.service_encryption_keys.notebooks, null) kms_key = var.service_encryption_keys.notebooks
no_public_ip = var.notebooks[each.key].internal_ip_only no_public_ip = var.notebooks[each.key].internal_ip_only
no_proxy_access = false no_proxy_access = false