Add schemas to Pubsub topic module.

Pubsub topics can now have
schemas (https://cloud.google.com/pubsub/docs/admin#schemas). This PR adds an
option to set the schema settings and create a new optional resource of type
`google_pubsub_schema` attached to the `google_pubsub_topic`.
This commit is contained in:
Israel Herraiz 2022-11-20 13:26:33 +01:00
parent f4960dd8fe
commit 54e9738c39
4 changed files with 79 additions and 1 deletions

View File

@ -1,6 +1,6 @@
# Google Cloud Pub/Sub Module
This module allows managing a single Pub/Sub topic, including multiple subscriptions and IAM bindings at the topic and subscriptions levels.
This module allows managing a single Pub/Sub topic, including multiple subscriptions and IAM bindings at the topic and subscriptions levels, as well as schemas.
## Examples
@ -20,6 +20,38 @@ module "pubsub" {
# tftest modules=1 resources=3
```
### Topic with schema
```hcl
module "topic_with_schema" {
source = "./fabric/modules/pubsub"
project_id = "my-project"
name = "my-topic"
schema = {
msg_encoding = "JSON"
schema_type = "AVRO"
definition = jsonencode({
"type" = "record",
"name" = "Avro",
"fields" : [{
"name" = "StringField",
"type" = "string"
},
{
"name" = "FloatField",
"type" = "float"
},
{
"name" = "BooleanField",
"type" = "boolean"
},
]
})
}
}
# tftest modules=1 resources=3
```
### Subscriptions
Subscriptions are defined with the `subscriptions` variable, allowing optional configuration of per-subscription defaults. Push subscriptions need extra configuration, shown in the following example.
@ -104,6 +136,7 @@ module "pubsub" {
| [message_retention_duration](variables.tf#L62) | Minimum duration to retain a message after it is published to the topic. | <code>string</code> | | <code>null</code> |
| [push_configs](variables.tf#L78) | Push subscription configurations. | <code title="map&#40;object&#40;&#123;&#10; attributes &#61; map&#40;string&#41;&#10; endpoint &#61; string&#10; oidc_token &#61; object&#40;&#123;&#10; audience &#61; string&#10; service_account_email &#61; string&#10; &#125;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [regions](variables.tf#L91) | List of regions used to set persistence policy. | <code>list&#40;string&#41;</code> | | <code>&#91;&#93;</code> |
| [schema](variables.tf#L118) | Topic schema. If set, all messages in this topic should follow this schema. | <code title="object&#40;&#123;&#10; schema_type &#61; string&#10; definition &#61; string&#10; msg_encoding &#61; optional&#40;string, &#34;ENCODING_UNSPECIFIED&#34;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [subscription_iam](variables.tf#L97) | IAM bindings for subscriptions in {SUBSCRIPTION => {ROLE => [MEMBERS]}} format. | <code>map&#40;map&#40;list&#40;string&#41;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [subscriptions](variables.tf#L103) | Topic subscriptions. Also define push configs for push subscriptions. If options is set to null subscription defaults will be used. Labels default to topic labels if set to null. | <code title="map&#40;object&#40;&#123;&#10; labels &#61; map&#40;string&#41;&#10; options &#61; object&#40;&#123;&#10; ack_deadline_seconds &#61; number&#10; message_retention_duration &#61; string&#10; retain_acked_messages &#61; bool&#10; expiration_policy_ttl &#61; string&#10; filter &#61; string&#10; &#125;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
@ -112,6 +145,8 @@ module "pubsub" {
| name | description | sensitive |
|---|---|:---:|
| [id](outputs.tf#L17) | Topic id. | |
| [schema](outputs.tf#L59) | Schema resource. | |
| [schema_id](outputs.tf#L51) | Schema resource id. | |
| [subscription_id](outputs.tf#L25) | Subscription ids. | |
| [subscriptions](outputs.tf#L35) | Subscription resources. | |
| [topic](outputs.tf#L43) | Topic resource. | |

View File

@ -35,6 +35,14 @@ locals {
}
}
resource "google_pubsub_schema" "default" {
count = var.schema == null ? 0 : 1
name = format("%s-%s", var.name, "schema")
type = var.schema.schema_type
definition = var.schema.definition
project = var.project_id
}
resource "google_pubsub_topic" "default" {
project = var.project_id
name = var.name
@ -48,6 +56,15 @@ resource "google_pubsub_topic" "default" {
allowed_persistence_regions = var.regions
}
}
depends_on = [google_pubsub_schema.default]
dynamic "schema_settings" {
for_each = var.schema == null ? [] : [""]
content {
schema = google_pubsub_schema.default[0].id
encoding = var.schema.msg_encoding
}
}
}
resource "google_pubsub_topic_iam_binding" "default" {

View File

@ -47,3 +47,19 @@ output "topic" {
google_pubsub_topic_iam_binding.default
]
}
output "schema_id" {
description = "Schema resource id."
value = google_pubsub_schema.default[0].id
depends_on = [
google_pubsub_schema.default
]
}
output "schema" {
description = "Schema resource."
value = google_pubsub_schema.default[0]
depends_on = [
google_pubsub_schema.default
]
}

View File

@ -114,3 +114,13 @@ variable "subscriptions" {
}))
default = {}
}
variable "schema" {
description = "Topic schema. If set, all messages in this topic should follow this schema."
type = object({
schema_type = string
definition = string
msg_encoding = optional(string, "ENCODING_UNSPECIFIED")
})
default = null
}