2020-05-11 08:51:00 -07:00
# Google Cloud Pub/Sub Module
2022-11-20 04:26:33 -08:00
This module allows managing a single Pub/Sub topic, including multiple subscriptions and IAM bindings at the topic and subscriptions levels, as well as schemas.
2020-05-11 08:51:00 -07:00
## Examples
### Simple topic with IAM
```hcl
module "pubsub" {
2022-09-06 08:46:09 -07:00
source = "./fabric/modules/pubsub"
2020-05-28 23:08:09 -07:00
project_id = "my-project"
2020-05-11 08:51:00 -07:00
name = "my-topic"
2020-11-05 12:28:34 -08:00
iam = {
2020-05-11 08:51:00 -07:00
"roles/pubsub.viewer" = ["group:foo@example.com"]
"roles/pubsub.subscriber" = ["user:user1@example.com"]
}
2020-05-28 23:07:08 -07:00
}
2023-04-14 08:14:02 -07:00
# tftest modules=1 resources=3 inventory=simple.yaml
2020-05-11 08:51:00 -07:00
```
2022-11-20 04:26:33 -08:00
### Topic with schema
```hcl
module "topic_with_schema" {
source = "./fabric/modules/pubsub"
project_id = "my-project"
name = "my-topic"
schema = {
msg_encoding = "JSON"
2022-12-16 03:53:56 -08:00
schema_type = "AVRO"
2022-11-20 04:26:33 -08:00
definition = jsonencode({
"type" = "record",
"name" = "Avro",
2022-11-20 07:36:36 -08:00
"fields" = [{
2022-11-20 04:26:33 -08:00
"name" = "StringField",
"type" = "string"
},
{
"name" = "FloatField",
"type" = "float"
},
{
"name" = "BooleanField",
"type" = "boolean"
},
]
})
}
}
2023-04-14 08:14:02 -07:00
# tftest modules=1 resources=2 inventory=schema.yaml
2022-11-20 04:26:33 -08:00
```
2020-05-11 08:51:00 -07:00
### Subscriptions
Subscriptions are defined with the `subscriptions` variable, allowing optional configuration of per-subscription defaults. Push subscriptions need extra configuration, shown in the following example.
```hcl
module "pubsub" {
2022-09-06 08:46:09 -07:00
source = "./fabric/modules/pubsub"
2020-11-06 00:36:32 -08:00
project_id = "my-project"
2020-05-11 08:51:00 -07:00
name = "my-topic"
subscriptions = {
2023-09-16 14:52:07 -07:00
test-pull = {}
2020-05-11 08:51:00 -07:00
test-pull-override = {
2023-09-16 14:52:07 -07:00
labels = { test = "override" }
retain_acked_messages = true
2020-05-11 08:51:00 -07:00
}
}
}
2023-04-14 08:14:02 -07:00
# tftest modules=1 resources=3 inventory=subscriptions.yaml
2020-05-11 08:51:00 -07:00
```
### Push subscriptions
Push subscriptions need extra configuration in the `push_configs` variable.
```hcl
module "pubsub" {
2022-09-06 08:46:09 -07:00
source = "./fabric/modules/pubsub"
2020-11-06 00:36:32 -08:00
project_id = "my-project"
2020-05-11 08:51:00 -07:00
name = "my-topic"
subscriptions = {
test-push = {
2023-09-16 14:52:07 -07:00
push = {
endpoint = "https://example.com/foo"
}
2020-05-11 08:51:00 -07:00
}
}
}
2022-01-28 11:15:35 -08:00
# tftest modules=1 resources=2
2020-05-11 08:51:00 -07:00
```
2022-11-21 08:43:17 -08:00
### BigQuery subscriptions
BigQuery subscriptions need extra configuration in the `bigquery_subscription_configs` variable.
```hcl
module "pubsub" {
source = "./fabric/modules/pubsub"
project_id = "my-project"
name = "my-topic"
subscriptions = {
test-bigquery = {
2023-09-16 14:52:07 -07:00
bigquery = {
table = "my_project_id:my_dataset.my_table"
use_topic_schema = true
write_metadata = false
drop_unknown_fields = true
}
2022-11-21 08:43:17 -08:00
}
}
}
# tftest modules=1 resources=2
```
2023-09-13 06:13:13 -07:00
### Cloud Storage subscriptions
Cloud Storage subscriptions need extra configuration in the `cloud_storage_subscription_configs` variable.
```hcl
module "pubsub" {
2023-09-13 10:53:05 -07:00
source = "./fabric/modules/pubsub"
2023-09-13 06:59:04 -07:00
project_id = "my-project"
2023-09-13 10:53:05 -07:00
name = "my-topic"
2023-09-13 06:13:13 -07:00
subscriptions = {
test-cloudstorage = {
2023-09-16 14:52:07 -07:00
cloud_storage = {
bucket = "my-bucket"
filename_prefix = "test_prefix"
filename_suffix = "test_suffix"
max_duration = "100s"
max_bytes = 1000
avro_config = {
write_metadata = true
}
2023-09-13 10:53:05 -07:00
}
2023-09-13 06:13:13 -07:00
}
}
}
# tftest modules=1 resources=2
```
2020-05-11 08:51:00 -07:00
### Subscriptions with IAM
```hcl
module "pubsub" {
2022-09-06 08:46:09 -07:00
source = "./fabric/modules/pubsub"
2020-11-06 00:36:32 -08:00
project_id = "my-project"
2020-05-11 08:51:00 -07:00
name = "my-topic"
subscriptions = {
test-1 = {
2023-09-16 14:52:07 -07:00
iam = {
"roles/pubsub.subscriber" = ["user:user1@example.com"]
}
2020-05-11 08:51:00 -07:00
}
}
}
2022-01-28 11:15:35 -08:00
# tftest modules=1 resources=3
2020-05-11 08:51:00 -07:00
```
<!-- BEGIN TFDOC -->
## Variables
| name | description | type | required | default |
2021-12-20 23:51:51 -08:00
|---|---|:---:|:---:|:---:|
2023-09-16 15:01:50 -07:00
| [name ](variables.tf#L73 ) | PubSub topic name. | < code > string</ code > | ✓ | |
| [project_id ](variables.tf#L78 ) | Project used for resources. | < code > string</ code > | ✓ | |
| [iam ](variables.tf#L17 ) | IAM bindings for topic in {ROLE => [MEMBERS]} format. | < code > map( list( string)) </ code > | | < code > {} </ code > |
| [iam_bindings ](variables.tf#L24 ) | Authoritative IAM bindings in {KEY => {role = ROLE, members = [], condition = {}}}. Keys are arbitrary. | < code title = "map(object({ members = list(string) role = string condition = optional(object({ expression = string title = string description = optional(string) })) }))" > map( object({…})) </ code > | | < code > {} </ code > |
| [iam_bindings_additive ](variables.tf#L39 ) | Keyring individual additive IAM bindings. Keys are arbitrary. | < code title = "map(object({ member = string role = string condition = optional(object({ expression = string title = string description = optional(string) })) }))" > map( object({…})) </ code > | | < code > {} </ code > |
| [kms_key ](variables.tf#L54 ) | KMS customer managed encryption key. | < code > string</ code > | | < code > null</ code > |
| [labels ](variables.tf#L60 ) | Labels. | < code > map( string) </ code > | | < code > {} </ code > |
| [message_retention_duration ](variables.tf#L67 ) | Minimum duration to retain a message after it is published to the topic. | < code > string</ code > | | < code > null</ code > |
| [regions ](variables.tf#L83 ) | List of regions used to set persistence policy. | < code > list( string) </ code > | | < code > [] </ code > |
| [schema ](variables.tf#L90 ) | Topic schema. If set, all messages in this topic should follow this schema. | < code title = "object({ definition = string msg_encoding = optional(string, "ENCODING_UNSPECIFIED") schema_type = string })" > object({…}) </ code > | | < code > null</ code > |
| [subscriptions ](variables.tf#L100 ) | Topic subscriptions. Also define push configs for push subscriptions. If options is set to null subscription defaults will be used. Labels default to topic labels if set to null. | < code title = "map(object({ labels = optional(map(string)) ack_deadline_seconds = optional(number) message_retention_duration = optional(string) retain_acked_messages = optional(bool, false) expiration_policy_ttl = optional(string) filter = optional(string) enable_message_ordering = optional(bool, false) enable_exactly_once_delivery = optional(bool, false) dead_letter_policy = optional(object({ topic = string max_delivery_attempts = optional(number) })) retry_policy = optional(object({ minimum_backoff = optional(number) maximum_backoff = optional(number) })) bigquery = optional(object({ table = string use_topic_schema = optional(bool, false) write_metadata = optional(bool, false) drop_unknown_fields = optional(bool, false) })) cloud_storage = optional(object({ bucket = string filename_prefix = optional(string) filename_suffix = optional(string) max_duration = optional(string) max_bytes = optional(number) avro_config = optional(object({ write_metadata = optional(bool, false) })) })) push = optional(object({ endpoint = string attributes = optional(map(string)) no_wrapper = optional(bool, false) oidc_token = optional(object({ audience = optional(string) service_account_email = string })) })) iam = optional(map(list(string)), {}) iam_bindings = optional(map(object({ members = list(string) role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) iam_bindings_additive = optional(map(object({ member = string role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) }))" > map( object({…})) </ code > | | < code > {} </ code > |
2020-05-11 08:51:00 -07:00
## Outputs
| name | description | sensitive |
|---|---|:---:|
2023-06-02 07:07:22 -07:00
| [id ](outputs.tf#L17 ) | Fully qualified topic id. | |
2023-09-16 15:01:50 -07:00
| [schema ](outputs.tf#L27 ) | Schema resource. | |
| [schema_id ](outputs.tf#L32 ) | Schema resource id. | |
| [subscription_id ](outputs.tf#L37 ) | Subscription ids. | |
| [subscriptions ](outputs.tf#L48 ) | Subscription resources. | |
| [topic ](outputs.tf#L57 ) | Topic resource. | |
2020-05-11 08:51:00 -07:00
<!-- END TFDOC -->