cloud-foundation-fabric/modules/pubsub

Google Cloud Pub/Sub Module

This module allows managing a single Pub/Sub topic, including multiple subscriptions and IAM bindings at the topic and subscriptions levels, as well as schemas.

Examples

Simple topic with IAM

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = "my-project"
  name       = "my-topic"
  iam = {
    "roles/pubsub.viewer"     = ["group:foo@example.com"]
    "roles/pubsub.subscriber" = ["user:user1@example.com"]
  }
}
# tftest modules=1 resources=3 inventory=simple.yaml

Topic with schema

module "topic_with_schema" {
  source     = "./fabric/modules/pubsub"
  project_id = "my-project"
  name       = "my-topic"
  schema = {
    msg_encoding = "JSON"
    schema_type  = "AVRO"
    definition = jsonencode({
      "type" = "record",
      "name" = "Avro",
      "fields" = [{
        "name" = "StringField",
        "type" = "string"
        },
        {
          "name" = "FloatField",
          "type" = "float"
        },
        {
          "name" = "BooleanField",
          "type" = "boolean"
        },
      ]
    })
  }
}
# tftest modules=1 resources=2 inventory=schema.yaml

Subscriptions

Subscriptions are defined with the subscriptions variable, allowing optional configuration of per-subscription defaults. Push subscriptions need extra configuration, shown in the following example.

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = "my-project"
  name       = "my-topic"
  subscriptions = {
    test-pull = null
    test-pull-override = {
      labels = { test = "override" }
      options = {
        ack_deadline_seconds       = null
        message_retention_duration = null
        retain_acked_messages      = true
        expiration_policy_ttl      = null
        filter                     = null
      }
    }
  }
}
# tftest modules=1 resources=3 inventory=subscriptions.yaml

Push subscriptions

Push subscriptions need extra configuration in the push_configs variable.

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = "my-project"
  name       = "my-topic"
  subscriptions = {
    test-push = null
  }
  push_configs = {
    test-push = {
      endpoint   = "https://example.com/foo"
      attributes = null
      oidc_token = null
    }
  }
}
# tftest modules=1 resources=2

BigQuery subscriptions

BigQuery subscriptions need extra configuration in the bigquery_subscription_configs variable.

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = "my-project"
  name       = "my-topic"
  subscriptions = {
    test-bigquery = null
  }
  bigquery_subscription_configs = {
    test-bigquery = {
      table               = "my_project_id:my_dataset.my_table"
      use_topic_schema    = true
      write_metadata      = false
      drop_unknown_fields = true
    }
  }
}
# tftest modules=1 resources=2

Cloud Storage subscriptions

Cloud Storage subscriptions need extra configuration in the cloud_storage_subscription_configs variable.

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = "my-project"
  name       = "my-topic"
  subscriptions = {
    test-cloudstorage = null
  }
  cloud_storage_subscription_configs = {
    test-cloudstorage = {
      bucket          = "my-bucket"
      filename_prefix = "test_prefix"
      filename_suffix = "test_suffix"
      max_duration    = "100s"
      max_bytes       = 1000
      avro_config = {
        write_metadata = true
      }
    }
  }
}
# tftest modules=1 resources=2

Subscriptions with IAM

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = "my-project"
  name       = "my-topic"
  subscriptions = {
    test-1 = null
    test-1 = null
  }
  subscription_iam = {
    test-1 = {
      "roles/pubsub.subscriber" = ["user:user1@ludomagno.net"]
    }
  }
}
# tftest modules=1 resources=3

Variables

name	description	type	required	default
name	PubSub topic name.	string	✓
project_id	Project used for resources.	string	✓
bigquery_subscription_configs	Configuration parameters for BigQuery subscriptions.	map(object({…}))		{}
cloud_storage_subscription_configs	Configuration parameters for Cloud Storage subscriptions.	map(object({…}))		{}
dead_letter_configs	Per-subscription dead letter policy configuration.	map(object({…}))		{}
defaults	Subscription defaults for options.	object({…})		{…}
iam	IAM bindings for topic in {ROLE => [MEMBERS]} format.	map(list(string))		{}
kms_key	KMS customer managed encryption key.	string		null
labels	Labels.	map(string)		{}
message_retention_duration	Minimum duration to retain a message after it is published to the topic.	string		null
push_configs	Push subscription configurations.	map(object({…}))		{}
regions	List of regions used to set persistence policy.	list(string)		[]
schema	Topic schema. If set, all messages in this topic should follow this schema.	object({…})		null
subscription_iam	IAM bindings for subscriptions in {SUBSCRIPTION => {ROLE => [MEMBERS]}} format.	map(map(list(string)))		{}
subscriptions	Topic subscriptions. Also define push configs for push subscriptions. If options is set to null subscription defaults will be used. Labels default to topic labels if set to null.	map(object({…}))		{}

Outputs

name	description	sensitive
id	Fully qualified topic id.
schema	Schema resource.
schema_id	Schema resource id.
subscription_id	Subscription ids.
subscriptions	Subscription resources.
topic	Topic resource.