cloud-foundation-fabric/modules/bigquery-dataset

Google Cloud Bigquery Module

This module allows managing a single BigQuery dataset, including access configuration, tables and views.

TODO

• check for dynamic values in tables and views
• add support for external tables

Examples

Simple dataset with access configuration

Access configuration defaults to using the separate google_bigquery_dataset_access resource, so as to leave the default dataset access rules untouched.

You can choose to manage the google_bigquery_dataset access rules instead via the dataset_access variable, but be sure to always have at least one OWNER access and to avoid duplicating accesses, or terraform apply will fail.

The access variables are split into access_roles and access_identities variables, so that dynamic values can be passed in for identities (eg a service account email generated by a different module or resource). The access_views variable is separate, so as to allow proper type constraints.

module "bigquery-dataset" {
  source     = "./modules/bigquery-dataset"
  project_id = "my-project"
  id          = "my-dataset"
  access_roles = {
    reader-group = { role = "READER", type = "group_by_email" }
    owner        = { role = "OWNER", type = "user_by_email" }
  }
  access_identities = {
    reader-group = "playground-test@ludomagno.net"
    owner        = "ludo@ludomagno.net"
  }
}

Dataset options

Dataset options are set via the options variable. all options must be specified, but a null value can be set to options that need to use defaults.

module "bigquery-dataset" {
  source     = "./modules/bigquery-dataset"
  project_id = "my-project"
  id         = "my-dataset"
  options = {
    default_table_expiration_ms     = 3600000
    default_partition_expiration_ms = null
    delete_contents_on_destroy      = false
  }
}

Tables and views

Tables are created via the tables variable, or the view variable for views. Support for external tables will be added in a future release.

module "bigquery-dataset" {
  source     = "./modules/bigquery-dataset"
  project_id = "my-project"
  id         = "my-dataset"
  tables = {
    table_a = {
      friendly_name = "Table a"
      labels        = {}
      options       = null
      partitioning  = null
      schema = file("table-a.json")
    }
  }
}

If partitioning is needed, populate the partitioning variable using either the time or range attribute.

module "bigquery-dataset" {
  source     = "./modules/bigquery-dataset"
  project_id = "my-project"
  id         = "my-dataset"
  tables = {
    table_a = {
      friendly_name = "Table a"
      labels        = {}
      options       = null
      partitioning = {
        field = null
        range = null # use start/end/interval for range
        time  = { type = "DAY", expiration_ms = null }
      }
      schema = file("table-a.json")
    }
  }
}

To create views use the view variable. If you're querying a table created by the same module terraform apply will initially fail and eventually succeed once the underlying table has been created. You can probably also use the module's output in the view's query to create a dependency on the table.

module "bigquery-dataset" {
  source     = "./modules/bigquery-dataset"
  project_id = "my-project"
  id         = "my-dataset"
  tables = {
    table_a = {
      friendly_name = "Table a"
      labels        = {}
      options       = null
      partitioning = {
        field = null
        range = null # use start/end/interval for range
        time  = { type = "DAY", expiration_ms = null }
      }
      schema = file("table-a.json")
    }
  }
  views = {
    view_a = {
      friendly_name  = "View a"
      labels         = {}
      query          = "SELECT * from `my-project.my-dataset.table_a`"
      use_legacy_sql = false
    }
  }
}

Variables

name	description	type	required	default
id	Dataset id.	string	✓
project_id	Id of the project where datasets will be created.	string	✓
access_identities	Map of access identities used for access roles with type different from view. A separate variable is needed as identities can be set to dynamic values.	map(string)		{}
access_roles	Map of access rules with role and identity type. Keys are arbitrary and only used to combine identities with each role. Valid types are domain, group_by_email, special_group, user_by_email, view.	map(object({...}))		{}
access_views	Map of view data for access roles with identity type equal to view. A separate variable is needed as identities can be set to dynamic values.	map(object({...}))		{}
dataset_access	Set access in the dataset resource instead of using separate resources.	bool		false
encryption_key	Self link of the KMS key that will be used to protect destination table.	string		null
friendly_name	Dataset friendly name.	string		null
labels	Dataset labels.	map(string)		{}
location	Dataset location.	string		EU
options	Dataset options.	object({...})		...
tables	Table definitions. Options and partitioning default to null. Partitioning can only use range or time, set the unused one to null.	map(object({...}))		{}
views	View definitions.	map(object({...}))		{}

Outputs

name	description	sensitive
dataset	Dataset resource.
dataset_id	Dataset id.
id	Fully qualified dataset id.
self_link	Dataset self link.
table_ids	Map of fully qualified table ids keyed by table ids.
tables	Table resources.
view_ids	Map of fully qualified view ids keyed by view ids.
views	View resources.