211 lines
9.5 KiB
Markdown
211 lines
9.5 KiB
Markdown
# Google Cloud Bigquery Module
|
|
|
|
This module allows managing a single BigQuery dataset, including access configuration, tables and views.
|
|
|
|
## TODO
|
|
|
|
- [ ] check for dynamic values in tables and views
|
|
- [ ] add support for external tables
|
|
|
|
## Examples
|
|
|
|
### Simple dataset with access configuration
|
|
|
|
Access configuration defaults to using the separate `google_bigquery_dataset_access` resource, so as to leave the default dataset access rules untouched.
|
|
|
|
You can choose to manage the `google_bigquery_dataset` access rules instead via the `dataset_access` variable, but be sure to always have at least one `OWNER` access and to avoid duplicating accesses, or `terraform apply` will fail.
|
|
|
|
The access variables are split into `access_roles` and `access_identities` variables, so that dynamic values can be passed in for identities (eg a service account email generated by a different module or resource). The `access_views` variable is separate, so as to allow proper type constraints.
|
|
|
|
```hcl
|
|
module "bigquery-dataset" {
|
|
source = "./modules/bigquery-dataset"
|
|
project_id = "my-project"
|
|
id = "my-dataset"
|
|
access = {
|
|
reader-group = { role = "READER", type = "group" }
|
|
owner = { role = "OWNER", type = "user" }
|
|
project_owners = { role = "OWNER", type = "special_group" }
|
|
view_1 = { role = "READER", type = "view" }
|
|
}
|
|
access_identities = {
|
|
reader-group = "playground-test@ludomagno.net"
|
|
owner = "ludo@ludomagno.net"
|
|
project_owners = "projectOwners"
|
|
view_1 = "my-project|my-dataset|my-table"
|
|
}
|
|
}
|
|
# tftest modules=1 resources=5
|
|
```
|
|
|
|
### IAM roles
|
|
|
|
Access configuration can also be specified via IAM instead of basic roles via the `iam` variable. When using IAM, basic roles cannot be used via the `access` family variables.
|
|
|
|
```hcl
|
|
module "bigquery-dataset" {
|
|
source = "./modules/bigquery-dataset"
|
|
project_id = "my-project"
|
|
id = "my-dataset"
|
|
iam = {
|
|
"roles/bigquery.dataOwner" = ["user:user1@example.org"]
|
|
}
|
|
}
|
|
# tftest modules=1 resources=2
|
|
```
|
|
|
|
roles/bigquery.dataOwner
|
|
|
|
### Dataset options
|
|
|
|
Dataset options are set via the `options` variable. all options must be specified, but a `null` value can be set to options that need to use defaults.
|
|
|
|
```hcl
|
|
module "bigquery-dataset" {
|
|
source = "./modules/bigquery-dataset"
|
|
project_id = "my-project"
|
|
id = "my-dataset"
|
|
options = {
|
|
default_table_expiration_ms = 3600000
|
|
default_partition_expiration_ms = null
|
|
delete_contents_on_destroy = false
|
|
}
|
|
}
|
|
# tftest modules=1 resources=1
|
|
```
|
|
|
|
### Tables and views
|
|
|
|
Tables are created via the `tables` variable, or the `view` variable for views. Support for external tables will be added in a future release.
|
|
|
|
```hcl
|
|
locals {
|
|
countries_schema = jsonencode([
|
|
{ name = "country", type = "STRING" },
|
|
{ name = "population", type = "INT64" },
|
|
])
|
|
}
|
|
|
|
module "bigquery-dataset" {
|
|
source = "./modules/bigquery-dataset"
|
|
project_id = "my-project"
|
|
id = "my_dataset"
|
|
tables = {
|
|
countries = {
|
|
friendly_name = "Countries"
|
|
labels = {}
|
|
options = null
|
|
partitioning = null
|
|
schema = local.countries_schema
|
|
deletion_protection = true
|
|
}
|
|
}
|
|
}
|
|
# tftest modules=1 resources=2
|
|
```
|
|
|
|
If partitioning is needed, populate the `partitioning` variable using either the `time` or `range` attribute.
|
|
|
|
```hcl
|
|
locals {
|
|
countries_schema = jsonencode([
|
|
{ name = "country", type = "STRING" },
|
|
{ name = "population", type = "INT64" },
|
|
])
|
|
}
|
|
|
|
module "bigquery-dataset" {
|
|
source = "./modules/bigquery-dataset"
|
|
project_id = "my-project"
|
|
id = "my-dataset"
|
|
tables = {
|
|
table_a = {
|
|
friendly_name = "Table a"
|
|
labels = {}
|
|
options = null
|
|
partitioning = {
|
|
field = null
|
|
range = null # use start/end/interval for range
|
|
time = { type = "DAY", expiration_ms = null }
|
|
}
|
|
schema = local.countries_schema
|
|
deletion_protection = true
|
|
}
|
|
}
|
|
}
|
|
# tftest modules=1 resources=2
|
|
```
|
|
|
|
To create views use the `view` variable. If you're querying a table created by the same module `terraform apply` will initially fail and eventually succeed once the underlying table has been created. You can probably also use the module's output in the view's query to create a dependency on the table.
|
|
|
|
```hcl
|
|
locals {
|
|
countries_schema = jsonencode([
|
|
{ name = "country", type = "STRING" },
|
|
{ name = "population", type = "INT64" },
|
|
])
|
|
}
|
|
|
|
module "bigquery-dataset" {
|
|
source = "./modules/bigquery-dataset"
|
|
project_id = "my-project"
|
|
id = "my_dataset"
|
|
tables = {
|
|
countries = {
|
|
friendly_name = "Countries"
|
|
labels = {}
|
|
options = null
|
|
partitioning = null
|
|
schema = local.countries_schema
|
|
deletion_protection = true
|
|
}
|
|
}
|
|
views = {
|
|
population = {
|
|
friendly_name = "Population"
|
|
labels = {}
|
|
query = "SELECT SUM(population) FROM my_dataset.countries"
|
|
use_legacy_sql = false
|
|
deletion_protection = true
|
|
}
|
|
}
|
|
}
|
|
|
|
# tftest modules=1 resources=3
|
|
```
|
|
<!-- BEGIN TFDOC -->
|
|
|
|
## Variables
|
|
|
|
| name | description | type | required | default |
|
|
|---|---|:---:|:---:|:---:|
|
|
| [id](variables.tf#L69) | Dataset id. | <code>string</code> | ✓ | |
|
|
| [project_id](variables.tf#L100) | Id of the project where datasets will be created. | <code>string</code> | ✓ | |
|
|
| [access](variables.tf#L17) | Map of access rules with role and identity type. Keys are arbitrary and must match those in the `access_identities` variable, types are `domain`, `group`, `special_group`, `user`, `view`. | <code title="map(object({ role = string type = string }))">map(object({…}))</code> | | <code>{}</code> |
|
|
| [access_identities](variables.tf#L33) | Map of access identities used for basic access roles. View identities have the format 'project_id|dataset_id|table_id'. | <code>map(string)</code> | | <code>{}</code> |
|
|
| [dataset_access](variables.tf#L39) | Set access in the dataset resource instead of using separate resources. | <code>bool</code> | | <code>false</code> |
|
|
| [description](variables.tf#L45) | Optional description. | <code>string</code> | | <code>"Terraform managed."</code> |
|
|
| [encryption_key](variables.tf#L51) | Self link of the KMS key that will be used to protect destination table. | <code>string</code> | | <code>null</code> |
|
|
| [friendly_name](variables.tf#L57) | Dataset friendly name. | <code>string</code> | | <code>null</code> |
|
|
| [iam](variables.tf#L63) | IAM bindings in {ROLE => [MEMBERS]} format. Mutually exclusive with the access_* variables used for basic roles. | <code>map(list(string))</code> | | <code>{}</code> |
|
|
| [labels](variables.tf#L74) | Dataset labels. | <code>map(string)</code> | | <code>{}</code> |
|
|
| [location](variables.tf#L80) | Dataset location. | <code>string</code> | | <code>"EU"</code> |
|
|
| [options](variables.tf#L86) | Dataset options. | <code title="object({ default_table_expiration_ms = number default_partition_expiration_ms = number delete_contents_on_destroy = bool })">object({…})</code> | | <code title="{ default_table_expiration_ms = null default_partition_expiration_ms = null delete_contents_on_destroy = false }">{…}</code> |
|
|
| [tables](variables.tf#L105) | Table definitions. Options and partitioning default to null. Partitioning can only use `range` or `time`, set the unused one to null. | <code title="map(object({ friendly_name = string labels = map(string) options = object({ clustering = list(string) encryption_key = string expiration_time = number }) partitioning = object({ field = string range = object({ end = number interval = number start = number }) time = object({ expiration_ms = number type = string }) }) schema = string deletion_protection = bool }))">map(object({…}))</code> | | <code>{}</code> |
|
|
| [views](variables.tf#L133) | View definitions. | <code title="map(object({ friendly_name = string labels = map(string) query = string use_legacy_sql = bool deletion_protection = bool }))">map(object({…}))</code> | | <code>{}</code> |
|
|
|
|
## Outputs
|
|
|
|
| name | description | sensitive |
|
|
|---|---|:---:|
|
|
| [dataset](outputs.tf#L17) | Dataset resource. | |
|
|
| [dataset_id](outputs.tf#L22) | Dataset id. | |
|
|
| [id](outputs.tf#L34) | Fully qualified dataset id. | |
|
|
| [self_link](outputs.tf#L46) | Dataset self link. | |
|
|
| [table_ids](outputs.tf#L58) | Map of fully qualified table ids keyed by table ids. | |
|
|
| [tables](outputs.tf#L63) | Table resources. | |
|
|
| [view_ids](outputs.tf#L68) | Map of fully qualified view ids keyed by view ids. | |
|
|
| [views](outputs.tf#L73) | View resources. | |
|
|
|
|
<!-- END TFDOC -->
|