151 lines
6.7 KiB
Markdown
151 lines
6.7 KiB
Markdown
|
# Google Cloud Bigquery Module
|
||
|
|
||
|
This module allows managing a single BigQuery dataset, including access configuration, tables and views.
|
||
|
|
||
|
## Examples
|
||
|
|
||
|
### Simple dataset with access configuration
|
||
|
|
||
|
Access configuration defaults to using incremental accesses, which add to the default ones set at dataset creation. You can use the `access_authoritative` variable to switch to authoritative mode and have full control over dataset-level access. Be sure to always have at least one `OWNER` access and to avoid duplicating accesses, or `terraform apply` will fail.
|
||
|
|
||
|
```hcl
|
||
|
module "bigquery-dataset" {
|
||
|
source = "./modules/bigquery-dataset"
|
||
|
project_id = "my-project
|
||
|
id = "my-dataset"
|
||
|
access = [
|
||
|
{
|
||
|
role = "OWNER"
|
||
|
identity_type = "group_by_email"
|
||
|
identity = "dataset-owners@example.com"
|
||
|
}
|
||
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
|
### Dataset options
|
||
|
|
||
|
Dataset options are set via the `options` variable. all options must be specified, but a `null` value can be set to options that need to use defaults.
|
||
|
|
||
|
```hcl
|
||
|
module "bigquery-dataset" {
|
||
|
source = "./modules/bigquery-dataset"
|
||
|
project_id = "my-project
|
||
|
id = "my-dataset"
|
||
|
options = {
|
||
|
default_table_expiration_ms = 3600000
|
||
|
default_partition_expiration_ms = null
|
||
|
delete_contents_on_destroy = false
|
||
|
}
|
||
|
}
|
||
|
```
|
||
|
|
||
|
### Tables and views
|
||
|
|
||
|
Tables are created via the `tables` variable, or the `view` variable for views. Support for external tables will be added in a future release.
|
||
|
|
||
|
```hcl
|
||
|
module "bigquery-dataset" {
|
||
|
source = "./modules/bigquery-dataset"
|
||
|
project_id = "my-project
|
||
|
id = "my-dataset"
|
||
|
tables = {
|
||
|
table_a = {
|
||
|
friendly_name = "Table a"
|
||
|
labels = {}
|
||
|
options = null
|
||
|
partitioning = null
|
||
|
schema = file("table-a.json")
|
||
|
}
|
||
|
}
|
||
|
}
|
||
|
```
|
||
|
|
||
|
If partitioning is needed, populate the `partitioning` variable using either the `time` or `range` attribute.
|
||
|
|
||
|
```hcl
|
||
|
module "bigquery-dataset" {
|
||
|
source = "./modules/bigquery-dataset"
|
||
|
project_id = "my-project
|
||
|
id = "my-dataset"
|
||
|
tables = {
|
||
|
table_a = {
|
||
|
friendly_name = "Table a"
|
||
|
labels = {}
|
||
|
options = null
|
||
|
partitioning = {
|
||
|
field = null
|
||
|
range = null # use start/end/interval for range
|
||
|
time = { type = "DAY", expiration_ms = null }
|
||
|
}
|
||
|
schema = file("table-a.json")
|
||
|
}
|
||
|
}
|
||
|
}
|
||
|
```
|
||
|
|
||
|
To create views use the `view` variable. If you're querying a table created by the same module `terraform apply` will initially fail and eventually succeed once the underlying table has been created. You can probably also use the module's output in the view's query to create a dependency on the table.
|
||
|
|
||
|
```hcl
|
||
|
module "bigquery-dataset" {
|
||
|
source = "./modules/bigquery-dataset"
|
||
|
project_id = "my-project
|
||
|
id = "my-dataset"
|
||
|
tables = {
|
||
|
table_a = {
|
||
|
friendly_name = "Table a"
|
||
|
labels = {}
|
||
|
options = null
|
||
|
partitioning = {
|
||
|
field = null
|
||
|
range = null # use start/end/interval for range
|
||
|
time = { type = "DAY", expiration_ms = null }
|
||
|
}
|
||
|
schema = file("table-a.json")
|
||
|
}
|
||
|
}
|
||
|
views = {
|
||
|
view_a = {
|
||
|
friendly_name = "View a"
|
||
|
labels = {}
|
||
|
query = "SELECT * from `my-project.my-dataset.table_a`"
|
||
|
use_legacy_sql = false
|
||
|
}
|
||
|
}
|
||
|
}
|
||
|
```
|
||
|
|
||
|
<!-- BEGIN TFDOC -->
|
||
|
## Variables
|
||
|
|
||
|
| name | description | type | required | default |
|
||
|
|---|---|:---: |:---:|:---:|
|
||
|
| id | Dataset id. | <code title="">string</code> | ✓ | |
|
||
|
| project_id | Id of the project where datasets will be created. | <code title="">string</code> | ✓ | |
|
||
|
| *access* | Dataset access rules keyed by role, valid identity types are `domain`, `group_by_email`, `special_group` and `user_by_email`. Mode can be controlled via the `access_authoritative` variable. | <code title="map(list(object({ identity_type = string identity = string })))">map(list(object({...})))</code> | | <code title="">{}</code> |
|
||
|
| *access_authoritative* | Use authoritative access instead of additive. | <code title="">bool</code> | | <code title="">false</code> |
|
||
|
| *access_views* | Dataset access rules for views. Mode can be controlled via the `access_authoritative` variable. | <code title="list(object({ project_id = string dataset_id = string table_id = string }))">list(object({...}))</code> | | <code title="">[]</code> |
|
||
|
| *encryption_key* | Self link of the KMS key that will be used to protect destination table. | <code title="">string</code> | | <code title="">null</code> |
|
||
|
| *friendly_name* | Dataset friendly name. | <code title="">string</code> | | <code title="">null</code> |
|
||
|
| *labels* | Dataset labels. | <code title="map(string)">map(string)</code> | | <code title="">{}</code> |
|
||
|
| *location* | Dataset location. | <code title="">string</code> | | <code title="">EU</code> |
|
||
|
| *options* | Dataset options. | <code title="object({ default_table_expiration_ms = number default_partition_expiration_ms = number delete_contents_on_destroy = bool })">object({...})</code> | | <code title="{ default_table_expiration_ms = null default_partition_expiration_ms = null delete_contents_on_destroy = false }">...</code> |
|
||
|
| *tables* | Table definitions. Options and partitioning default to null. Partitioning can only use `range` or `time`, set the unused one to null. | <code title="map(object({ friendly_name = string labels = map(string) options = object({ clustering = list(string) encryption_key = string expiration_time = number }) partitioning = object({ field = string range = object({ end = number interval = number start = number }) time = object({ expiration_ms = number type = string }) }) schema = string }))">map(object({...}))</code> | | <code title="">{}</code> |
|
||
|
| *views* | View definitions. | <code title="map(object({ friendly_name = string labels = map(string) query = string use_legacy_sql = bool }))">map(object({...}))</code> | | <code title="">{}</code> |
|
||
|
|
||
|
## Outputs
|
||
|
|
||
|
| name | description | sensitive |
|
||
|
|---|---|:---:|
|
||
|
| dataset | Dataset resource. | |
|
||
|
| dataset_id | Dataset full id. | |
|
||
|
| id | Dataset id. | |
|
||
|
| self_link | Dataset self link. | |
|
||
|
| tables | Table resources. | |
|
||
|
| views | View resources. | |
|
||
|
<!-- END TFDOC -->
|
||
|
|
||
|
## TODO
|
||
|
|
||
|
- [ ] add support for tables
|