cloud-foundation-fabric/blueprints/factories/bigquery-factory

Google Cloud BQ Factory

This module allows creation and management of BigQuery datasets and views as well as tables by defining them in well formatted yaml files.

Yaml abstraction for BQ can simplify users onboarding and also makes creation of tables easier compared to HCL.

Subfolders distinguish between views and tables and ensures easier navigation for users.

This factory is based on the BQ dataset module which currently only supports tables and views. As soon as external table and materialized view support is added, factory will be enhanced accordingly.

You can create as many files as you like, the code will loop through it and create the required variables in order to execute everything accordingly.

Example

Terraform code

module "bq" {
  source = "github.com/GoogleCloudPlatform/cloud-foundation-fabric/modules/bigquery-dataset"

  for_each   = local.output
  project_id = var.project_id
  id         = each.key
  views      = try(each.value.views, null)
  tables     = try(each.value.tables, null)
}
# tftest skip

Configuration Structure

base_folder
│
├── tables
│   ├── table_a.yaml
│   ├── table_b.yaml
├── views
│   ├── view_a.yaml
│   ├── view_b.yaml

YAML structure and definition formatting

Tables

Table definition to be placed in a set of yaml files in the corresponding subfolder. Structure should look as following:

dataset: # required name of the dataset the table is to be placed in
table: # required descriptive name of the table
schema: # required schema in JSON FORMAT Example: [{name: "test", type: "STRING"},{name: "test2", type: "INT64"}]
labels: # not required, defaults to {}, Example: {"a":"thisislabela","b":"thisislabelb"}
use_legacy_sql: boolean # not required, defaults to false
deletion_protection: boolean # not required, defaults to false

Views

View definition to be placed in a set of yaml files in the corresponding subfolder. Structure should look as following:

dataset: # required, name of the dataset the view is to be placed in
view: # required, descriptive name of the view
query: # required, SQL Query for the view in quotes
labels: # not required, defaults to {}, Example: {"a":"thisislabela","b":"thisislabelb"}
use_legacy_sql: bool # not required, defaults to false
deletion_protection: bool # not required, defaults to false

Variables

name	description	type	required	default
project_id	Project ID	string	✓
tables_dir	Relative path for the folder storing table data.	string	✓
views_dir	Relative path for the folder storing view data.	string	✓

TODO

• add external table support
• add materialized view support