History

Ludovico Magnocavallo 6941313c7d Factories refactor (#1843 ) * factories refactor doc * Adds file schema and filesystem organization * Update 20231106-factories.md * move factories out of blueprints and create new factories README * align factory in billing-account module * align factory in dataplex-datascan module * align factory in billing-account module * align factory in net-firewall-policy module * align factory in dns-response-policy module * align factory in net-vpc-firewall module * align factory in net-vpc module * align factory variable names in FAST * remove decentralized firewall blueprint * bump terraform version * bump module versions * update top-level READMEs * move project factory to modules * fix variable names and tests * tfdoc * remove changelog link * add project factory to top-level README * fix cludrun eventarc diff * fix README * fix cludrun eventarc diff --------- Co-authored-by: Simone Ruffilli <sruffilli@google.com>		2024-02-26 10:16:52 +00:00
..
README.md	Factories refactor (#1843 )	2024-02-26 10:16:52 +00:00
main.tf	Factories refactor (#1843 )	2024-02-26 10:16:52 +00:00
variables.tf	Factories refactor (#1843 )	2024-02-26 10:16:52 +00:00

README.md

Google Cloud BQ Factory

This module allows creation and management of BigQuery datasets tables and views by defining them in well-formatted YAML files. YAML abstraction for BQ can simplify users onboarding and also makes creation of tables easier compared to HCL.

This factory is based on the BQ dataset module which currently only supports tables and views. As soon as external table and materialized view support is added, this factory will be enhanced accordingly.

You can create as many files as you like, the code will loop through it and create everything accordingly.

Example

Terraform code

In this section we show how to create tables and views from a file structure similar to the one shown below.

bigquery
│
├── tables
│   ├── table_a.yaml
│   ├── table_b.yaml
├── views
│   ├── view_a.yaml
│   ├── view_b.yaml

First we create the table definition in bigquery/tables/countries.yaml.

# tftest-file id=table path=bigquery/tables/countries.yaml
dataset: my_dataset
table: countries
deletion_protection: true
labels:
  env: prod
schema:
  - name: country
    type: STRING
  - name: population
    type: INT64

And a view in bigquery/views/population.yaml.

# tftest-file id=view path=bigquery/views/population.yaml
dataset: my_dataset
view: department
query: SELECT SUM(population) from  my_dataset.countries
labels:
  env: prod

With this file structure, we can use the factory as follows:

module "bq" {
  source      = "./fabric/modules/__experimental_deprecated/bigquery-factory"
  project_id  = var.project_id
  tables_path = "bigquery/tables"
  views_path  = "bigquery/views"
}
# tftest modules=2 resources=3 files=table,view

Variables

name	description	type	required
project_id	Project ID.	`string`	✓
tables_path	Relative path for the folder storing table data.	`string`	✓
views_path	Relative path for the folder storing view data.	`string`	✓

TODO

add external table support
add materialized view support