cloud-foundation-fabric/modules/data-catalog-tag

Google Cloud Data Catalog Tag Module

This module allows managing Data Catalog Tag on GCP resources such as BigQuery Datasets, Tables or columns.

TODO

• Add support for entries different than Bigquery resources.
• Add support to BOOL when issue is fixed.
• Add support to RICHTEXT when issue is fixed.

Examples

Dataset Tag

module "data-catalog-tag" {
  source = "./fabric/modules/data-catalog-tag"
  tags = {
    "landing/countries" = {
      project_id = "project-data-product"
      parent     = "projects/project-data-product/datasets/landing"
      location   = "europe-west1"
      template   = "projects/project-datagov/locations/europe-west1/tagTemplates/demo"
      fields = {
        source = {
          string_value = "DB-1"
        }
        datetime = {
          timestamp_value = "2024-02-03T06:50:50.91Z"
        }
        num = {
          double_value = 4.3
        }
        pii = {
          enum_value = "NONE"
        }
      }
    }
  }
}
# tftest modules=1 resources=1

Table Tag

module "data-catalog-tag" {
  source = "./fabric/modules/data-catalog-tag"
  tags = {
    "landing/countries" = {
      project_id = "project-data-product"
      parent     = "projects/project-data-product/datasets/landing/tables/countries"
      location   = "europe-west1"
      template   = "projects/project-datagov/locations/europe-west1/tagTemplates/demo"
      fields = {
        source = {
          string_value = "DB-1 Table-A"
        }
      }
    }
  }
}
# tftest modules=1 resources=1

Column Tag

module "data-catalog-tag" {
  source = "./fabric/modules/data-catalog-tag"
  tags = {
    "landing/countries" = {
      project_id = "project-data-product"
      parent     = "projects/project-data-product/datasets/landing/tables/countries"
      column     = "country"
      location   = "europe-west1"
      template   = "projects/project-datagov/locations/europe-west1/tagTemplates/demo"
      fields = {
        source = {
          string_value = "DB-1 Table-A Column-B"
        }
      }
    }
  }
}
# tftest modules=1 resources=1

Factory

Similarly to other modules, a rules factory (see Resource Factories) is also included here to allow tags management via descriptive configuration files.

Factory configuration is via one optional attributes in the factory_config_path variable specifying the path where tags files are stored.

Factory tags are merged with rules declared in code, with the latter taking precedence where both use the same key.

This is an example of a simple factory:

module "data-catalog-tag" {
  source = "./fabric/modules/data-catalog-tag"
  tags = {
    "landing/countries" = {
      project_id = "project-data-product"
      parent     = "projects/project-data-product/datasets/landing/tables/countries"
      column     = "country"
      location   = "europe-west1"
      template   = "projects/project-datagov/locations/europe-west1/tagTemplates/demo"
      fields = {
        source = {
          string_value = "DB-1 Table-A Column-B"
        }
      }
    }
  }
  factories_config = {
    tags = "data"
  }
}
# tftest modules=1 resources=2 files=demo_tag

# tftest-file id=demo_tag path=data/tag_1.yaml

project_id: project-data-product
parent: projects/project-data-product/datasets/exposure
location: europe-west1
template: projects/project-datagov/locations/europe-west1/tagTemplates/test
fields:
  owner_email: 
    string_value: example@example.com
  num: 
    double_value: 5
  pii: 
    enum_value: NONE

Variables

name	description	type	required	default
factories_config	Paths to data files and folders that enable factory functionality.	object({…})		{}
tags	Tags definitions in the form {TAG => TAG_DEFINITION}.	map(object({…}))		{}

Outputs

name	description	sensitive
data_catalog_tag_ids	Data catalog tag ids.