cloud-foundation-fabric/modules/dataproc

Google Cloud Dataproc

This module Manages a Google 'Cloud Dataproc' cluster resource, including IAM.

Examples

Simple

module "processing-dp-cluster-2" {
  source     = "./fabric/modules/dataproc"
  project_id = "my-project"
  name       = "my-cluster"
  region     = "europe-west1"
}
# tftest modules=1 resources=1

Cluster configuration

module "processing-dp-cluster" {
  source     = "./fabric/modules/dataproc"
  project_id = "my-project"
  name       = "my-cluster"
  region     = "europe-west1"
  prefix     = "prefix"
  dataproc_config = {
    cluster_config = {
      gce_cluster_config = {
        subnetwork             = "https://www.googleapis.com/compute/v1/projects/PROJECT/regions/europe-west1/subnetworks/SUBNET"
        zone                   = "europe-west1-b"
        service_account        = ""
        service_account_scopes = ["cloud-platform"]
        internal_ip_only       = true
      }
    }
  }
}
# tftest modules=1 resources=1

IAM Examples

IAM is managed via several variables that implement different levels of control:

• group_iam and iam configure authoritative bindings that manage individual roles exclusively, mapping to the google_dataproc_cluster_iam_binding resource
• iam_additive configure additive bindings that only manage individual role/member pairs, mapping to the google_dataproc_cluster_iam_member resource

Authorative IAM

module "processing-dp-cluster" {
  source     = "./fabric/modules/dataproc"
  project_id = "my-project"
  name       = "my-cluster"
  region     = "europe-west1"
  prefix     = "prefix"
  iam_additive = {
    "roles/dataproc.viewer" = [
      "serviceAccount:service-account@PROJECT_ID.iam.gserviceaccount.com"
    ]
  }
}
# tftest modules=1 resources=2

Additive IAM

module "processing-dp-cluster" {
  source     = "./fabric/modules/dataproc"
  project_id = "my-project"
  name       = "my-cluster"
  region     = "europe-west1"
  prefix     = "prefix"
  group_iam = {
    "gcp-data-engineers@example.net" = [
      "roles/dataproc.viewer"
    ]
  }
}
# tftest modules=1 resources=2

Variables

name	description	type	required	default
name	Cluster name.	string	✓
project_id	Project ID.	string	✓
region	Dataproc region.	string	✓
dataproc_config	Dataproc cluster config.	object({…})		{}
group_iam	Authoritative IAM binding for organization groups, in {GROUP_EMAIL => [ROLES]} format. Group emails need to be static. Can be used in combination with the iam variable.	map(list(string))		{}
iam	IAM bindings in {ROLE => [MEMBERS]} format.	map(list(string))		{}
iam_additive	IAM additive bindings in {ROLE => [MEMBERS]} format.	map(list(string))		{}
labels	The resource labels for instance to use to annotate any related underlying resources, such as Compute Engine VMs.	map(string)		{}
prefix	Optional prefix used to generate project id and name.	string		null
service_account	Service account to set on the Dataproc cluster.	string		null