cloud-foundation-fabric/modules/dataproc

Google Cloud Dataproc

This module Manages a Google Cloud Dataproc cluster resource, including IAM.

TODO

• Add support for Cloud Dataproc autoscaling policy.

Examples

Simple

module "processing-dp-cluster-2" {
  source     = "./fabric/modules/dataproc"
  project_id = "my-project"
  name       = "my-cluster"
  region     = "europe-west1"
}
# tftest modules=1 resources=1

Cluster configuration

To set cluster configuration use the 'dataproc_config.cluster_config' variable.

module "processing-dp-cluster" {
  source     = "./fabric/modules/dataproc"
  project_id = "my-project"
  name       = "my-cluster"
  region     = "europe-west1"
  prefix     = "prefix"
  dataproc_config = {
    cluster_config = {
      gce_cluster_config = {
        subnetwork             = "https://www.googleapis.com/compute/v1/projects/PROJECT/regions/europe-west1/subnetworks/SUBNET"
        zone                   = "europe-west1-b"
        service_account        = ""
        service_account_scopes = ["cloud-platform"]
        internal_ip_only       = true
      }
    }
  }
}
# tftest modules=1 resources=1

Cluster with CMEK encryption

To set cluster configuration use the Customer Managed Encryption key, set dataproc_config.encryption_config. variable. The Compute Engine service agent and the Cloud Storage service agent need to have CryptoKey Encrypter/Decrypter role on they configured KMS key (Documentation).

module "processing-dp-cluster" {
  source     = "./fabric/modules/dataproc"
  project_id = "my-project"
  name       = "my-cluster"
  region     = "europe-west1"
  prefix     = "prefix"
  dataproc_config = {
    cluster_config = {
      gce_cluster_config = {
        subnetwork             = "https://www.googleapis.com/compute/v1/projects/PROJECT/regions/europe-west1/subnetworks/SUBNET"
        zone                   = "europe-west1-b"
        service_account        = ""
        service_account_scopes = ["cloud-platform"]
        internal_ip_only       = true
      }
    }
    encryption_config = {
      kms_key_name = "projects/project-id/locations/region/keyRings/key-ring-name/cryptoKeys/key-name"
    }
  }
}
# tftest modules=1 resources=1

IAM Examples

IAM is managed via several variables that implement different levels of control:

• group_iam and iam configure authoritative bindings that manage individual roles exclusively, mapping to the google_dataproc_cluster_iam_binding resource
• iam_additive configure additive bindings that only manage individual role/member pairs, mapping to the google_dataproc_cluster_iam_member resource

Authoritative IAM

The iam variable is based on role keys and is typically used for service accounts, or where member values can be dynamic and would create potential problems in the underlying for_each cycle.

module "processing-dp-cluster" {
  source     = "./fabric/modules/dataproc"
  project_id = "my-project"
  name       = "my-cluster"
  region     = "europe-west1"
  prefix     = "prefix"
  iam = {
    "roles/dataproc.viewer" = [
      "serviceAccount:service-account@PROJECT_ID.iam.gserviceaccount.com"
    ]
  }
}
# tftest modules=1 resources=2

The group_iam variable uses group email addresses as keys and is a convenient way to assign roles to humans following Google's best practices. The end result is readable code that also serves as documentation.

module "processing-dp-cluster" {
  source     = "./fabric/modules/dataproc"
  project_id = "my-project"
  name       = "my-cluster"
  region     = "europe-west1"
  prefix     = "prefix"
  group_iam = {
    "gcp-data-engineers@example.net" = [
      "roles/dataproc.viewer"
    ]
  }
}
# tftest modules=1 resources=2

Additive IAM

Additive IAM is typically used where bindings for specific roles are controlled by different modules or in different Terraform stages. One example is when the cluster is created by one team but a different team manages access.

module "processing-dp-cluster" {
  source     = "./fabric/modules/dataproc"
  project_id = "my-project"
  name       = "my-cluster"
  region     = "europe-west1"
  prefix     = "prefix"
  iam_additive = {
    "roles/dataproc.viewer" = [
      "serviceAccount:service-account@PROJECT_ID.iam.gserviceaccount.com"
    ]
  }
}
# tftest modules=1 resources=2

Variables

name	description	type	required	default
name	Cluster name.	string	✓
project_id	Project ID.	string	✓
region	Dataproc region.	string	✓
dataproc_config	Dataproc cluster config.	object({…})		{}
group_iam	Authoritative IAM binding for organization groups, in {GROUP_EMAIL => [ROLES]} format. Group emails need to be static. Can be used in combination with the iam variable.	map(list(string))		{}
iam	IAM bindings in {ROLE => [MEMBERS]} format.	map(list(string))		{}
iam_additive	IAM additive bindings in {ROLE => [MEMBERS]} format.	map(list(string))		{}
labels	The resource labels for instance to use to annotate any related underlying resources, such as Compute Engine VMs.	map(string)		{}
prefix	Optional prefix used to generate project id and name.	string		null
service_account	Service account to set on the Dataproc cluster.	string		null

Outputs

name	description	sensitive
bucket_names	List of bucket names which have been assigned to the cluster.
http_ports	The map of port descriptions to URLs.
id	Fully qualified cluster id.
instance_names	List of instance names which have been assigned to the cluster.
name	The name of the cluster.