865ef463bb
* add support for new attributes to bigquery module * bump provider versions |
||
---|---|---|
.. | ||
README.md | ||
iam.tf | ||
main.tf | ||
outputs.tf | ||
variables.tf | ||
versions.tf |
README.md
Google Cloud Dataproc
This module Manages a Google Cloud Dataproc cluster resource, including IAM.
TODO
- Add support for Cloud Dataproc autoscaling policy.
Examples
Simple
module "processing-dp-cluster-2" {
source = "./fabric/modules/dataproc"
project_id = "my-project"
name = "my-cluster"
region = "europe-west1"
}
# tftest modules=1 resources=1
Cluster configuration
To set cluster configuration use the 'dataproc_config.cluster_config' variable.
module "processing-dp-cluster" {
source = "./fabric/modules/dataproc"
project_id = "my-project"
name = "my-cluster"
region = "europe-west1"
prefix = "prefix"
dataproc_config = {
cluster_config = {
gce_cluster_config = {
subnetwork = "https://www.googleapis.com/compute/v1/projects/PROJECT/regions/europe-west1/subnetworks/SUBNET"
zone = "europe-west1-b"
service_account = ""
service_account_scopes = ["cloud-platform"]
internal_ip_only = true
}
}
}
}
# tftest modules=1 resources=1
Cluster with CMEK encryption
To set cluster configuration use the Customer Managed Encryption key, set dataproc_config.encryption_config.
variable. The Compute Engine service agent and the Cloud Storage service agent need to have CryptoKey Encrypter/Decrypter
role on they configured KMS key (Documentation).
module "processing-dp-cluster" {
source = "./fabric/modules/dataproc"
project_id = "my-project"
name = "my-cluster"
region = "europe-west1"
prefix = "prefix"
dataproc_config = {
cluster_config = {
gce_cluster_config = {
subnetwork = "https://www.googleapis.com/compute/v1/projects/PROJECT/regions/europe-west1/subnetworks/SUBNET"
zone = "europe-west1-b"
service_account = ""
service_account_scopes = ["cloud-platform"]
internal_ip_only = true
}
}
encryption_config = {
kms_key_name = "projects/project-id/locations/region/keyRings/key-ring-name/cryptoKeys/key-name"
}
}
}
# tftest modules=1 resources=1
IAM Examples
IAM is managed via several variables that implement different levels of control:
group_iam
andiam
configure authoritative bindings that manage individual roles exclusively, mapping to thegoogle_dataproc_cluster_iam_binding
resourceiam_additive
configure additive bindings that only manage individual role/member pairs, mapping to thegoogle_dataproc_cluster_iam_member
resource
Authoritative IAM
The iam variable is based on role keys and is typically used for service accounts, or where member values can be dynamic and would create potential problems in the underlying for_each cycle.
module "processing-dp-cluster" {
source = "./fabric/modules/dataproc"
project_id = "my-project"
name = "my-cluster"
region = "europe-west1"
prefix = "prefix"
iam = {
"roles/dataproc.viewer" = [
"serviceAccount:service-account@PROJECT_ID.iam.gserviceaccount.com"
]
}
}
# tftest modules=1 resources=2
The group_iam variable uses group email addresses as keys and is a convenient way to assign roles to humans following Google's best practices. The end result is readable code that also serves as documentation.
module "processing-dp-cluster" {
source = "./fabric/modules/dataproc"
project_id = "my-project"
name = "my-cluster"
region = "europe-west1"
prefix = "prefix"
group_iam = {
"gcp-data-engineers@example.net" = [
"roles/dataproc.viewer"
]
}
}
# tftest modules=1 resources=2
Additive IAM
Additive IAM is typically used where bindings for specific roles are controlled by different modules or in different Terraform stages. One example is when the cluster is created by one team but a different team manages access.
module "processing-dp-cluster" {
source = "./fabric/modules/dataproc"
project_id = "my-project"
name = "my-cluster"
region = "europe-west1"
prefix = "prefix"
iam_additive = {
"roles/dataproc.viewer" = [
"serviceAccount:service-account@PROJECT_ID.iam.gserviceaccount.com"
]
}
}
# tftest modules=1 resources=2
Variables
name | description | type | required | default |
---|---|---|---|---|
name | Cluster name. | string |
✓ | |
project_id | Project ID. | string |
✓ | |
region | Dataproc region. | string |
✓ | |
dataproc_config | Dataproc cluster config. | object({…}) |
{} |
|
group_iam | Authoritative IAM binding for organization groups, in {GROUP_EMAIL => [ROLES]} format. Group emails need to be static. Can be used in combination with the iam variable. |
map(list(string)) |
{} |
|
iam | IAM bindings in {ROLE => [MEMBERS]} format. | map(list(string)) |
{} |
|
iam_additive | IAM additive bindings in {ROLE => [MEMBERS]} format. | map(list(string)) |
{} |
|
labels | The resource labels for instance to use to annotate any related underlying resources, such as Compute Engine VMs. | map(string) |
{} |
|
prefix | Optional prefix used to generate project id and name. | string |
null |
|
service_account | Service account to set on the Dataproc cluster. | string |
null |
Outputs
name | description | sensitive |
---|---|---|
bucket_names | List of bucket names which have been assigned to the cluster. | |
http_ports | The map of port descriptions to URLs. | |
instance_names | List of instance names which have been assigned to the cluster. | |
name | The name of the cluster. |