2023-02-27 13:14:07 -08:00
# Google Cloud Dataproc
2023-02-27 13:23:45 -08:00
This module Manages a Google ['Cloud Dataproc' ](https://cloud.google.com/dataproc ) cluster resource, including IAM.
2023-02-27 13:14:07 -08:00
## Examples
### Simple
```hcl
module "processing-dp-cluster-2" {
source = "./fabric/modules/dataproc"
project_id = "my-project"
name = "my-cluster"
region = "europe-west1"
}
# tftest modules=1 resources=1
```
### Cluster configuration
```hcl
module "processing-dp-cluster" {
source = "./fabric/modules/dataproc"
project_id = "my-project"
name = "my-cluster"
region = "europe-west1"
prefix = "prefix"
dataproc_config = {
cluster_config = {
gce_cluster_config = {
subnetwork = "https://www.googleapis.com/compute/v1/projects/PROJECT/regions/europe-west1/subnetworks/SUBNET"
zone = "europe-west1-b"
service_account = ""
service_account_scopes = ["cloud-platform"]
internal_ip_only = true
}
}
}
}
# tftest modules=1 resources=1
```
## IAM Examples
IAM is managed via several variables that implement different levels of control:
- `group_iam` and `iam` configure authoritative bindings that manage individual roles exclusively, mapping to the `google_dataproc_cluster_iam_binding` resource
- `iam_additive` configure additive bindings that only manage individual role/member pairs, mapping to the `google_dataproc_cluster_iam_member` resource
### Authorative IAM
```hcl
module "processing-dp-cluster" {
source = "./fabric/modules/dataproc"
project_id = "my-project"
name = "my-cluster"
region = "europe-west1"
prefix = "prefix"
iam_additive = {
"roles/dataproc.viewer" = [
"serviceAccount:service-account@PROJECT_ID.iam.gserviceaccount.com"
]
}
}
# tftest modules=1 resources=2
```
### Additive IAM
```hcl
module "processing-dp-cluster" {
source = "./fabric/modules/dataproc"
project_id = "my-project"
name = "my-cluster"
region = "europe-west1"
prefix = "prefix"
group_iam = {
"gcp-data-engineers@example.net" = [
"roles/dataproc.viewer"
]
}
}
# tftest modules=1 resources=2
```
<!-- BEGIN TFDOC -->
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
2023-02-27 13:23:45 -08:00
| [name ](variables.tf#L211 ) | Cluster name. | < code > string</ code > | ✓ | |
| [project_id ](variables.tf#L226 ) | Project ID. | < code > string</ code > | ✓ | |
| [region ](variables.tf#L231 ) | Dataproc region. | < code > string</ code > | ✓ | |
| [dataproc_config ](variables.tf#L17 ) | Dataproc cluster config. | < code title = "object({ graceful_decommission_timeout &# 61 ; optional &# 40 ; string , null &# 41 ;&# 10 ; cluster_config &# 61 ; optional &# 40 ; object &# 40 ;&# 123 ;&# 10 ; staging_bucket &# 61 ; optional &# 40 ; string , null &# 41 ;&# 10 ; temp_bucket &# 61 ; optional &# 40 ; string , null &# 41 ;&# 10 ; gce_cluster_config &# 61 ; optional &# 40 ; object &# 40 ;&# 123 ;&# 10 ; zone &# 61 ; optional &# 40 ; string , null &# 41 ;&# 10 ; network &# 61 ; optional &# 40 ; string , null &# 41 ;&# 10 ; subnetwork &# 61 ; optional &# 40 ; string , null &# 41 ;&# 10 ; service_account &# 61 ; optional &# 40 ; string , null &# 41 ;&# 10 ; service_account_scopes &# 61 ; optional &# 40 ; list &# 40 ; string &# 41 ;, null &# 41 ;&# 10 ; tags &# 61 ; optional &# 40 ; list &# 40 ; string &# 41 ;, &# 91 ;&# 93 ;&# 41 ;&# 10 ; internal_ip_only &# 61 ; optional &# 40 ; bool , null &# 41 ;&# 10 ; metadata &# 61 ; optional &# 40 ; map &# 40 ; string &# 41 ;, &# 123 ;&# 125 ;&# 41 ;&# 10 ; reservation_affinity &# 61 ; optional &# 40 ; object &# 40 ;&# 123 ;&# 10 ; consume_reservation_type &# 61 ; string &# 10 ; key &# 61 ; string &# 10 ; values &# 61 ; string &# 10 ; &# 125 ;&# 41 ;, null &# 41 ;&# 10 ; node_group_affinity &# 61 ; optional &# 40 ; object &# 40 ;&# 123 ;&# 10 ; node_group_uri &# 61 ; string &# 10 ; &# 125 ;&# 41 ;, null &# 41 ;&# 10 ;&# 10 ;&# 10 ; shielded_instance_config &# 61 ; optional &# 40 ; object &# 40 ;&# 123 ;&# 10 ; enable_secure_boot &# 61 ; bool &# 10 ; enable_vtpm &# 61 ; bool &# 10 ; enable_integrity_monitoring &# 61 ; bool &# 10 ; &# 125 ;&# 41 ;, null &# 41 ;&# 10 ; &# 125 ;&# 41 ;, null &# 41 ;&# 10 ; master_config &# 61 ; optional &# 40 ; object &# 40 ;&# 123 ;&# 10 ; num_instances &# 61 ; number &# 10 ; machine_type &# 61 ; string &# 10 ; min_cpu_platform &# 61 ; string &# 10 ; disk_config &# 61 ; optional &# 40 ; object &# 40 ;&# 123 ;&# 10 ; boot_disk_type &# 61 ; string &# 10 ; boot_disk_size_gb &# 61 ; number &# 10 ; num_local_ssds &# 61 ; number &# 10 ; &# 125 ;&# 41 ;, null &# 41 ;&# 10 ; accelerators &# 61 ; optional &# 40 ; object &# 40 ;&# 123 ;&# 10 ; accelerator_type &# 61 ; string &# 10 ; accelerator_count &# 61 ; number &# 10 ; &# 125 ;&# 41 ;, null &# 41 ;&# 10 ; &# 125 ;&# 41 ;, null &# 41 ;&# 10 ; worker_config &# 61 ; optional &# 40 ; object &# 40 ;&# 123 ;&# 10 ; num_instances &# 61 ; number &# 10 ; machine_type &# 61 ; string &# 10 ; min_cpu_platform &# 61 ; string &# 10 ; disk_config &# 61 ; optional &# 40 ; object &# 40 ;&# 123 ;&# 10 ; boot_disk_type &# 61 ; string &# 10 ; boot_disk_size_gb &# 61 ; number &# 10 ; num_local_ssds &# 61 ; number &# 10 ; &# 125 ;&# 41 ;, null &# 41 ;&# 10 ; image_uri &# 61 ; string &# 10 ; accelerators &# 61 ; optional &# 40 ; object &# 40 ;&# 123 ;&# 10 ; accelerator_type &# 61 ; string &# 10 ; accelerator_count &# 61 ; number &# 10 ; &# 125 ;&# 41 ;, null &# 41 ;&# 10 ; &# 125 ;&# 41 ;, null &# 41 ;&# 10 ; preemptible_worker_config &# 61 ; optional &# 40 ; object &# 40 ;&# 123 ;&# 10 ; num_instances &# 61 ; number &# 10 ; preemptibility &# 61 ; string &# 10 ; disk_config &# 61 ; optional &# 40 ; object &# 40 ;&# 123 ;&# 10 ; boot_disk_type &# 61 ; string &# 10 ; boot_disk_size_gb &# 61 ; number &# 10 ; num_local_ssds &# 61 ; number &# 10 ; &# 125 ;&# 41 ;, null &# 41 ;&# 10 ; &# 125 ;&# 41 ;, null &# 41 ;&# 10 ; software_config &# 61 ; optional &# 40 ; object &# 40 ;&# 123 ;&# 10 ; image_version &# 61 ; string &# 10 ; override_properties &# 61 ; list &# 40 ; map &# 40 ; string &# 41 ;&# 41 ;&# 10 ; optional_components &# 61 ; list &# 40 ; string &# 41 ;&# 10 ; &# 125 ;&# 41 ;, null &# 41 ;&# 10 ; security_config &# 61 ; optional &# 40 ; object &# 40 ;&# 123 ;&# 10 ; kerberos_config &# 61 ; object &# 40 ;&# 123 ;&# 10 ; cross_realm_trust_admin_server &# 61 ; optional &# 40 ; string , null &# 41 ;&# 10 ; cross_realm_trust_kdc &# 61 ; optional &# 40 ; string , null &# 41 ;&# 10 ; cross_realm_trust_realm &# 61 ; optional &# 40 ; string , null &# 41 ;&# 10 ; cross_realm_trust_shared_password_uri &# 61 ; optional &# 40 ; string , null &# 41 ;&# 10 ;
| [group_iam ](variables.tf#L184 ) | Authoritative IAM binding for organization groups, in {GROUP_EMAIL => [ROLES]} format. Group emails need to be static. Can be used in combination with the `iam` variable. | < code > map( list( string)) </ code > | | < code > {} </ code > |
| [iam ](variables.tf#L191 ) | IAM bindings in {ROLE => [MEMBERS]} format. | < code > map( list( string)) </ code > | | < code > {} </ code > |
| [iam_additive ](variables.tf#L198 ) | IAM additive bindings in {ROLE => [MEMBERS]} format. | < code > map( list( string)) </ code > | | < code > {} </ code > |
| [labels ](variables.tf#L205 ) | The resource labels for instance to use to annotate any related underlying resources, such as Compute Engine VMs. | < code > map( string) </ code > | | < code > {} </ code > |
| [prefix ](variables.tf#L216 ) | Optional prefix used to generate project id and name. | < code > string</ code > | | < code > null</ code > |
| [service_account ](variables.tf#L236 ) | Service account to set on the Dataproc cluster. | < code > string</ code > | | < code > null</ code > |
2023-02-27 13:14:07 -08:00
<!-- END TFDOC -->
