105 lines
12 KiB
Markdown
105 lines
12 KiB
Markdown
|
# Google Cloud Dataproc
|
||
|
|
||
|
This module Manages a Cloud Dataproc cluster resource, including IAM.
|
||
|
|
||
|
allows simple management of ['Google Data Fusion'](https://cloud.google.com/data-fusion) instances. It supports creating Basic or Enterprise, public or private instances.
|
||
|
|
||
|
## Examples
|
||
|
|
||
|
### Simple
|
||
|
|
||
|
```hcl
|
||
|
module "processing-dp-cluster-2" {
|
||
|
source = "./fabric/modules/dataproc"
|
||
|
project_id = "my-project"
|
||
|
name = "my-cluster"
|
||
|
region = "europe-west1"
|
||
|
}
|
||
|
# tftest modules=1 resources=1
|
||
|
```
|
||
|
|
||
|
### Cluster configuration
|
||
|
|
||
|
```hcl
|
||
|
module "processing-dp-cluster" {
|
||
|
source = "./fabric/modules/dataproc"
|
||
|
project_id = "my-project"
|
||
|
name = "my-cluster"
|
||
|
region = "europe-west1"
|
||
|
prefix = "prefix"
|
||
|
dataproc_config = {
|
||
|
cluster_config = {
|
||
|
gce_cluster_config = {
|
||
|
subnetwork = "https://www.googleapis.com/compute/v1/projects/PROJECT/regions/europe-west1/subnetworks/SUBNET"
|
||
|
zone = "europe-west1-b"
|
||
|
service_account = ""
|
||
|
service_account_scopes = ["cloud-platform"]
|
||
|
internal_ip_only = true
|
||
|
}
|
||
|
}
|
||
|
}
|
||
|
}
|
||
|
# tftest modules=1 resources=1
|
||
|
```
|
||
|
|
||
|
## IAM Examples
|
||
|
|
||
|
IAM is managed via several variables that implement different levels of control:
|
||
|
|
||
|
- `group_iam` and `iam` configure authoritative bindings that manage individual roles exclusively, mapping to the `google_dataproc_cluster_iam_binding` resource
|
||
|
- `iam_additive` configure additive bindings that only manage individual role/member pairs, mapping to the `google_dataproc_cluster_iam_member` resource
|
||
|
|
||
|
### Authorative IAM
|
||
|
|
||
|
```hcl
|
||
|
module "processing-dp-cluster" {
|
||
|
source = "./fabric/modules/dataproc"
|
||
|
project_id = "my-project"
|
||
|
name = "my-cluster"
|
||
|
region = "europe-west1"
|
||
|
prefix = "prefix"
|
||
|
iam_additive = {
|
||
|
"roles/dataproc.viewer" = [
|
||
|
"serviceAccount:service-account@PROJECT_ID.iam.gserviceaccount.com"
|
||
|
]
|
||
|
}
|
||
|
}
|
||
|
# tftest modules=1 resources=2
|
||
|
```
|
||
|
|
||
|
### Additive IAM
|
||
|
|
||
|
```hcl
|
||
|
module "processing-dp-cluster" {
|
||
|
source = "./fabric/modules/dataproc"
|
||
|
project_id = "my-project"
|
||
|
name = "my-cluster"
|
||
|
region = "europe-west1"
|
||
|
prefix = "prefix"
|
||
|
group_iam = {
|
||
|
"gcp-data-engineers@example.net" = [
|
||
|
"roles/dataproc.viewer"
|
||
|
]
|
||
|
}
|
||
|
}
|
||
|
# tftest modules=1 resources=2
|
||
|
```
|
||
|
<!-- BEGIN TFDOC -->
|
||
|
|
||
|
## Variables
|
||
|
|
||
|
| name | description | type | required | default |
|
||
|
|---|---|:---:|:---:|:---:|
|
||
|
| [name](variables.tf#L50) | Cluster name. | <code>string</code> | ✓ | |
|
||
|
| [project_id](variables.tf#L65) | Project ID. | <code>string</code> | ✓ | |
|
||
|
| [region](variables.tf#L70) | Dataproc region. | <code>string</code> | ✓ | |
|
||
|
| [dataproc_config](variables.tf#L75) | Dataproc cluster config. | <code title="object({ graceful_decommission_timeout = optional(string, null) cluster_config = optional(object({ staging_bucket = optional(string, null) temp_bucket = optional(string, null) gce_cluster_config = optional(object({ zone = optional(string, null) network = optional(string, null) subnetwork = optional(string, null) service_account = optional(string, null) service_account_scopes = optional(list(string), null) tags = optional(list(string), []) internal_ip_only = optional(bool, null) metadata = optional(map(string), {}) reservation_affinity = optional(object({ consume_reservation_type = string key = string values = string }), null) node_group_affinity = optional(object({ node_group_uri = string }), null) shielded_instance_config = optional(object({ enable_secure_boot = bool enable_vtpm = bool enable_integrity_monitoring = bool }), null) }), null) master_config = optional(object({ num_instances = number machine_type = string min_cpu_platform = string disk_config = optional(object({ boot_disk_type = string boot_disk_size_gb = number num_local_ssds = number }), null) accelerators = optional(object({ accelerator_type = string accelerator_count = number }), null) }), null) worker_config = optional(object({ num_instances = number machine_type = string min_cpu_platform = string disk_config = optional(object({ boot_disk_type = string boot_disk_size_gb = number num_local_ssds = number }), null) image_uri = string accelerators = optional(object({ accelerator_type = string accelerator_count = number }), null) }), null) preemptible_worker_config = optional(object({ num_instances = number preemptibility = string disk_config = optional(object({ boot_disk_type = string boot_disk_size_gb = number num_local_ssds = number }), null) }), null) software_config = optional(object({ image_version = string override_properties = list(map(string)) optional_components = list(string) }), null) security_config = optional(object({ kerberos_config = object({ cross_realm_trust_admin_server = optional(string, null) cross_realm_trust_kdc = optional(string, null) cross_realm_trust_realm = optional(string, null) cross_realm_trust_shared_password_uri = optional(string, null)
|
||
|
| [group_iam](variables.tf#L23) | Authoritative IAM binding for organization groups, in {GROUP_EMAIL => [ROLES]} format. Group emails need to be static. Can be used in combination with the `iam` variable. | <code>map(list(string))</code> | | <code>{}</code> |
|
||
|
| [iam](variables.tf#L30) | IAM bindings in {ROLE => [MEMBERS]} format. | <code>map(list(string))</code> | | <code>{}</code> |
|
||
|
| [iam_additive](variables.tf#L37) | IAM additive bindings in {ROLE => [MEMBERS]} format. | <code>map(list(string))</code> | | <code>{}</code> |
|
||
|
| [labels](variables.tf#L44) | The resource labels for instance to use to annotate any related underlying resources, such as Compute Engine VMs. | <code>map(string)</code> | | <code>{}</code> |
|
||
|
| [prefix](variables.tf#L55) | Optional prefix used to generate project id and name. | <code>string</code> | | <code>null</code> |
|
||
|
| [service_account](variables.tf#L17) | Service account to set on the Dataproc cluster. | <code>string</code> | | <code>null</code> |
|
||
|
|
||
|
<!-- END TFDOC -->
|