documentation WIP
This commit is contained in:
Ludovico Magnocavallo
parent
6af9953a49
commit
a16cf9e2a8
|
|
@ -1,26 +1,6 @@
|
|||
# GKE Multitenant Module
|
||||
|
||||
TODO: add description and diagram
|
||||
|
||||
<p align="center">
|
||||
<img src="diagram.png" alt="GKE multitenant">
|
||||
</p>
|
||||
|
||||
## Design overview and choices
|
||||
|
||||
TODO
|
||||
|
||||
### Cluster and nodepool configuration
|
||||
|
||||
TODO
|
||||
|
||||
### Fleet management
|
||||
|
||||
TODO
|
||||
|
||||
### Variable configuration
|
||||
|
||||
TODO
|
||||
TODO: add brief explanation and refer back to dev folder?
|
||||
|
||||
<!-- TFDOC OPTS files:1 show_extra:1 -->
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
|
|
|||
|
|
@ -43,7 +43,7 @@ module "gke-cluster" {
|
|||
horizontal_pod_autoscaling = true
|
||||
config_connector_config = true
|
||||
kalm_config = false
|
||||
gcp_filestore_csi_driver_config = false
|
||||
gcp_filestore_csi_driver_config = each.value.overrides.gcp_filestore_csi_driver_config
|
||||
gke_backup_agent_config = false
|
||||
# enable only if enable_dataplane_v2 is changed to false below
|
||||
network_policy_config = false
|
||||
|
|
|
|||
|
|
@ -10,34 +10,33 @@ TODO: add description and diagram
|
|||
|
||||
TODO
|
||||
|
||||
### Cluster and nodepool configuration
|
||||
## How to run this stage
|
||||
|
||||
Clusters and nodepools configuration comes with variables that let you define common configuration (cluster_default) and define the single cluster details when needed. This stage is designed with multi-tenancy in mind, this means that the GKE clusters within this stage will be similar more often than not.
|
||||
TODO
|
||||
|
||||
```
|
||||
cluster_defaults = {
|
||||
cloudrun_config = false
|
||||
database_encryption_key = null
|
||||
gcp_filestore_csi_driver_config = false
|
||||
master_authorized_ranges = {
|
||||
rfc1918_1 = "10.0.0.0/8"
|
||||
rfc1918_2 = "172.16.0.0/12"
|
||||
rfc1918_3 = "192.168.0.0/16"
|
||||
on_prem = "100.xxx.xxx.xxx/xx"
|
||||
}
|
||||
max_pods_per_node = 110
|
||||
pod_security_policy = false
|
||||
release_channel = "STABLE"
|
||||
vertical_pod_autoscaling = false
|
||||
}
|
||||
```
|
||||
The variable cluster_defaults is used as a shared configuration between all the clusters within the projects, clusters variable instead can be used to define the required specific details:
|
||||
### Providers configuration
|
||||
|
||||
```
|
||||
TODO
|
||||
|
||||
### Variable configuration
|
||||
|
||||
#### Cluster and nodepools
|
||||
|
||||
This stage is designed with multi-tenancy in mind, and the expectation is that GKE clusters will mostly share a common set of defaults. Stage variables are designed to support this approach for both clusters and nodepools:
|
||||
|
||||
- the `cluster_default` variable allows defining common defaults for cluster
|
||||
- the `clusters` variable is used to declare the actual GKE clusters and allows overriding defaults on a per-cluster basis
|
||||
- the `nodepool_defaults` variable allows definining common defaults for nodepools
|
||||
- the `nodepools` variable is used to declare cluster nodepools and allows overriding defaults on a per-cluster basis
|
||||
|
||||
This is an example of that shows the use of the above variables:
|
||||
|
||||
```hcl
|
||||
# the `cluster_defaults` variable defaults are used and not shown here
|
||||
clusters = {
|
||||
"gke-1" = {
|
||||
"gke-00" = {
|
||||
cluster_autoscaling = null
|
||||
description = "gke-1"
|
||||
description = "gke-00"
|
||||
dns_domain = null
|
||||
location = "europe-west1"
|
||||
labels = {}
|
||||
|
|
@ -45,13 +44,13 @@ clusters = {
|
|||
master_range = "172.17.16.0/28"
|
||||
pods = "pods"
|
||||
services = "services"
|
||||
subnet = "https://www.googleapis.com/compute/v1/projects/XXXX-dev-net-spoke-0/regions/europe-west1/subnetworks/dev-gke-nodes-ew1"
|
||||
subnet = local.vpc.subnet_self_links["europe-west3/gke-dev-0"]
|
||||
}
|
||||
overrides = null
|
||||
}
|
||||
"gke-2" = {
|
||||
"gke-01" = {
|
||||
cluster_autoscaling = null
|
||||
description = "gke-2"
|
||||
description = "gke-01"
|
||||
dns_domain = null
|
||||
location = "europe-west3"
|
||||
labels = {}
|
||||
|
|
@ -59,19 +58,25 @@ clusters = {
|
|||
master_range = "172.17.17.0/28"
|
||||
pods = "pods"
|
||||
services = "services"
|
||||
subnet = "https://www.googleapis.com/compute/v1/projects/XXXX-dev-net-spoke-0/regions/europe-west3/subnetworks/dev-gke-nodes-ew3"
|
||||
subnet = local.vpc.subnet_self_links["europe-west3/gke-dev-0"]
|
||||
}
|
||||
overrides = {
|
||||
cloudrun_config = false
|
||||
database_encryption_key = null
|
||||
gcp_filestore_csi_driver_config = true
|
||||
master_authorized_ranges = {
|
||||
rfc1918_1 = "10.0.0.0/8"
|
||||
}
|
||||
max_pods_per_node = 64
|
||||
pod_security_policy = true
|
||||
release_channel = "STABLE"
|
||||
vertical_pod_autoscaling = false
|
||||
}
|
||||
overrides = null
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
The same design principle used for the clusters variable is used to define the nodepool and attach them to the right GKE cluster (previously defined)
|
||||
|
||||
```
|
||||
nodepools = {
|
||||
"gke-1" = {
|
||||
"np-001" = {
|
||||
"gke-0" = {
|
||||
"gke-00-000" = {
|
||||
initial_node_count = 1
|
||||
node_count = 1
|
||||
node_type = "n2-standard-4"
|
||||
|
|
@ -79,21 +84,27 @@ nodepools = {
|
|||
spot = false
|
||||
}
|
||||
}
|
||||
"gke-2" = {
|
||||
"np-002" = {
|
||||
"gke-1" = {
|
||||
"gke-01-000" = {
|
||||
initial_node_count = 1
|
||||
node_count = 1
|
||||
node_type = "n2-standard-4"
|
||||
overrides = null
|
||||
overrides = {
|
||||
image_type = "UBUNTU_CONTAINERD"
|
||||
max_pods_per_node = 64
|
||||
node_locations = []
|
||||
node_tags = []
|
||||
node_taints = []
|
||||
}
|
||||
spot = true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
On the top of all the clusters configuration, the variable authenticator_security_group can be used to define the google group that should be used within Google Groups for RBAC feature as authenticator security group.
|
||||
There are two additional variables that influence cluster configuration: `authenticator_security_group` to configure Google Groups for RBAC, `dns_domain` to configure Cloud DNS for GKE.
|
||||
|
||||
### Fleet management
|
||||
#### Fleet management
|
||||
|
||||
Fleet management is achieved by the configuration of the fleet_configmanagement_templates, fleet_configmanagement_clusters and fleet_features variables exposed by the module in _module. In details fleet_features lets you activate the fleet features, fleet_configmanagement_templates lets you define one o more fleet configmanagement configuration template to be activated onto one or more GKE clusters. Configured features and settings can be applied to clusters by leveraging fleet_configmanagement_clusters where a single template can be applied to one or more clusters.
|
||||
|
||||
|
|
@ -142,7 +153,6 @@ fleet_features = {
|
|||
}
|
||||
```
|
||||
|
||||
|
||||
## How to run this stage
|
||||
|
||||
This stage is meant to be executed after "foundational stages" (i.e., stages [`00-bootstrap`](../../00-bootstrap), [`01-resman`](../../01-resman), 02-networking (either [VPN](../../02-networking-vpn) or [NVA](../../02-networking-nva)) and [`02-security`](../../02-security)) have been run.
|
||||
|
|
@ -176,25 +186,25 @@ TODO
|
|||
|---|---|:---:|:---:|:---:|:---:|
|
||||
| [automation](variables.tf#L21) | Automation resources created by the bootstrap stage. | <code title="object({ outputs_bucket = string })">object({…})</code> | ✓ | | <code>00-bootstrap</code> |
|
||||
| [billing_account](variables.tf#L35) | Billing account id and organization id ('nnnnnnnn' or null). | <code title="object({ id = string organization_id = number })">object({…})</code> | ✓ | | <code>00-bootstrap</code> |
|
||||
| [clusters](variables.tf#L74) | | <code title="map(object({ cluster_autoscaling = object({ cpu_min = number cpu_max = number memory_min = number memory_max = number }) description = string dns_domain = string labels = map(string) location = string net = object({ master_range = string pods = string services = string subnet = string }) overrides = object({ cloudrun_config = bool database_encryption_key = string master_authorized_ranges = map(string) max_pods_per_node = number pod_security_policy = bool release_channel = string vertical_pod_autoscaling = bool gcp_filestore_csi_driver_config = bool }) }))">map(object({…}))</code> | ✓ | | |
|
||||
| [folder_ids](variables.tf#L176) | Folders to be used for the networking resources in folders/nnnnnnnnnnn format. If null, folder will be created. | <code title="object({ gke-dev = string })">object({…})</code> | ✓ | | <code>01-resman</code> |
|
||||
| [host_project_ids](variables.tf#L198) | Host project for the shared VPC. | <code title="object({ dev-spoke-0 = string })">object({…})</code> | ✓ | | <code>02-networking</code> |
|
||||
| [nodepools](variables.tf#L230) | | <code title="map(map(object({ node_count = number node_type = string initial_node_count = number overrides = object({ image_type = string max_pods_per_node = number node_locations = list(string) node_tags = list(string) node_taints = list(string) }) spot = bool })))">map(map(object({…})))</code> | ✓ | | |
|
||||
| [prefix](variables.tf#L253) | Prefix used for resources that need unique names. | <code>string</code> | ✓ | | |
|
||||
| [vpc_self_links](variables.tf#L265) | Self link for the shared VPC. | <code title="object({ dev-spoke-0 = string })">object({…})</code> | ✓ | | <code>02-networking</code> |
|
||||
| [clusters](variables.tf#L73) | | <code title="map(object({ cluster_autoscaling = object({ cpu_min = number cpu_max = number memory_min = number memory_max = number }) description = string dns_domain = string labels = map(string) location = string net = object({ master_range = string pods = string services = string subnet = string }) overrides = object({ cloudrun_config = bool database_encryption_key = string master_authorized_ranges = map(string) max_pods_per_node = number pod_security_policy = bool release_channel = string vertical_pod_autoscaling = bool gcp_filestore_csi_driver_config = bool }) }))">map(object({…}))</code> | ✓ | | |
|
||||
| [folder_ids](variables.tf#L175) | Folders to be used for the networking resources in folders/nnnnnnnnnnn format. If null, folder will be created. | <code title="object({ gke-dev = string })">object({…})</code> | ✓ | | <code>01-resman</code> |
|
||||
| [host_project_ids](variables.tf#L197) | Host project for the shared VPC. | <code title="object({ dev-spoke-0 = string })">object({…})</code> | ✓ | | <code>02-networking</code> |
|
||||
| [nodepools](variables.tf#L229) | | <code title="map(map(object({ node_count = number node_type = string initial_node_count = number overrides = object({ image_type = string max_pods_per_node = number node_locations = list(string) node_tags = list(string) node_taints = list(string) }) spot = bool })))">map(map(object({…})))</code> | ✓ | | |
|
||||
| [prefix](variables.tf#L252) | Prefix used for resources that need unique names. | <code>string</code> | ✓ | | |
|
||||
| [vpc_self_links](variables.tf#L264) | Self link for the shared VPC. | <code title="object({ dev-spoke-0 = string })">object({…})</code> | ✓ | | <code>02-networking</code> |
|
||||
| [authenticator_security_group](variables.tf#L29) | Optional group used for Groups for GKE. | <code>string</code> | | <code>null</code> | |
|
||||
| [cluster_defaults](variables.tf#L44) | Default values for optional cluster configurations. | <code title="object({ cloudrun_config = bool database_encryption_key = string master_authorized_ranges = map(string) max_pods_per_node = number pod_security_policy = bool release_channel = string vertical_pod_autoscaling = bool gcp_filestore_csi_driver_config = bool })">object({…})</code> | | <code title="{ cloudrun_config = false database_encryption_key = null master_authorized_ranges = { rfc1918_1 = "10.0.0.0/8" rfc1918_2 = "172.16.0.0/12" rfc1918_3 = "192.168.0.0/16" } max_pods_per_node = 110 pod_security_policy = false release_channel = "STABLE" vertical_pod_autoscaling = false gcp_filestore_csi_driver_config = false }">{…}</code> | |
|
||||
| [dns_domain](variables.tf#L107) | Domain name used for clusters, prefixed by each cluster name. Leave null to disable Cloud DNS for GKE. | <code>string</code> | | <code>null</code> | |
|
||||
| [fleet_configmanagement_clusters](variables.tf#L113) | Config management features enabled on specific sets of member clusters, in config name => [cluster name] format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [fleet_configmanagement_templates](variables.tf#L121) | Sets of config management configurations that can be applied to member clusters, in config name => {options} format. | <code title="map(object({ binauthz = bool config_sync = object({ git = object({ gcp_service_account_email = string https_proxy = string policy_dir = string secret_type = string sync_branch = string sync_repo = string sync_rev = string sync_wait_secs = number }) prevent_drift = string source_format = string }) hierarchy_controller = object({ enable_hierarchical_resource_quota = bool enable_pod_tree_labels = bool }) policy_controller = object({ audit_interval_seconds = number exemptable_namespaces = list(string) log_denies_enabled = bool referential_rules_enabled = bool template_library_installed = bool }) version = string }))">map(object({…}))</code> | | <code>{}</code> | |
|
||||
| [fleet_features](variables.tf#L156) | Enable and configue fleet features. Set to null to disable GKE Hub if fleet workload identity is not used. | <code title="object({ appdevexperience = bool configmanagement = bool identityservice = bool multiclusteringress = string multiclusterservicediscovery = bool servicemesh = bool })">object({…})</code> | | <code>null</code> | |
|
||||
| [fleet_workload_identity](variables.tf#L169) | Use Fleet Workload Identity for clusters. Enables GKE Hub if set to true. | <code>bool</code> | | <code>true</code> | |
|
||||
| [group_iam](variables.tf#L184) | Project-level authoritative IAM bindings for groups in {GROUP_EMAIL => [ROLES]} format. Use group emails as keys, list of roles as values. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [iam](variables.tf#L191) | Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [labels](variables.tf#L206) | Project-level labels. | <code>map(string)</code> | | <code>{}</code> | |
|
||||
| [nodepool_defaults](variables.tf#L212) | | <code title="object({ image_type = string max_pods_per_node = number node_locations = list(string) node_tags = list(string) node_taints = list(string) })">object({…})</code> | | <code title="{ image_type = "COS_CONTAINERD" max_pods_per_node = 110 node_locations = null node_tags = null node_taints = [] }">{…}</code> | |
|
||||
| [outputs_location](variables.tf#L247) | Path where providers, tfvars files, and lists for the following stages are written. Leave empty to disable. | <code>string</code> | | <code>null</code> | |
|
||||
| [project_services](variables.tf#L258) | Additional project services to enable. | <code>list(string)</code> | | <code>[]</code> | |
|
||||
| [dns_domain](variables.tf#L106) | Domain name used for clusters, prefixed by each cluster name. Leave null to disable Cloud DNS for GKE. | <code>string</code> | | <code>null</code> | |
|
||||
| [fleet_configmanagement_clusters](variables.tf#L112) | Config management features enabled on specific sets of member clusters, in config name => [cluster name] format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [fleet_configmanagement_templates](variables.tf#L120) | Sets of config management configurations that can be applied to member clusters, in config name => {options} format. | <code title="map(object({ binauthz = bool config_sync = object({ git = object({ gcp_service_account_email = string https_proxy = string policy_dir = string secret_type = string sync_branch = string sync_repo = string sync_rev = string sync_wait_secs = number }) prevent_drift = string source_format = string }) hierarchy_controller = object({ enable_hierarchical_resource_quota = bool enable_pod_tree_labels = bool }) policy_controller = object({ audit_interval_seconds = number exemptable_namespaces = list(string) log_denies_enabled = bool referential_rules_enabled = bool template_library_installed = bool }) version = string }))">map(object({…}))</code> | | <code>{}</code> | |
|
||||
| [fleet_features](variables.tf#L155) | Enable and configue fleet features. Set to null to disable GKE Hub if fleet workload identity is not used. | <code title="object({ appdevexperience = bool configmanagement = bool identityservice = bool multiclusteringress = string multiclusterservicediscovery = bool servicemesh = bool })">object({…})</code> | | <code>null</code> | |
|
||||
| [fleet_workload_identity](variables.tf#L168) | Use Fleet Workload Identity for clusters. Enables GKE Hub if set to true. | <code>bool</code> | | <code>true</code> | |
|
||||
| [group_iam](variables.tf#L183) | Project-level authoritative IAM bindings for groups in {GROUP_EMAIL => [ROLES]} format. Use group emails as keys, list of roles as values. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [iam](variables.tf#L190) | Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [labels](variables.tf#L205) | Project-level labels. | <code>map(string)</code> | | <code>{}</code> | |
|
||||
| [nodepool_defaults](variables.tf#L211) | | <code title="object({ image_type = string max_pods_per_node = number node_locations = list(string) node_tags = list(string) node_taints = list(string) })">object({…})</code> | | <code title="{ image_type = "COS_CONTAINERD" max_pods_per_node = 110 node_locations = null node_tags = null node_taints = [] }">{…}</code> | |
|
||||
| [outputs_location](variables.tf#L246) | Path where providers, tfvars files, and lists for the following stages are written. Leave empty to disable. | <code>string</code> | | <code>null</code> | |
|
||||
| [project_services](variables.tf#L257) | Additional project services to enable. | <code>list(string)</code> | | <code>[]</code> | |
|
||||
|
||||
## Outputs
|
||||
|
||||
|
|
|
|||
|
|
@ -54,7 +54,6 @@ variable "cluster_defaults" {
|
|||
gcp_filestore_csi_driver_config = bool
|
||||
})
|
||||
default = {
|
||||
# TODO: review defaults
|
||||
cloudrun_config = false
|
||||
database_encryption_key = null
|
||||
# binary_authorization = false
|
||||
|
|
|
|||
Loading…
Reference in New Issue