Merge pull request #700 from GoogleCloudPlatform/fast/gke2
FAST: GKE multitenant infrastructure
This commit is contained in:
Julio Castillo
GitHub
commit
86cd08e283
|
|
@ -97,8 +97,7 @@ module "cluster" {
|
|||
master_ipv4_cidr_block = var.master_cidr_block
|
||||
master_global_access = false
|
||||
}
|
||||
enable_binary_authorization = true
|
||||
workload_identity = true
|
||||
workload_identity = true
|
||||
}
|
||||
|
||||
module "cluster_nodepool" {
|
||||
|
|
|
|||
|
|
@ -0,0 +1,12 @@
|
|||
# GKE and Serverless examples
|
||||
|
||||
The examples in this folder show implement **end-to-end scenarios** for GKE or Serveless topologies that show how to automate common configurations or leverage specific products.
|
||||
|
||||
They are meant to be used as minimal but complete starting points to create actual infrastructure, and as playgrounds to experiment with Google Cloud features.
|
||||
|
||||
## Examples
|
||||
|
||||
### Multitenant GKE fleet
|
||||
|
||||
<a href="./multitenant-fleet/" title="GKE multitenant fleet"><img src="./multitenant-fleet/diagram.png" align="left" width="280px"></a> This [example](./multitenant-fleet/) allows simple centralized management of similar sets of GKE clusters and their nodepools in a single project, and optional fleet management via GKE Hub templated configurations.
|
||||
<br clear="left">
|
||||
|
|
@ -0,0 +1,359 @@
|
|||
# GKE Multitenant Example
|
||||
|
||||
This example presents an opinionated architecture to handle multiple homogeneous GKE clusters. The general idea behind this example is to deploy a single project hosting multiple clusters leveraging several useful GKE features.
|
||||
|
||||
The pattern used in this design is useful, for example, in cases where multiple clusters host/support the same workloads, such as in the case of a multi-regional deployment. Furthermore, combined with Anthos Config Sync and proper RBAC, this architecture can be used to host multiple tenants (e.g. teams, applications) sharing the clusters.
|
||||
|
||||
This example is used as part of the [FAST GKE stage](../../../fast/stages/03-gke-multitenant/) but it can also be used independently if desired.
|
||||
|
||||
<p align="center">
|
||||
<img src="diagram.png" alt="GKE multitenant">
|
||||
</p>
|
||||
|
||||
The overall architecture is based on the following design decisions:
|
||||
|
||||
- All clusters are assumed to be [private](https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters), therefore only [VPC-native clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips) are supported.
|
||||
- Logging and monitoring configured to use Cloud Operations for system components and user workloads.
|
||||
- [GKE metering](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-usage-metering) enabled by default and stored in a bigquery dataset created within the project.
|
||||
- Optional [GKE Fleet](https://cloud.google.com/kubernetes-engine/docs/fleets-overview) support with the possibility to enable any of the following features:
|
||||
- [Fleet workload identity](https://cloud.google.com/anthos/fleet-management/docs/use-workload-identity)
|
||||
- [Anthos Config Management](https://cloud.google.com/anthos-config-management/docs/overview)
|
||||
- [Anthos Service Mesh](https://cloud.google.com/service-mesh/docs/overview)
|
||||
- [Anthos Identity Service](https://cloud.google.com/anthos/identity/setup/fleet)
|
||||
- [Multi-cluster services](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-services)
|
||||
- [Multi-cluster ingress](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-ingress).
|
||||
- Support for [Config Sync](https://cloud.google.com/anthos-config-management/docs/config-sync-overview), [Hierarchy Controller](https://cloud.google.com/anthos-config-management/docs/concepts/hierarchy-controller), and [Policy Controller](https://cloud.google.com/anthos-config-management/docs/concepts/policy-controller) when using Anthos Config Management.
|
||||
- [Groups for GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/google-groups-rbac) can be enabled to facilitate the creation of flexible RBAC policies referencing group principals.
|
||||
- Support for [application layer secret encryption](https://cloud.google.com/kubernetes-engine/docs/how-to/encrypting-secrets).
|
||||
- Support to customize peering configuration of the control plane VPC (e.g. to import/export routes to the peered network)
|
||||
- Some features are enabled by default in all clusters:
|
||||
- [Intranode visibility](https://cloud.google.com/kubernetes-engine/docs/how-to/intranode-visibility)
|
||||
- [Dataplane v2](https://cloud.google.com/kubernetes-engine/docs/concepts/dataplane-v2)
|
||||
- [Shielded GKE nodes](https://cloud.google.com/kubernetes-engine/docs/how-to/shielded-gke-nodes)
|
||||
- [Workload identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity)
|
||||
- [Node local DNS cache](https://cloud.google.com/kubernetes-engine/docs/how-to/nodelocal-dns-cache)
|
||||
- [Use of the GCE persistent disk CSI driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver)
|
||||
- Node [auto-upgrade](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-upgrades) and [auto-repair](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair) for all node pools
|
||||
|
||||
<!--
|
||||
- [GKE subsetting for L4 internal load balancers](https://cloud.google.com/kubernetes-engine/docs/concepts/service-load-balancer#subsetting) enabled by default in all clusters
|
||||
-->
|
||||
|
||||
## Basic usage
|
||||
|
||||
The following example shows how to deploy a single cluster and a single node pool
|
||||
|
||||
```hcl
|
||||
module "gke" {
|
||||
source = "./fabric/examples/gke-serverless/multitenant-fleet/"
|
||||
project_id = var.project_id
|
||||
billing_account_id = var.billing_account_id
|
||||
folder_id = var.folder_id
|
||||
prefix = "myprefix"
|
||||
vpc_config = {
|
||||
host_project_id = "my-host-project-id"
|
||||
vpc_self_link = "projects/my-host-project-id/global/networks/my-network"
|
||||
}
|
||||
|
||||
authenticator_security_group = "gke-rbac-base@example.com"
|
||||
group_iam = {
|
||||
"gke-admin@example.com" = [
|
||||
"roles/container.admin"
|
||||
]
|
||||
}
|
||||
iam = {
|
||||
"roles/container.clusterAdmin" = [
|
||||
"cicd@my-cicd-project.iam.gserviceaccount.com"
|
||||
]
|
||||
}
|
||||
|
||||
clusters = {
|
||||
mycluster = {
|
||||
cluster_autoscaling = null
|
||||
description = "My cluster"
|
||||
dns_domain = null
|
||||
location = "europe-west1"
|
||||
labels = {}
|
||||
net = {
|
||||
master_range = "172.17.16.0/28"
|
||||
pods = "pods"
|
||||
services = "services"
|
||||
subnet = "projects/my-host-project-id/regions/europe-west1/subnetworks/mycluster-subnet"
|
||||
}
|
||||
overrides = null
|
||||
}
|
||||
}
|
||||
nodepools = {
|
||||
mycluster = {
|
||||
mynodepool = {
|
||||
initial_node_count = 1
|
||||
node_count = 1
|
||||
node_type = "n2-standard-4"
|
||||
overrides = null
|
||||
spot = false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
# tftest modules=1 resources=0
|
||||
```
|
||||
|
||||
## Creating Multiple Clusters
|
||||
|
||||
The following example shows how to deploy two clusters with different configurations.
|
||||
|
||||
The first cluster `cluster-euw1` defines the mandatory configuration parameters (description, location, network setup) and inherits the some defaults from the `cluster_defaults` and `nodepool_deaults` variables. These two variables are used whenever the `override` key of the `clusters` and `nodepools` variables are set to `null`.
|
||||
|
||||
On the other hand, the second cluster (`cluster-euw3`) defines its own configuration by providing a value to the `overrides` key.
|
||||
|
||||
|
||||
```hcl
|
||||
module "gke" {
|
||||
source = "./fabric/examples/gke-serverless/multitenant-fleet/"
|
||||
project_id = var.project_id
|
||||
billing_account_id = var.billing_account_id
|
||||
folder_id = var.folder_id
|
||||
prefix = "myprefix"
|
||||
vpc_config = {
|
||||
host_project_id = "my-host-project-id"
|
||||
vpc_self_link = "projects/my-host-project-id/global/networks/my-network"
|
||||
}
|
||||
clusters = {
|
||||
cluster-euw1 = {
|
||||
cluster_autoscaling = null
|
||||
description = "Cluster for europ-west1"
|
||||
dns_domain = null
|
||||
location = "europe-west1"
|
||||
labels = {}
|
||||
net = {
|
||||
master_range = "172.17.16.0/28"
|
||||
pods = "pods"
|
||||
services = "services"
|
||||
subnet = "projects/my-host-project-id/regions/europe-west1/subnetworks/euw1-subnet"
|
||||
}
|
||||
overrides = null
|
||||
}
|
||||
cluster-euw3 = {
|
||||
cluster_autoscaling = null
|
||||
description = "Cluster for europe-west3"
|
||||
dns_domain = null
|
||||
location = "europe-west3"
|
||||
labels = {}
|
||||
net = {
|
||||
master_range = "172.17.17.0/28"
|
||||
pods = "pods"
|
||||
services = "services"
|
||||
subnet = "projects/my-host-project-id/regions/europe-west3/subnetworks/euw3-subnet"
|
||||
}
|
||||
overrides = {
|
||||
cloudrun_config = false
|
||||
database_encryption_key = null
|
||||
gcp_filestore_csi_driver_config = true
|
||||
master_authorized_ranges = {
|
||||
rfc1918_1 = "10.0.0.0/8"
|
||||
}
|
||||
max_pods_per_node = 64
|
||||
pod_security_policy = true
|
||||
release_channel = "STABLE"
|
||||
vertical_pod_autoscaling = false
|
||||
}
|
||||
}
|
||||
}
|
||||
nodepools = {
|
||||
cluster-euw1 = {
|
||||
pool-euw1 = {
|
||||
initial_node_count = 1
|
||||
node_count = 1
|
||||
node_type = "n2-standard-4"
|
||||
overrides = null
|
||||
spot = false
|
||||
}
|
||||
}
|
||||
cluster-euw3 = {
|
||||
pool-euw3 = {
|
||||
initial_node_count = 1
|
||||
node_count = 1
|
||||
node_type = "n2-standard-4"
|
||||
overrides = {
|
||||
image_type = "UBUNTU_CONTAINERD"
|
||||
max_pods_per_node = 64
|
||||
node_locations = []
|
||||
node_tags = []
|
||||
node_taints = []
|
||||
}
|
||||
spot = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
# tftest modules=1 resources=0
|
||||
```
|
||||
|
||||
## Multiple clusters with GKE Fleet
|
||||
|
||||
This example deploys two clusters and configures several GKE Fleet features:
|
||||
|
||||
- Enables [multi-cluster ingress](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-ingress) and sets the configuration cluster to be `cluster-eu1`.
|
||||
- Enables [Multi-cluster services](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-services) and assigns the [required roles](https://cloud.google.com/kubernetes-engine/docs/how-to/multi-cluster-services#authenticating) to its service accounts.
|
||||
- A `default` Config Management template is created with binary authorization, config sync enabled with a git repository, hierarchy controller, and policy controller.
|
||||
- The two clusters are configured to use the `default` Config Management template.
|
||||
|
||||
```hcl
|
||||
module "gke" {
|
||||
source = "./fabric/examples/gke-serverless/multitenant-fleet/"
|
||||
project_id = var.project_id
|
||||
billing_account_id = var.billing_account_id
|
||||
folder_id = var.folder_id
|
||||
prefix = "myprefix"
|
||||
vpc_config = {
|
||||
host_project_id = "my-host-project-id"
|
||||
vpc_self_link = "projects/my-host-project-id/global/networks/my-network"
|
||||
}
|
||||
clusters = {
|
||||
cluster-euw1 = {
|
||||
cluster_autoscaling = null
|
||||
description = "Cluster for europe-west1"
|
||||
dns_domain = null
|
||||
location = "europe-west1"
|
||||
labels = {}
|
||||
net = {
|
||||
master_range = "172.17.16.0/28"
|
||||
pods = "pods"
|
||||
services = "services"
|
||||
subnet = "projects/my-host-project-id/regions/europe-west1/subnetworks/euw1-subnet"
|
||||
}
|
||||
overrides = null
|
||||
}
|
||||
cluster-euw3 = {
|
||||
cluster_autoscaling = null
|
||||
description = "Cluster for europe-west3"
|
||||
dns_domain = null
|
||||
location = "europe-west3"
|
||||
labels = {}
|
||||
net = {
|
||||
master_range = "172.17.17.0/28"
|
||||
pods = "pods"
|
||||
services = "services"
|
||||
subnet = "projects/my-host-project-id/regions/europe-west3/subnetworks/euw3-subnet"
|
||||
}
|
||||
overrides = null
|
||||
}
|
||||
}
|
||||
nodepools = {
|
||||
cluster-euw1 = {
|
||||
pool-euw1 = {
|
||||
initial_node_count = 1
|
||||
node_count = 1
|
||||
node_type = "n2-standard-4"
|
||||
overrides = null
|
||||
spot = false
|
||||
}
|
||||
}
|
||||
cluster-euw3 = {
|
||||
pool-euw3 = {
|
||||
initial_node_count = 1
|
||||
node_count = 1
|
||||
node_type = "n2-standard-4"
|
||||
overrides = null
|
||||
spot = true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fleet_features = {
|
||||
appdevexperience = false
|
||||
configmanagement = true
|
||||
identityservice = true
|
||||
multiclusteringress = "cluster-euw1"
|
||||
multiclusterservicediscovery = true
|
||||
servicemesh = true
|
||||
}
|
||||
fleet_workload_identity = true
|
||||
fleet_configmanagement_templates = {
|
||||
default = {
|
||||
binauthz = true
|
||||
config_sync = {
|
||||
git = {
|
||||
gcp_service_account_email = null
|
||||
https_proxy = null
|
||||
policy_dir = "configsync"
|
||||
secret_type = "none"
|
||||
source_format = "hierarchy"
|
||||
sync_branch = "main"
|
||||
sync_repo = "https://github.com/myorg/myrepo"
|
||||
sync_rev = null
|
||||
sync_wait_secs = null
|
||||
}
|
||||
prevent_drift = true
|
||||
source_format = "hierarchy"
|
||||
}
|
||||
hierarchy_controller = {
|
||||
enable_hierarchical_resource_quota = true
|
||||
enable_pod_tree_labels = true
|
||||
}
|
||||
policy_controller = {
|
||||
audit_interval_seconds = 30
|
||||
exemptable_namespaces = ["kube-system"]
|
||||
log_denies_enabled = true
|
||||
referential_rules_enabled = true
|
||||
template_library_installed = true
|
||||
}
|
||||
version = "1.10.2"
|
||||
}
|
||||
}
|
||||
fleet_configmanagement_clusters = {
|
||||
default = ["cluster-euw1", "cluster-euw3"]
|
||||
}
|
||||
}
|
||||
|
||||
# tftest modules=1 resources=0
|
||||
```
|
||||
|
||||
<!-- TFDOC OPTS files:1 show_extra:1 -->
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Files
|
||||
|
||||
| name | description | modules |
|
||||
|---|---|---|
|
||||
| [gke-clusters.tf](./gke-clusters.tf) | None | <code>gke-cluster</code> |
|
||||
| [gke-hub.tf](./gke-hub.tf) | None | <code>gke-hub</code> |
|
||||
| [gke-nodepools.tf](./gke-nodepools.tf) | None | <code>gke-nodepool</code> |
|
||||
| [main.tf](./main.tf) | Module-level locals and resources. | <code>bigquery-dataset</code> · <code>project</code> |
|
||||
| [outputs.tf](./outputs.tf) | Output variables. | |
|
||||
| [variables.tf](./variables.tf) | Module variables. | |
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default | producer |
|
||||
|---|---|:---:|:---:|:---:|:---:|
|
||||
| [billing_account_id](variables.tf#L23) | Billing account id. | <code>string</code> | ✓ | | |
|
||||
| [clusters](variables.tf#L57) | | <code title="map(object({ cluster_autoscaling = object({ cpu_min = number cpu_max = number memory_min = number memory_max = number }) description = string dns_domain = string labels = map(string) location = string net = object({ master_range = string pods = string services = string subnet = string }) overrides = object({ cloudrun_config = bool database_encryption_key = string master_authorized_ranges = map(string) max_pods_per_node = number pod_security_policy = bool release_channel = string vertical_pod_autoscaling = bool gcp_filestore_csi_driver_config = bool }) }))">map(object({…}))</code> | ✓ | | |
|
||||
| [folder_id](variables.tf#L158) | Folder used for the GKE project in folders/nnnnnnnnnnn format. | <code>string</code> | ✓ | | |
|
||||
| [nodepools](variables.tf#L201) | | <code title="map(map(object({ node_count = number node_type = string initial_node_count = number overrides = object({ image_type = string max_pods_per_node = number node_locations = list(string) node_tags = list(string) node_taints = list(string) }) spot = bool })))">map(map(object({…})))</code> | ✓ | | |
|
||||
| [prefix](variables.tf#L231) | Prefix used for resources that need unique names. | <code>string</code> | ✓ | | |
|
||||
| [project_id](variables.tf#L236) | ID of the project that will contain all the clusters. | <code>string</code> | ✓ | | |
|
||||
| [vpc_config](variables.tf#L248) | Shared VPC project and VPC details. | <code title="object({ host_project_id = string vpc_self_link = string })">object({…})</code> | ✓ | | |
|
||||
| [authenticator_security_group](variables.tf#L17) | Optional group used for Groups for GKE. | <code>string</code> | | <code>null</code> | |
|
||||
| [cluster_defaults](variables.tf#L28) | Default values for optional cluster configurations. | <code title="object({ cloudrun_config = bool database_encryption_key = string master_authorized_ranges = map(string) max_pods_per_node = number pod_security_policy = bool release_channel = string vertical_pod_autoscaling = bool gcp_filestore_csi_driver_config = bool })">object({…})</code> | | <code title="{ cloudrun_config = false database_encryption_key = null master_authorized_ranges = { rfc1918_1 = "10.0.0.0/8" rfc1918_2 = "172.16.0.0/12" rfc1918_3 = "192.168.0.0/16" } max_pods_per_node = 110 pod_security_policy = false release_channel = "STABLE" vertical_pod_autoscaling = false gcp_filestore_csi_driver_config = false }">{…}</code> | |
|
||||
| [dns_domain](variables.tf#L90) | Domain name used for clusters, prefixed by each cluster name. Leave null to disable Cloud DNS for GKE. | <code>string</code> | | <code>null</code> | |
|
||||
| [fleet_configmanagement_clusters](variables.tf#L96) | Config management features enabled on specific sets of member clusters, in config name => [cluster name] format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [fleet_configmanagement_templates](variables.tf#L103) | Sets of config management configurations that can be applied to member clusters, in config name => {options} format. | <code title="map(object({ binauthz = bool config_sync = object({ git = object({ gcp_service_account_email = string https_proxy = string policy_dir = string secret_type = string sync_branch = string sync_repo = string sync_rev = string sync_wait_secs = number }) prevent_drift = string source_format = string }) hierarchy_controller = object({ enable_hierarchical_resource_quota = bool enable_pod_tree_labels = bool }) policy_controller = object({ audit_interval_seconds = number exemptable_namespaces = list(string) log_denies_enabled = bool referential_rules_enabled = bool template_library_installed = bool }) version = string }))">map(object({…}))</code> | | <code>{}</code> | |
|
||||
| [fleet_features](variables.tf#L138) | Enable and configue fleet features. Set to null to disable GKE Hub if fleet workload identity is not used. | <code title="object({ appdevexperience = bool configmanagement = bool identityservice = bool multiclusteringress = string multiclusterservicediscovery = bool servicemesh = bool })">object({…})</code> | | <code>null</code> | |
|
||||
| [fleet_workload_identity](variables.tf#L151) | Use Fleet Workload Identity for clusters. Enables GKE Hub if set to true. | <code>bool</code> | | <code>false</code> | |
|
||||
| [group_iam](variables.tf#L163) | Project-level IAM bindings for groups. Use group emails as keys, list of roles as values. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [iam](variables.tf#L170) | Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [labels](variables.tf#L177) | Project-level labels. | <code>map(string)</code> | | <code>{}</code> | |
|
||||
| [nodepool_defaults](variables.tf#L183) | | <code title="object({ image_type = string max_pods_per_node = number node_locations = list(string) node_tags = list(string) node_taints = list(string) })">object({…})</code> | | <code title="{ image_type = "COS_CONTAINERD" max_pods_per_node = 110 node_locations = null node_tags = null node_taints = [] }">{…}</code> | |
|
||||
| [peering_config](variables.tf#L218) | Configure peering with the control plane VPC. Requires compute.networks.updatePeering. Set to null if you don't want to update the default peering configuration. | <code title="object({ export_routes = bool import_routes = bool })">object({…})</code> | | <code title="{ export_routes = true // TODO(jccb) is there any situation where the control plane VPC would export any routes? import_routes = false }">{…}</code> | |
|
||||
| [project_services](variables.tf#L241) | Additional project services to enable. | <code>list(string)</code> | | <code>[]</code> | |
|
||||
|
||||
## Outputs
|
||||
|
||||
| name | description | sensitive | consumers |
|
||||
|---|---|:---:|---|
|
||||
| [cluster_ids](outputs.tf#L22) | Cluster ids. | | |
|
||||
| [clusters](outputs.tf#L17) | Cluster resources. | | |
|
||||
| [project_id](outputs.tf#L29) | GKE project id. | | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 43 KiB |
|
|
@ -0,0 +1,116 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
locals {
|
||||
clusters = {
|
||||
for name, config in var.clusters :
|
||||
name => merge(config, {
|
||||
overrides = coalesce(config.overrides, var.cluster_defaults)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
module "gke-cluster" {
|
||||
source = "../../../modules/gke-cluster"
|
||||
for_each = local.clusters
|
||||
name = each.key
|
||||
project_id = module.gke-project-0.project_id
|
||||
description = each.value.description
|
||||
location = each.value.location
|
||||
network = var.vpc_config.vpc_self_link
|
||||
subnetwork = each.value.net.subnet
|
||||
secondary_range_pods = each.value.net.pods
|
||||
secondary_range_services = each.value.net.services
|
||||
labels = each.value.labels
|
||||
addons = {
|
||||
cloudrun_config = each.value.overrides.cloudrun_config
|
||||
dns_cache_config = true
|
||||
http_load_balancing = true
|
||||
gce_persistent_disk_csi_driver_config = true
|
||||
horizontal_pod_autoscaling = true
|
||||
config_connector_config = true
|
||||
kalm_config = false
|
||||
gcp_filestore_csi_driver_config = each.value.overrides.gcp_filestore_csi_driver_config
|
||||
gke_backup_agent_config = false
|
||||
# enable only if enable_dataplane_v2 is changed to false below
|
||||
network_policy_config = false
|
||||
istio_config = {
|
||||
enabled = false
|
||||
tls = false
|
||||
}
|
||||
}
|
||||
# change these here for all clusters if absolutely needed
|
||||
authenticator_security_group = var.authenticator_security_group
|
||||
enable_dataplane_v2 = true
|
||||
enable_l4_ilb_subsetting = false
|
||||
enable_intranode_visibility = true
|
||||
enable_shielded_nodes = true
|
||||
workload_identity = true
|
||||
private_cluster_config = {
|
||||
enable_private_nodes = true
|
||||
enable_private_endpoint = false
|
||||
master_ipv4_cidr_block = each.value.net.master_range
|
||||
master_global_access = true
|
||||
}
|
||||
dns_config = each.value.dns_domain == null ? null : {
|
||||
cluster_dns = "CLOUD_DNS"
|
||||
cluster_dns_scope = "VPC_SCOPE"
|
||||
cluster_dns_domain = "${each.key}.${var.dns_domain}"
|
||||
}
|
||||
logging_config = ["SYSTEM_COMPONENTS", "WORKLOADS"]
|
||||
monitoring_config = ["SYSTEM_COMPONENTS", "WORKLOADS"]
|
||||
|
||||
peering_config = var.peering_config == null ? null : {
|
||||
export_routes = var.peering_config.export_routes
|
||||
import_routes = var.peering_config.import_routes
|
||||
project_id = var.vpc_config.host_project_id
|
||||
}
|
||||
resource_usage_export_config = {
|
||||
enabled = true
|
||||
dataset = module.gke-dataset-resource-usage.dataset_id
|
||||
}
|
||||
# TODO: the attributes below are "primed" from project-level defaults
|
||||
# in locals, merge defaults with cluster-level stuff
|
||||
# TODO(jccb): change fabric module
|
||||
database_encryption = (
|
||||
each.value.overrides.database_encryption_key == null
|
||||
? {
|
||||
enabled = false
|
||||
state = null
|
||||
key_name = null
|
||||
}
|
||||
: {
|
||||
enabled = true
|
||||
state = "ENCRYPTED"
|
||||
key_name = each.value.overrides.database_encryption_key
|
||||
}
|
||||
)
|
||||
default_max_pods_per_node = each.value.overrides.max_pods_per_node
|
||||
master_authorized_ranges = each.value.overrides.master_authorized_ranges
|
||||
pod_security_policy = each.value.overrides.pod_security_policy
|
||||
release_channel = each.value.overrides.release_channel
|
||||
vertical_pod_autoscaling = each.value.overrides.vertical_pod_autoscaling
|
||||
# dynamic "cluster_autoscaling" {
|
||||
# for_each = each.value.cluster_autoscaling == null ? {} : { 1 = 1 }
|
||||
# content {
|
||||
# enabled = true
|
||||
# cpu_min = each.value.cluster_autoscaling.cpu_min
|
||||
# cpu_max = each.value.cluster_autoscaling.cpu_max
|
||||
# memory_min = each.value.cluster_autoscaling.memory_min
|
||||
# memory_max = each.value.cluster_autoscaling.memory_max
|
||||
# }
|
||||
# }
|
||||
}
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
locals {
|
||||
fleet_enabled = (
|
||||
var.fleet_features != null || var.fleet_workload_identity
|
||||
)
|
||||
fleet_mcs_enabled = (
|
||||
try(var.fleet_features.multiclusterservicediscovery, false) == true
|
||||
)
|
||||
}
|
||||
|
||||
module "gke-hub" {
|
||||
source = "../../../modules/gke-hub"
|
||||
count = local.fleet_enabled ? 1 : 0
|
||||
project_id = module.gke-project-0.project_id
|
||||
clusters = {
|
||||
for cluster_id in keys(var.clusters) :
|
||||
cluster_id => module.gke-cluster[cluster_id].id
|
||||
}
|
||||
features = var.fleet_features
|
||||
configmanagement_templates = var.fleet_configmanagement_templates
|
||||
configmanagement_clusters = var.fleet_configmanagement_clusters
|
||||
workload_identity_clusters = (
|
||||
var.fleet_workload_identity ? keys(var.clusters) : []
|
||||
)
|
||||
|
||||
depends_on = [
|
||||
module.gke-nodepool
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,66 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
locals {
|
||||
nodepools = merge([
|
||||
for cluster, nodepools in var.nodepools : {
|
||||
for nodepool, config in nodepools :
|
||||
"${cluster}/${nodepool}" => merge(config, {
|
||||
name = nodepool
|
||||
cluster = cluster
|
||||
overrides = coalesce(config.overrides, var.nodepool_defaults)
|
||||
})
|
||||
}
|
||||
]...)
|
||||
}
|
||||
|
||||
module "gke-nodepool" {
|
||||
source = "../../../modules/gke-nodepool"
|
||||
for_each = local.nodepools
|
||||
name = each.value.name
|
||||
project_id = module.gke-project-0.project_id
|
||||
cluster_name = module.gke-cluster[each.value.cluster].name
|
||||
location = module.gke-cluster[each.value.cluster].location
|
||||
initial_node_count = each.value.initial_node_count
|
||||
node_machine_type = each.value.node_type
|
||||
node_spot = each.value.spot
|
||||
|
||||
node_count = each.value.node_count
|
||||
# node_count = (
|
||||
# each.value.autoscaling_config == null ? each.value.node_count : null
|
||||
# )
|
||||
# dynamic "autoscaling_config" {
|
||||
# for_each = each.value.autoscaling_config == null ? {} : { 1 = 1 }
|
||||
# content {
|
||||
# min_node_count = each.value.autoscaling_config.min_node_count
|
||||
# max_node_count = each.value.autoscaling_config.max_node_count
|
||||
# }
|
||||
# }
|
||||
|
||||
# overrides
|
||||
node_locations = each.value.overrides.node_locations
|
||||
max_pods_per_node = each.value.overrides.max_pods_per_node
|
||||
node_image_type = each.value.overrides.image_type
|
||||
node_tags = each.value.overrides.node_tags
|
||||
node_taints = each.value.overrides.node_taints
|
||||
|
||||
management_config = {
|
||||
auto_repair = true
|
||||
auto_upgrade = true
|
||||
}
|
||||
|
||||
node_service_account_create = true
|
||||
}
|
||||
|
|
@ -0,0 +1,86 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
module "gke-project-0" {
|
||||
source = "../../../modules/project"
|
||||
billing_account = var.billing_account_id
|
||||
name = var.project_id
|
||||
parent = var.folder_id
|
||||
prefix = var.prefix
|
||||
group_iam = var.group_iam
|
||||
labels = var.labels
|
||||
iam = merge(var.iam, {
|
||||
"roles/gkehub.serviceAgent" = [
|
||||
"serviceAccount:${module.gke-project-0.service_accounts.robots.fleet}"
|
||||
] }
|
||||
)
|
||||
services = concat(
|
||||
[
|
||||
"anthos.googleapis.com",
|
||||
"anthosconfigmanagement.googleapis.com",
|
||||
"cloudresourcemanager.googleapis.com",
|
||||
"container.googleapis.com",
|
||||
"dns.googleapis.com",
|
||||
"gkeconnect.googleapis.com",
|
||||
"gkehub.googleapis.com",
|
||||
"iam.googleapis.com",
|
||||
"multiclusteringress.googleapis.com",
|
||||
"multiclusterservicediscovery.googleapis.com",
|
||||
"stackdriver.googleapis.com",
|
||||
"trafficdirector.googleapis.com"
|
||||
],
|
||||
var.project_services
|
||||
)
|
||||
service_config = {
|
||||
disable_on_destroy = false
|
||||
disable_dependent_services = false
|
||||
}
|
||||
shared_vpc_service_config = {
|
||||
attach = true
|
||||
host_project = var.vpc_config.host_project_id
|
||||
service_identity_iam = merge({
|
||||
"roles/compute.networkUser" = [
|
||||
"cloudservices", "container-engine"
|
||||
]
|
||||
"roles/container.hostServiceAgentUser" = [
|
||||
"container-engine"
|
||||
]
|
||||
},
|
||||
!local.fleet_mcs_enabled ? {} : {
|
||||
"roles/multiclusterservicediscovery.serviceAgent" = ["gke-mcs"]
|
||||
"roles/compute.networkViewer" = ["gke-mcs-importer"]
|
||||
})
|
||||
}
|
||||
# specify project-level org policies here if you need them
|
||||
# policy_boolean = {
|
||||
# "constraints/compute.disableGuestAttributesAccess" = true
|
||||
# }
|
||||
# policy_list = {
|
||||
# "constraints/compute.trustedImageProjects" = {
|
||||
# inherit_from_parent = null
|
||||
# suggested_value = null
|
||||
# status = true
|
||||
# values = ["projects/fl01-prod-iac-core-0"]
|
||||
# }
|
||||
# }
|
||||
}
|
||||
|
||||
module "gke-dataset-resource-usage" {
|
||||
source = "../../../modules/bigquery-dataset"
|
||||
project_id = module.gke-project-0.project_id
|
||||
id = "gke_resource_usage"
|
||||
friendly_name = "GKE resource usage."
|
||||
}
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# tfdoc:file:description Output variables.
|
||||
|
||||
output "clusters" {
|
||||
description = "Cluster resources."
|
||||
value = module.gke-cluster
|
||||
}
|
||||
|
||||
output "cluster_ids" {
|
||||
description = "Cluster ids."
|
||||
value = {
|
||||
for k, v in module.gke-cluster : k => v.id
|
||||
}
|
||||
}
|
||||
|
||||
output "project_id" {
|
||||
description = "GKE project id."
|
||||
value = module.gke-project-0.project_id
|
||||
}
|
||||
|
|
@ -0,0 +1,254 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
variable "authenticator_security_group" {
|
||||
description = "Optional group used for Groups for GKE."
|
||||
type = string
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "billing_account_id" {
|
||||
description = "Billing account id."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "cluster_defaults" {
|
||||
description = "Default values for optional cluster configurations."
|
||||
type = object({
|
||||
cloudrun_config = bool
|
||||
database_encryption_key = string
|
||||
master_authorized_ranges = map(string)
|
||||
max_pods_per_node = number
|
||||
pod_security_policy = bool
|
||||
release_channel = string
|
||||
vertical_pod_autoscaling = bool
|
||||
gcp_filestore_csi_driver_config = bool
|
||||
})
|
||||
default = {
|
||||
# TODO: review defaults
|
||||
cloudrun_config = false
|
||||
database_encryption_key = null
|
||||
master_authorized_ranges = {
|
||||
rfc1918_1 = "10.0.0.0/8"
|
||||
rfc1918_2 = "172.16.0.0/12"
|
||||
rfc1918_3 = "192.168.0.0/16"
|
||||
}
|
||||
max_pods_per_node = 110
|
||||
pod_security_policy = false
|
||||
release_channel = "STABLE"
|
||||
vertical_pod_autoscaling = false
|
||||
gcp_filestore_csi_driver_config = false
|
||||
}
|
||||
}
|
||||
|
||||
variable "clusters" {
|
||||
description = ""
|
||||
type = map(object({
|
||||
cluster_autoscaling = object({
|
||||
cpu_min = number
|
||||
cpu_max = number
|
||||
memory_min = number
|
||||
memory_max = number
|
||||
})
|
||||
description = string
|
||||
dns_domain = string
|
||||
labels = map(string)
|
||||
location = string
|
||||
net = object({
|
||||
master_range = string
|
||||
pods = string
|
||||
services = string
|
||||
subnet = string
|
||||
})
|
||||
overrides = object({
|
||||
cloudrun_config = bool
|
||||
database_encryption_key = string
|
||||
# binary_authorization = bool
|
||||
master_authorized_ranges = map(string)
|
||||
max_pods_per_node = number
|
||||
pod_security_policy = bool
|
||||
release_channel = string
|
||||
vertical_pod_autoscaling = bool
|
||||
gcp_filestore_csi_driver_config = bool
|
||||
})
|
||||
}))
|
||||
}
|
||||
|
||||
variable "dns_domain" {
|
||||
description = "Domain name used for clusters, prefixed by each cluster name. Leave null to disable Cloud DNS for GKE."
|
||||
type = string
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "fleet_configmanagement_clusters" {
|
||||
description = "Config management features enabled on specific sets of member clusters, in config name => [cluster name] format."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "fleet_configmanagement_templates" {
|
||||
description = "Sets of config management configurations that can be applied to member clusters, in config name => {options} format."
|
||||
type = map(object({
|
||||
binauthz = bool
|
||||
config_sync = object({
|
||||
git = object({
|
||||
gcp_service_account_email = string
|
||||
https_proxy = string
|
||||
policy_dir = string
|
||||
secret_type = string
|
||||
sync_branch = string
|
||||
sync_repo = string
|
||||
sync_rev = string
|
||||
sync_wait_secs = number
|
||||
})
|
||||
prevent_drift = string
|
||||
source_format = string
|
||||
})
|
||||
hierarchy_controller = object({
|
||||
enable_hierarchical_resource_quota = bool
|
||||
enable_pod_tree_labels = bool
|
||||
})
|
||||
policy_controller = object({
|
||||
audit_interval_seconds = number
|
||||
exemptable_namespaces = list(string)
|
||||
log_denies_enabled = bool
|
||||
referential_rules_enabled = bool
|
||||
template_library_installed = bool
|
||||
})
|
||||
version = string
|
||||
}))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "fleet_features" {
|
||||
description = "Enable and configue fleet features. Set to null to disable GKE Hub if fleet workload identity is not used."
|
||||
type = object({
|
||||
appdevexperience = bool
|
||||
configmanagement = bool
|
||||
identityservice = bool
|
||||
multiclusteringress = string
|
||||
multiclusterservicediscovery = bool
|
||||
servicemesh = bool
|
||||
})
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "fleet_workload_identity" {
|
||||
description = "Use Fleet Workload Identity for clusters. Enables GKE Hub if set to true."
|
||||
type = bool
|
||||
default = false
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "folder_id" {
|
||||
description = "Folder used for the GKE project in folders/nnnnnnnnnnn format."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "group_iam" {
|
||||
description = "Project-level IAM bindings for groups. Use group emails as keys, list of roles as values."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "iam" {
|
||||
description = "Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "labels" {
|
||||
description = "Project-level labels."
|
||||
type = map(string)
|
||||
default = {}
|
||||
}
|
||||
|
||||
variable "nodepool_defaults" {
|
||||
description = ""
|
||||
type = object({
|
||||
image_type = string
|
||||
max_pods_per_node = number
|
||||
node_locations = list(string)
|
||||
node_tags = list(string)
|
||||
node_taints = list(string)
|
||||
})
|
||||
default = {
|
||||
image_type = "COS_CONTAINERD"
|
||||
max_pods_per_node = 110
|
||||
node_locations = null
|
||||
node_tags = null
|
||||
node_taints = []
|
||||
}
|
||||
}
|
||||
|
||||
variable "nodepools" {
|
||||
description = ""
|
||||
type = map(map(object({
|
||||
node_count = number
|
||||
node_type = string
|
||||
initial_node_count = number
|
||||
overrides = object({
|
||||
image_type = string
|
||||
max_pods_per_node = number
|
||||
node_locations = list(string)
|
||||
node_tags = list(string)
|
||||
node_taints = list(string)
|
||||
})
|
||||
spot = bool
|
||||
})))
|
||||
}
|
||||
|
||||
variable "peering_config" {
|
||||
description = "Configure peering with the control plane VPC. Requires compute.networks.updatePeering. Set to null if you don't want to update the default peering configuration."
|
||||
type = object({
|
||||
export_routes = bool
|
||||
import_routes = bool
|
||||
})
|
||||
default = {
|
||||
export_routes = true
|
||||
// TODO(jccb) is there any situation where the control plane VPC would export any routes?
|
||||
import_routes = false
|
||||
}
|
||||
}
|
||||
|
||||
variable "prefix" {
|
||||
description = "Prefix used for resources that need unique names."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "project_id" {
|
||||
description = "ID of the project that will contain all the clusters."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "project_services" {
|
||||
description = "Additional project services to enable."
|
||||
type = list(string)
|
||||
default = []
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "vpc_config" {
|
||||
description = "Shared VPC project and VPC details."
|
||||
type = object({
|
||||
host_project_id = string
|
||||
vpc_self_link = string
|
||||
})
|
||||
}
|
||||
|
|
@ -461,20 +461,20 @@ The remaining configuration is manual, as it regards the repositories themselves
|
|||
| name | description | type | required | default | producer |
|
||||
|---|---|:---:|:---:|:---:|:---:|
|
||||
| [billing_account](variables.tf#L17) | Billing account id and organization id ('nnnnnnnn' or null). | <code title="object({ id = string organization_id = number })">object({…})</code> | ✓ | | |
|
||||
| [organization](variables.tf#L196) | Organization details. | <code title="object({ domain = string id = number customer_id = string })">object({…})</code> | ✓ | | |
|
||||
| [prefix](variables.tf#L211) | Prefix used for resources that need unique names. Use 9 characters or less. | <code>string</code> | ✓ | | |
|
||||
| [organization](variables.tf#L198) | Organization details. | <code title="object({ domain = string id = number customer_id = string })">object({…})</code> | ✓ | | |
|
||||
| [prefix](variables.tf#L213) | Prefix used for resources that need unique names. Use 9 characters or less. | <code>string</code> | ✓ | | |
|
||||
| [bootstrap_user](variables.tf#L25) | Email of the nominal user running this stage for the first time. | <code>string</code> | | <code>null</code> | |
|
||||
| [cicd_repositories](variables.tf#L31) | CI/CD repository configuration. Identity providers reference keys in the `federated_identity_providers` variable. Set to null to disable, or set individual repositories to null if not needed. | <code title="object({ bootstrap = object({ branch = string identity_provider = string name = string type = string }) cicd = object({ branch = string identity_provider = string name = string type = string }) resman = object({ branch = string identity_provider = string name = string type = string }) })">object({…})</code> | | <code>null</code> | |
|
||||
| [custom_role_names](variables.tf#L83) | Names of custom roles defined at the org level. | <code title="object({ organization_iam_admin = string service_project_network_admin = string })">object({…})</code> | | <code title="{ organization_iam_admin = "organizationIamAdmin" service_project_network_admin = "serviceProjectNetworkAdmin" }">{…}</code> | |
|
||||
| [fast_features](variables.tf#L95) | Selective control for top-level FAST features. | <code title="object({ data_platform = bool project_factory = bool sandbox = bool teams = bool })">object({…})</code> | | <code title="{ data_platform = true project_factory = true sandbox = true teams = true }">{…}</code> | |
|
||||
| [federated_identity_providers](variables.tf#L112) | Workload Identity Federation pools. The `cicd_repositories` variable references keys here. | <code title="map(object({ attribute_condition = string issuer = string custom_settings = object({ issuer_uri = string allowed_audiences = list(string) }) }))">map(object({…}))</code> | | <code>{}</code> | |
|
||||
| [groups](variables.tf#L126) | Group names to grant organization-level permissions. | <code>map(string)</code> | | <code title="{ gcp-billing-admins = "gcp-billing-admins", gcp-devops = "gcp-devops", gcp-network-admins = "gcp-network-admins" gcp-organization-admins = "gcp-organization-admins" gcp-security-admins = "gcp-security-admins" gcp-support = "gcp-support" }">{…}</code> | |
|
||||
| [iam](variables.tf#L140) | Organization-level custom IAM settings in role => [principal] format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [iam_additive](variables.tf#L146) | Organization-level custom IAM settings in role => [principal] format for non-authoritative bindings. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [locations](variables.tf#L152) | Optional locations for GCS, BigQuery, and logging buckets created here. | <code title="object({ bq = string gcs = string logging = string pubsub = list(string) })">object({…})</code> | | <code title="{ bq = "EU" gcs = "EU" logging = "global" pubsub = [] }">{…}</code> | |
|
||||
| [log_sinks](variables.tf#L171) | Org-level log sinks, in name => {type, filter} format. | <code title="map(object({ filter = string type = string }))">map(object({…}))</code> | | <code title="{ audit-logs = { filter = "logName:\"/logs/cloudaudit.googleapis.com%2Factivity\" OR logName:\"/logs/cloudaudit.googleapis.com%2Fsystem_event\"" type = "bigquery" } vpc-sc = { filter = "protoPayload.metadata.@type=\"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata\"" type = "bigquery" } }">{…}</code> | |
|
||||
| [outputs_location](variables.tf#L205) | Enable writing provider, tfvars and CI/CD workflow files to local filesystem. Leave null to disable | <code>string</code> | | <code>null</code> | |
|
||||
| [project_parent_ids](variables.tf#L221) | Optional parents for projects created here in folders/nnnnnnn format. Null values will use the organization as parent. | <code title="object({ automation = string billing = string logging = string })">object({…})</code> | | <code title="{ automation = null billing = null logging = null }">{…}</code> | |
|
||||
| [fast_features](variables.tf#L95) | Selective control for top-level FAST features. | <code title="object({ data_platform = bool gke = bool project_factory = bool sandbox = bool teams = bool })">object({…})</code> | | <code title="{ data_platform = true gke = true project_factory = true sandbox = true teams = true }">{…}</code> | |
|
||||
| [federated_identity_providers](variables.tf#L114) | Workload Identity Federation pools. The `cicd_repositories` variable references keys here. | <code title="map(object({ attribute_condition = string issuer = string custom_settings = object({ issuer_uri = string allowed_audiences = list(string) }) }))">map(object({…}))</code> | | <code>{}</code> | |
|
||||
| [groups](variables.tf#L128) | Group names to grant organization-level permissions. | <code>map(string)</code> | | <code title="{ gcp-billing-admins = "gcp-billing-admins", gcp-devops = "gcp-devops", gcp-network-admins = "gcp-network-admins" gcp-organization-admins = "gcp-organization-admins" gcp-security-admins = "gcp-security-admins" gcp-support = "gcp-support" }">{…}</code> | |
|
||||
| [iam](variables.tf#L142) | Organization-level custom IAM settings in role => [principal] format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [iam_additive](variables.tf#L148) | Organization-level custom IAM settings in role => [principal] format for non-authoritative bindings. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [locations](variables.tf#L154) | Optional locations for GCS, BigQuery, and logging buckets created here. | <code title="object({ bq = string gcs = string logging = string pubsub = list(string) })">object({…})</code> | | <code title="{ bq = "EU" gcs = "EU" logging = "global" pubsub = [] }">{…}</code> | |
|
||||
| [log_sinks](variables.tf#L173) | Org-level log sinks, in name => {type, filter} format. | <code title="map(object({ filter = string type = string }))">map(object({…}))</code> | | <code title="{ audit-logs = { filter = "logName:\"/logs/cloudaudit.googleapis.com%2Factivity\" OR logName:\"/logs/cloudaudit.googleapis.com%2Fsystem_event\"" type = "bigquery" } vpc-sc = { filter = "protoPayload.metadata.@type=\"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata\"" type = "bigquery" } }">{…}</code> | |
|
||||
| [outputs_location](variables.tf#L207) | Enable writing provider, tfvars and CI/CD workflow files to local filesystem. Leave null to disable | <code>string</code> | | <code>null</code> | |
|
||||
| [project_parent_ids](variables.tf#L223) | Optional parents for projects created here in folders/nnnnnnn format. Null values will use the organization as parent. | <code title="object({ automation = string billing = string logging = string })">object({…})</code> | | <code title="{ automation = null billing = null logging = null }">{…}</code> | |
|
||||
|
||||
## Outputs
|
||||
|
||||
|
|
|
|||
|
|
@ -68,6 +68,7 @@ module "automation-project" {
|
|||
"cloudresourcemanager.googleapis.com",
|
||||
"container.googleapis.com",
|
||||
"compute.googleapis.com",
|
||||
"container.googleapis.com",
|
||||
"essentialcontacts.googleapis.com",
|
||||
"iam.googleapis.com",
|
||||
"iamcredentials.googleapis.com",
|
||||
|
|
|
|||
|
|
@ -173,6 +173,13 @@ module "organization" {
|
|||
]
|
||||
(var.custom_role_names.service_project_network_admin) = [
|
||||
"compute.globalOperations.get",
|
||||
# compute.networks.updatePeering and compute.networks.get are
|
||||
# used by automation service accounts who manage service
|
||||
# projects where peering creation might be needed (e.g. GKE). If
|
||||
# you remove them your network administrators should create
|
||||
# peerings for service projects
|
||||
"compute.networks.updatePeering",
|
||||
"compute.networks.get",
|
||||
"compute.organizations.disableXpnResource",
|
||||
"compute.organizations.enableXpnResource",
|
||||
"compute.projects.get",
|
||||
|
|
|
|||
|
|
@ -96,12 +96,14 @@ variable "fast_features" {
|
|||
description = "Selective control for top-level FAST features."
|
||||
type = object({
|
||||
data_platform = bool
|
||||
gke = bool
|
||||
project_factory = bool
|
||||
sandbox = bool
|
||||
teams = bool
|
||||
})
|
||||
default = {
|
||||
data_platform = true
|
||||
gke = true
|
||||
project_factory = true
|
||||
sandbox = true
|
||||
teams = true
|
||||
|
|
|
|||
|
|
@ -158,6 +158,7 @@ Due to its simplicity, this stage lends itself easily to customizations: adding
|
|||
|---|---|---|---|
|
||||
| [billing.tf](./billing.tf) | Billing resources for external billing use cases. | <code>organization</code> | <code>google_billing_account_iam_member</code> |
|
||||
| [branch-data-platform.tf](./branch-data-platform.tf) | Data Platform stages resources. | <code>folder</code> · <code>gcs</code> · <code>iam-service-account</code> | |
|
||||
| [branch-gke.tf](./branch-gke.tf) | GKE multitenant stage resources. | <code>folder</code> · <code>gcs</code> · <code>iam-service-account</code> | |
|
||||
| [branch-networking.tf](./branch-networking.tf) | Networking stage resources. | <code>folder</code> · <code>gcs</code> · <code>iam-service-account</code> | |
|
||||
| [branch-project-factory.tf](./branch-project-factory.tf) | Project factory stage resources. | <code>gcs</code> · <code>iam-service-account</code> | |
|
||||
| [branch-sandbox.tf](./branch-sandbox.tf) | Sandbox stage resources. | <code>folder</code> · <code>gcs</code> · <code>iam-service-account</code> | |
|
||||
|
|
@ -180,30 +181,31 @@ Due to its simplicity, this stage lends itself easily to customizations: adding
|
|||
|---|---|:---:|:---:|:---:|:---:|
|
||||
| [automation](variables.tf#L20) | Automation resources created by the bootstrap stage. | <code title="object({ outputs_bucket = string project_id = string project_number = string federated_identity_pool = string federated_identity_providers = map(object({ issuer = string issuer_uri = string name = string principal_tpl = string principalset_tpl = string })) })">object({…})</code> | ✓ | | <code>00-bootstrap</code> |
|
||||
| [billing_account](variables.tf#L38) | Billing account id and organization id ('nnnnnnnn' or null). | <code title="object({ id = string organization_id = number })">object({…})</code> | ✓ | | <code>00-bootstrap</code> |
|
||||
| [organization](variables.tf#L177) | Organization details. | <code title="object({ domain = string id = number customer_id = string })">object({…})</code> | ✓ | | <code>00-bootstrap</code> |
|
||||
| [prefix](variables.tf#L201) | Prefix used for resources that need unique names. Use 9 characters or less. | <code>string</code> | ✓ | | <code>00-bootstrap</code> |
|
||||
| [organization](variables.tf#L179) | Organization details. | <code title="object({ domain = string id = number customer_id = string })">object({…})</code> | ✓ | | <code>00-bootstrap</code> |
|
||||
| [prefix](variables.tf#L203) | Prefix used for resources that need unique names. Use 9 characters or less. | <code>string</code> | ✓ | | <code>00-bootstrap</code> |
|
||||
| [cicd_repositories](variables.tf#L47) | CI/CD repository configuration. Identity providers reference keys in the `automation.federated_identity_providers` variable. Set to null to disable, or set individual repositories to null if not needed. | <code title="object({ data_platform_dev = object({ branch = string identity_provider = string name = string type = string }) data_platform_prod = object({ branch = string identity_provider = string name = string type = string }) networking = object({ branch = string identity_provider = string name = string type = string }) project_factory_dev = object({ branch = string identity_provider = string name = string type = string }) project_factory_prod = object({ branch = string identity_provider = string name = string type = string }) security = object({ branch = string identity_provider = string name = string type = string }) })">object({…})</code> | | <code>null</code> | |
|
||||
| [custom_roles](variables.tf#L117) | Custom roles defined at the org level, in key => id format. | <code title="object({ service_project_network_admin = string })">object({…})</code> | | <code>null</code> | <code>00-bootstrap</code> |
|
||||
| [fast_features](variables.tf#L126) | Selective control for top-level FAST features. | <code title="object({ data_platform = bool project_factory = bool sandbox = bool teams = bool })">object({…})</code> | | <code title="{ data_platform = true project_factory = true sandbox = true teams = true }">{…}</code> | <code>00-bootstrap</code> |
|
||||
| [groups](variables.tf#L144) | Group names to grant organization-level permissions. | <code>map(string)</code> | | <code title="{ gcp-billing-admins = "gcp-billing-admins", gcp-devops = "gcp-devops", gcp-network-admins = "gcp-network-admins" gcp-organization-admins = "gcp-organization-admins" gcp-security-admins = "gcp-security-admins" gcp-support = "gcp-support" }">{…}</code> | <code>00-bootstrap</code> |
|
||||
| [locations](variables.tf#L159) | Optional locations for GCS, BigQuery, and logging buckets created here. | <code title="object({ bq = string gcs = string logging = string pubsub = list(string) })">object({…})</code> | | <code title="{ bq = "EU" gcs = "EU" logging = "global" pubsub = [] }">{…}</code> | <code>00-bootstrap</code> |
|
||||
| [organization_policy_configs](variables.tf#L187) | Organization policies customization. | <code title="object({ allowed_policy_member_domains = list(string) })">object({…})</code> | | <code>null</code> | |
|
||||
| [outputs_location](variables.tf#L195) | Enable writing provider, tfvars and CI/CD workflow files to local filesystem. Leave null to disable | <code>string</code> | | <code>null</code> | |
|
||||
| [tag_names](variables.tf#L212) | Customized names for resource management tags. | <code title="object({ context = string environment = string })">object({…})</code> | | <code title="{ context = "context" environment = "environment" }">{…}</code> | |
|
||||
| [team_folders](variables.tf#L229) | Team folders to be created. Format is described in a code comment. | <code title="map(object({ descriptive_name = string group_iam = map(list(string)) impersonation_groups = list(string) }))">map(object({…}))</code> | | <code>null</code> | |
|
||||
| [fast_features](variables.tf#L126) | Selective control for top-level FAST features. | <code title="object({ data_platform = bool gke = bool project_factory = bool sandbox = bool teams = bool })">object({…})</code> | | <code title="{ data_platform = true gke = true project_factory = true sandbox = true teams = true }">{…}</code> | <code>00-bootstrap</code> |
|
||||
| [groups](variables.tf#L146) | Group names to grant organization-level permissions. | <code>map(string)</code> | | <code title="{ gcp-billing-admins = "gcp-billing-admins", gcp-devops = "gcp-devops", gcp-network-admins = "gcp-network-admins" gcp-organization-admins = "gcp-organization-admins" gcp-security-admins = "gcp-security-admins" gcp-support = "gcp-support" }">{…}</code> | <code>00-bootstrap</code> |
|
||||
| [locations](variables.tf#L161) | Optional locations for GCS, BigQuery, and logging buckets created here. | <code title="object({ bq = string gcs = string logging = string pubsub = list(string) })">object({…})</code> | | <code title="{ bq = "EU" gcs = "EU" logging = "global" pubsub = [] }">{…}</code> | <code>00-bootstrap</code> |
|
||||
| [organization_policy_configs](variables.tf#L189) | Organization policies customization. | <code title="object({ allowed_policy_member_domains = list(string) })">object({…})</code> | | <code>null</code> | |
|
||||
| [outputs_location](variables.tf#L197) | Enable writing provider, tfvars and CI/CD workflow files to local filesystem. Leave null to disable | <code>string</code> | | <code>null</code> | |
|
||||
| [tag_names](variables.tf#L214) | Customized names for resource management tags. | <code title="object({ context = string environment = string })">object({…})</code> | | <code title="{ context = "context" environment = "environment" }">{…}</code> | |
|
||||
| [team_folders](variables.tf#L231) | Team folders to be created. Format is described in a code comment. | <code title="map(object({ descriptive_name = string group_iam = map(list(string)) impersonation_groups = list(string) }))">map(object({…}))</code> | | <code>null</code> | |
|
||||
|
||||
## Outputs
|
||||
|
||||
| name | description | sensitive | consumers |
|
||||
|---|---|:---:|---|
|
||||
| [cicd_repositories](outputs.tf#L171) | WIF configuration for CI/CD repositories. | | |
|
||||
| [dataplatform](outputs.tf#L185) | Data for the Data Platform stage. | | |
|
||||
| [networking](outputs.tf#L201) | Data for the networking stage. | | |
|
||||
| [project_factories](outputs.tf#L210) | Data for the project factories stage. | | |
|
||||
| [providers](outputs.tf#L226) | Terraform provider files for this stage and dependent stages. | ✓ | <code>02-networking</code> · <code>02-security</code> · <code>03-dataplatform</code> · <code>xx-sandbox</code> · <code>xx-teams</code> |
|
||||
| [sandbox](outputs.tf#L233) | Data for the sandbox stage. | | <code>xx-sandbox</code> |
|
||||
| [security](outputs.tf#L247) | Data for the networking stage. | | <code>02-security</code> |
|
||||
| [teams](outputs.tf#L257) | Data for the teams stage. | | |
|
||||
| [tfvars](outputs.tf#L270) | Terraform variable files for the following stages. | ✓ | |
|
||||
| [cicd_repositories](outputs.tf#L188) | WIF configuration for CI/CD repositories. | | |
|
||||
| [dataplatform](outputs.tf#L202) | Data for the Data Platform stage. | | |
|
||||
| [gke_multitenant](outputs.tf#L274) | Data for the GKE multitenant stage. | | <code>03-gke-multitenant</code> |
|
||||
| [networking](outputs.tf#L218) | Data for the networking stage. | | |
|
||||
| [project_factories](outputs.tf#L227) | Data for the project factories stage. | | |
|
||||
| [providers](outputs.tf#L243) | Terraform provider files for this stage and dependent stages. | ✓ | <code>02-networking</code> · <code>02-security</code> · <code>03-dataplatform</code> · <code>xx-sandbox</code> · <code>xx-teams</code> |
|
||||
| [sandbox](outputs.tf#L250) | Data for the sandbox stage. | | <code>xx-sandbox</code> |
|
||||
| [security](outputs.tf#L264) | Data for the networking stage. | | <code>02-security</code> |
|
||||
| [teams](outputs.tf#L295) | Data for the teams stage. | | |
|
||||
| [tfvars](outputs.tf#L308) | Terraform variable files for the following stages. | ✓ | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
|
|
|||
|
|
@ -25,6 +25,8 @@ locals {
|
|||
],
|
||||
local.branch_optional_sa_lists.dp-dev,
|
||||
local.branch_optional_sa_lists.dp-prod,
|
||||
local.branch_optional_sa_lists.gke-dev,
|
||||
local.branch_optional_sa_lists.gke-prod,
|
||||
local.branch_optional_sa_lists.pf-dev,
|
||||
local.branch_optional_sa_lists.pf-prod,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -56,7 +56,8 @@ module "branch-dp-dev-folder" {
|
|||
}
|
||||
tag_bindings = {
|
||||
context = try(
|
||||
module.organization.tag_values["${var.tag_names.environment}/development"].id, null
|
||||
module.organization.tag_values["${var.tag_names.environment}/development"].id,
|
||||
null
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -82,7 +83,8 @@ module "branch-dp-prod-folder" {
|
|||
}
|
||||
tag_bindings = {
|
||||
context = try(
|
||||
module.organization.tag_values["${var.tag_names.environment}/production"].id, null
|
||||
module.organization.tag_values["${var.tag_names.environment}/production"].id,
|
||||
null
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,158 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description GKE multitenant stage resources.
|
||||
|
||||
moved {
|
||||
from = module.branch-gke-folder
|
||||
to = module.branch-gke-folder.0
|
||||
}
|
||||
|
||||
module "branch-gke-folder" {
|
||||
source = "../../../modules/folder"
|
||||
count = var.fast_features.gke ? 1 : 0
|
||||
parent = "organizations/${var.organization.id}"
|
||||
name = "GKE"
|
||||
tag_bindings = {
|
||||
context = try(
|
||||
module.organization.tag_values["${var.tag_names.context}/gke"].id, null
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
moved {
|
||||
from = module.branch-gke-dev-folder
|
||||
to = module.branch-gke-dev-folder.0
|
||||
}
|
||||
|
||||
module "branch-gke-dev-folder" {
|
||||
source = "../../../modules/folder"
|
||||
count = var.fast_features.gke ? 1 : 0
|
||||
parent = module.branch-gke-folder.0.id
|
||||
name = "Development"
|
||||
iam = {
|
||||
"roles/owner" = [module.branch-gke-dev-sa.0.iam_email]
|
||||
"roles/logging.admin" = [module.branch-gke-dev-sa.0.iam_email]
|
||||
"roles/resourcemanager.folderAdmin" = [module.branch-gke-dev-sa.0.iam_email]
|
||||
"roles/resourcemanager.projectCreator" = [module.branch-gke-dev-sa.0.iam_email]
|
||||
"roles/compute.xpnAdmin" = [module.branch-gke-dev-sa.0.iam_email]
|
||||
}
|
||||
tag_bindings = {
|
||||
context = try(
|
||||
module.organization.tag_values["${var.tag_names.environment}/development"].id,
|
||||
null
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
moved {
|
||||
from = module.branch-gke-prod-folder
|
||||
to = module.branch-gke-prod-folder.0
|
||||
}
|
||||
|
||||
module "branch-gke-prod-folder" {
|
||||
source = "../../../modules/folder"
|
||||
count = var.fast_features.gke ? 1 : 0
|
||||
parent = module.branch-gke-folder.0.id
|
||||
name = "Production"
|
||||
iam = {
|
||||
"roles/owner" = [module.branch-gke-prod-sa.0.iam_email]
|
||||
"roles/logging.admin" = [module.branch-gke-prod-sa.0.iam_email]
|
||||
"roles/resourcemanager.folderAdmin" = [module.branch-gke-prod-sa.0.iam_email]
|
||||
"roles/resourcemanager.projectCreator" = [module.branch-gke-prod-sa.0.iam_email]
|
||||
"roles/compute.xpnAdmin" = [module.branch-gke-prod-sa.0.iam_email]
|
||||
}
|
||||
tag_bindings = {
|
||||
context = try(
|
||||
module.organization.tag_values["${var.tag_names.environment}/production"].id,
|
||||
null
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
moved {
|
||||
from = module.branch-gke-dev-sa
|
||||
to = module.branch-gke-dev-sa.0
|
||||
}
|
||||
|
||||
module "branch-gke-dev-sa" {
|
||||
source = "../../../modules/iam-service-account"
|
||||
count = var.fast_features.gke ? 1 : 0
|
||||
project_id = var.automation.project_id
|
||||
name = "dev-resman-gke-0"
|
||||
description = "Terraform gke multitenant dev service account."
|
||||
prefix = var.prefix
|
||||
iam = {
|
||||
"roles/iam.serviceAccountTokenCreator" = ["group:${local.groups.gcp-devops}"]
|
||||
}
|
||||
iam_storage_roles = {
|
||||
(var.automation.outputs_bucket) = ["roles/storage.admin"]
|
||||
}
|
||||
}
|
||||
|
||||
moved {
|
||||
from = module.branch-gke-prod-sa
|
||||
to = module.branch-gke-prod-sa.0
|
||||
}
|
||||
|
||||
module "branch-gke-prod-sa" {
|
||||
source = "../../../modules/iam-service-account"
|
||||
count = var.fast_features.gke ? 1 : 0
|
||||
project_id = var.automation.project_id
|
||||
name = "prod-resman-gke-0"
|
||||
description = "Terraform gke multitenant prod service account."
|
||||
prefix = var.prefix
|
||||
iam = {
|
||||
"roles/iam.serviceAccountTokenCreator" = ["group:${local.groups.gcp-devops}"]
|
||||
}
|
||||
iam_storage_roles = {
|
||||
(var.automation.outputs_bucket) = ["roles/storage.admin"]
|
||||
}
|
||||
}
|
||||
|
||||
moved {
|
||||
from = module.branch-gke-dev-gcs
|
||||
to = module.branch-gke-dev-gcs.0
|
||||
}
|
||||
|
||||
module "branch-gke-dev-gcs" {
|
||||
source = "../../../modules/gcs"
|
||||
count = var.fast_features.gke ? 1 : 0
|
||||
project_id = var.automation.project_id
|
||||
name = "dev-resman-gke-0"
|
||||
prefix = var.prefix
|
||||
versioning = true
|
||||
iam = {
|
||||
"roles/storage.objectAdmin" = [module.branch-gke-dev-sa.0.iam_email]
|
||||
}
|
||||
}
|
||||
|
||||
moved {
|
||||
from = module.branch-gke-prod-gcs
|
||||
to = module.branch-gke-prod-gcs.0
|
||||
}
|
||||
|
||||
module "branch-gke-prod-gcs" {
|
||||
source = "../../../modules/gcs"
|
||||
count = var.fast_features.gke ? 1 : 0
|
||||
project_id = var.automation.project_id
|
||||
name = "prod-resman-gke-0"
|
||||
prefix = var.prefix
|
||||
versioning = true
|
||||
iam = {
|
||||
"roles/storage.objectAdmin" = [module.branch-gke-prod-sa.0.iam_email]
|
||||
}
|
||||
}
|
||||
|
|
@ -52,12 +52,14 @@ module "branch-network-prod-folder" {
|
|||
iam = {
|
||||
(local.custom_roles.service_project_network_admin) = concat(
|
||||
local.branch_optional_sa_lists.dp-prod,
|
||||
local.branch_optional_sa_lists.gke-prod,
|
||||
local.branch_optional_sa_lists.pf-prod,
|
||||
)
|
||||
}
|
||||
tag_bindings = {
|
||||
environment = try(
|
||||
module.organization.tag_values["${var.tag_names.environment}/production"].id, null
|
||||
module.organization.tag_values["${var.tag_names.environment}/production"].id,
|
||||
null
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -69,12 +71,14 @@ module "branch-network-dev-folder" {
|
|||
iam = {
|
||||
(local.custom_roles.service_project_network_admin) = concat(
|
||||
local.branch_optional_sa_lists.dp-dev,
|
||||
local.branch_optional_sa_lists.gke-dev,
|
||||
local.branch_optional_sa_lists.pf-dev,
|
||||
)
|
||||
}
|
||||
tag_bindings = {
|
||||
environment = try(
|
||||
module.organization.tag_values["${var.tag_names.environment}/development"].id, null
|
||||
module.organization.tag_values["${var.tag_names.environment}/development"].id,
|
||||
null
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -27,10 +27,12 @@ locals {
|
|||
billing_org = var.billing_account.organization_id == var.organization.id
|
||||
billing_org_ext = !local.billing_ext && !local.billing_org
|
||||
branch_optional_sa_lists = {
|
||||
dp-dev = compact([try(module.branch-dp-dev-sa.0.iam_email, "")])
|
||||
dp-prod = compact([try(module.branch-dp-prod-sa.0.iam_email, "")])
|
||||
pf-dev = compact([try(module.branch-pf-dev-sa.0.iam_email, "")])
|
||||
pf-prod = compact([try(module.branch-pf-prod-sa.0.iam_email, "")])
|
||||
dp-dev = compact([try(module.branch-dp-dev-sa.0.iam_email, "")])
|
||||
dp-prod = compact([try(module.branch-dp-prod-sa.0.iam_email, "")])
|
||||
gke-dev = compact([try(module.branch-gke-dev-sa.0.iam_email, "")])
|
||||
gke-prod = compact([try(module.branch-gke-prod-sa.0.iam_email, "")])
|
||||
pf-dev = compact([try(module.branch-pf-dev-sa.0.iam_email, "")])
|
||||
pf-prod = compact([try(module.branch-pf-prod-sa.0.iam_email, "")])
|
||||
}
|
||||
cicd_repositories = {
|
||||
for k, v in coalesce(var.cicd_repositories, {}) : k => v
|
||||
|
|
|
|||
|
|
@ -66,6 +66,8 @@ module "organization" {
|
|||
],
|
||||
local.branch_optional_sa_lists.dp-dev,
|
||||
local.branch_optional_sa_lists.dp-prod,
|
||||
local.branch_optional_sa_lists.gke-dev,
|
||||
local.branch_optional_sa_lists.gke-prod,
|
||||
local.branch_optional_sa_lists.pf-dev,
|
||||
local.branch_optional_sa_lists.pf-prod,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ locals {
|
|||
tf_providers_file = "03-data-platform-prod-providers.tf"
|
||||
tf_var_files = local.cicd_workflow_var_files.stage_3
|
||||
}
|
||||
# TODO(jccb): add gke here
|
||||
networking = {
|
||||
service_account = try(module.branch-network-sa-cicd.0.email, null)
|
||||
tf_providers_file = "02-networking-providers.tf"
|
||||
|
|
@ -64,6 +65,8 @@ locals {
|
|||
{
|
||||
data-platform-dev = try(module.branch-dp-dev-folder.0.id, null)
|
||||
data-platform-prod = try(module.branch-dp-prod-folder.0.id, null)
|
||||
gke-dev = try(module.branch-gke-dev-folder.0.id, null)
|
||||
gke-prod = try(module.branch-gke-prod-folder.0.id, null)
|
||||
networking = module.branch-network-folder.id
|
||||
networking-dev = module.branch-network-dev-folder.id
|
||||
networking-prod = module.branch-network-prod-folder.id
|
||||
|
|
@ -109,6 +112,18 @@ locals {
|
|||
sa = module.branch-dp-prod-sa.0.email
|
||||
})
|
||||
},
|
||||
!var.fast_features.gke ? {} : {
|
||||
"03-gke-dev" = templatefile(local._tpl_providers, {
|
||||
bucket = module.branch-gke-dev-gcs.0.name
|
||||
name = "gke-dev"
|
||||
sa = module.branch-gke-dev-sa.0.email
|
||||
})
|
||||
"03-gke-prod" = templatefile(local._tpl_providers, {
|
||||
bucket = module.branch-gke-prod-gcs.0.name
|
||||
name = "gke-prod"
|
||||
sa = module.branch-gke-prod-sa.0.email
|
||||
})
|
||||
},
|
||||
!var.fast_features.project_factory ? {} : {
|
||||
"03-project-factory-dev" = templatefile(local._tpl_providers, {
|
||||
bucket = module.branch-pf-dev-gcs.0.name
|
||||
|
|
@ -150,6 +165,8 @@ locals {
|
|||
{
|
||||
data-platform-dev = try(module.branch-dp-dev-sa.0.email, null)
|
||||
data-platform-prod = try(module.branch-dp-prod-sa.0.email, null)
|
||||
gke-dev = try(module.branch-gke-dev-sa.0.email, null)
|
||||
gke-prod = try(module.branch-gke-prod-sa.0.email, null)
|
||||
networking = module.branch-network-sa.email
|
||||
project-factory-dev = try(module.branch-pf-dev-sa.0.email, null)
|
||||
project-factory-prod = try(module.branch-pf-prod-sa.0.email, null)
|
||||
|
|
@ -254,6 +271,27 @@ output "security" {
|
|||
}
|
||||
}
|
||||
|
||||
output "gke_multitenant" {
|
||||
# tfdoc:output:consumers 03-gke-multitenant
|
||||
description = "Data for the GKE multitenant stage."
|
||||
value = (
|
||||
var.fast_features.gke
|
||||
? {
|
||||
"dev" = {
|
||||
folder = module.branch-gke-dev-folder.0.id
|
||||
gcs_bucket = module.branch-gke-dev-gcs.0.name
|
||||
service_account = module.branch-gke-dev-sa.0.email
|
||||
}
|
||||
"prod" = {
|
||||
folder = module.branch-gke-prod-folder.0.id
|
||||
gcs_bucket = module.branch-gke-prod-gcs.0.name
|
||||
service_account = module.branch-gke-prod-sa.0.email
|
||||
}
|
||||
}
|
||||
: {}
|
||||
)
|
||||
}
|
||||
|
||||
output "teams" {
|
||||
description = "Data for the teams stage."
|
||||
value = {
|
||||
|
|
|
|||
|
|
@ -128,12 +128,14 @@ variable "fast_features" {
|
|||
description = "Selective control for top-level FAST features."
|
||||
type = object({
|
||||
data_platform = bool
|
||||
gke = bool
|
||||
project_factory = bool
|
||||
sandbox = bool
|
||||
teams = bool
|
||||
})
|
||||
default = {
|
||||
data_platform = true
|
||||
gke = true
|
||||
project_factory = true
|
||||
sandbox = true
|
||||
teams = true
|
||||
|
|
|
|||
|
|
@ -379,8 +379,8 @@ DNS configurations are centralised in the `dns-*.tf` files. Spokes delegate DNS
|
|||
| [psa_ranges](variables.tf#L142) | IP ranges used for Private Service Access (e.g. CloudSQL). | <code title="object({ dev = object({ ranges = map(string) routes = object({ export = bool import = bool }) }) prod = object({ ranges = map(string) routes = object({ export = bool import = bool }) }) })">object({…})</code> | | <code>null</code> | |
|
||||
| [region_trigram](variables.tf#L183) | Short names for GCP regions. | <code>map(string)</code> | | <code title="{ europe-west1 = "ew1" europe-west3 = "ew3" }">{…}</code> | |
|
||||
| [router_configs](variables.tf#L192) | Configurations for CRs and onprem routers. | <code title="map(object({ adv = object({ custom = list(string) default = bool }) asn = number }))">map(object({…}))</code> | | <code title="{ landing-trusted-ew1 = { asn = "64512" adv = null } landing-trusted-ew4 = { asn = "64512" adv = null } }">{…}</code> | |
|
||||
| [service_accounts](variables.tf#L215) | Automation service accounts in name => email format. | <code title="object({ data-platform-dev = string data-platform-prod = string project-factory-dev = string project-factory-prod = string })">object({…})</code> | | <code>null</code> | <code>01-resman</code> |
|
||||
| [vpn_onprem_configs](variables.tf#L227) | VPN gateway configuration for onprem interconnection. | <code title="map(object({ adv = object({ default = bool custom = list(string) }) peer_external_gateway = object({ redundancy_type = string interfaces = list(object({ id = number ip_address = string })) }) tunnels = list(object({ peer_asn = number peer_external_gateway_interface = number secret = string session_range = string vpn_gateway_interface = number })) }))">map(object({…}))</code> | | <code title="{ landing-trusted-ew1 = { adv = { default = false custom = [ "cloud_dns", "googleapis_private", "googleapis_restricted", "gcp_all" ] } peer_external_gateway = { redundancy_type = "SINGLE_IP_INTERNALLY_REDUNDANT" interfaces = [ { id = 0, ip_address = "8.8.8.8" }, ] } tunnels = [ { peer_asn = 65534 peer_external_gateway_interface = 0 secret = "foobar" session_range = "169.254.1.0/30" vpn_gateway_interface = 0 }, { peer_asn = 65534 peer_external_gateway_interface = 0 secret = "foobar" session_range = "169.254.1.4/30" vpn_gateway_interface = 1 } ] } landing-trusted-ew4 = { adv = { default = false custom = [ "cloud_dns", "googleapis_private", "googleapis_restricted", "gcp_all" ] } peer_external_gateway = { redundancy_type = "SINGLE_IP_INTERNALLY_REDUNDANT" interfaces = [ { id = 0, ip_address = "8.8.8.8" }, ] } tunnels = [ { peer_asn = 65534 peer_external_gateway_interface = 0 secret = "foobar" session_range = "169.254.1.0/30" vpn_gateway_interface = 0 }, { peer_asn = 65534 peer_external_gateway_interface = 0 secret = "foobar" session_range = "169.254.1.4/30" vpn_gateway_interface = 1 } ] } }">{…}</code> | |
|
||||
| [service_accounts](variables.tf#L215) | Automation service accounts in name => email format. | <code title="object({ data-platform-dev = string data-platform-prod = string gke-dev = string gke-prod = string project-factory-dev = string project-factory-prod = string })">object({…})</code> | | <code>null</code> | <code>01-resman</code> |
|
||||
| [vpn_onprem_configs](variables.tf#L229) | VPN gateway configuration for onprem interconnection. | <code title="map(object({ adv = object({ default = bool custom = list(string) }) peer_external_gateway = object({ redundancy_type = string interfaces = list(object({ id = number ip_address = string })) }) tunnels = list(object({ peer_asn = number peer_external_gateway_interface = number secret = string session_range = string vpn_gateway_interface = number })) }))">map(object({…}))</code> | | <code title="{ landing-trusted-ew1 = { adv = { default = false custom = [ "cloud_dns", "googleapis_private", "googleapis_restricted", "gcp_all" ] } peer_external_gateway = { redundancy_type = "SINGLE_IP_INTERNALLY_REDUNDANT" interfaces = [ { id = 0, ip_address = "8.8.8.8" }, ] } tunnels = [ { peer_asn = 65534 peer_external_gateway_interface = 0 secret = "foobar" session_range = "169.254.1.0/30" vpn_gateway_interface = 0 }, { peer_asn = 65534 peer_external_gateway_interface = 0 secret = "foobar" session_range = "169.254.1.4/30" vpn_gateway_interface = 1 } ] } landing-trusted-ew4 = { adv = { default = false custom = [ "cloud_dns", "googleapis_private", "googleapis_restricted", "gcp_all" ] } peer_external_gateway = { redundancy_type = "SINGLE_IP_INTERNALLY_REDUNDANT" interfaces = [ { id = 0, ip_address = "8.8.8.8" }, ] } tunnels = [ { peer_asn = 65534 peer_external_gateway_interface = 0 secret = "foobar" session_range = "169.254.1.0/30" vpn_gateway_interface = 0 }, { peer_asn = 65534 peer_external_gateway_interface = 0 secret = "foobar" session_range = "169.254.1.4/30" vpn_gateway_interface = 1 } ] } }">{…}</code> | |
|
||||
|
||||
## Outputs
|
||||
|
||||
|
|
|
|||
|
|
@ -31,7 +31,9 @@ locals {
|
|||
stage3_sas_delegated_grants = [
|
||||
"roles/composer.sharedVpcAgent",
|
||||
"roles/compute.networkUser",
|
||||
"roles/compute.networkViewer",
|
||||
"roles/container.hostServiceAgentUser",
|
||||
"roles/multiclusterservicediscovery.serviceAgent",
|
||||
"roles/vpcaccess.user",
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -41,7 +41,8 @@ module "dev-spoke-project" {
|
|||
metric_scopes = [module.landing-project.project_id]
|
||||
iam = {
|
||||
"roles/dns.admin" = compact([
|
||||
try(local.service_accounts.project-factory-dev, null)
|
||||
try(local.service_accounts.gke-dev, null),
|
||||
try(local.service_accounts.project-factory-dev, null),
|
||||
])
|
||||
}
|
||||
}
|
||||
|
|
@ -128,6 +129,7 @@ resource "google_project_iam_binding" "dev_spoke_project_iam_delegated" {
|
|||
members = compact([
|
||||
try(local.service_accounts.data-platform-dev, null),
|
||||
try(local.service_accounts.project-factory-dev, null),
|
||||
try(local.service_accounts.gke-dev, null),
|
||||
])
|
||||
condition {
|
||||
title = "dev_stage3_sa_delegated_grants"
|
||||
|
|
|
|||
|
|
@ -41,7 +41,8 @@ module "prod-spoke-project" {
|
|||
metric_scopes = [module.landing-project.project_id]
|
||||
iam = {
|
||||
"roles/dns.admin" = compact([
|
||||
try(local.service_accounts.project-factory-prod, null)
|
||||
try(local.service_accounts.gke-prod, null),
|
||||
try(local.service_accounts.project-factory-prod, null),
|
||||
])
|
||||
}
|
||||
}
|
||||
|
|
@ -128,6 +129,7 @@ resource "google_project_iam_binding" "prod_spoke_project_iam_delegated" {
|
|||
members = compact([
|
||||
try(local.service_accounts.data-platform-prod, null),
|
||||
try(local.service_accounts.project-factory-prod, null),
|
||||
try(local.service_accounts.gke-prod, null),
|
||||
])
|
||||
condition {
|
||||
title = "prod_stage3_sa_delegated_grants"
|
||||
|
|
|
|||
|
|
@ -218,6 +218,8 @@ variable "service_accounts" {
|
|||
type = object({
|
||||
data-platform-dev = string
|
||||
data-platform-prod = string
|
||||
gke-dev = string
|
||||
gke-prod = string
|
||||
project-factory-dev = string
|
||||
project-factory-prod = string
|
||||
})
|
||||
|
|
|
|||
|
|
@ -151,8 +151,8 @@ There are two broad sets of variables you will need to fill in:
|
|||
|
||||
To avoid the tedious job of filling in the first group of variables with values derived from other stages' outputs, the same mechanism used above for the provider configuration can be used to leverage pre-configured `.tfvars` files.
|
||||
|
||||
If you have set a valid value for `outputs_location` in the bootstrap and in the resman stage, simply link the relevant `*.auto.tfvars.json` files from this stage's folder in the path you specified.
|
||||
The `*` above is set to the name of the stage that produced it, except for `globals.auto.tfvars.json` which is also generated by the bootstrap stage, containing global values compiled manually for the bootstrap stage.
|
||||
If you have set a valid value for `outputs_location` in the bootstrap and in the resman stage, simply link the relevant `*.auto.tfvars.json` files from this stage's folder in the path you specified.
|
||||
The `*` above is set to the name of the stage that produced it, except for `globals.auto.tfvars.json` which is also generated by the bootstrap stage, containing global values compiled manually for the bootstrap stage.
|
||||
For this stage, link the following files:
|
||||
|
||||
```bash
|
||||
|
|
@ -303,8 +303,8 @@ DNS configurations are centralised in the `dns-*.tf` files. Spokes delegate DNS
|
|||
| [psa_ranges](variables.tf#L129) | IP ranges used for Private Service Access (e.g. CloudSQL). | <code title="object({ dev = object({ ranges = map(string) routes = object({ export = bool import = bool }) }) prod = object({ ranges = map(string) routes = object({ export = bool import = bool }) }) })">object({…})</code> | | <code>null</code> | |
|
||||
| [region_trigram](variables.tf#L166) | Short names for GCP regions. | <code>map(string)</code> | | <code title="{ europe-west1 = "ew1" europe-west3 = "ew3" }">{…}</code> | |
|
||||
| [router_onprem_configs](variables.tf#L175) | Configurations for routers used for onprem connectivity. | <code title="map(object({ adv = object({ custom = list(string) default = bool }) asn = number }))">map(object({…}))</code> | | <code title="{ landing-ew1 = { asn = "65533" adv = null } }">{…}</code> | |
|
||||
| [service_accounts](variables.tf#L193) | Automation service accounts in name => email format. | <code title="object({ data-platform-dev = string data-platform-prod = string project-factory-dev = string project-factory-prod = string })">object({…})</code> | | <code>null</code> | <code>01-resman</code> |
|
||||
| [vpn_onprem_configs](variables.tf#L205) | VPN gateway configuration for onprem interconnection. | <code title="map(object({ adv = object({ default = bool custom = list(string) }) peer_external_gateway = object({ redundancy_type = string interfaces = list(object({ id = number ip_address = string })) }) tunnels = list(object({ peer_asn = number peer_external_gateway_interface = number secret = string session_range = string vpn_gateway_interface = number })) }))">map(object({…}))</code> | | <code title="{ landing-ew1 = { adv = { default = false custom = [ "cloud_dns", "googleapis_private", "googleapis_restricted", "gcp_all" ] } peer_external_gateway = { redundancy_type = "SINGLE_IP_INTERNALLY_REDUNDANT" interfaces = [ { id = 0, ip_address = "8.8.8.8" }, ] } tunnels = [ { peer_asn = 65534 peer_external_gateway_interface = 0 secret = "foobar" session_range = "169.254.1.0/30" vpn_gateway_interface = 0 }, { peer_asn = 65534 peer_external_gateway_interface = 0 secret = "foobar" session_range = "169.254.1.4/30" vpn_gateway_interface = 1 } ] } }">{…}</code> | |
|
||||
| [service_accounts](variables.tf#L193) | Automation service accounts in name => email format. | <code title="object({ data-platform-dev = string data-platform-prod = string gke-dev = string gke-prod = string project-factory-dev = string project-factory-prod = string })">object({…})</code> | | <code>null</code> | <code>01-resman</code> |
|
||||
| [vpn_onprem_configs](variables.tf#L207) | VPN gateway configuration for onprem interconnection. | <code title="map(object({ adv = object({ default = bool custom = list(string) }) peer_external_gateway = object({ redundancy_type = string interfaces = list(object({ id = number ip_address = string })) }) tunnels = list(object({ peer_asn = number peer_external_gateway_interface = number secret = string session_range = string vpn_gateway_interface = number })) }))">map(object({…}))</code> | | <code title="{ landing-ew1 = { adv = { default = false custom = [ "cloud_dns", "googleapis_private", "googleapis_restricted", "gcp_all" ] } peer_external_gateway = { redundancy_type = "SINGLE_IP_INTERNALLY_REDUNDANT" interfaces = [ { id = 0, ip_address = "8.8.8.8" }, ] } tunnels = [ { peer_asn = 65534 peer_external_gateway_interface = 0 secret = "foobar" session_range = "169.254.1.0/30" vpn_gateway_interface = 0 }, { peer_asn = 65534 peer_external_gateway_interface = 0 secret = "foobar" session_range = "169.254.1.4/30" vpn_gateway_interface = 1 } ] } }">{…}</code> | |
|
||||
|
||||
## Outputs
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,8 @@
|
|||
# skip boilerplate check
|
||||
|
||||
region: europe-west1
|
||||
description: Default subnet for prod gke nodes
|
||||
ip_cidr_range: 10.64.0.0/24
|
||||
secondary_ip_range:
|
||||
pods: 100.64.0.0/16
|
||||
services: 192.168.1.0/24
|
||||
|
|
@ -28,7 +28,9 @@ locals {
|
|||
stage3_sas_delegated_grants = [
|
||||
"roles/composer.sharedVpcAgent",
|
||||
"roles/compute.networkUser",
|
||||
"roles/compute.networkViewer",
|
||||
"roles/container.hostServiceAgentUser",
|
||||
"roles/multiclusterservicediscovery.serviceAgent",
|
||||
"roles/vpcaccess.user",
|
||||
]
|
||||
service_accounts = {
|
||||
|
|
|
|||
|
|
@ -42,7 +42,8 @@ module "dev-spoke-project" {
|
|||
metric_scopes = [module.landing-project.project_id]
|
||||
iam = {
|
||||
"roles/dns.admin" = compact([
|
||||
try(local.service_accounts.project-factory-dev, null)
|
||||
try(local.service_accounts.gke-dev, null),
|
||||
try(local.service_accounts.project-factory-dev, null),
|
||||
])
|
||||
}
|
||||
}
|
||||
|
|
@ -105,6 +106,7 @@ resource "google_project_iam_binding" "dev_spoke_project_iam_delegated" {
|
|||
members = compact([
|
||||
try(local.service_accounts.data-platform-dev, null),
|
||||
try(local.service_accounts.project-factory-dev, null),
|
||||
try(local.service_accounts.gke-dev, null),
|
||||
])
|
||||
condition {
|
||||
title = "dev_stage3_sa_delegated_grants"
|
||||
|
|
|
|||
|
|
@ -42,7 +42,8 @@ module "prod-spoke-project" {
|
|||
metric_scopes = [module.landing-project.project_id]
|
||||
iam = {
|
||||
"roles/dns.admin" = compact([
|
||||
try(local.service_accounts.project-factory-prod, null)
|
||||
try(local.service_accounts.gke-prod, null),
|
||||
try(local.service_accounts.project-factory-prod, null),
|
||||
])
|
||||
}
|
||||
}
|
||||
|
|
@ -105,6 +106,7 @@ resource "google_project_iam_binding" "prod_spoke_project_iam_delegated" {
|
|||
members = compact([
|
||||
try(local.service_accounts.data-platform-prod, null),
|
||||
try(local.service_accounts.project-factory-prod, null),
|
||||
try(local.service_accounts.gke-prod, null),
|
||||
])
|
||||
condition {
|
||||
title = "prod_stage3_sa_delegated_grants"
|
||||
|
|
|
|||
|
|
@ -196,6 +196,8 @@ variable "service_accounts" {
|
|||
type = object({
|
||||
data-platform-dev = string
|
||||
data-platform-prod = string
|
||||
gke-dev = string
|
||||
gke-prod = string
|
||||
project-factory-dev = string
|
||||
project-factory-prod = string
|
||||
})
|
||||
|
|
|
|||
|
|
@ -328,8 +328,8 @@ DNS configurations are centralised in the `dns-*.tf` files. Spokes delegate DNS
|
|||
| [region_trigram](variables.tf#L166) | Short names for GCP regions. | <code>map(string)</code> | | <code title="{ europe-west1 = "ew1" europe-west3 = "ew3" }">{…}</code> | |
|
||||
| [router_onprem_configs](variables.tf#L175) | Configurations for routers used for onprem connectivity. | <code title="map(object({ adv = object({ custom = list(string) default = bool }) asn = number }))">map(object({…}))</code> | | <code title="{ landing-ew1 = { asn = "65533" adv = null } }">{…}</code> | |
|
||||
| [router_spoke_configs](variables-vpn.tf#L18) | Configurations for routers used for internal connectivity. | <code title="map(object({ adv = object({ custom = list(string) default = bool }) asn = number }))">map(object({…}))</code> | | <code title="{ landing-ew1 = { asn = "64512", adv = null } landing-ew4 = { asn = "64512", adv = null } spoke-dev-ew1 = { asn = "64513", adv = null } spoke-dev-ew4 = { asn = "64513", adv = null } spoke-prod-ew1 = { asn = "64514", adv = null } spoke-prod-ew4 = { asn = "64514", adv = null } }">{…}</code> | |
|
||||
| [service_accounts](variables.tf#L193) | Automation service accounts in name => email format. | <code title="object({ data-platform-dev = string data-platform-prod = string project-factory-dev = string project-factory-prod = string })">object({…})</code> | | <code>null</code> | <code>01-resman</code> |
|
||||
| [vpn_onprem_configs](variables.tf#L205) | VPN gateway configuration for onprem interconnection. | <code title="map(object({ adv = object({ default = bool custom = list(string) }) peer_external_gateway = object({ redundancy_type = string interfaces = list(object({ id = number ip_address = string })) }) tunnels = list(object({ peer_asn = number peer_external_gateway_interface = number secret = string session_range = string vpn_gateway_interface = number })) }))">map(object({…}))</code> | | <code title="{ landing-ew1 = { adv = { default = false custom = [ "cloud_dns", "googleapis_private", "googleapis_restricted", "gcp_all" ] } peer_external_gateway = { redundancy_type = "SINGLE_IP_INTERNALLY_REDUNDANT" interfaces = [ { id = 0, ip_address = "8.8.8.8" }, ] } tunnels = [ { peer_asn = 65534 peer_external_gateway_interface = 0 secret = "foobar" session_range = "169.254.1.0/30" vpn_gateway_interface = 0 }, { peer_asn = 65534 peer_external_gateway_interface = 0 secret = "foobar" session_range = "169.254.1.4/30" vpn_gateway_interface = 1 } ] } }">{…}</code> | |
|
||||
| [service_accounts](variables.tf#L193) | Automation service accounts in name => email format. | <code title="object({ data-platform-dev = string data-platform-prod = string gke-dev = string gke-prod = string project-factory-dev = string project-factory-prod = string })">object({…})</code> | | <code>null</code> | <code>01-resman</code> |
|
||||
| [vpn_onprem_configs](variables.tf#L207) | VPN gateway configuration for onprem interconnection. | <code title="map(object({ adv = object({ default = bool custom = list(string) }) peer_external_gateway = object({ redundancy_type = string interfaces = list(object({ id = number ip_address = string })) }) tunnels = list(object({ peer_asn = number peer_external_gateway_interface = number secret = string session_range = string vpn_gateway_interface = number })) }))">map(object({…}))</code> | | <code title="{ landing-ew1 = { adv = { default = false custom = [ "cloud_dns", "googleapis_private", "googleapis_restricted", "gcp_all" ] } peer_external_gateway = { redundancy_type = "SINGLE_IP_INTERNALLY_REDUNDANT" interfaces = [ { id = 0, ip_address = "8.8.8.8" }, ] } tunnels = [ { peer_asn = 65534 peer_external_gateway_interface = 0 secret = "foobar" session_range = "169.254.1.0/30" vpn_gateway_interface = 0 }, { peer_asn = 65534 peer_external_gateway_interface = 0 secret = "foobar" session_range = "169.254.1.4/30" vpn_gateway_interface = 1 } ] } }">{…}</code> | |
|
||||
| [vpn_spoke_configs](variables-vpn.tf#L37) | VPN gateway configuration for spokes. | <code title="map(object({ default = bool custom = list(string) }))">map(object({…}))</code> | | <code title="{ landing-ew1 = { default = false custom = ["rfc_1918_10", "rfc_1918_172", "rfc_1918_192"] } landing-ew4 = { default = false custom = ["rfc_1918_10", "rfc_1918_172", "rfc_1918_192"] } dev-ew1 = { default = false custom = ["gcp_dev"] } prod-ew1 = { default = false custom = ["gcp_prod"] } prod-ew4 = { default = false custom = ["gcp_prod"] } }">{…}</code> | |
|
||||
|
||||
## Outputs
|
||||
|
|
|
|||
|
|
@ -0,0 +1,8 @@
|
|||
# skip boilerplate check
|
||||
|
||||
region: europe-west1
|
||||
description: Default subnet for prod gke nodes
|
||||
ip_cidr_range: 10.64.0.0/24
|
||||
secondary_ip_range:
|
||||
pods: 100.64.0.0/16
|
||||
services: 192.168.1.0/24
|
||||
|
|
@ -28,7 +28,9 @@ locals {
|
|||
stage3_sas_delegated_grants = [
|
||||
"roles/composer.sharedVpcAgent",
|
||||
"roles/compute.networkUser",
|
||||
"roles/compute.networkViewer",
|
||||
"roles/container.hostServiceAgentUser",
|
||||
"roles/multiclusterservicediscovery.serviceAgent",
|
||||
"roles/vpcaccess.user",
|
||||
]
|
||||
service_accounts = {
|
||||
|
|
|
|||
|
|
@ -42,7 +42,8 @@ module "dev-spoke-project" {
|
|||
metric_scopes = [module.landing-project.project_id]
|
||||
iam = {
|
||||
"roles/dns.admin" = compact([
|
||||
try(local.service_accounts.project-factory-dev, null)
|
||||
try(local.service_accounts.gke-dev, null),
|
||||
try(local.service_accounts.project-factory-dev, null),
|
||||
])
|
||||
}
|
||||
}
|
||||
|
|
@ -105,6 +106,7 @@ resource "google_project_iam_binding" "dev_spoke_project_iam_delegated" {
|
|||
members = compact([
|
||||
try(local.service_accounts.data-platform-dev, null),
|
||||
try(local.service_accounts.project-factory-dev, null),
|
||||
try(local.service_accounts.gke-dev, null),
|
||||
])
|
||||
condition {
|
||||
title = "dev_stage3_sa_delegated_grants"
|
||||
|
|
|
|||
|
|
@ -42,7 +42,8 @@ module "prod-spoke-project" {
|
|||
metric_scopes = [module.landing-project.project_id]
|
||||
iam = {
|
||||
"roles/dns.admin" = compact([
|
||||
try(local.service_accounts.project-factory-prod, null)
|
||||
try(local.service_accounts.gke-prod, null),
|
||||
try(local.service_accounts.project-factory-prod, null),
|
||||
])
|
||||
}
|
||||
}
|
||||
|
|
@ -105,6 +106,7 @@ resource "google_project_iam_binding" "prod_spoke_project_iam_delegated" {
|
|||
members = compact([
|
||||
try(local.service_accounts.data-platform-prod, null),
|
||||
try(local.service_accounts.project-factory-prod, null),
|
||||
try(local.service_accounts.gke-prod, null),
|
||||
])
|
||||
condition {
|
||||
title = "prod_stage3_sa_delegated_grants"
|
||||
|
|
|
|||
|
|
@ -196,6 +196,8 @@ variable "service_accounts" {
|
|||
type = object({
|
||||
data-platform-dev = string
|
||||
data-platform-prod = string
|
||||
gke-dev = string
|
||||
gke-prod = string
|
||||
project-factory-dev = string
|
||||
project-factory-prod = string
|
||||
})
|
||||
|
|
|
|||
|
|
@ -158,7 +158,7 @@ You can find examples in the `[demo](../../../../examples/data-solutions/data-pl
|
|||
|
||||
| name | description | modules | resources |
|
||||
|---|---|---|---|
|
||||
| [main.tf](./main.tf) | Data Platformy. | <code>data-platform-foundations</code> | |
|
||||
| [main.tf](./main.tf) | Data Platform. | <code>data-platform-foundations</code> | |
|
||||
| [outputs.tf](./outputs.tf) | Output variables. | | <code>google_storage_bucket_object</code> · <code>local_file</code> |
|
||||
| [variables.tf](./variables.tf) | Terraform Variables. | | |
|
||||
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@
|
|||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description Data Platformy.
|
||||
# tfdoc:file:description Data Platform.
|
||||
|
||||
module "data-platform" {
|
||||
source = "../../../../examples/data-solutions/data-platform-foundations"
|
||||
|
|
|
|||
|
|
@ -0,0 +1,9 @@
|
|||
# GKE Multitenant stage
|
||||
|
||||
This directory contains a stage that can be used to centralize management of GKE multinenant clusters.
|
||||
|
||||
The Terraform code follows the same general approach used for the [project factory](../03-project-factory/) and [data platform](../03-data-platform/) stages, where a "fat module" contains the stage code and is used by thin code wrappers that localize it for each environment or specialized configuration:
|
||||
|
||||
The [`dev` folder](./dev/) contains an example setup for a generic development environment, and can be used as-is or cloned to implement other environments, or more specialized setups
|
||||
|
||||
Refer to [the `dev` documentation](./dev/README.md) configuration details, and to [the `gke-serverless` documentation](../../../examples/gke-serverless/multitenant-fleet) for the architectural design and decisions taken.
|
||||
|
|
@ -0,0 +1,174 @@
|
|||
# GKE Multitenant
|
||||
|
||||
This stage allows creation and management of a fleet of GKE multitenant clusters, optionally leveraging GKE Hub to configure additional features. It's designed to be replicated once for every homogeneous set of clusters, either per environment or with more granularity as needed (e.g. teams or sets of teams sharing similar requirements).
|
||||
|
||||
The following diagram illustrates the high-level design of created resources, which can be adapted to specific requirements via variables:
|
||||
|
||||
<p align="center">
|
||||
<img src="diagram.png" alt="GKE multitenant">
|
||||
</p>
|
||||
|
||||
## Design overview and choices
|
||||
|
||||
> The detailed architecture of the underlying resources is explained in the documentation of [GKE multitenant module](../../../../examples/gke-serverless/multitenant-fleet/README.md).
|
||||
|
||||
This stage creates a project containing and as many clusters and node pools as requested by the user through the [variables](#variables) explained below. The GKE clusters are created with the with the following setup:
|
||||
|
||||
- All clusters are assumed to be [private](https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters), therefore only [VPC-native clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips) are supported.
|
||||
- Logging and monitoring configured to use Cloud Operations for system components and user workloads.
|
||||
- [GKE metering](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-usage-metering) enabled by default and stored in a bigquery dataset created within the project.
|
||||
- Optional [GKE Fleet](https://cloud.google.com/kubernetes-engine/docs/fleets-overview) support with the possibility to enable any of the following features:
|
||||
- [Fleet workload identity](https://cloud.google.com/anthos/fleet-management/docs/use-workload-identity)
|
||||
- [Anthos Config Management](https://cloud.google.com/anthos-config-management/docs/overview)
|
||||
- [Anthos Service Mesh](https://cloud.google.com/service-mesh/docs/overview)
|
||||
- [Anthos Identity Service](https://cloud.google.com/anthos/identity/setup/fleet)
|
||||
- [Multi-cluster services](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-services)
|
||||
- [Multi-cluster ingress](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-ingress).
|
||||
- Support for [Config Sync](https://cloud.google.com/anthos-config-management/docs/config-sync-overview), [Hierarchy Controller](https://cloud.google.com/anthos-config-management/docs/concepts/hierarchy-controller), and [Policy Controller](https://cloud.google.com/anthos-config-management/docs/concepts/policy-controller) when using Anthos Config Management.
|
||||
- [Groups for GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/google-groups-rbac) can be enabled to facilitate the creation of flexible RBAC policies referencing group principals.
|
||||
- Support for [application layer secret encryption](https://cloud.google.com/kubernetes-engine/docs/how-to/encrypting-secrets).
|
||||
- Support to customize peering configuration of the control plane VPC (e.g. to import/export routes to the peered network)
|
||||
- Some features are enabled by default in all clusters:
|
||||
- [Intranode visibility](https://cloud.google.com/kubernetes-engine/docs/how-to/intranode-visibility)
|
||||
- [Dataplane v2](https://cloud.google.com/kubernetes-engine/docs/concepts/dataplane-v2)
|
||||
- [Shielded GKE nodes](https://cloud.google.com/kubernetes-engine/docs/how-to/shielded-gke-nodes)
|
||||
- [Workload identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity)
|
||||
- [Node local DNS cache](https://cloud.google.com/kubernetes-engine/docs/how-to/nodelocal-dns-cache)
|
||||
- [Use of the GCE persistent disk CSI driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver)
|
||||
- Node [auto-upgrade](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-upgrades) and [auto-repair](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair) for all node pools
|
||||
|
||||
|
||||
## How to run this stage
|
||||
|
||||
This stage is meant to be executed after "foundational stages" (i.e., stages [`00-bootstrap`](../../00-bootstrap), [`01-resman`](../../01-resman), 02-networking (either [VPN](../../02-networking-vpn) or [NVA](../../02-networking-nva)) and [`02-security`](../../02-security)) have been run.
|
||||
|
||||
It's of course possible to run this stage in isolation, by making sure the architectural prerequisites are satisfied (e.g., networking), and that the Service Account running the stage is granted the roles/permissions below:
|
||||
|
||||
- on the organization or network folder level
|
||||
- `roles/xpnAdmin` or a custom role which includes the following permissions
|
||||
- `compute.organizations.enableXpnResource`,
|
||||
- `compute.organizations.disableXpnResource`,
|
||||
- `compute.subnetworks.setIamPolicy`,
|
||||
- on each folder where projects are created
|
||||
- `roles/logging.admin`
|
||||
- `roles/owner`
|
||||
- `roles/resourcemanager.folderAdmin`
|
||||
- `roles/resourcemanager.projectCreator`
|
||||
- `roles/xpnAdmin`
|
||||
- on the host project for the Shared VPC
|
||||
- `roles/browser`
|
||||
- `roles/compute.viewer`
|
||||
- on the organization or billing account
|
||||
- `roles/billing.admin`
|
||||
|
||||
The VPC host project, VPC and subnets should already exist.
|
||||
|
||||
### Providers configuration
|
||||
|
||||
If you're running this on top of FAST, you should run the following commands to create the providers file, and populate the required variables from the previous stage.
|
||||
|
||||
```bash
|
||||
# Variable `outputs_location` is set to `~/fast-config` in stage 01-resman
|
||||
$ cd fabric-fast/stages/03-gke-multitenant/dev
|
||||
ln -s ~/fast-config/providers/03-gke-dev-providers.tf .
|
||||
```
|
||||
|
||||
### Variable configuration
|
||||
|
||||
There are two broad sets of variables you will need to fill in:
|
||||
|
||||
- variables shared by other stages (organization id, billing account id, etc.), or derived from a resource managed by a different stage (folder id, automation project id, etc.)
|
||||
- variables specific to resources managed by this stage
|
||||
|
||||
#### Variables passed in from other stages
|
||||
|
||||
To avoid the tedious job of filling in the first group of variables with values derived from other stages' outputs, the same mechanism used above for the provider configuration can be used to leverage pre-configured `.tfvars` files.
|
||||
|
||||
If you configured a valid path for `outputs_location` in the bootstrap and networking stage, simply link the relevant `terraform-*.auto.tfvars.json` files from this stage's outputs folder (under the path you specified), where the `*` above is set to the name of the stage that produced it. For this stage, a single `.tfvars` file is available:
|
||||
|
||||
```bash
|
||||
# Variable `outputs_location` is set to `~/fast-config`
|
||||
ln -s ~/fast-config/tfvars/00-bootstrap.auto.tfvars.json .
|
||||
ln -s ~/fast-config/tfvars/01-resman.auto.tfvars.json .
|
||||
ln -s ~/fast-config/tfvars/02-networking.auto.tfvars.json .
|
||||
```
|
||||
|
||||
If you're not using FAST, refer to the [Variables](#variables) table at the bottom of this document for a full list of variables, their origin (e.g., a stage or specific to this one), and descriptions explaining their meaning.
|
||||
|
||||
#### Cluster and node pools
|
||||
|
||||
This stage is designed with multi-tenancy in mind, and the expectation is that GKE clusters will mostly share a common set of defaults. Variables are designed to support this approach for both clusters and node pools:
|
||||
|
||||
- the `cluster_default` variable allows defining common defaults for all clusters
|
||||
- the `clusters` variable is used to declare the actual GKE clusters and allows overriding defaults on a per-cluster basis
|
||||
- the `nodepool_defaults` variable allows definining common defaults for all node pools
|
||||
- the `nodepools` variable is used to declare cluster node pools and allows overriding defaults on a per-cluster basis
|
||||
|
||||
There are two additional variables that influence cluster configuration: `authenticator_security_group` to configure [Google Groups for RBAC](https://cloud.google.com/kubernetes-engine/docs/how-to/google-groups-rbac), `dns_domain` to configure [Cloud DNS for GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/cloud-dns).
|
||||
|
||||
#### Fleet management
|
||||
|
||||
Fleet management is entirely optional, and uses three separate variables:
|
||||
|
||||
- `fleet_features`: specifies the [GKE fleet](https://cloud.google.com/anthos/fleet-management/docs/fleet-concepts#fleet-enabled-components) features you want activate
|
||||
- `fleet_configmanagement_templates`: defines configuration templates for specific sets of features ([Config Management](https://cloud.google.com/anthos-config-management/docs/how-to/install-anthos-config-management) currently)
|
||||
- `fleet_configmanagement_clusters`: specifies which clusters are managed by fleet features, and the optional Config Management template for each cluster
|
||||
- `fleet_workload_identity`: to enables optional centralized [Workload Identity](https://cloud.google.com/anthos/fleet-management/docs/use-workload-identity)
|
||||
|
||||
Leave all these variables unset (or set to `null`) to disable fleet management.
|
||||
|
||||
## Running Terraform
|
||||
|
||||
Once the [provider](#providers-configuration) and [variable](#variable-configuration) configuration is complete, you can apply this stage:
|
||||
|
||||
```bash
|
||||
terraform init
|
||||
terraform apply
|
||||
```
|
||||
|
||||
<!-- TFDOC OPTS files:1 show_extra:1 -->
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Files
|
||||
|
||||
| name | description | modules | resources |
|
||||
|---|---|---|---|
|
||||
| [main.tf](./main.tf) | GKE multitenant for development environment. | <code>multitenant-fleet</code> | |
|
||||
| [outputs.tf](./outputs.tf) | Output variables. | | <code>google_storage_bucket_object</code> · <code>local_file</code> |
|
||||
| [variables.tf](./variables.tf) | Module variables. | | |
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default | producer |
|
||||
|---|---|:---:|:---:|:---:|:---:|
|
||||
| [automation](variables.tf#L21) | Automation resources created by the bootstrap stage. | <code title="object({ outputs_bucket = string })">object({…})</code> | ✓ | | <code>00-bootstrap</code> |
|
||||
| [billing_account](variables.tf#L35) | Billing account id and organization id ('nnnnnnnn' or null). | <code title="object({ id = string organization_id = number })">object({…})</code> | ✓ | | <code>00-bootstrap</code> |
|
||||
| [clusters](variables.tf#L73) | | <code title="map(object({ cluster_autoscaling = object({ cpu_min = number cpu_max = number memory_min = number memory_max = number }) description = string dns_domain = string labels = map(string) location = string net = object({ master_range = string pods = string services = string subnet = string }) overrides = object({ cloudrun_config = bool database_encryption_key = string master_authorized_ranges = map(string) max_pods_per_node = number pod_security_policy = bool release_channel = string vertical_pod_autoscaling = bool gcp_filestore_csi_driver_config = bool }) }))">map(object({…}))</code> | ✓ | | |
|
||||
| [folder_ids](variables.tf#L175) | Folders to be used for the networking resources in folders/nnnnnnnnnnn format. If null, folder will be created. | <code title="object({ gke-dev = string })">object({…})</code> | ✓ | | <code>01-resman</code> |
|
||||
| [host_project_ids](variables.tf#L197) | Host project for the shared VPC. | <code title="object({ dev-spoke-0 = string })">object({…})</code> | ✓ | | <code>02-networking</code> |
|
||||
| [nodepools](variables.tf#L229) | | <code title="map(map(object({ node_count = number node_type = string initial_node_count = number overrides = object({ image_type = string max_pods_per_node = number node_locations = list(string) node_tags = list(string) node_taints = list(string) }) spot = bool })))">map(map(object({…})))</code> | ✓ | | |
|
||||
| [prefix](variables.tf#L252) | Prefix used for resources that need unique names. | <code>string</code> | ✓ | | |
|
||||
| [vpc_self_links](variables.tf#L264) | Self link for the shared VPC. | <code title="object({ dev-spoke-0 = string })">object({…})</code> | ✓ | | <code>02-networking</code> |
|
||||
| [authenticator_security_group](variables.tf#L29) | Optional group used for Groups for GKE. | <code>string</code> | | <code>null</code> | |
|
||||
| [cluster_defaults](variables.tf#L44) | Default values for optional cluster configurations. | <code title="object({ cloudrun_config = bool database_encryption_key = string master_authorized_ranges = map(string) max_pods_per_node = number pod_security_policy = bool release_channel = string vertical_pod_autoscaling = bool gcp_filestore_csi_driver_config = bool })">object({…})</code> | | <code title="{ cloudrun_config = false database_encryption_key = null master_authorized_ranges = { rfc1918_1 = "10.0.0.0/8" rfc1918_2 = "172.16.0.0/12" rfc1918_3 = "192.168.0.0/16" } max_pods_per_node = 110 pod_security_policy = false release_channel = "STABLE" vertical_pod_autoscaling = false gcp_filestore_csi_driver_config = false }">{…}</code> | |
|
||||
| [dns_domain](variables.tf#L106) | Domain name used for clusters, prefixed by each cluster name. Leave null to disable Cloud DNS for GKE. | <code>string</code> | | <code>null</code> | |
|
||||
| [fleet_configmanagement_clusters](variables.tf#L112) | Config management features enabled on specific sets of member clusters, in config name => [cluster name] format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [fleet_configmanagement_templates](variables.tf#L120) | Sets of config management configurations that can be applied to member clusters, in config name => {options} format. | <code title="map(object({ binauthz = bool config_sync = object({ git = object({ gcp_service_account_email = string https_proxy = string policy_dir = string secret_type = string sync_branch = string sync_repo = string sync_rev = string sync_wait_secs = number }) prevent_drift = string source_format = string }) hierarchy_controller = object({ enable_hierarchical_resource_quota = bool enable_pod_tree_labels = bool }) policy_controller = object({ audit_interval_seconds = number exemptable_namespaces = list(string) log_denies_enabled = bool referential_rules_enabled = bool template_library_installed = bool }) version = string }))">map(object({…}))</code> | | <code>{}</code> | |
|
||||
| [fleet_features](variables.tf#L155) | Enable and configue fleet features. Set to null to disable GKE Hub if fleet workload identity is not used. | <code title="object({ appdevexperience = bool configmanagement = bool identityservice = bool multiclusteringress = string multiclusterservicediscovery = bool servicemesh = bool })">object({…})</code> | | <code>null</code> | |
|
||||
| [fleet_workload_identity](variables.tf#L168) | Use Fleet Workload Identity for clusters. Enables GKE Hub if set to true. | <code>bool</code> | | <code>false</code> | |
|
||||
| [group_iam](variables.tf#L183) | Project-level authoritative IAM bindings for groups in {GROUP_EMAIL => [ROLES]} format. Use group emails as keys, list of roles as values. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [iam](variables.tf#L190) | Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [labels](variables.tf#L205) | Project-level labels. | <code>map(string)</code> | | <code>{}</code> | |
|
||||
| [nodepool_defaults](variables.tf#L211) | | <code title="object({ image_type = string max_pods_per_node = number node_locations = list(string) node_tags = list(string) node_taints = list(string) })">object({…})</code> | | <code title="{ image_type = "COS_CONTAINERD" max_pods_per_node = 110 node_locations = null node_tags = null node_taints = [] }">{…}</code> | |
|
||||
| [outputs_location](variables.tf#L246) | Path where providers, tfvars files, and lists for the following stages are written. Leave empty to disable. | <code>string</code> | | <code>null</code> | |
|
||||
| [project_services](variables.tf#L257) | Additional project services to enable. | <code>list(string)</code> | | <code>[]</code> | |
|
||||
|
||||
## Outputs
|
||||
|
||||
| name | description | sensitive | consumers |
|
||||
|---|---|:---:|---|
|
||||
| [cluster_ids](outputs.tf#L63) | Cluster ids. | | |
|
||||
| [clusters](outputs.tf#L57) | Cluster resources. | ✓ | |
|
||||
| [project_id](outputs.tf#L68) | GKE project id. | | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 43 KiB |
|
|
@ -0,0 +1,43 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description GKE multitenant for development environment.
|
||||
|
||||
module "gke-multitenant" {
|
||||
source = "../../../../examples/gke-serverless/multitenant-fleet"
|
||||
billing_account_id = var.billing_account.id
|
||||
folder_id = var.folder_ids.gke-dev
|
||||
project_id = "gke-clusters-0"
|
||||
group_iam = var.group_iam
|
||||
iam = var.iam
|
||||
labels = merge(var.labels, { environment = "dev" })
|
||||
prefix = "${var.prefix}-dev"
|
||||
project_services = var.project_services
|
||||
vpc_config = {
|
||||
host_project_id = var.host_project_ids.dev-spoke-0
|
||||
vpc_self_link = var.vpc_self_links.dev-spoke-0
|
||||
}
|
||||
cluster_defaults = var.cluster_defaults
|
||||
nodepool_defaults = var.nodepool_defaults
|
||||
clusters = var.clusters
|
||||
nodepools = var.nodepools
|
||||
authenticator_security_group = var.authenticator_security_group
|
||||
dns_domain = var.dns_domain
|
||||
fleet_configmanagement_clusters = var.fleet_configmanagement_clusters
|
||||
fleet_configmanagement_templates = var.fleet_configmanagement_templates
|
||||
fleet_features = var.fleet_features
|
||||
fleet_workload_identity = var.fleet_workload_identity
|
||||
}
|
||||
|
|
@ -0,0 +1,71 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# tfdoc:file:description Output variables.
|
||||
|
||||
locals {
|
||||
tfvars = {
|
||||
clusters = module.gke-multitenant.cluster_ids
|
||||
project_ids = {
|
||||
gke-dev = module.gke-multitenant.project_id
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# generate tfvars file for subsequent stages
|
||||
|
||||
resource "local_file" "tfvars" {
|
||||
for_each = var.outputs_location == null ? {} : { 1 = 1 }
|
||||
file_permission = "0644"
|
||||
filename = "${pathexpand(var.outputs_location)}/tfvars/03-gke-dev.auto.tfvars.json"
|
||||
content = jsonencode(local.tfvars)
|
||||
}
|
||||
|
||||
resource "google_storage_bucket_object" "tfvars" {
|
||||
bucket = var.automation.outputs_bucket
|
||||
name = "tfvars/03-gke-dev.auto.tfvars.json"
|
||||
content = jsonencode(local.tfvars)
|
||||
}
|
||||
|
||||
# outputs
|
||||
|
||||
output "clusters" {
|
||||
description = "Cluster resources."
|
||||
value = module.gke-multitenant.clusters
|
||||
sensitive = true
|
||||
}
|
||||
|
||||
output "cluster_ids" {
|
||||
description = "Cluster ids."
|
||||
value = module.gke-multitenant.cluster_ids
|
||||
}
|
||||
|
||||
output "project_id" {
|
||||
description = "GKE project id."
|
||||
value = module.gke-multitenant.project_id
|
||||
}
|
||||
|
|
@ -0,0 +1,270 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# we deal with one env here
|
||||
# 1 project, m clusters
|
||||
# cloud dns for gke?
|
||||
|
||||
variable "automation" {
|
||||
# tfdoc:variable:source 00-bootstrap
|
||||
description = "Automation resources created by the bootstrap stage."
|
||||
type = object({
|
||||
outputs_bucket = string
|
||||
})
|
||||
}
|
||||
|
||||
variable "authenticator_security_group" {
|
||||
description = "Optional group used for Groups for GKE."
|
||||
type = string
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "billing_account" {
|
||||
# tfdoc:variable:source 00-bootstrap
|
||||
description = "Billing account id and organization id ('nnnnnnnn' or null)."
|
||||
type = object({
|
||||
id = string
|
||||
organization_id = number
|
||||
})
|
||||
}
|
||||
|
||||
variable "cluster_defaults" {
|
||||
description = "Default values for optional cluster configurations."
|
||||
type = object({
|
||||
cloudrun_config = bool
|
||||
database_encryption_key = string
|
||||
master_authorized_ranges = map(string)
|
||||
max_pods_per_node = number
|
||||
pod_security_policy = bool
|
||||
release_channel = string
|
||||
vertical_pod_autoscaling = bool
|
||||
gcp_filestore_csi_driver_config = bool
|
||||
})
|
||||
default = {
|
||||
cloudrun_config = false
|
||||
database_encryption_key = null
|
||||
# binary_authorization = false
|
||||
master_authorized_ranges = {
|
||||
rfc1918_1 = "10.0.0.0/8"
|
||||
rfc1918_2 = "172.16.0.0/12"
|
||||
rfc1918_3 = "192.168.0.0/16"
|
||||
}
|
||||
max_pods_per_node = 110
|
||||
pod_security_policy = false
|
||||
release_channel = "STABLE"
|
||||
vertical_pod_autoscaling = false
|
||||
gcp_filestore_csi_driver_config = false
|
||||
}
|
||||
}
|
||||
|
||||
variable "clusters" {
|
||||
description = ""
|
||||
type = map(object({
|
||||
cluster_autoscaling = object({
|
||||
cpu_min = number
|
||||
cpu_max = number
|
||||
memory_min = number
|
||||
memory_max = number
|
||||
})
|
||||
description = string
|
||||
dns_domain = string
|
||||
labels = map(string)
|
||||
location = string
|
||||
net = object({
|
||||
master_range = string
|
||||
pods = string
|
||||
services = string
|
||||
subnet = string
|
||||
})
|
||||
overrides = object({
|
||||
cloudrun_config = bool
|
||||
database_encryption_key = string
|
||||
# binary_authorization = bool
|
||||
master_authorized_ranges = map(string)
|
||||
max_pods_per_node = number
|
||||
pod_security_policy = bool
|
||||
release_channel = string
|
||||
vertical_pod_autoscaling = bool
|
||||
gcp_filestore_csi_driver_config = bool
|
||||
})
|
||||
}))
|
||||
}
|
||||
|
||||
variable "dns_domain" {
|
||||
description = "Domain name used for clusters, prefixed by each cluster name. Leave null to disable Cloud DNS for GKE."
|
||||
type = string
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "fleet_configmanagement_clusters" {
|
||||
description = "Config management features enabled on specific sets of member clusters, in config name => [cluster name] format."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
|
||||
variable "fleet_configmanagement_templates" {
|
||||
description = "Sets of config management configurations that can be applied to member clusters, in config name => {options} format."
|
||||
type = map(object({
|
||||
binauthz = bool
|
||||
config_sync = object({
|
||||
git = object({
|
||||
gcp_service_account_email = string
|
||||
https_proxy = string
|
||||
policy_dir = string
|
||||
secret_type = string
|
||||
sync_branch = string
|
||||
sync_repo = string
|
||||
sync_rev = string
|
||||
sync_wait_secs = number
|
||||
})
|
||||
prevent_drift = string
|
||||
source_format = string
|
||||
})
|
||||
hierarchy_controller = object({
|
||||
enable_hierarchical_resource_quota = bool
|
||||
enable_pod_tree_labels = bool
|
||||
})
|
||||
policy_controller = object({
|
||||
audit_interval_seconds = number
|
||||
exemptable_namespaces = list(string)
|
||||
log_denies_enabled = bool
|
||||
referential_rules_enabled = bool
|
||||
template_library_installed = bool
|
||||
})
|
||||
version = string
|
||||
}))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "fleet_features" {
|
||||
description = "Enable and configue fleet features. Set to null to disable GKE Hub if fleet workload identity is not used."
|
||||
type = object({
|
||||
appdevexperience = bool
|
||||
configmanagement = bool
|
||||
identityservice = bool
|
||||
multiclusteringress = string
|
||||
multiclusterservicediscovery = bool
|
||||
servicemesh = bool
|
||||
})
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "fleet_workload_identity" {
|
||||
description = "Use Fleet Workload Identity for clusters. Enables GKE Hub if set to true."
|
||||
type = bool
|
||||
default = false
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "folder_ids" {
|
||||
# tfdoc:variable:source 01-resman
|
||||
description = "Folders to be used for the networking resources in folders/nnnnnnnnnnn format. If null, folder will be created."
|
||||
type = object({
|
||||
gke-dev = string
|
||||
})
|
||||
}
|
||||
|
||||
variable "group_iam" {
|
||||
description = "Project-level authoritative IAM bindings for groups in {GROUP_EMAIL => [ROLES]} format. Use group emails as keys, list of roles as values."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "iam" {
|
||||
description = "Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "host_project_ids" {
|
||||
# tfdoc:variable:source 02-networking
|
||||
description = "Host project for the shared VPC."
|
||||
type = object({
|
||||
dev-spoke-0 = string
|
||||
})
|
||||
}
|
||||
|
||||
variable "labels" {
|
||||
description = "Project-level labels."
|
||||
type = map(string)
|
||||
default = {}
|
||||
}
|
||||
|
||||
variable "nodepool_defaults" {
|
||||
description = ""
|
||||
type = object({
|
||||
image_type = string
|
||||
max_pods_per_node = number
|
||||
node_locations = list(string)
|
||||
node_tags = list(string)
|
||||
node_taints = list(string)
|
||||
})
|
||||
default = {
|
||||
image_type = "COS_CONTAINERD"
|
||||
max_pods_per_node = 110
|
||||
node_locations = null
|
||||
node_tags = null
|
||||
node_taints = []
|
||||
}
|
||||
}
|
||||
|
||||
variable "nodepools" {
|
||||
description = ""
|
||||
type = map(map(object({
|
||||
node_count = number
|
||||
node_type = string
|
||||
initial_node_count = number
|
||||
overrides = object({
|
||||
image_type = string
|
||||
max_pods_per_node = number
|
||||
node_locations = list(string)
|
||||
node_tags = list(string)
|
||||
node_taints = list(string)
|
||||
})
|
||||
spot = bool
|
||||
})))
|
||||
}
|
||||
|
||||
variable "outputs_location" {
|
||||
description = "Path where providers, tfvars files, and lists for the following stages are written. Leave empty to disable."
|
||||
type = string
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "prefix" {
|
||||
description = "Prefix used for resources that need unique names."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "project_services" {
|
||||
description = "Additional project services to enable."
|
||||
type = list(string)
|
||||
default = []
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "vpc_self_links" {
|
||||
# tfdoc:variable:source 02-networking
|
||||
description = "Self link for the shared VPC."
|
||||
type = object({
|
||||
dev-spoke-0 = string
|
||||
})
|
||||
}
|
||||
|
|
@ -53,7 +53,7 @@ It's of course possible to run this stage in isolation, by making sure the archi
|
|||
|
||||
### Providers configuration
|
||||
|
||||
If you're running this on top of Fast, you should run the following commands to create the providers file, and populate the required variables from the previous stage.
|
||||
If you're running this on top of FAST, you should run the following commands to create the providers file, and populate the required variables from the previous stage.
|
||||
|
||||
```bash
|
||||
# Variable `outputs_location` is set to `~/fast-config` in stage 01-resman
|
||||
|
|
|
|||
|
|
@ -68,13 +68,13 @@ module "cluster-1" {
|
|||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| [location](variables.tf#L161) | Cluster zone or region. | <code>string</code> | ✓ | |
|
||||
| [name](variables.tf#L228) | Cluster name. | <code>string</code> | ✓ | |
|
||||
| [network](variables.tf#L233) | Name or self link of the VPC used for the cluster. Use the self link for Shared VPC. | <code>string</code> | ✓ | |
|
||||
| [project_id](variables.tf#L277) | Cluster project id. | <code>string</code> | ✓ | |
|
||||
| [secondary_range_pods](variables.tf#L300) | Subnet secondary range name used for pods. | <code>string</code> | ✓ | |
|
||||
| [secondary_range_services](variables.tf#L305) | Subnet secondary range name used for services. | <code>string</code> | ✓ | |
|
||||
| [subnetwork](variables.tf#L310) | VPC subnetwork name or self link. | <code>string</code> | ✓ | |
|
||||
| [location](variables.tf#L155) | Cluster zone or region. | <code>string</code> | ✓ | |
|
||||
| [name](variables.tf#L222) | Cluster name. | <code>string</code> | ✓ | |
|
||||
| [network](variables.tf#L227) | Name or self link of the VPC used for the cluster. Use the self link for Shared VPC. | <code>string</code> | ✓ | |
|
||||
| [project_id](variables.tf#L271) | Cluster project id. | <code>string</code> | ✓ | |
|
||||
| [secondary_range_pods](variables.tf#L294) | Subnet secondary range name used for pods. | <code>string</code> | ✓ | |
|
||||
| [secondary_range_services](variables.tf#L299) | Subnet secondary range name used for services. | <code>string</code> | ✓ | |
|
||||
| [subnetwork](variables.tf#L304) | VPC subnetwork name or self link. | <code>string</code> | ✓ | |
|
||||
| [addons](variables.tf#L17) | Addons enabled in the cluster (true means enabled). | <code title="object({ cloudrun_config = bool dns_cache_config = bool horizontal_pod_autoscaling = bool http_load_balancing = bool istio_config = object({ enabled = bool tls = bool }) network_policy_config = bool gce_persistent_disk_csi_driver_config = bool gcp_filestore_csi_driver_config = bool config_connector_config = bool kalm_config = bool gke_backup_agent_config = bool })">object({…})</code> | | <code title="{ cloudrun_config = false dns_cache_config = false horizontal_pod_autoscaling = true http_load_balancing = true istio_config = { enabled = false tls = false } network_policy_config = false gce_persistent_disk_csi_driver_config = false gcp_filestore_csi_driver_config = false config_connector_config = false kalm_config = false gke_backup_agent_config = false }">{…}</code> |
|
||||
| [authenticator_security_group](variables.tf#L53) | RBAC security group for Google Groups for GKE, format is gke-security-groups@yourdomain.com. | <code>string</code> | | <code>null</code> |
|
||||
| [cluster_autoscaling](variables.tf#L59) | Enable and configure limits for Node Auto-Provisioning with Cluster Autoscaler. | <code title="object({ enabled = bool cpu_min = number cpu_max = number memory_min = number memory_max = number })">object({…})</code> | | <code title="{ enabled = false cpu_min = 0 cpu_max = 0 memory_min = 0 memory_max = 0 }">{…}</code> |
|
||||
|
|
@ -83,29 +83,28 @@ module "cluster-1" {
|
|||
| [description](variables.tf#L97) | Cluster description. | <code>string</code> | | <code>null</code> |
|
||||
| [dns_config](variables.tf#L103) | Configuration for Using Cloud DNS for GKE. | <code title="object({ cluster_dns = string cluster_dns_scope = string cluster_dns_domain = string })">object({…})</code> | | <code>null</code> |
|
||||
| [enable_autopilot](variables.tf#L113) | Create cluster in autopilot mode. With autopilot there's no need to create node-pools and some features are not supported (e.g. setting default_max_pods_per_node). | <code>bool</code> | | <code>false</code> |
|
||||
| [enable_binary_authorization](variables.tf#L119) | Enable Google Binary Authorization. | <code>bool</code> | | <code>null</code> |
|
||||
| [enable_dataplane_v2](variables.tf#L125) | Enable Dataplane V2 on the cluster, will disable network_policy addons config. | <code>bool</code> | | <code>false</code> |
|
||||
| [enable_intranode_visibility](variables.tf#L131) | Enable intra-node visibility to make same node pod to pod traffic visible. | <code>bool</code> | | <code>null</code> |
|
||||
| [enable_l4_ilb_subsetting](variables.tf#L137) | Enable L4ILB Subsetting. | <code>bool</code> | | <code>null</code> |
|
||||
| [enable_shielded_nodes](variables.tf#L143) | Enable Shielded Nodes features on all nodes in this cluster. | <code>bool</code> | | <code>null</code> |
|
||||
| [enable_tpu](variables.tf#L149) | Enable Cloud TPU resources in this cluster. | <code>bool</code> | | <code>null</code> |
|
||||
| [labels](variables.tf#L155) | Cluster resource labels. | <code>map(string)</code> | | <code>null</code> |
|
||||
| [logging_config](variables.tf#L166) | Logging configuration (enabled components). | <code>list(string)</code> | | <code>null</code> |
|
||||
| [logging_service](variables.tf#L172) | Logging service (disable with an empty string). | <code>string</code> | | <code>"logging.googleapis.com/kubernetes"</code> |
|
||||
| [maintenance_config](variables.tf#L178) | Maintenance window configuration. | <code title="object({ daily_maintenance_window = object({ start_time = string }) recurring_window = object({ start_time = string end_time = string recurrence = string }) maintenance_exclusion = list(object({ exclusion_name = string start_time = string end_time = string })) })">object({…})</code> | | <code title="{ daily_maintenance_window = { start_time = "03:00" } recurring_window = null maintenance_exclusion = [] }">{…}</code> |
|
||||
| [master_authorized_ranges](variables.tf#L204) | External Ip address ranges that can access the Kubernetes cluster master through HTTPS. | <code>map(string)</code> | | <code>{}</code> |
|
||||
| [min_master_version](variables.tf#L210) | Minimum version of the master, defaults to the version of the most recent official release. | <code>string</code> | | <code>null</code> |
|
||||
| [monitoring_config](variables.tf#L216) | Monitoring configuration (enabled components). | <code>list(string)</code> | | <code>null</code> |
|
||||
| [monitoring_service](variables.tf#L222) | Monitoring service (disable with an empty string). | <code>string</code> | | <code>"monitoring.googleapis.com/kubernetes"</code> |
|
||||
| [node_locations](variables.tf#L238) | Zones in which the cluster's nodes are located. | <code>list(string)</code> | | <code>[]</code> |
|
||||
| [notification_config](variables.tf#L244) | GKE Cluster upgrade notifications via PubSub. | <code>bool</code> | | <code>false</code> |
|
||||
| [peering_config](variables.tf#L250) | Configure peering with the master VPC for private clusters. | <code title="object({ export_routes = bool import_routes = bool project_id = string })">object({…})</code> | | <code>null</code> |
|
||||
| [pod_security_policy](variables.tf#L260) | Enable the PodSecurityPolicy feature. | <code>bool</code> | | <code>null</code> |
|
||||
| [private_cluster_config](variables.tf#L266) | Enable and configure private cluster, private nodes must be true if used. | <code title="object({ enable_private_nodes = bool enable_private_endpoint = bool master_ipv4_cidr_block = string master_global_access = bool })">object({…})</code> | | <code>null</code> |
|
||||
| [release_channel](variables.tf#L282) | Release channel for GKE upgrades. | <code>string</code> | | <code>null</code> |
|
||||
| [resource_usage_export_config](variables.tf#L288) | Configure the ResourceUsageExportConfig feature. | <code title="object({ enabled = bool dataset = string })">object({…})</code> | | <code title="{ enabled = null dataset = null }">{…}</code> |
|
||||
| [vertical_pod_autoscaling](variables.tf#L315) | Enable the Vertical Pod Autoscaling feature. | <code>bool</code> | | <code>null</code> |
|
||||
| [workload_identity](variables.tf#L321) | Enable the Workload Identity feature. | <code>bool</code> | | <code>true</code> |
|
||||
| [enable_dataplane_v2](variables.tf#L119) | Enable Dataplane V2 on the cluster, will disable network_policy addons config. | <code>bool</code> | | <code>false</code> |
|
||||
| [enable_intranode_visibility](variables.tf#L125) | Enable intra-node visibility to make same node pod to pod traffic visible. | <code>bool</code> | | <code>null</code> |
|
||||
| [enable_l4_ilb_subsetting](variables.tf#L131) | Enable L4ILB Subsetting. | <code>bool</code> | | <code>null</code> |
|
||||
| [enable_shielded_nodes](variables.tf#L137) | Enable Shielded Nodes features on all nodes in this cluster. | <code>bool</code> | | <code>null</code> |
|
||||
| [enable_tpu](variables.tf#L143) | Enable Cloud TPU resources in this cluster. | <code>bool</code> | | <code>null</code> |
|
||||
| [labels](variables.tf#L149) | Cluster resource labels. | <code>map(string)</code> | | <code>null</code> |
|
||||
| [logging_config](variables.tf#L160) | Logging configuration (enabled components). | <code>list(string)</code> | | <code>null</code> |
|
||||
| [logging_service](variables.tf#L166) | Logging service (disable with an empty string). | <code>string</code> | | <code>"logging.googleapis.com/kubernetes"</code> |
|
||||
| [maintenance_config](variables.tf#L172) | Maintenance window configuration. | <code title="object({ daily_maintenance_window = object({ start_time = string }) recurring_window = object({ start_time = string end_time = string recurrence = string }) maintenance_exclusion = list(object({ exclusion_name = string start_time = string end_time = string })) })">object({…})</code> | | <code title="{ daily_maintenance_window = { start_time = "03:00" } recurring_window = null maintenance_exclusion = [] }">{…}</code> |
|
||||
| [master_authorized_ranges](variables.tf#L198) | External Ip address ranges that can access the Kubernetes cluster master through HTTPS. | <code>map(string)</code> | | <code>{}</code> |
|
||||
| [min_master_version](variables.tf#L204) | Minimum version of the master, defaults to the version of the most recent official release. | <code>string</code> | | <code>null</code> |
|
||||
| [monitoring_config](variables.tf#L210) | Monitoring configuration (enabled components). | <code>list(string)</code> | | <code>null</code> |
|
||||
| [monitoring_service](variables.tf#L216) | Monitoring service (disable with an empty string). | <code>string</code> | | <code>"monitoring.googleapis.com/kubernetes"</code> |
|
||||
| [node_locations](variables.tf#L232) | Zones in which the cluster's nodes are located. | <code>list(string)</code> | | <code>[]</code> |
|
||||
| [notification_config](variables.tf#L238) | GKE Cluster upgrade notifications via PubSub. | <code>bool</code> | | <code>false</code> |
|
||||
| [peering_config](variables.tf#L244) | Configure peering with the master VPC for private clusters. | <code title="object({ export_routes = bool import_routes = bool project_id = string })">object({…})</code> | | <code>null</code> |
|
||||
| [pod_security_policy](variables.tf#L254) | Enable the PodSecurityPolicy feature. | <code>bool</code> | | <code>null</code> |
|
||||
| [private_cluster_config](variables.tf#L260) | Enable and configure private cluster, private nodes must be true if used. | <code title="object({ enable_private_nodes = bool enable_private_endpoint = bool master_ipv4_cidr_block = string master_global_access = bool })">object({…})</code> | | <code>null</code> |
|
||||
| [release_channel](variables.tf#L276) | Release channel for GKE upgrades. | <code>string</code> | | <code>null</code> |
|
||||
| [resource_usage_export_config](variables.tf#L282) | Configure the ResourceUsageExportConfig feature. | <code title="object({ enabled = bool dataset = string })">object({…})</code> | | <code title="{ enabled = null dataset = null }">{…}</code> |
|
||||
| [vertical_pod_autoscaling](variables.tf#L309) | Enable the Vertical Pod Autoscaling feature. | <code>bool</code> | | <code>null</code> |
|
||||
| [workload_identity](variables.tf#L315) | Enable the Workload Identity feature. | <code>bool</code> | | <code>true</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
|
|
|
|||
|
|
@ -43,7 +43,6 @@ resource "google_container_cluster" "cluster" {
|
|||
monitoring_service = var.monitoring_config == null ? var.monitoring_service : null
|
||||
resource_labels = var.labels
|
||||
default_max_pods_per_node = var.enable_autopilot ? null : var.default_max_pods_per_node
|
||||
enable_binary_authorization = var.enable_binary_authorization
|
||||
enable_intranode_visibility = var.enable_intranode_visibility
|
||||
enable_l4_ilb_subsetting = var.enable_l4_ilb_subsetting
|
||||
enable_shielded_nodes = var.enable_shielded_nodes
|
||||
|
|
|
|||
|
|
@ -116,12 +116,6 @@ variable "enable_autopilot" {
|
|||
default = false
|
||||
}
|
||||
|
||||
variable "enable_binary_authorization" {
|
||||
description = "Enable Google Binary Authorization."
|
||||
type = bool
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "enable_dataplane_v2" {
|
||||
description = "Enable Dataplane V2 on the cluster, will disable network_policy addons config."
|
||||
type = bool
|
||||
|
|
|
|||
|
|
@ -22,10 +22,10 @@ module "project" {
|
|||
name = "gkehub-test"
|
||||
parent = "folders/12345"
|
||||
services = [
|
||||
"container.googleapis.com",
|
||||
"gkehub.googleapis.com",
|
||||
"gkeconnect.googleapis.com",
|
||||
"anthosconfigmanagement.googleapis.com",
|
||||
"container.googleapis.com",
|
||||
"gkeconnect.googleapis.com",
|
||||
"gkehub.googleapis.com",
|
||||
"multiclusteringress.googleapis.com",
|
||||
"multiclusterservicediscovery.googleapis.com",
|
||||
"mesh.googleapis.com"
|
||||
|
|
|
|||
|
|
@ -84,7 +84,7 @@ resource "google_container_node_pool" "nodepool" {
|
|||
location = var.location
|
||||
name = var.name
|
||||
|
||||
initial_node_count = var.initial_node_count
|
||||
initial_node_count = var.node_count == null ? var.initial_node_count : null // (dmarzi) TOFIX
|
||||
max_pods_per_node = var.max_pods_per_node
|
||||
node_count = var.autoscaling_config == null ? var.node_count : null
|
||||
node_locations = var.node_locations
|
||||
|
|
|
|||
|
|
@ -40,7 +40,9 @@ locals {
|
|||
fleet = "service-%s@gcp-sa-gkehub"
|
||||
gae-flex = "service-%s@gae-api-prod"
|
||||
# TODO: deprecate gcf
|
||||
gcf = "service-%s@gcf-admin-robot"
|
||||
gcf = "service-%s@gcf-admin-robot"
|
||||
# TODO: jit?
|
||||
gke-mcs = "service-%s@gcp-sa-mcsd"
|
||||
monitoring-notifications = "service-%s@gcp-sa-monitoring-notification"
|
||||
pubsub = "service-%s@gcp-sa-pubsub"
|
||||
secretmanager = "service-%s@gcp-sa-secretmanager"
|
||||
|
|
@ -55,10 +57,15 @@ locals {
|
|||
service_account_cloud_services = (
|
||||
"${local.project.number}@cloudservices.gserviceaccount.com"
|
||||
)
|
||||
service_accounts_robots = {
|
||||
for k, v in local._service_accounts_robot_services :
|
||||
k => "${format(v, local.project.number)}.iam.gserviceaccount.com"
|
||||
}
|
||||
service_accounts_robots = merge(
|
||||
{
|
||||
for k, v in local._service_accounts_robot_services :
|
||||
k => "${format(v, local.project.number)}.iam.gserviceaccount.com"
|
||||
},
|
||||
{
|
||||
gke-mcs-importer = "${local.project.project_id}.svc.id.goog[gke-mcs/gke-mcs-importer]"
|
||||
}
|
||||
)
|
||||
service_accounts_jit_services = [
|
||||
"cloudasset.googleapis.com",
|
||||
"gkehub.googleapis.com",
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ variable "folder_id" {
|
|||
}
|
||||
|
||||
variable "project_id" {
|
||||
default = "projects/project-id"
|
||||
default = "project-id"
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
|
|
|
|||
|
|
@ -35,6 +35,8 @@ module "stage" {
|
|||
service_accounts = {
|
||||
data-platform-dev = "string"
|
||||
data-platform-prod = "string"
|
||||
gke-dev = "string"
|
||||
gke-prod = "string"
|
||||
project-factory-dev = "string"
|
||||
project-factory-prod = "string"
|
||||
}
|
||||
|
|
|
|||
|
|
@ -40,6 +40,8 @@ module "stage" {
|
|||
service_accounts = {
|
||||
data-platform-dev = "string"
|
||||
data-platform-prod = "string"
|
||||
gke-dev = "string"
|
||||
gke-prod = "string"
|
||||
project-factory-dev = "string"
|
||||
project-factory-prod = "string"
|
||||
}
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@ def test_vpn_peering_parity(e2e_plan_runner):
|
|||
for VPN and VPC peering resources'''
|
||||
_, plan_peering = e2e_plan_runner(fixture_path=FIXTURE_PEERING)
|
||||
_, plan_vpn = e2e_plan_runner(fixture_path=FIXTURE_VPN)
|
||||
|
||||
ddiff = DeepDiff(plan_vpn, plan_peering, ignore_order=True,
|
||||
group_by='address', view='tree')
|
||||
|
||||
|
|
|
|||
|
|
@ -40,6 +40,8 @@ module "stage" {
|
|||
service_accounts = {
|
||||
data-platform-dev = "string"
|
||||
data-platform-prod = "string"
|
||||
gke-dev = "string"
|
||||
gke-prod = "string"
|
||||
project-factory-dev = "string"
|
||||
project-factory-prod = "string"
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in New Issue