zecfoundation
/
cloud-foundation-fabric
mirror of https://github.com/ZcashFoundation/cloud-foundation-fabric.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add support for internal service account to GKE nodepool module (#156) 

* add support for internal service account to GKE nodepool module, fixes #62

* update shared vpc example to use internally managed service account

* update shared vpc example to use internally managed service account

* update hub and spoke peering example to use renamed gke nodepool variables
This commit is contained in:
Ludovico Magnocavallo 2020-11-07 10:48:12 +01:00 committed by GitHub
parent c31764fa7e
commit b3ae7c9454
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
13 changed files with 284 additions and 82 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

53
modules/gke-nodepool/README.md
View File

@ -4,6 +4,10 @@ This module allows simplified creation and management of individual GKE nodepool
## Example usage
### Module defaults
If no specific node configuration is set via variables, the module uses the provider's defaults only setting OAuth scopes to a minimal working set (devstorage read-only, logging and monitoring write) and the node machine type to `n1-standard-1`. The service account set by the provider in this case is the GCE default service account.
```hcl
module "cluster-1-nodepool-1" {
source = "../modules/gke-nodepool"
@ -14,6 +18,21 @@ module "cluster-1-nodepool-1" {
}
```
### Internally managed service account
To have the module auto-create a service account for the nodes, set the `node_service_account_create` variable to `true`. When a service account is created by the module, OAuth scopes are set to `cloud-platform` by default. The service account resource and email (in both plain and IAM formats) are then available in outputs to assign IAM roles from your own code.
```hcl
module "cluster-1-nodepool-1" {
source = "../modules/gke-nodepool"
project_id = "myproject"
cluster_name = "cluster-1"
location = "europe-west1-b"
name = "nodepool-1"
node_service_account_create = true
}
```
<!-- BEGIN TFDOC -->
## Variables
@ -28,23 +47,24 @@ module "cluster-1-nodepool-1" {
| *management_config* | Optional node management configuration. | <code title="object&#40;&#123;&#10;auto_repair &#61; bool&#10;auto_upgrade &#61; bool&#10;&#125;&#41;">object({...})</code> | | <code title="">null</code> |
| *max_pods_per_node* | Maximum number of pods per node. | <code title="">number</code> | | <code title="">null</code> |
| *name* | Optional nodepool name. | <code title="">string</code> | | <code title="">null</code> |
| *node_config_disk_size* | Node disk size, defaults to 100GB. | <code title="">number</code> | | <code title="">100</code> |
| *node_config_disk_type* | Node disk type, defaults to pd-standard. | <code title="">string</code> | | <code title="">pd-standard</code> |
| *node_config_guest_accelerator* | Map of type and count of attached accelerator cards. | <code title="map&#40;number&#41;">map(number)</code> | | <code title="">{}</code> |
| *node_config_image_type* | Nodes image type. | <code title="">string</code> | | <code title="">null</code> |
| *node_config_labels* | Kubernetes labels attached to nodes. | <code title="map&#40;string&#41;">map(string)</code> | | <code title="">{}</code> |
| *node_config_local_ssd_count* | Number of local SSDs attached to nodes. | <code title="">number</code> | | <code title="">0</code> |
| *node_config_machine_type* | Nodes machine type. | <code title="">string</code> | | <code title="">n1-standard-1</code> |
| *node_config_metadata* | Metadata key/value pairs assigned to nodes. Set disable-legacy-endpoints to true when using this variable. | <code title="map&#40;string&#41;">map(string)</code> | | <code title="">null</code> |
| *node_config_min_cpu_platform* | Minimum CPU platform for nodes. | <code title="">string</code> | | <code title="">null</code> |
| *node_config_oauth_scopes* | Set of Google API scopes for the nodes service account. Include logging-write, monitoring, and storage-ro when using this variable. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">["logging-write", "monitoring", "monitoring-write", "storage-ro"]</code> |
| *node_config_preemptible* | Use preemptible VMs for nodes. | <code title="">bool</code> | | <code title="">null</code> |
| *node_config_sandbox_config* | GKE Sandbox configuration. Needs image_type set to COS_CONTAINERD and node_version set to 1.12.7-gke.17 when using this variable. | <code title="">string</code> | | <code title="">null</code> |
| *node_config_service_account* | Service account used for nodes. | <code title="">string</code> | | <code title="">null</code> |
| *node_config_shielded_instance_config* | Shielded instance options. | <code title="object&#40;&#123;&#10;enable_secure_boot &#61; bool&#10;enable_integrity_monitoring &#61; bool&#10;&#125;&#41;">object({...})</code> | | <code title="">null</code> |
| *node_config_tags* | Network tags applied to nodes. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">null</code> |
| *node_count* | Number of nodes per instance group, can be updated after creation. Ignored when autoscaling is set. | <code title="">number</code> | | <code title="">null</code> |
| *node_disk_size* | Node disk size, defaults to 100GB. | <code title="">number</code> | | <code title="">100</code> |
| *node_disk_type* | Node disk type, defaults to pd-standard. | <code title="">string</code> | | <code title="">pd-standard</code> |
| *node_guest_accelerator* | Map of type and count of attached accelerator cards. | <code title="map&#40;number&#41;">map(number)</code> | | <code title="">{}</code> |
| *node_image_type* | Nodes image type. | <code title="">string</code> | | <code title="">null</code> |
| *node_labels* | Kubernetes labels attached to nodes. | <code title="map&#40;string&#41;">map(string)</code> | | <code title="">{}</code> |
| *node_local_ssd_count* | Number of local SSDs attached to nodes. | <code title="">number</code> | | <code title="">0</code> |
| *node_locations* | Optional list of zones in which nodes should be located. Uses cluster locations if unset. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">null</code> |
| *node_machine_type* | Nodes machine type. | <code title="">string</code> | | <code title="">n1-standard-1</code> |
| *node_metadata* | Metadata key/value pairs assigned to nodes. Set disable-legacy-endpoints to true when using this variable. | <code title="map&#40;string&#41;">map(string)</code> | | <code title="">null</code> |
| *node_min_cpu_platform* | Minimum CPU platform for nodes. | <code title="">string</code> | | <code title="">null</code> |
| *node_preemptible* | Use preemptible VMs for nodes. | <code title="">bool</code> | | <code title="">null</code> |
| *node_sandbox_config* | GKE Sandbox configuration. Needs image_type set to COS_CONTAINERD and node_version set to 1.12.7-gke.17 when using this variable. | <code title="">string</code> | | <code title="">null</code> |
| *node_service_account* | Service account email. Unused if service account is auto-created. | <code title="">string</code> | | <code title="">null</code> |
| *node_service_account_create* | Auto-create service account. | <code title="">bool</code> | | <code title="">false</code> |
| *node_service_account_scopes* | Scopes applied to service account. Default to: 'cloud-platform' when creating a service account; 'devstorage.read_only', 'logging.write', 'monitoring.write' otherwise. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">[]</code> |
| *node_shielded_instance_config* | Shielded instance options. | <code title="object&#40;&#123;&#10;enable_secure_boot &#61; bool&#10;enable_integrity_monitoring &#61; bool&#10;&#125;&#41;">object({...})</code> | | <code title="">null</code> |
| *node_tags* | Network tags applied to nodes. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">null</code> |
| *upgrade_config* | Optional node upgrade configuration. | <code title="object&#40;&#123;&#10;max_surge &#61; number&#10;max_unavailable &#61; number&#10;&#125;&#41;">object({...})</code> | | <code title="">null</code> |
| *workload_metadata_config* | Metadata configuration to expose to workloads on the node pool. | <code title="">string</code> | | <code title="">GKE_METADATA_SERVER</code> |
@ -53,4 +73,7 @@ module "cluster-1-nodepool-1" {
| name | description | sensitive |
|---|---|:---:|
| name | Nodepool name. | |
| service_account | Service account resource. | |
| service_account_email | Service account email. | |
| service_account_iam_email | Service account email. | |
<!-- END TFDOC -->

66
modules/gke-nodepool/main.tf
View File

@ -14,6 +14,38 @@
* limitations under the License.
*/
locals {
service_account_email = (
var.node_service_account_create
? (
length(google_service_account.service_account) > 0
? google_service_account.service_account[0].email
: null
)
: var.node_service_account
)
service_account_scopes = (
length(var.node_service_account_scopes) > 0
? var.node_service_account_scopes
: (
var.node_service_account_create
? ["https://www.googleapis.com/auth/cloud-platform"]
: [
"https://www.googleapis.com/auth/devstorage.read_only",
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring.write"
]
)
)
}
resource "google_service_account" "service_account" {
count = var.node_service_account_create ? 1 : 0
project = var.project_id
account_id = "tf-gke-${var.cluster_name}-${var.name}"
display_name = "Terraform GKE ${var.cluster_name} ${var.name}."
}
resource "google_container_node_pool" "nodepool" {
provider = google-beta
@ -29,21 +61,21 @@ resource "google_container_node_pool" "nodepool" {
version = var.gke_version
node_config {
disk_size_gb = var.node_config_disk_size
disk_type = var.node_config_disk_type
image_type = var.node_config_image_type
labels = var.node_config_labels
local_ssd_count = var.node_config_local_ssd_count
machine_type = var.node_config_machine_type
metadata = var.node_config_metadata
min_cpu_platform = var.node_config_min_cpu_platform
oauth_scopes = var.node_config_oauth_scopes
preemptible = var.node_config_preemptible
service_account = var.node_config_service_account
tags = var.node_config_tags
disk_size_gb = var.node_disk_size
disk_type = var.node_disk_type
image_type = var.node_image_type
labels = var.node_labels
local_ssd_count = var.node_local_ssd_count
machine_type = var.node_machine_type
metadata = var.node_metadata
min_cpu_platform = var.node_min_cpu_platform
oauth_scopes = local.service_account_scopes
preemptible = var.node_preemptible
service_account = local.service_account_email
tags = var.node_tags
dynamic guest_accelerator {
for_each = var.node_config_guest_accelerator
for_each = var.node_guest_accelerator
iterator = config
content {
type = config.key
@ -53,8 +85,8 @@ resource "google_container_node_pool" "nodepool" {
dynamic sandbox_config {
for_each = (
var.node_config_sandbox_config != null
? [var.node_config_sandbox_config]
var.node_sandbox_config != null
? [var.node_sandbox_config]
: []
)
iterator = config
@ -65,8 +97,8 @@ resource "google_container_node_pool" "nodepool" {
dynamic shielded_instance_config {
for_each = (
var.node_config_shielded_instance_config != null
? [var.node_config_shielded_instance_config]
var.node_shielded_instance_config != null
? [var.node_shielded_instance_config]
: []
)
iterator = config

22
modules/gke-nodepool/outputs.tf
View File

@ -18,3 +18,25 @@ output "name" {
description = "Nodepool name."
value = google_container_node_pool.nodepool.name
}
output "service_account" {
description = "Service account resource."
value = (
var.node_service_account_create
? google_service_account.service_account[0]
: null
)
}
output "service_account_email" {
description = "Service account email."
value = local.service_account_email
}
output "service_account_iam_email" {
description = "Service account email."
value = join("", [
"serviceAccount:",
local.service_account_email == null ? "" : local.service_account_email
])
}

52
modules/gke-nodepool/variables.tf
View File

@ -66,85 +66,93 @@ variable "name" {
default = null
}
variable "node_config_disk_size" {
variable "node_disk_size" {
description = "Node disk size, defaults to 100GB."
type = number
default = 100
}
variable "node_config_disk_type" {
variable "node_disk_type" {
description = "Node disk type, defaults to pd-standard."
type = string
default = "pd-standard"
}
variable "node_config_guest_accelerator" {
variable "node_guest_accelerator" {
description = "Map of type and count of attached accelerator cards."
type = map(number)
default = {}
}
variable "node_config_image_type" {
variable "node_image_type" {
description = "Nodes image type."
type = string
default = null
}
variable "node_config_labels" {
variable "node_labels" {
description = "Kubernetes labels attached to nodes."
type = map(string)
default = {}
}
variable "node_config_local_ssd_count" {
variable "node_local_ssd_count" {
description = "Number of local SSDs attached to nodes."
type = number
default = 0
}
variable "node_config_machine_type" {
variable "node_machine_type" {
description = "Nodes machine type."
type = string
default = "n1-standard-1"
}
variable "node_config_metadata" {
variable "node_metadata" {
description = "Metadata key/value pairs assigned to nodes. Set disable-legacy-endpoints to true when using this variable."
type = map(string)
default = null
}
variable "node_config_min_cpu_platform" {
variable "node_min_cpu_platform" {
description = "Minimum CPU platform for nodes."
type = string
default = null
}
variable "node_config_oauth_scopes" {
description = "Set of Google API scopes for the nodes service account. Include logging-write, monitoring, and storage-ro when using this variable."
type = list(string)
default = ["logging-write", "monitoring", "monitoring-write", "storage-ro"]
}
variable "node_config_preemptible" {
variable "node_preemptible" {
description = "Use preemptible VMs for nodes."
type = bool
default = null
}
variable "node_config_sandbox_config" {
variable "node_sandbox_config" {
description = "GKE Sandbox configuration. Needs image_type set to COS_CONTAINERD and node_version set to 1.12.7-gke.17 when using this variable."
type = string
default = null
}
variable "node_config_service_account" {
description = "Service account used for nodes."
variable "node_service_account" {
description = "Service account email. Unused if service account is auto-created."
type = string
default = null
}
variable "node_config_shielded_instance_config" {
variable "node_service_account_create" {
description = "Auto-create service account."
type = bool
default = false
}
# scopes and scope aliases list
# https://cloud.google.com/sdk/gcloud/reference/compute/instances/create#--scopes
variable "node_service_account_scopes" {
description = "Scopes applied to service account. Default to: 'cloud-platform' when creating a service account; 'devstorage.read_only', 'logging.write', 'monitoring.write' otherwise."
type = list(string)
default = []
}
variable "node_shielded_instance_config" {
description = "Shielded instance options."
type = object({
enable_secure_boot = bool
@ -153,13 +161,13 @@ variable "node_config_shielded_instance_config" {
default = null
}
variable "node_config_tags" {
variable "node_tags" {
description = "Network tags applied to nodes."
type = list(string)
default = null
}
# variable "node_config_taint" {
# variable "node_taint" {
# description = "Kubernetes taints applied to nodes."
# type = string
# default = null

12
networking/hub-and-spoke-peering/main.tf
View File

@ -220,12 +220,12 @@ module "cluster-1" {
}
module "cluster-1-nodepool-1" {
source = "../../modules/gke-nodepool"
name = "nodepool-1"
project_id = var.project_id
location = module.cluster-1.location
cluster_name = module.cluster-1.name
node_config_service_account = module.service-account-gke-node.email
source = "../../modules/gke-nodepool"
name = "nodepool-1"
project_id = var.project_id
location = module.cluster-1.location
cluster_name = module.cluster-1.name
node_service_account = module.service-account-gke-node.email
}
# roles assigned via this module use non-authoritative IAM bindings at the

1
networking/shared-vpc-gke/README.md
View File

@ -39,7 +39,6 @@ There's a minor glitch that can surface running `terraform destroy`, where the s
|---|---|:---:|
| gke_clusters | GKE clusters information. | |
| projects | Project ids. | |
| service_accounts | GCE and GKE service accounts. | |
| vms | GCE VMs. | |
| vpc | Shared VPC. | |
<!-- END TFDOC -->

15
networking/shared-vpc-gke/main.tf
View File

@ -74,8 +74,8 @@ module "project-svc-gke" {
}
iam = {
"roles/container.developer" = [module.vm-bastion.service_account_iam_email],
"roles/logging.logWriter" = [module.service-account-gke-node.iam_email],
"roles/monitoring.metricWriter" = [module.service-account-gke-node.iam_email],
"roles/logging.logWriter" = [module.cluster-1-nodepool-1.service_account_iam_email],
"roles/monitoring.metricWriter" = [module.cluster-1-nodepool-1.service_account_iam_email],
"roles/owner" = var.owners_gke
}
}
@ -220,14 +220,5 @@ module "cluster-1-nodepool-1" {
project_id = module.project-svc-gke.project_id
location = module.cluster-1.location
cluster_name = module.cluster-1.name
node_config_service_account = module.service-account-gke-node.email
}
# roles assigned via this module use non-authoritative IAM bindings at the
# project level, with no risk of conflicts with pre-existing roles
module "service-account-gke-node" {
source = "../../modules/iam-service-account"
project_id = module.project-svc-gke.project_id
name = "gke-node"
node_service_account_create = true
}

8
networking/shared-vpc-gke/outputs.tf
View File

@ -28,14 +28,6 @@ output "projects" {
}
}
output "service_accounts" {
description = "GCE and GKE service accounts."
value = {
bastion = module.vm-bastion.service_account_email
gke_node = module.service-account-gke-node.email
}
}
output "vpc" {
description = "Shared VPC."
value = {

13
tests/modules/gke_nodepool/__init__.py Normal file
View File

@ -0,0 +1,13 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

26
tests/modules/gke_nodepool/fixture/main.tf Normal file
View File

@ -0,0 +1,26 @@
/**
* Copyright 2020 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
module "test" {
source = "../../../../modules/gke-nodepool"
project_id = "my-project"
cluster_name = "cluster-1"
location = "europe-west1-b"
name = "nodepool-1"
node_service_account = var.node_service_account
node_service_account_create = var.node_service_account_create
node_service_account_scopes = var.node_service_account_scopes
}

32
tests/modules/gke_nodepool/fixture/variables.tf Normal file
View File

@ -0,0 +1,32 @@
/**
* Copyright 2020 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
variable "node_service_account" {
type = string
default = null
}
variable "node_service_account_create" {
type = bool
default = false
}
# scopes and scope aliases list
# https://cloud.google.com/sdk/gcloud/reference/compute/instances/create#--scopes
variable "node_service_account_scopes" {
type = list(string)
default = []
}

64
tests/modules/gke_nodepool/test_plan.py Normal file
View File

@ -0,0 +1,64 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import pytest
FIXTURES_DIR = os.path.join(os.path.dirname(__file__), 'fixture')
OAUTH_SCOPE = ['https://www.googleapis.com/auth/cloud-platform']
OAUTH_SCOPES = [
'https://www.googleapis.com/auth/devstorage.read_only',
'https://www.googleapis.com/auth/logging.write',
'https://www.googleapis.com/auth/monitoring.write']
def test_defaults(plan_runner):
"Test resources created with variable defaults."
_, resources = plan_runner(FIXTURES_DIR)
assert len(resources) == 1
node_config = resources[0]['values']['node_config'][0]
assert node_config['oauth_scopes'] == OAUTH_SCOPES
assert 'service_account' not in node_config
def test_external_sa(plan_runner):
"Test resources created with externally managed sa."
_, resources = plan_runner(
FIXTURES_DIR, node_service_account='foo@example.org')
assert len(resources) == 1
node_config = resources[0]['values']['node_config'][0]
assert node_config['oauth_scopes'] == OAUTH_SCOPES
assert node_config['service_account'] == 'foo@example.org'
def test_external_scopes(plan_runner):
"Test resources created with externally defined scopes."
oauth_scopes = '["https://www.googleapis.com/auth/cloud-platform"]'
_, resources = plan_runner(
FIXTURES_DIR, node_service_account_scopes=oauth_scopes)
assert len(resources) == 1
node_config = resources[0]['values']['node_config'][0]
assert node_config['oauth_scopes'] == OAUTH_SCOPE
assert 'service_account' not in node_config
def test_internal_sa(plan_runner):
"Test resources created with internally managed sa."
_, resources = plan_runner(FIXTURES_DIR, node_service_account_create='true')
assert len(resources) == 2
node_config = resources[0]['values']['node_config'][0]
assert node_config['oauth_scopes'] == OAUTH_SCOPE
assert 'service_account' not in node_config

2
tests/networking/shared_vpc_gke/test_plan.py
View File

@ -23,5 +23,5 @@ FIXTURES_DIR = os.path.join(os.path.dirname(__file__), 'fixture')
def test_resources(e2e_plan_runner):
"Test that plan works and the numbers of resources is as expected."
modules, resources = e2e_plan_runner(FIXTURES_DIR)
assert len(modules) == 11
assert len(modules) == 10
assert len(resources) == 43