zecfoundation
/
cloud-foundation-fabric
mirror of https://github.com/ZcashFoundation/cloud-foundation-fabric.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into serverless-program

This commit is contained in:
Ludovico Magnocavallo 2023-02-25 07:41:10 +01:00 committed by GitHub
parent 5f9955b5fe df9e332b9c
commit b1e37f630c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
10 changed files with 540 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
blueprints/networking/README.md
View File

@ -82,3 +82,11 @@ The emulated on-premises environment can be used to test access to different ser
It is meant to be used as a starting point for most Shared VPC configurations, and to be integrated to the above blueprints where Shared VPC is needed in more complex network topologies.
<br clear="left">
### Exposing applications to the internet via GCLB and Private Service Connect
<a href="./psc-glb-and-armor/" title="Shared VPC with GKE"><img src="./psc-glb-and-armor/diagram.png" align="left" width="280px"></a> This [blueprint](./psc-glb-and-armor/) shows how to configure an external Google Cloud Load Balancer, with a simple Cloud Armor rule to protect against DDOS attacks, to provide an external endpoint to an application provided by another team via Private Service Connect (PSC).
It is meant to be used as a starting point for users that want to explore PSC to reduce some of the complexity in their network setup.
<br clear="left">

127
blueprints/networking/psc-glb-and-armor/README.md Normal file
View File

@ -0,0 +1,127 @@
# HTTP Load Balancer with Cloud Armor and Private Service Connect
## Introduction
This blueprint contains all necessary Terraform code to configure HTTP load balancing and Google’s advanced WAF security tool (Cloud Armor) on top to securely deploy an application, provided by another team.
This tutorial is general enough to fit in a variety of use-cases, from hosting a mobile app's backend to deploy proprietary workloads at scale.
## Architecture
<p align="center"> <img src="diagram.png" width="700"> </p>
The main components that we would be setting up are (to learn more about these products, click on the hyperlinks):
* [Cloud Armor](https://cloud.google.com/armor) - Google Cloud Armor is the web-application firewall (WAF) and DDoS mitigation service that helps users defend their web apps and services at Google scale at the edge of Google’s network.
* [Cloud Load Balancer](https://cloud.google.com/load-balancing) - When your app usage spikes, it is important to scale, optimize and secure the app. Cloud Load Balancing is a fully distributed solution that balances user traffic to multiple backends to avoid congestion, reduce latency and increase security. Some important features it offers that we use here are:
* Single global anycast IP and autoscaling - CLB acts as a frontend to all your backend instances across all regions. It provides cross-region load balancing, automatic multi-region failover and scales to support increase in resources.
* Global Forwarding Rule - To route traffic to different regions, global load balancers use global forwarding rules, which bind the global IP address and a single target proxy.
* Target Proxy - For external HTTP(S) load balancers, proxies route incoming requests to a URL map. This is essentially how you can handle the connections.
* URL Map - URL Maps are used to route requests to a backend service based on the rules that you define for the host and path of an incoming URL.
* Backend Service - A Backend Service defines CLB distributes traffic. The backend service configuration consists of a set of values - protocols to connect to backends, session settings, health checks and timeouts.
* Health Check - Health check is a method provided to determine if the corresponding backends respond to traffic. Health checks connect to backends on a configurable, periodic basis. Each connection attempt is called a probe. Google Cloud records the success or failure of each probe.
* [Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect) - Private Service Connect. Private Service Connect allows private consumption of services across VPC networks that belong to different groups, teams, projects, or organizations. You can publish and consume services using IP addresses that you define and that are internal to your VPC network. No VPC peering or VPN is needed between the consumer and produce of the service.
* [Cloud Run](https://cloud.google.com/run) - Cloud Run is a fully managed container runtime on Google Cloud. In this blueprint it's used to host [httpbin](http://httpbin.org/), which is a simple HTTP server application. The setup the blueprint showcases will also work with GCE or GKE, but might need small adjustments.
This architecture is ideal, if the external Load Balancer and the application are managed by different teams. The application team configures the application using the Internal Load Balancer, they have all the freedom how they evolve and change their implementation. The load balancer team, configures the load balancer and assures security using Cloud Armor.
In a real world implementation, the IaC code might be split into two separate repositories, and the application team raises changes to the load balancer using a pull request. Also the service attachment in the producer project is configured to allow all connections via `connection_preference = "ACCEPT_AUTOMATIC"` in a real world setup you would want to use a manual white listing instead.
The Terraform IaC also deploys a VM to the producer VPC network, this VM is only needed for the internal Layer 7 Load Balancer to properly work with Cloud Run, but has no logical function in the setup, no traffic is routed via this machine.
## Setup
This solution assumes you already have two projects created and set up where you wish to host these resources. If not, and you would like for the projects to create a new project as well, please refer to the [github repository](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/tree/master/blueprints/data-solutions/gcs-to-bq-with-least-privileges) for instructions.
### Prerequisites
* Have an [organization](https://cloud.google.com/resource-manager/docs/creating-managing-organization) set up in Google cloud.
* Have a [billing account](https://cloud.google.com/billing/docs/how-to/manage-billing-account) set up.
* Have two existing [projects](https://cloud.google.com/resource-manager/docs/creating-managing-projects) with [billing enabled](https://cloud.google.com/billing/docs/how-to/modify-project).
### Roles & Permissions
In order to spin up this architecture, you will need to be a user with the “__Project owner__” [IAM](https://cloud.google.com/iam) role on the existing project:
Note: To grant a user a role, take a look at the [Granting and Revoking Access](https://cloud.google.com/iam/docs/granting-changing-revoking-access#grant-single-role) documentation.
### Spinning up the architecture
#### Step 1: Cloning the repository
Click on the button below, sign in if required and when the prompt appears, click on “confirm”.
[![Open Cloudshell](../../../assets/images/cloud-shell-button.png)](https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2Fterraform-google-modules%2Fcloud-foundation-fabric&cloudshell_print=cloud-shell-readme.txt&cloudshell_working_dir=blueprints%2Fnetworking%2Fpsc-glb-and-armor)
This will clone the repository to your cloud shell and a screen like this one will appear:
![cloud_shell](../glb-and-armor/cloud_shell.png)
Before we deploy the architecture, you will need the following information:
* The __project IDs__.
#### Step 2: Deploying the resources
1. After cloning the repo, and going through the prerequisites, head back to the cloud shell editor.
2. Make sure you’re in the following directory. if not, you can change your directory to it via the ‘cd’ command:
cloudshell_open/cloud-foundation-fabric/blueprints/cloud-operations/glb_and_armor
3. Run the following command to initialize the terraform working directory:
terraform init
4. Copy the following command into a console and replace __[consumer-project-id]__ and __[produce-project-id]__ with your project’s IDs. Then run the following command to run the terraform script and create all relevant resources for this architecture:
terraform apply -var consumer_project_id=[consumer-project-id] -var producer_project_id=[producer-project-id]
The resource creation will take a few minutes… but when it’s complete, you should see an output stating the command completed successfully with a list of the created resources.
__Congratulations__! You have successfully deployed an HTTP Load Balancer with Cloud Armor security and a PSC based backend server.
## Testing your architecture
You can simply invoke the service by calling
curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" -H "Content-Type: application/json" http://$LB_IP
## Cleaning up your environment
The easiest way to remove all the deployed resources is to run the following command in Cloud Shell:
terraform destroy
The above command will delete the associated resources so there will be no billable charges made afterwards.
<!-- BEGIN TFDOC -->
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [consumer_project_id](variables.tf#L17) | The consumer project, in which the GCLB and Cloud Armor should be created. | <code>string</code> | ✓ | |
| [prefix](variables.tf#L22) | Prefix used for resource names. | <code>string</code> | ✓ | |
| [producer_project_id](variables.tf#L31) | The producer project, in which the ILB, PSC Service Attachment and Cloud Run service should be created. | <code>string</code> | ✓ | |
| [project_create](variables.tf#L36) | Create project instead of using an existing one. | <code>bool</code> | | <code>false</code> |
| [region](variables.tf#L42) | The GCP region in which the resources should be deployed. | <code>string</code> | | <code>&#34;europe-west1&#34;</code> |
| [zone](variables.tf#L48) | The GCP zone for the VM. | <code>string</code> | | <code>&#34;europe-west1-b&#34;</code> |
## Outputs
| name | description | sensitive |
|---|---|:---:|
| [lb_ip](outputs.tf#L17) | Load balancer IP address. | |
<!-- END TFDOC -->
## Test
```hcl
module "psc-glb-and-armor-test" {
source = "./fabric/blueprints/networking/psc-glb-and-armor"
prefix = "test"
project_create = true
consumer_project_id = "project-1"
producer_project_id = "project-2"
}
# tftest modules=3 resources=31
```

98
blueprints/networking/psc-glb-and-armor/consumer.tf Normal file
View File

@ -0,0 +1,98 @@
/**
* Copyright 2023 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
module "consumer_project" {
source = "../../../modules/project"
name = var.consumer_project_id
project_create = var.project_create
services = [
"iam.googleapis.com",
"compute.googleapis.com",
]
}
resource "google_compute_region_network_endpoint_group" "psc_neg" {
name = "psc-neg"
region = var.region
project = module.consumer_project.project_id
network_endpoint_type = "PRIVATE_SERVICE_CONNECT"
psc_target_service = google_compute_service_attachment.psc_ilb_service_attachment.self_link
network = "default"
subnetwork = "default"
}
resource "google_compute_global_forwarding_rule" "default" {
project = module.consumer_project.project_id
name = "global-rule"
load_balancing_scheme = "EXTERNAL_MANAGED"
target = google_compute_target_http_proxy.default.id
port_range = "80"
}
resource "google_compute_target_http_proxy" "default" {
project = module.consumer_project.project_id
name = "target-proxy"
description = "a description"
url_map = google_compute_url_map.default.id
}
resource "google_compute_url_map" "default" {
project = module.consumer_project.project_id
name = "url-map-target-proxy"
description = "A simple URL Map, routing all traffic to the PSC NEG"
default_service = google_compute_backend_service.default.id
host_rule {
hosts = ["*"]
path_matcher = "allpaths"
}
path_matcher {
name = "allpaths"
default_service = google_compute_backend_service.default.id
path_rule {
paths = ["/*"]
service = google_compute_backend_service.default.id
}
}
}
resource "google_compute_security_policy" "policy" {
provider = google-beta
project = module.consumer_project.project_id
name = "ddos-protection"
adaptive_protection_config {
layer_7_ddos_defense_config {
enable = true
}
}
}
resource "google_compute_backend_service" "default" {
provider = google-beta
project = module.consumer_project.project_id
name = "backend"
load_balancing_scheme = "EXTERNAL_MANAGED"
protocol = "HTTPS"
security_policy = google_compute_security_policy.policy.id
backend {
group = google_compute_region_network_endpoint_group.psc_neg.id
balancing_mode = "UTILIZATION"
capacity_scaler = 1.0
}
}

BIN
blueprints/networking/psc-glb-and-armor/diagram.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 201 KiB

20
blueprints/networking/psc-glb-and-armor/outputs.tf Normal file
View File

@ -0,0 +1,20 @@
/**
* Copyright 2023 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
output "lb_ip" {
description = "Load balancer IP address."
value = google_compute_global_forwarding_rule.default.ip_address
}

231
blueprints/networking/psc-glb-and-armor/producer.tf Normal file
View File

@ -0,0 +1,231 @@
/**
* Copyright 2023 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
module "producer_project" {
source = "../../../modules/project"
name = var.producer_project_id
project_create = var.project_create
services = [
"iam.googleapis.com",
"run.googleapis.com",
"compute.googleapis.com",
]
}
resource "google_service_account" "app" {
project = module.producer_project.project_id
account_id = "example-app"
display_name = "Example App Service Account"
}
resource "google_cloud_run_service" "app" {
name = "example-app"
location = var.region
project = module.producer_project.project_id
template {
spec {
containers {
image = "kennethreitz/httpbin:latest"
ports {
container_port = 80
}
}
service_account_name = google_service_account.app.email
}
}
autogenerate_revision_name = true
traffic {
percent = 100
latest_revision = true
}
metadata {
annotations = {
"run.googleapis.com/ingress" = "internal-and-cloud-load-balancing"
}
}
}
resource "google_compute_region_network_endpoint_group" "neg" {
name = "example-app-neg"
network_endpoint_type = "SERVERLESS"
region = var.region
project = module.producer_project.project_id
cloud_run {
service = google_cloud_run_service.app.name
}
}
resource "google_compute_forwarding_rule" "psc_ilb_target_service" {
name = "producer-forwarding-rule"
region = var.region
project = module.producer_project.project_id
load_balancing_scheme = "INTERNAL_MANAGED"
port_range = "443"
allow_global_access = true
target = google_compute_region_target_https_proxy.default.id
network = google_compute_network.psc_ilb_network.name
subnetwork = google_compute_subnetwork.ilb_subnetwork.name
}
resource "google_compute_region_target_https_proxy" "default" {
name = "l7-ilb-target-http-proxy"
provider = google-beta
region = var.region
project = module.producer_project.project_id
url_map = google_compute_region_url_map.default.id
ssl_certificates = [google_compute_region_ssl_certificate.default.id]
}
resource "google_compute_region_ssl_certificate" "default" {
region = var.region
project = module.producer_project.project_id
name = "my-certificate"
private_key = tls_private_key.example.private_key_pem
certificate = tls_self_signed_cert.example.cert_pem
}
resource "google_compute_region_url_map" "default" {
name = "l7-ilb-regional-url-map"
provider = google-beta
region = var.region
project = module.producer_project.project_id
default_service = google_compute_region_backend_service.producer_service_backend.id
}
resource "tls_private_key" "example" {
algorithm = "RSA"
rsa_bits = 2048
}
resource "tls_self_signed_cert" "example" {
private_key_pem = tls_private_key.example.private_key_pem
subject {
common_name = "app.example.com"
organization = "Org"
}
validity_period_hours = 12
allowed_uses = [
"key_encipherment",
"digital_signature",
"server_auth",
]
}
resource "google_compute_region_backend_service" "producer_service_backend" {
name = "producer-service"
region = var.region
project = module.producer_project.project_id
load_balancing_scheme = "INTERNAL_MANAGED"
protocol = "HTTPS"
backend {
group = google_compute_region_network_endpoint_group.neg.id
balancing_mode = "UTILIZATION"
capacity_scaler = 1.0
}
}
resource "google_compute_network" "psc_ilb_network" {
name = "psc-ilb-network"
auto_create_subnetworks = false
project = module.producer_project.project_id
}
resource "google_compute_subnetwork" "ilb_subnetwork" {
name = "ilb-subnetwork"
region = var.region
project = module.producer_project.project_id
network = google_compute_network.psc_ilb_network.id
ip_cidr_range = "10.0.0.0/16"
purpose = "INTERNAL_HTTPS_LOAD_BALANCER"
role = "ACTIVE"
}
resource "google_compute_subnetwork" "psc_private_subnetwork" {
name = "psc-private-subnetwork"
region = var.region
project = module.producer_project.project_id
network = google_compute_network.psc_ilb_network.id
ip_cidr_range = "10.3.0.0/16"
purpose = "PRIVATE"
role = "ACTIVE"
}
resource "google_compute_subnetwork" "psc_ilb_nat" {
name = "psc-ilb-nat"
region = var.region
project = module.producer_project.project_id
network = google_compute_network.psc_ilb_network.id
purpose = "PRIVATE_SERVICE_CONNECT"
ip_cidr_range = "10.1.0.0/16"
}
resource "google_compute_subnetwork" "vms" {
name = "vms"
region = var.region
project = module.producer_project.project_id
network = google_compute_network.psc_ilb_network.id
ip_cidr_range = "10.4.0.0/16"
}
resource "google_compute_service_attachment" "psc_ilb_service_attachment" {
name = "my-psc-ilb"
region = var.region
project = module.producer_project.project_id
description = "A service attachment configured with Terraform"
enable_proxy_protocol = false
connection_preference = "ACCEPT_AUTOMATIC"
nat_subnets = [google_compute_subnetwork.psc_ilb_nat.id]
target_service = google_compute_forwarding_rule.psc_ilb_target_service.id
}
resource "google_service_account" "noop" {
project = module.producer_project.project_id
account_id = "noop-sa"
display_name = "Service Account for NOOP VM"
}
resource "google_compute_instance" "noop-vm" {
project = module.producer_project.project_id
name = "noop-ilb-vm"
machine_type = "e2-medium"
zone = var.zone
boot_disk {
initialize_params {
image = "debian-cloud/debian-11"
}
}
network_interface {
network = google_compute_network.psc_ilb_network.id
subnetwork = google_compute_subnetwork.vms.id
}
service_account {
email = google_service_account.noop.email
scopes = []
}
}

52
blueprints/networking/psc-glb-and-armor/variables.tf Normal file
View File

@ -0,0 +1,52 @@
/**
* Copyright 2023 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
variable "consumer_project_id" {
description = "The consumer project, in which the GCLB and Cloud Armor should be created."
type = string
}
variable "prefix" {
description = "Prefix used for resource names."
type = string
validation {
condition = var.prefix != ""
error_message = "Prefix cannot be empty."
}
}
variable "producer_project_id" {
description = "The producer project, in which the ILB, PSC Service Attachment and Cloud Run service should be created."
type = string
}
variable "project_create" {
description = "Create project instead of using an existing one."
type = bool
default = false
}
variable "region" {
description = "The GCP region in which the resources should be deployed."
type = string
default = "europe-west1"
}
variable "zone" {
description = "The GCP zone for the VM."
type = string
default = "europe-west1-b"
}

2
modules/gke-hub/README.md
View File

@ -314,7 +314,7 @@ module "hub" {
]
}
# tftest modules=8 resources=31
# tftest modules=8 resources=32
```
<!-- BEGIN TFDOC -->

1
modules/project/README.md
View File

@ -189,6 +189,7 @@ This table lists all affected services and roles that you need to grant to servi
| cloudasset.googleapis.com | cloudasset | roles/cloudasset.serviceAgent |
| cloudbuild.googleapis.com | cloudbuild | roles/cloudbuild.builds.builder |
| gkehub.googleapis.com | fleet | roles/gkehub.serviceAgent |
| meshconfig.googleapis.com | servicemesh | roles/anthosservicemesh.serviceAgent |
| multiclusteringress.googleapis.com | multicluster-ingress | roles/multiclusteringress.serviceAgent |
| pubsub.googleapis.com | pubsub | roles/pubsub.serviceAgent |
| sqladmin.googleapis.com | sqladmin | roles/cloudsql.serviceAgent |

2
modules/project/service-accounts.tf
View File

@ -50,6 +50,7 @@ locals {
notebooks = "service-%s@gcp-sa-notebooks"
pubsub = "service-%s@gcp-sa-pubsub"
secretmanager = "service-%s@gcp-sa-secretmanager"
servicemesh = "service-%s@gcp-sa-servicemesh"
sql = "service-%s@gcp-sa-cloud-sql"
sqladmin = "service-%s@gcp-sa-cloud-sql"
storage = "service-%s@gs-project-accounts"
@ -81,6 +82,7 @@ locals {
"gkehub.googleapis.com", # grant roles/gkehub.serviceAgent to fleet
"multiclusteringress.googleapis.com", # grant roles/multiclusteringress.serviceAgent to multicluster-ingress
"pubsub.googleapis.com", # grant roles/pubsub.serviceAgent to pubsub
"meshconfig.googleapis.com", # grant roles/anthosservicemesh.serviceAgent to meshconfig
"secretmanager.googleapis.com", # no grants needed
"sqladmin.googleapis.com", # grant roles/cloudsql.serviceAgent to sqladmin (TODO: verify)
]