zecfoundation
/
cloud-foundation-fabric
mirror of https://github.com/ZcashFoundation/cloud-foundation-fabric.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'fast-dev-dp' of https://github.com/GoogleCloudPlatform/cloud-foundation-fabric into fast-dev-dp

This commit is contained in:
Lorenzo Caggioni 2022-02-02 18:34:55 +01:00
parent b38b9b2d57 fafbe5c1c8
commit 22739f43ea
57 changed files with 1443 additions and 133 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
examples/cloud-operations/README.md
View File

@ -50,5 +50,10 @@ The example's feed tracks changes to Google Compute instances, and the Cloud Fun
This [example](./onprem-sa-key-management) shows how to manage IAM Service Account Keys by manually generating a key pair and uploading the public part of the key to GCP.
s
<br clear="left">
## Migrate for Compute Engine (v5)
<a href="./vm-migration" title="Packer image builder"><img src="./vm-migration/host-target-projects/diagram.png" align="left" width="280px"></a> This set of [examples](./vm-migration) shows how to deploy Migrate for Compute Engine (v5) on top of existing Cloud Foundations on different scenarios. An example on how to deploy the M4CE connector on VMWare ESXi is also part of the examples.
<br clear="left">

34
examples/cloud-operations/vm-migration/README.md Normal file
View File

@ -0,0 +1,34 @@
# Migrate for Compute Engine (v5) examples
The examples in this folder implement **Migrate for Compute Engine (v5)** environments for the main migration scenarios like the ones with host and target project, or with shared VPC.
They are meant to be used as minimal but complete starting points to create migration environment **on top of existing cloud foundations**, and as playgrounds to experiment with specific Google Cloud features.
## Examples
### M4CE on a single project
<a href="./single-project/" title="M4CE with single project"><img src="./single-project/diagram.png" align="left" width="280px"></a> This [example](./single-project/) implements a simple environment for Migrate for Compute Engine (v5) where both the API backend and the migration target environment are deployed on a single GCP project.
This example represents the easiest sceario to implement a Migrate for Compute Engine (v5) enviroment suitable for small migrations on simple enviroments or for product showcases.
<br clear="left">
### M4CE with host and target projects
<a href="./host-target-projects/" title="M4CE with host and target projects"><img src="./host-target-projects/diagram.png" align="left" width="280px"></a> This [example](./host-target-projects/) implements a Migrate for Compute Engine (v5) host and target projects topology where the API backend and access grants are implemented on the host project while workloads are migrated on a different target project.
This example shows a complex scenario where Migrate for Compute Engine (v5) can be deployed on top of and existing HUB and SPOKE topology and the migration target projects are deployed with platform foundations.
<br clear="left">
### M4CE with Host and Target Projects and Shared VPC
<a href="./host-target-sharedvpc/" title="M4CE with host and target projects and shared VPC"><img src="./host-target-sharedvpc/diagram.png" align="left" width="280px"></a> This [example](./host-target-sharedvpc/) implements a Migrate for Compute Engine (v5) host and target projects topology as described above with the support of shared VPC.
The example shows how to implement a Migrate for Compute Engine (v5) environment on top of an existing shared VPC topology where the shared VPC service projects are the target projects for the migration.
<br clear="left">
### ESXi Connector
<a href="./esxi/" title="M4CE ESXi connector"><img src="./esxi/diagram.png" align="left" width="280px"></a> This [example](./esxi/) implements a Migrate for Compute Engine (v5) environment on a VMWare ESXi cluster as source for VM migrations.
The example shows how to deploy the Migrate for Compute Engine (v5) connector and implement all the security prerequisites for the migration to GCP.

43
examples/cloud-operations/vm-migration/esxi/README.md Normal file
View File

@ -0,0 +1,43 @@
# M4CE(v5) - ESXi Connector
This example deploys a virtual machine from an OVA image and the security prerequisites to run the Migrate for Compute Engine (v5) [connector](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/migrate-connector) on VMWare ESXi.
The example is designed to deploy the M4CE (v5) connector on and existing VMWare environment. The [network configuration](https://cloud.google.com/migrate/compute-engine/docs/5.0/concepts/architecture#migration_architecture) required to allow the communication of the migrate connetor to the GCP API is not included in this example.
This is the high level diagram:
![High-level diagram](diagram.png "High-level diagram")
## Managed resources and services
This sample creates several distinct groups of resources:
- virtual machine
- [M4CE migrate connector](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/migrate-connector#installing_the_migrate_connector)
- IAM
- [vCenter user role](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/migrate-connector#step-1)
<!-- BEGIN TFDOC -->
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [m4ce_ssh_public_key](variables.tf#L43) | Filesystem path to the public key for the SSH login | <code>string</code> | ✓ | |
| [vcenter_password](variables.tf#L48) | VCenter user password. | <code>string</code> | ✓ | |
| [vsphere_environment](variables.tf#L53) | VMVware VSphere connection parameters | <code title="object&#40;&#123;&#10; vcenter_ip &#61; string&#10; vcenter_user &#61; string&#10; data_center &#61; string&#10; resource_pool &#61; string&#10; host_ip &#61; string&#10; datastore &#61; string&#10; virtual_net &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | |
| [m4ce_appliance_properties](variables.tf#L15) | M4CE connector OVA image configuration parameters | <code title="object&#40;&#123;&#10; hostname &#61; string&#10; ip0 &#61; string&#10; netmask0 &#61; string&#10; gateway &#61; string&#10; DNS &#61; string&#10; proxy &#61; string&#10; route0 &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; &#34;hostname&#34; &#61; &#34;gcp-m4ce-connector&#34;&#10; &#34;ip0&#34; &#61; &#34;0.0.0.0&#34;&#10; &#34;netmask0&#34; &#61; &#34;0.0.0.0&#34;&#10; &#34;gateway&#34; &#61; &#34;0.0.0.0&#34;&#10; &#34;DNS&#34; &#61; &#34;&#34;&#10; &#34;proxy&#34; &#61; &#34;&#34;&#10; &#34;route0&#34; &#61; &#34;&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
| [m4ce_connector_ovf_url](variables.tf#L37) | http URL to the public M4CE connector OVA image | <code>string</code> | | <code>&#34;https:&#47;&#47;storage.googleapis.com&#47;vmmigration-public-artifacts&#47;migrate-connector-2-0-1663.ova&#34;</code> |
<!-- END TFDOC -->
## Manual Steps
Once this example is deployed a VCenter user has to be created and binded to the M4CE role in order to allow the connector access the VMWare resources.
The user can be created manually through the VCenter web interface or througt GOV commandline if it is available:
```bash
export GOVC_URL=<VCENTER_URL> (eg. https://192.168.1.100/sdk)
export GOVC_USERNAME=<VCENTER_ADMIN_USER> (eg. administrator@example.local)
export GOVC_PASSWORD=<PASSWORD>
export GOVC_INSECURE=true
govc sso.user.create -p <USER_PASSWORD> -R gcp-m4ce-role gcp-m4ce-user
govc permissions.set -principal gcp-m4ce-user@example.local -propagate=true -role gcp-m4ce-role
```

20
examples/cloud-operations/vm-migration/esxi/backend.tf.sample Normal file
View File

@ -0,0 +1,20 @@
# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
terraform {
backend "gcs" {
bucket = ""
}
}

BIN
examples/cloud-operations/vm-migration/esxi/diagram.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

58
examples/cloud-operations/vm-migration/esxi/main.tf Normal file
View File

@ -0,0 +1,58 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
locals {
rsa-public-key = file(var.m4ce_ssh_public_key)
}
resource "vsphere_role" "gcp-m4ce-role" {
name = "gcp-m4ce-role"
role_privileges = [
"Global.DisableMethods",
"Global.EnableMethods",
"VirtualMachine.Config.ChangeTracking",
"VirtualMachine.Interact.PowerOff",
"VirtualMachine.Provisioning.DiskRandomRead",
"VirtualMachine.Provisioning.GetVmFiles",
"VirtualMachine.State.CreateSnapshot",
"VirtualMachine.State.RemoveSnapshot"
]
}
resource "vsphere_virtual_machine" "gcp-m4ce-connector" {
name = var.m4ce_appliance_properties.hostname
resource_pool_id = data.vsphere_resource_pool.vsphere_pool.id
datastore_id = data.vsphere_datastore.vsphere_datastore.id
host_system_id = data.vsphere_host.vsphere_host.id
datacenter_id = data.vsphere_datacenter.vsphere_dc.id
num_cpus = 4
memory = 16384
network_interface {
network_id = data.vsphere_network.vsphere_network.id
}
wait_for_guest_net_timeout = 0
wait_for_guest_ip_timeout = 0
scsi_type = "lsilogic-sas"
ovf_deploy {
remote_ovf_url = var.m4ce_connector_ovf_url
}
vapp {
properties = merge({ "public-keys" = local.rsa-public-key }, var.m4ce_appliance_properties)
}
}

20
examples/cloud-operations/vm-migration/esxi/provider.tf Normal file
View File

@ -0,0 +1,20 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
provider "vsphere" {
user = var.vsphere_environment.vcenter_user
password = var.vcenter_password
vsphere_server = var.vsphere_environment.vcenter_ip
allow_unverified_ssl = true
}

66
examples/cloud-operations/vm-migration/esxi/variables.tf Normal file
View File

@ -0,0 +1,66 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
variable "m4ce_appliance_properties" {
description = "M4CE connector OVA image configuration parameters"
type = object({
hostname = string
ip0 = string
netmask0 = string
gateway = string
DNS = string
proxy = string
route0 = string
})
default = {
"hostname" = "gcp-m4ce-connector"
"ip0" = "0.0.0.0"
"netmask0" = "0.0.0.0"
"gateway" = "0.0.0.0"
"DNS" = ""
"proxy" = ""
"route0" = ""
}
}
variable "m4ce_connector_ovf_url" {
description = "http URL to the public M4CE connector OVA image"
type = string
default = "https://storage.googleapis.com/vmmigration-public-artifacts/migrate-connector-2-0-1663.ova"
}
variable "m4ce_ssh_public_key" {
description = "Filesystem path to the public key for the SSH login"
type = string
}
variable "vcenter_password" {
description = "VCenter user password."
type = string
}
variable "vsphere_environment" {
description = "VMVware VSphere connection parameters"
type = object({
vcenter_ip = string
vcenter_user = string
data_center = string
resource_pool = string
host_ip = string
datastore = string
virtual_net = string
})
}

37
examples/cloud-operations/vm-migration/esxi/vsphere.tf Normal file
View File

@ -0,0 +1,37 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
data "vsphere_datacenter" "vsphere_dc" {
name = var.vsphere_environment.data_center
}
data "vsphere_resource_pool" "vsphere_pool" {
name = var.vsphere_environment.resource_pool
datacenter_id = data.vsphere_datacenter.vsphere_dc.id
}
data "vsphere_host" "vsphere_host" {
name = var.vsphere_environment.host_ip
datacenter_id = data.vsphere_datacenter.vsphere_dc.id
}
data "vsphere_datastore" "vsphere_datastore" {
name = var.vsphere_environment.datastore
datacenter_id = data.vsphere_datacenter.vsphere_dc.id
}
data "vsphere_network" "vsphere_network" {
name = var.vsphere_environment.virtual_net
datacenter_id = data.vsphere_datacenter.vsphere_dc.id
}

40
examples/cloud-operations/vm-migration/host-target-projects/README.md Normal file
View File

@ -0,0 +1,40 @@
# M4CE(v5) - Host and Target Projects
This example creates a Migrate for Compute Engine (v5) environment deployed on an host project with multiple [target projects](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#identifying_your_host_project).
The example is designed to implement a M4CE (v5) environment on-top of complex migration landing environments where VMs have to be migrated to multiple target projects. It also includes the IAM wiring needed to make such scenarios work.
This is the high level diagram:
![High-level diagram](diagram.png "High-level diagram")
## Managed resources and services
This sample creates\updates several distinct groups of resources:
- projects
- Deploy M4CE host project with [required services](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#enabling_required_services_on_the_host_project) on a new or existing project.
- M4CE target project prerequisites deployed on existing projects.
- IAM
- Create a [service account](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/migrate-connector#step-3) used at runtime by the M4CE connector for data replication
- Grant [migration admin roles](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#using_predefined_roles) to provided user accounts
- Grant [migration viewer role](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#using_predefined_roles) to provided user accounts
<!-- BEGIN TFDOC -->
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [migration_admin_users](variables.tf#L15) | List of users authorized to create a new M4CE sources and perform all other migration operations, in IAM format | <code>list&#40;string&#41;</code> | ✓ | |
| [migration_target_projects](variables.tf#L20) | List of target projects for m4ce workload migrations | <code>list&#40;string&#41;</code> | ✓ | |
| [migration_viewer_users](variables.tf#L25) | List of users authorized to retrive information about M4CE in the Google Cloud Console, in IAM format | <code>list&#40;string&#41;</code> | | <code>&#91;&#93;</code> |
| [project_create](variables.tf#L31) | Parameters for the creation of the new project to host the M4CE backend | <code title="object&#40;&#123;&#10; billing_account_id &#61; string&#10; parent &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [project_name](variables.tf#L40) | Name of an existing project or of the new project assigned as M4CE host project | <code>string</code> | | <code>&#34;m4ce-host-project-000&#34;</code> |
## Outputs
| name | description | sensitive |
|---|---|:---:|
| [m4ce_gmanaged_service_account](outputs.tf#L15) | Google managed service account created automatically during the migrate connector registration.. It is used by M4CE to perform activities on target projects | |
<!-- END TFDOC -->

20
examples/cloud-operations/vm-migration/host-target-projects/backend.tf.sample Normal file
View File

@ -0,0 +1,20 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
terraform {
backend "gcs" {
bucket = ""
}
}

BIN
examples/cloud-operations/vm-migration/host-target-projects/diagram.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 34 KiB

73
examples/cloud-operations/vm-migration/host-target-projects/main.tf Normal file
View File

@ -0,0 +1,73 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
module "host-project" {
source = "../../../../modules/project"
billing_account = (var.project_create != null
? var.project_create.billing_account_id
: null
)
name = var.project_name
parent = (var.project_create != null
? var.project_create.parent
: null
)
services = [
"cloudresourcemanager.googleapis.com",
"compute.googleapis.com",
"iam.googleapis.com",
"logging.googleapis.com",
"servicemanagement.googleapis.com",
"servicecontrol.googleapis.com",
"vmmigration.googleapis.com",
]
project_create = var.project_create != null
iam_additive = {
"roles/iam.serviceAccountKeyAdmin" = var.migration_admin_users,
"roles/iam.serviceAccountCreator" = var.migration_admin_users,
"roles/vmmigration.admin" = var.migration_admin_users,
"roles/vmmigration.viewer" = var.migration_viewer_users,
}
}
module "m4ce-service-account" {
source = "../../../../modules/iam-service-account"
project_id = module.host-project.project_id
name = "m4ce-sa"
generate_key = true
}
module "target-projects" {
for_each = toset(var.migration_target_projects)
source = "../../../../modules/project"
name = each.key
project_create = false
services = [
"servicemanagement.googleapis.com",
"servicecontrol.googleapis.com",
"iam.googleapis.com",
"cloudresourcemanager.googleapis.com",
"compute.googleapis.com"
]
iam_additive = {
"roles/resourcemanager.projectIamAdmin" = var.migration_admin_users,
"roles/compute.viewer" = var.migration_admin_users,
"roles/iam.serviceAccountUser" = var.migration_admin_users
}
}

18
examples/cloud-operations/vm-migration/host-target-projects/outputs.tf Normal file
View File

@ -0,0 +1,18 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
output "m4ce_gmanaged_service_account" {
description = "Google managed service account created automatically during the migrate connector registration.. It is used by M4CE to perform activities on target projects"
value = "serviceAccount:service-${module.host-project.number}@gcp-sa-vmmigration.iam.gserviceaccount.com"
}

44
examples/cloud-operations/vm-migration/host-target-projects/variables.tf Normal file
View File

@ -0,0 +1,44 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
variable "migration_admin_users" {
description = "List of users authorized to create a new M4CE sources and perform all other migration operations, in IAM format"
type = list(string)
}
variable "migration_target_projects" {
description = "List of target projects for m4ce workload migrations"
type = list(string)
}
variable "migration_viewer_users" {
description = "List of users authorized to retrive information about M4CE in the Google Cloud Console, in IAM format"
type = list(string)
default = []
}
variable "project_create" {
description = "Parameters for the creation of the new project to host the M4CE backend"
type = object({
billing_account_id = string
parent = string
})
default = null
}
variable "project_name" {
description = "Name of an existing project or of the new project assigned as M4CE host project"
type = string
default = "m4ce-host-project-000"
}

44
examples/cloud-operations/vm-migration/host-target-sharedvpc/README.md Normal file
View File

@ -0,0 +1,44 @@
# M4CE(v5) - Host and Target Projects with Shared VPC
This example creates a Migrate for Compute Engine (v5) environment deployed on an host project with multiple [target projects](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#identifying_your_host_project) and shared VPCs.
The example is designed to implement a M4CE (v5) environment on-top of complex migration landing environment where VMs have to be migrated to multiple target projects. In this example targets are alse service projects for a shared VPC. It also includes the IAM wiring needed to make such scenarios work.
This is the high level diagram:
![High-level diagram](diagram.png "High-level diagram")
## Managed resources and services
This sample creates\update several distinct groups of resources:
- projects
- M4CE host project with [required services](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#enabling_required_services_on_the_host_project) deployed on a new or existing project.
- M4CE target project prerequisites deployed on existing projects.
- IAM
- Create a [service account](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/migrate-connector#step-3) used at runtime by the M4CE connector for data replication
- Grant [migration admin roles](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#using_predefined_roles) to provided user accounts.
- Grant [migration viewer role](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#using_predefined_roles) to provided user accounts.
- Grant [roles on shared VPC](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/target-project#configure-permissions) to migration admins
<!-- BEGIN TFDOC -->
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [migration_admin_users](variables.tf#L15) | List of users authorized to create a new M4CE sources and perform all other migration operations, in IAM format | <code>list&#40;string&#41;</code> | ✓ | |
| [migration_target_projects](variables.tf#L20) | List of target projects for m4ce workload migrations | <code>list&#40;string&#41;</code> | ✓ | |
| [sharedvpc_host_projects](variables.tf#L45) | List of host projects that share a VPC with the selected target projects | <code>list&#40;string&#41;</code> | ✓ | |
| [migration_viewer_users](variables.tf#L25) | List of users authorized to retrive information about M4CE in the Google Cloud Console, in IAM format | <code>list&#40;string&#41;</code> | | <code>&#91;&#93;</code> |
| [project_create](variables.tf#L30) | Parameters for the creation of the new project to host the M4CE backend | <code title="object&#40;&#123;&#10; billing_account_id &#61; string&#10; parent &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [project_name](variables.tf#L39) | Name of an existing project or of the new project assigned as M4CE host project | <code>string</code> | | <code>&#34;m4ce-host-project-000&#34;</code> |
## Outputs
| name | description | sensitive |
|---|---|:---:|
| [m4ce_gmanaged_service_account](outputs.tf#L15) | Google managed service account created automatically during the migrate connector registration. It is used by M4CE to perform activities on target projects | |
<!-- END TFDOC -->
## Manual Steps
Once this example is deployed the M4CE [m4ce_gmanaged_service_account](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/target-sa-compute-engine#configuring_the_default_service_account) has to be configured to grant the access to the shared VPC and allow the deploy of Compute Engine instances as the result of the migration.

20
examples/cloud-operations/vm-migration/host-target-sharedvpc/backend.tf.sample Normal file
View File

@ -0,0 +1,20 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
terraform {
backend "gcs" {
bucket = ""
}
}

BIN
examples/cloud-operations/vm-migration/host-target-sharedvpc/diagram.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 34 KiB

84
examples/cloud-operations/vm-migration/host-target-sharedvpc/main.tf Normal file
View File

@ -0,0 +1,84 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
module "host-project" {
source = "../../../../modules/project"
billing_account = (var.project_create != null
? var.project_create.billing_account_id
: null
)
name = var.project_name
parent = (var.project_create != null
? var.project_create.parent
: null
)
services = [
"cloudresourcemanager.googleapis.com",
"compute.googleapis.com",
"iam.googleapis.com",
"logging.googleapis.com",
"servicemanagement.googleapis.com",
"servicecontrol.googleapis.com",
"vmmigration.googleapis.com",
]
project_create = var.project_create != null
iam_additive = {
"roles/iam.serviceAccountKeyAdmin" = var.migration_admin_users,
"roles/iam.serviceAccountCreator" = var.migration_admin_users,
"roles/vmmigration.admin" = var.migration_admin_users,
"roles/vmmigration.viewer" = var.migration_viewer_users,
}
}
module "m4ce-service-account" {
source = "../../../../modules/iam-service-account"
project_id = module.host-project.project_id
name = "m4ce-sa"
}
module "target-projects" {
for_each = toset(var.migration_target_projects)
source = "../../../../modules/project"
name = each.key
project_create = false
services = [
"cloudresourcemanager.googleapis.com",
"compute.googleapis.com",
"iam.googleapis.com",
"servicemanagement.googleapis.com",
"servicecontrol.googleapis.com",
]
iam_additive = {
"roles/resourcemanager.projectIamAdmin" = var.migration_admin_users,
"roles/iam.serviceAccountUser" = var.migration_admin_users,
}
}
module "sharedvpc_host_project" {
for_each = toset(var.sharedvpc_host_projects)
source = "../../../../modules/project"
name = each.key
project_create = false
iam_additive = {
"roles/compute.viewer" = var.migration_admin_users,
}
}

18
examples/cloud-operations/vm-migration/host-target-sharedvpc/outputs.tf Normal file
View File

@ -0,0 +1,18 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
output "m4ce_gmanaged_service_account" {
description = "Google managed service account created automatically during the migrate connector registration. It is used by M4CE to perform activities on target projects"
value = "serviceAccount:service-${module.host-project.number}@gcp-sa-vmmigration.iam.gserviceaccount.com"
}

48
examples/cloud-operations/vm-migration/host-target-sharedvpc/variables.tf Normal file
View File

@ -0,0 +1,48 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
variable "migration_admin_users" {
description = "List of users authorized to create a new M4CE sources and perform all other migration operations, in IAM format"
type = list(string)
}
variable "migration_target_projects" {
description = "List of target projects for m4ce workload migrations"
type = list(string)
}
variable "migration_viewer_users" {
description = "List of users authorized to retrive information about M4CE in the Google Cloud Console, in IAM format"
type = list(string)
default = []
}
variable "project_create" {
description = "Parameters for the creation of the new project to host the M4CE backend"
type = object({
billing_account_id = string
parent = string
})
default = null
}
variable "project_name" {
description = "Name of an existing project or of the new project assigned as M4CE host project"
type = string
default = "m4ce-host-project-000"
}
variable "sharedvpc_host_projects" {
description = "List of host projects that share a VPC with the selected target projects"
type = list(string)
}

41
examples/cloud-operations/vm-migration/single-project/README.md Normal file
View File

@ -0,0 +1,41 @@
# M4CE(v5) - Single Project
This sample creates a simple M4CE (v5) environment deployed on a signle [host project](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#identifying_your_host_project).
The example is designed for quick tests or product demos where it is required to setup a simple and minimal M4CE (v5) environment. It also includes the IAM wiring needed to make such scenarios work.
This is the high level diagram:
![High-level diagram](diagram.png "High-level diagram")
## Managed resources and services
This sample creates several distinct groups of resources:
- projects
- M4CE host project with [required services](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#enabling_required_services_on_the_host_project) deployed on a new or existing project.
- networking
- Default VPC network
- IAM
- One [service account](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/migrate-connector#step-3) used at runtime by the M4CE connector for data replication
- Grant [migration admin roles](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#using_predefined_roles) to admin user accounts
- Grant [migration viewer role](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#using_predefined_roles) to viewer user accounts
<!-- BEGIN TFDOC -->
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [migration_admin_users](variables.tf#L15) | List of users authorized to create a new M4CE sources and perform all other migration operations, in IAM format | <code>list&#40;string&#41;</code> | ✓ | |
| [migration_viewer_users](variables.tf#L20) | List of users authorized to retrive information about M4CE in the Google Cloud Console, in IAM format | <code>list&#40;string&#41;</code> | | <code>&#91;&#93;</code> |
| [project_create](variables.tf#L26) | Parameters for the creation of the new project to host the M4CE backend | <code title="object&#40;&#123;&#10; billing_account_id &#61; string&#10; parent &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [project_name](variables.tf#L35) | Name of an existing project or of the new project assigned as M4CE host an target project | <code>string</code> | | <code>&#34;m4ce-host-project-000&#34;</code> |
| [vpc_config](variables.tf#L41) | Parameters to create a simple VPC on the M4CE project | <code title="object&#40;&#123;&#10; ip_cidr_range &#61; string,&#10; region &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; ip_cidr_range &#61; &#34;10.200.0.0&#47;20&#34;,&#10; region &#61; &#34;us-west2&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
## Outputs
| name | description | sensitive |
|---|---|:---:|
| [m4ce_gmanaged_service_account](outputs.tf#L15) | Google managed service account created automatically during the migrate connector registration. It is used by M4CE to perform activities on target projects | |
<!-- END TFDOC -->

20
examples/cloud-operations/vm-migration/single-project/backend.tf.sample Normal file
View File

@ -0,0 +1,20 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
terraform {
backend "gcs" {
bucket = ""
}
}

BIN
examples/cloud-operations/vm-migration/single-project/diagram.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 26 KiB

90
examples/cloud-operations/vm-migration/single-project/main.tf Normal file
View File

@ -0,0 +1,90 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
module "landing-project" {
source = "../../../../modules/project"
billing_account = (var.project_create != null
? var.project_create.billing_account_id
: null
)
name = var.project_name
parent = (var.project_create != null
? var.project_create.parent
: null
)
services = [
"cloudresourcemanager.googleapis.com",
"compute.googleapis.com",
"iam.googleapis.com",
"logging.googleapis.com",
"networkconnectivity.googleapis.com",
"servicemanagement.googleapis.com",
"servicecontrol.googleapis.com",
"vmmigration.googleapis.com"
]
project_create = var.project_create != null
iam_additive = {
"roles/iam.serviceAccountKeyAdmin" = var.migration_admin_users,
"roles/iam.serviceAccountCreator" = var.migration_admin_users,
"roles/vmmigration.admin" = var.migration_admin_users,
"roles/vmmigration.viewer" = var.migration_viewer_users
}
}
module "m4ce-service-account" {
source = "../../../../modules/iam-service-account"
project_id = module.landing-project.project_id
name = "m4ce-sa"
generate_key = true
}
module "landing-vpc" {
source = "../../../../modules/net-vpc"
project_id = module.landing-project.project_id
name = "landing-vpc"
subnets = [
{
ip_cidr_range = var.vpc_config.ip_cidr_range
name = "landing-vpc-${var.vpc_config.region}"
region = var.vpc_config.region
secondary_ip_range = {}
}
]
}
module "landing-vpc-firewall" {
source = "../../../../modules/net-vpc-firewall"
project_id = module.landing-project.project_id
network = module.landing-vpc.name
admin_ranges = []
http_source_ranges = []
https_source_ranges = []
ssh_source_ranges = []
custom_rules = {
allow-ssh = {
description = "Allow SSH from IAP"
direction = "INGRESS"
action = "allow"
sources = []
ranges = ["35.235.240.0/20"]
targets = []
use_service_accounts = false
rules = [{ protocol = "tcp", ports = ["22"] }]
extra_attributes = {}
}
}
}

18
examples/cloud-operations/vm-migration/single-project/outputs.tf Normal file
View File

@ -0,0 +1,18 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
output "m4ce_gmanaged_service_account" {
description = "Google managed service account created automatically during the migrate connector registration. It is used by M4CE to perform activities on target projects"
value = "serviceAccount:service-${module.landing-project.number}@gcp-sa-vmmigration.iam.gserviceaccount.com"
}

51
examples/cloud-operations/vm-migration/single-project/variables.tf Normal file
View File

@ -0,0 +1,51 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
variable "migration_admin_users" {
description = "List of users authorized to create a new M4CE sources and perform all other migration operations, in IAM format"
type = list(string)
}
variable "migration_viewer_users" {
description = "List of users authorized to retrive information about M4CE in the Google Cloud Console, in IAM format"
type = list(string)
default = []
}
variable "project_create" {
description = "Parameters for the creation of the new project to host the M4CE backend"
type = object({
billing_account_id = string
parent = string
})
default = null
}
variable "project_name" {
description = "Name of an existing project or of the new project assigned as M4CE host an target project"
type = string
default = "m4ce-host-project-000"
}
variable "vpc_config" {
description = "Parameters to create a simple VPC on the M4CE project"
type = object({
ip_cidr_range = string,
region = string
})
default = {
ip_cidr_range = "10.200.0.0/20",
region = "us-west2"
}
}

29
examples/data-solutions/gcs-to-bq-with-least-privileges/README.md
View File

@ -3,17 +3,18 @@
This example creates the infrastructure needed to run a [Cloud Dataflow](https://cloud.google.com/dataflow) pipeline to import data from [GCS](https://cloud.google.com/storage) to [Bigquery](https://cloud.google.com/bigquery). The example will create different service accounts with least privileges on resources. To run the pipeline, users listed in `data_eng_principals` can impersonate all those service accounts.
The solution will use:
- internal IPs for GCE and Cloud Dataflow instances
- Cloud NAT to let resources egress to the Internet, to run system updates and install packages
- rely on [Service Account Impersonation](https://cloud.google.com/iam/docs/impersonating-service-accounts) to avoid the use of service account keys
- Service Accounts with least privilege on each resource
- (Optional) CMEK encription for GCS bucket, DataFlow instances and BigQuery tables
The example is designed to match real-world use cases with a minimum amount of resources and some compromise listed below. It can be used as a starting point for more complex scenarios.
- internal IPs for GCE and Cloud Dataflow instances
- Cloud NAT to let resources egress to the Internet, to run system updates and install packages
- rely on [Service Account Impersonation](https://cloud.google.com/iam/docs/impersonating-service-accounts) to avoid the use of service account keys
- Service Accounts with least privilege on each resource
- (Optional) CMEK encription for GCS bucket, DataFlow instances and BigQuery tables
The example is designed to match real-world use cases with a minimum amount of resources and some compromises listed below. It can be used as a starting point for more complex scenarios.
This is the high level diagram:
![GCS to Biquery High-level diagram](diagram.png "GCS to Biquery High-level diagram")
## Move to real use case consideration
In the example we implemented some compromise to keep the example minimal and easy to read. On a real word use case, you may evaluate the option to:
- Configure a Shared-VPC
@ -93,13 +94,13 @@ We need to create 3 file:
- A `person_udf.js` containing the UDF javascript file used by the Dataflow template.
- A `person_schema.json` file containing the table schema used to import the CSV.
You can find an example of those file in the folder `./data-demo`. You can copy the example files in the GCS bucket using the command returned in the terraform output as `command-01-gcs`. Below an example:
You can find an example of those file in the folder `./data-demo`. You can copy the example files in the GCS bucket using the command returned in the terraform output as `command_01_gcs`. Below an example:
```bash
gsutil -i gcs-landing@PROJECT.iam.gserviceaccount.com cp data-demo/* gs://LANDING_BUCKET
```
We can now run the Dataflow pipeline using the `gcloud` returned in the terraform output as `command-02-dataflow`. Below an example:
We can now run the Dataflow pipeline using the `gcloud` returned in the terraform output as `command_02_dataflow`. Below an example:
```bash
gcloud --impersonate-service-account=orch-test@PROJECT.iam.gserviceaccount.com dataflow jobs run test_batch_01 \
@ -119,7 +120,7 @@ outputTable=PROJECT:datalake.person,\
bigQueryLoadingTemporaryDirectory=gs://PREFIX-df-tmp
```
You can check data imported into Google BigQuery using the command returned in the terraform output as `command-03-bq`. Below an example:
You can check data imported into Google BigQuery using the command returned in the terraform output as `command_03_bq`. Below an example:
```
bq query --use_legacy_sql=false 'SELECT * FROM `PROJECT.datalake.person` LIMIT 1000'
@ -144,10 +145,10 @@ bq query --use_legacy_sql=false 'SELECT * FROM `PROJECT.datalake.person` LIMIT 1
|---|---|:---:|
| [bq_tables](outputs.tf#L15) | Bigquery Tables. | |
| [buckets](outputs.tf#L20) | GCS bucket Cloud KMS crypto keys. | |
| [command-01-gcs](outputs.tf#L43) | gcloud command to copy data into the created bucket impersonating the service account. | |
| [command-02-dataflow](outputs.tf#L48) | Command to run Dataflow template impersonating the service account. | |
| [command-03-bq](outputs.tf#L70) | BigQuery command to query imported data. | |
| [command_01_gcs](outputs.tf#L43) | gcloud command to copy data into the created bucket impersonating the service account. | |
| [command_02_dataflow](outputs.tf#L48) | Command to run Dataflow template impersonating the service account. | |
| [command_03_bq](outputs.tf#L69) | BigQuery command to query imported data. | |
| [project_id](outputs.tf#L28) | Project id. | |
| [serviceaccount](outputs.tf#L33) | Service account. | |
| [service_accounts](outputs.tf#L33) | Service account. | |
<!-- END TFDOC -->

2
examples/data-solutions/gcs-to-bq-with-least-privileges/bigquery.tftpl Normal file
View File

@ -0,0 +1,2 @@
bq query --project_id=${project_id} --use_legacy_sql=false \
'SELECT * FROM `${project_id}.${bigquery_dataset}.${bigquery_table}` LIMIT ${sql_limit}'

18
examples/data-solutions/gcs-to-bq-with-least-privileges/dataflow.tftpl Normal file
View File

@ -0,0 +1,18 @@
gcloud \
--impersonate-service-account=${sa_orch_email} \
dataflow jobs run test_batch_01 \
--gcs-location gs://dataflow-templates/latest/GCS_Text_to_BigQuery \
--project ${project_id} \
--region ${region} \
--disable-public-ips \
--subnetwork ${subnet} \
--staging-location ${gcs_df_stg} \
--service-account-email ${sa_df_email} \
%{ if cmek_encryption }--dataflow-kms-key=${kms_key_df} %{ endif } \
--parameters \
javascriptTextTransformFunctionName=transform,\
JSONPath=${data_schema_file},\
javascriptTextTransformGcsPath=${data_udf_file},\
inputFilePattern=${data_file},\
outputTable=${project_id}:${bigquery_dataset}.${bigquery_table},\
bigQueryLoadingTemporaryDirectory=${gcs_df_tmp}

24
examples/data-solutions/gcs-to-bq-with-least-privileges/datastorage.tf
View File

@ -12,10 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
###############################################################################
# GCS #
###############################################################################
module "gcs-data" {
source = "../../../modules/gcs"
project_id = module.project.project_id
@ -23,7 +19,7 @@ module "gcs-data" {
name = "data"
location = var.region
storage_class = "REGIONAL"
encryption_key = var.cmek_encryption ? try(module.kms[0].keys.key-gcs.id, null) : null
encryption_key = var.cmek_encryption ? module.kms[0].keys.key-gcs.id : null
force_destroy = true
}
@ -34,22 +30,20 @@ module "gcs-df-tmp" {
name = "df-tmp"
location = var.region
storage_class = "REGIONAL"
encryption_key = var.cmek_encryption ? try(module.kms[0].keys.key-gcs.id, null) : null
encryption_key = var.cmek_encryption ? module.kms[0].keys.key-gcs.id : null
force_destroy = true
}
###############################################################################
# BQ #
###############################################################################
module "bigquery-dataset" {
source = "../../../modules/bigquery-dataset"
project_id = module.project.project_id
id = "datalake"
location = var.region
# Define Tables in Terraform for the porpuse of the example.
# Probably in a production environment you would handle Tables creation in a
# separate Terraform State or using a different tool/pipeline (for example: Dataform).
# Note: we define tables in Terraform for the purpose of this
# example. A production environment would probably handle table
# creation in a separate terraform pipeline or using a different
# tool (for example: Dataform)
tables = {
person = {
friendly_name = "Person. Dataflow import."
@ -64,10 +58,10 @@ module "bigquery-dataset" {
deletion_protection = false
options = {
clustering = null
encryption_key = var.cmek_encryption ? try(module.kms[0].keys.key-bq.id, null) : null
encryption_key = var.cmek_encryption ? module.kms[0].keys.key-bq.id : null
expiration_time = null
}
}
}
encryption_key = var.cmek_encryption ? try(module.kms[0].keys.key-bq.id, null) : null
encryption_key = var.cmek_encryption ? module.kms[0].keys.key-bq.id : null
}

2
examples/data-solutions/gcs-to-bq-with-least-privileges/kms.tf
View File

@ -17,7 +17,7 @@ module "kms" {
source = "../../../modules/kms"
project_id = module.project.project_id
keyring = {
name = "${var.prefix}-keyring",
name = "${var.prefix}-keyring"
location = var.region
}
keys = {

16
examples/data-solutions/gcs-to-bq-with-least-privileges/main.tf
View File

@ -58,19 +58,18 @@ locals {
module.service-account-orch.iam_email,
]
"roles/iam.serviceAccountTokenCreator" = concat(
var.data_eng_principals,
)
"roles/viewer" = concat(
var.data_eng_principals
)
# Dataflow roles
"roles/dataflow.admin" = concat([
module.service-account-orch.iam_email,
], var.data_eng_principals
"roles/dataflow.admin" = concat(
[module.service-account-orch.iam_email],
var.data_eng_principals
)
"roles/dataflow.worker" = [
module.service-account-df.iam_email,
]
"roles/dataflow.developer" = var.data_eng_principals
"roles/compute.viewer" = var.data_eng_principals
# network roles
"roles/compute.networkUser" = [
module.service-account-df.iam_email,
@ -79,10 +78,6 @@ locals {
}
}
###############################################################################
# Projects #
###############################################################################
module "project" {
source = "../../../modules/project"
name = var.project_id
@ -101,6 +96,7 @@ module "project" {
"storage.googleapis.com",
"storage-component.googleapis.com",
]
# additive IAM bindings avoid disrupting bindings in existing project
iam = var.project_create != null ? local.iam : {}
iam_additive = var.project_create == null ? local.iam : {}

52
examples/data-solutions/gcs-to-bq-with-least-privileges/outputs.tf
View File

@ -30,7 +30,7 @@ output "project_id" {
value = module.project.project_id
}
output "serviceaccount" {
output "service_accounts" {
description = "Service account."
value = {
bq = module.service-account-bq.email
@ -40,36 +40,38 @@ output "serviceaccount" {
}
}
output "command-01-gcs" {
output "command_01_gcs" {
description = "gcloud command to copy data into the created bucket impersonating the service account."
value = "gsutil -i ${module.service-account-landing.email} cp data-demo/* ${module.gcs-data.url}"
}
output "command-02-dataflow" {
output "command_02_dataflow" {
description = "Command to run Dataflow template impersonating the service account."
value = <<EOT
gcloud --impersonate-service-account=${module.service-account-orch.email} dataflow jobs run test_batch_01 \
--gcs-location gs://dataflow-templates/latest/GCS_Text_to_BigQuery \
--project ${module.project.project_id} \
--region ${var.region} \
--disable-public-ips \
--subnetwork ${module.vpc.subnets[format("%s/%s", var.region, "subnet")].self_link} \
--staging-location ${module.gcs-df-tmp.url} \
--service-account-email ${module.service-account-df.email} \
${var.cmek_encryption ? format("--dataflow-kms-key=%s", module.kms[0].key_ids.key-df) : ""} \
--parameters \
javascriptTextTransformFunctionName=transform,\
JSONPath=${module.gcs-data.url}/person_schema.json,\
javascriptTextTransformGcsPath=${module.gcs-data.url}/person_udf.js,\
inputFilePattern=${module.gcs-data.url}/person.csv,\
outputTable=${module.project.project_id}:${module.bigquery-dataset.dataset_id}.${module.bigquery-dataset.tables["person"].table_id},\
bigQueryLoadingTemporaryDirectory=${module.gcs-df-tmp.url}
EOT
value = templatefile("${path.module}/dataflow.tftpl", {
sa_orch_email = module.service-account-orch.email
project_id = module.project.project_id
region = var.region
subnet = module.vpc.subnets["${var.region}/subnet"].self_link
gcs_df_stg = format("%s/%s", module.gcs-df-tmp.url, "stg")
sa_df_email = module.service-account-df.email
cmek_encryption = var.cmek_encryption
kms_key_df = var.cmek_encryption ? module.kms[0].key_ids.key-df : null
gcs_data = module.gcs-data.url
data_schema_file = format("%s/%s", module.gcs-data.url, "person_schema.json")
data_udf_file = format("%s/%s", module.gcs-data.url, "person_udf.js")
data_file = format("%s/%s", module.gcs-data.url, "person.csv")
bigquery_dataset = module.bigquery-dataset.dataset_id
bigquery_table = module.bigquery-dataset.tables["person"].table_id
gcs_df_tmp = format("%s/%s", module.gcs-df-tmp.url, "tmp")
})
}
output "command-03-bq" {
output "command_03_bq" {
description = "BigQuery command to query imported data."
value = <<EOT
bq query --project_id=${module.project.project_id} --use_legacy_sql=false 'SELECT * FROM `${module.project.project_id}.${module.bigquery-dataset.dataset_id}.${module.bigquery-dataset.tables["person"].table_id}` LIMIT 1000'"
EOT
value = templatefile("${path.module}/bigquery.tftpl", {
project_id = module.project.project_id
bigquery_dataset = module.bigquery-dataset.dataset_id
bigquery_table = module.bigquery-dataset.tables["person"].table_id
sql_limit = 1000
})
}

4
examples/data-solutions/gcs-to-bq-with-least-privileges/serviceaccounts.tf
View File

@ -12,10 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
###############################################################################
# Service Accounts #
###############################################################################
module "service-account-bq" {
source = "../../../modules/iam-service-account"
project_id = module.project.project_id

4
examples/data-solutions/gcs-to-bq-with-least-privileges/terraform.tfvars.sample
View File

@ -1,3 +1,3 @@
data_eng_principals = ["user:data-eng@domain.com"]
project_id = "datalake-001"
prefix = "prefix"
project_id = "datalake-001"
prefix = "prefix"

4
examples/data-solutions/gcs-to-bq-with-least-privileges/vpc.tf
View File

@ -12,10 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
###############################################################################
# Networking #
###############################################################################
module "vpc" {
source = "../../../modules/net-vpc"
project_id = module.project.project_id

12
fast/stages/00-bootstrap/README.md
View File

@ -162,7 +162,7 @@ At any time during the life of this stage, you can configure it to automatically
Automatic generation of files is disabled by default. To enable the mechanism, set the `outputs_location` variable to a valid path on a local filesystem, e.g.
```hcl
outputs_location = "../../configs"
outputs_location = "../../config"
```
Once the variable is set, `apply` will generate and manage providers and variables files, including the initial one used for this stage after the first run. You can then link these files in the relevant stages, instead of manually transfering outputs from one stage, to Terraform variables in another.
@ -190,7 +190,9 @@ Below is the outline of the output files generated by this stage:
### Running the stage
The first `apply` run as a user needs a special runtime variable, so that the user roles are preserved when setting IAM bindings:
Before running `init` and `apply`, check your environment so no extra variables that might influence authentication are present (e.g. `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT`). In general you should use user application credentials, and FAST will then take care to provision automation identities and configure impersonation for you.
When running the first `apply` as a user, you need to pass a special runtime variable so that the user roles are preserved when setting IAM bindings.
```bash
terraform init
@ -201,13 +203,15 @@ terraform apply \
Once the initial `apply` completes successfully, configure a remote backend using the new GCS bucket, and impersonation on the automation service account for this stage. To do this, you can use the generated `providers.tf` file if you have configured output files as described above, or extract its contents from Terraform's output, then migrate state with `terraform init`:
```bash
# if using output files via the outputs_location variable
ln -s [path set in outputs_location]/00-bootstrap/* ./
# if using output files via the outputs_location and set to `../../config`
ln -s ../../config/00-bootstrap/* ./
# or from outputs if not using output files
terraform output -json providers | jq -r '.["00-bootstrap"]' \
> providers.tf
# migrate state to GCS bucket configured in providers file
terraform init -migrate-state
# run terraform apply to remo user iam binding
terraform apply
```
## Customizations

10
fast/stages/01-resman/README.md
View File

@ -53,8 +53,8 @@ To simplify setup, the previous stage pre-configures a valid providers file in i
If you have set a valid value for `outputs_location` in the bootstrap stage, simply link the relevant `providers.tf` file from this stage's folder in the path you specified:
```bash
# `outputs_location` is set to `../../configs/example`
ln -s ../../configs/example/01-resman/providers.tf
# `outputs_location` is set to `../../config`
ln -s ../../config/01-resman/providers.tf
```
If you have not configured `outputs_location` in bootstrap, you can derive the providers file from that stage's outputs:
@ -65,6 +65,8 @@ terraform output -json providers | jq -r '.["01-resman"]' \
> ../01-resman/providers.tf
```
If you want to continue to rely on `outputs_location` logic, create a `terraform.tfvars` file and configure it as deacribed [here](../00-bootstrap/#output-files-and-cross-stage-variables).
### Variable configuration
There are two broad sets of variables you will need to fill in:
@ -77,8 +79,8 @@ To avoid the tedious job of filling in the first group of variable with values d
If you configured a valid path for `outputs_location` in the bootstrap stage, simply link the relevant `terraform-*.auto.tfvars.json` files from this stage's outputs folder (under the path you specified), where the `*` above is set to the name of the stage that produced it. For this stage, a single `.tfvars` file is avalaible:
```bash
# `outputs_location` is set to `../../configs/example`
ln -s ../../configs/example/01-resman/terraform-bootstrap.auto.tfvars.json
# `outputs_location` is set to `../../config`
ln -s ../../config/01-resman/terraform-bootstrap.auto.tfvars.json
```
A second set of variables is specific to this stage, they are all optional so if you need to customize them, create an extra `terraform.tfvars` file.

12
fast/stages/02-networking/README.md
View File

@ -136,8 +136,8 @@ To simplify setup, the previous stage pre-configures a valid providers file in i
If you have set a valid value for `outputs_location` in the bootstrap stage, simply link the relevant `providers.tf` file from this stage's folder in the path you specified:
```bash
# `outputs_location` is set to `../../configs/example`
ln -s ../../configs/example/02-networking/providers.tf
# `outputs_location` is set to `../../config`
ln -s ../../config/02-networking/providers.tf
```
If you have not configured `outputs_location` in bootstrap, you can derive the providers file from that stage's outputs:
@ -160,11 +160,13 @@ To avoid the tedious job of filling in the first group of variables with values
If you have set a valid value for `outputs_location` in the bootstrap and in the resman stage, simply link the relevant `terraform-*.auto.tfvars.json` files from this stage's folder in the path you specified, where the `*` above is set to the name of the stage that produced it. For this stage, a single `.tfvars` file is available:
```bash
# `outputs_location` is set to `../../configs/example`
ln -s ../../configs/example/02-networking/terraform-bootstrap.auto.tfvars.json
ln -s ../../configs/example/02-networking/terraform-resman.auto.tfvars.json
# `outputs_location` is set to `../../config`
ln -s ../../config/02-networking/terraform-bootstrap.auto.tfvars.json
ln -s ../../config/02-networking/terraform-resman.auto.tfvars.json
```
If you want to continue to rely on `outputs_location` logic, create a `terraform.tfvars` file and configure it as deacribed [here](../00-bootstrap/#output-files-and-cross-stage-variables).
Please refer to the [Variables](#variables) table below for a map of the variable origins, and to the sections below on how to adapt this stage to your networking configuration.
### VPCs

12
fast/stages/02-security/README.md
View File

@ -57,8 +57,8 @@ To simplify setup, the previous stage pre-configures a valid providers file in i
If you have set a valid value for `outputs_location` in the resource management stage, simply link the relevant `providers.tf` file from this stage's folder in the path you specified:
```bash
# `outputs_location` is set to `../../configs/example`
ln -s ../../configs/example/02-security/providers.tf
# `outputs_location` is set to `../../config`
ln -s ../../config/02-security/providers.tf
```
If you have not configured `outputs_location` in resource management, you can derive the providers file from that stage's outputs:
@ -69,6 +69,8 @@ terraform output -json providers | jq -r '.["02-security"]' \
> ../02-security/providers.tf
```
If you want to continue to rely on `outputs_location` logic, create a `terraform.tfvars` file and configure it as deacribed [here](../00-bootstrap/#output-files-and-cross-stage-variables).
### Variable configuration
There are two broad sets of variables you will need to fill in:
@ -81,9 +83,9 @@ To avoid the tedious job of filling in the first group of variables with values
If you configured a valid path for `outputs_location` in the previous stages, simply link the relevant `terraform-*.auto.tfvars.json` files from this stage's output folder (under the path you specified), where the `*` above is set to the name of the stage that produced it. For this stage, two `.tfvars` files are available:
```bash
# `outputs_location` is set to `../../configs/example`
ln -s ../../configs/example/02-security/terraform-bootstrap.auto.tfvars.json
ln -s ../../configs/example/02-security/terraform-resman.auto.tfvars.json
# `outputs_location` is set to `../../config`
ln -s ../../config/02-security/terraform-bootstrap.auto.tfvars.json
ln -s ../../config/02-security/terraform-resman.auto.tfvars.json
```
A second set of optional variables is specific to this stage. If you need to customize them, create an extra `terraform.tfvars` file.

74
fast/stages/03-project-factory/prod/README.md
View File

@ -15,51 +15,50 @@ Projects for each environment across different teams are created by dedicated se
The project factory takes care of the following activities:
* Project creation
* API/Services enablement
* Service accounts creation
* IAM roles assignment for groups and service accounts
* KMS keys roles assignment
* Shared VPC attachment and subnets IAM binding
* DNS zones creation and visibility configuration
* Project-level org policies definition
* Billing setup (billing account attachment and budget configuration)
* Essential contacts definition (for [budget alerts](https://cloud.google.com/billing/docs/how-to/budgets) and [important notifications](https://cloud.google.com/resource-manager/docs/managing-notification-contacts?hl=en))
- Project creation
- API/Services enablement
- Service accounts creation
- IAM roles assignment for groups and service accounts
- KMS keys roles assignment
- Shared VPC attachment and subnets IAM binding
- DNS zones creation and visibility configuration
- Project-level org policies definition
- Billing setup (billing account attachment and budget configuration)
- Essential contacts definition (for [budget alerts](https://cloud.google.com/billing/docs/how-to/budgets) and [important notifications](https://cloud.google.com/resource-manager/docs/managing-notification-contacts?hl=en))
## How to run this stage
This stage is meant to be executed after "foundational stages" (i.e., stages [`00-bootstrap`](../../00-bootstrap), [`01-resman`](../../01-resman), [`02-networking`](../../02-networking) and [`02-security`](../../02-security)) have been run.
It's of course possible to run this stage in isolation, by making sure the architectural prerequisites are satisfied (e.g., networking), and that the Service Account running the stage is granted the roles/permissions below:
* One service account per environment, each with appropriate permissions
* at the organization level a custom role for networking operations including the following permissions
* `"compute.organizations.enableXpnResource"`,
* `"compute.organizations.disableXpnResource"`,
* `"compute.subnetworks.setIamPolicy"`,
* `"dns.networks.bindPrivateDNSZone"`
* and role `"roles/orgpolicy.policyAdmin"`
* on each folder where projects are created
* `"roles/logging.admin"`
* `"roles/owner"`
* `"roles/resourcemanager.folderAdmin"`
* `"roles/resourcemanager.projectCreator"`
* on the host project for the Shared VPC
* `"roles/browser"`
* `"roles/compute.viewer"`
* `"roles/dns.admin"`
* If networking is used (e.g., for VMs, GKE Clusters or AppEngine flex), VPC Host projects and their subnets should exist when creating projects
* If per-environment DNS sub-zones are required, one "root" zone per environment should exist when creating projects (e.g., prod.gcp.example.com.)
- One service account per environment, each with appropriate permissions
- at the organization level a custom role for networking operations including the following permissions
- `"compute.organizations.enableXpnResource"`,
- `"compute.organizations.disableXpnResource"`,
- `"compute.subnetworks.setIamPolicy"`,
- `"dns.networks.bindPrivateDNSZone"`
- and role `"roles/orgpolicy.policyAdmin"`
- on each folder where projects are created
- `"roles/logging.admin"`
- `"roles/owner"`
- `"roles/resourcemanager.folderAdmin"`
- `"roles/resourcemanager.projectCreator"`
- on the host project for the Shared VPC
- `"roles/browser"`
- `"roles/compute.viewer"`
- `"roles/dns.admin"`
- If networking is used (e.g., for VMs, GKE Clusters or AppEngine flex), VPC Host projects and their subnets should exist when creating projects
- If per-environment DNS sub-zones are required, one "root" zone per environment should exist when creating projects (e.g., prod.gcp.example.com.)
### Providers configuration
If you're running this on top of Fast, you should run the following commands to create the providers file, and populate the required variables from the previous stage.
```bash
# Variable `outputs_location` is set to `../../configs/example` in stage 01-resman
# Variable `outputs_location` is set to `../../config` in stage 01-resman
$ cd fabric-fast/stages/03-project-factory/prod
ln -s ../../../configs/example/03-project-factory-prod/providers.tf
ln -s ../../../config/03-project-factory-prod/providers.tf
```
### Variable configuration
@ -74,18 +73,17 @@ To avoid the tedious job of filling in the first group of variables with values
If you configured a valid path for `outputs_location` in the bootstrap and networking stage, simply link the relevant `terraform-*.auto.tfvars.json` files from this stage's outputs folder (under the path you specified), where the `*` above is set to the name of the stage that produced it. For this stage, a single `.tfvars` file is available:
```bash
# Variable `outputs_location` is set to `../../configs/example` in stages 01-bootstrap and 02-networking
ln -s ../../../configs/example/03-project-factory-prod/terraform-bootstrap.auto.tfvars.json
ln -s ../../../configs/example/03-project-factory-prod/terraform-networking.auto.tfvars.json
# Variable `outputs_location` is set to `../../config` in stages 01-bootstrap and 02-networking
ln -s ../../../config/03-project-factory-prod/terraform-bootstrap.auto.tfvars.json
ln -s ../../../config/03-project-factory-prod/terraform-networking.auto.tfvars.json
```
If you're not using Fast, refer to the [Variables](#variables) table at the bottom of this document for a full list of variables, their origin (e.g., a stage or specific to this one), and descriptions explaining their meaning.
Besides the values above, a project factory takes 2 additional inputs:
Besides the values above, a project factory takes 2 additional inputs:
* `data/defaults.yaml`, manually configured by adapting the [`prod/data/defaults.yaml.sample`](./prod/data/defaults.yaml.sample), which defines per-environment default values e.g., for billing alerts and labels.
* `data/projects/*.yaml`, one file per project (optionally grouped in folders), which configures each project. A [`prod/data/projects/project.yaml.sample`](./prod/data/projects/project.yaml.sample) is provided as reference and documentation for the schema. Projects will be named after the filename, e.g., `fast-prod-lab0.yaml` will create project `fast-prod-lab0`.
- `data/defaults.yaml`, manually configured by adapting the [`prod/data/defaults.yaml.sample`](./prod/data/defaults.yaml.sample), which defines per-environment default values e.g., for billing alerts and labels.
- `data/projects/*.yaml`, one file per project (optionally grouped in folders), which configures each project. A [`prod/data/projects/project.yaml.sample`](./prod/data/projects/project.yaml.sample) is provided as reference and documentation for the schema. Projects will be named after the filename, e.g., `fast-prod-lab0.yaml` will create project `fast-prod-lab0`.
Once the configuration is complete, run the project factory by running

19
modules/README.md
View File

@ -8,6 +8,25 @@ Authoritative IAM bindings are primarily used (e.g. `google_storage_bucket_iam_b
Specific modules also offer support for non-authoritative bindings (e.g. `google_storage_bucket_iam_member` for service accounts), to allow granular permission management on resources that they don't manage directly.
These modules are not necessarily backward compatible. Changes breaking compatibility in modules are marked by major releases (but not all major releases contain breaking changes). Please be mindful when upgrading Fabric modules in existing Terraform setups, and always try to use versioned references in module sources so you can easily revert back to a previous version. Since the introduction of the `moved` block in Terraform we try to use it whenever possible to make updates non-breaking, but that does not cover all changes we might need to make.
These modules are used in the examples included in this repository. If you are using any of those examples in your own Terraform configuration, make sure that you are using the same version for all the modules, and switch module sources to GitHub format using references. The recommended approach to working with Fabric modules is the following:
- Fork the repository and own the fork. This will allow you to:
- Evolve the existing modules.
- Create your own modules.
- Sync from the upstream repository to get all the updates.
- Use GitHub sources with refs to reference the modules in your fork. See an example below:
```
module "project" {
source = "github.com/my-fork/cloud-foundation-fabric.git//modules/project?ref=v12.0.0"
name = "my-project"
billing_account = "123456-123456-123456"
parent = "organizations/123456"
}
```
## Foundational modules
- [billing budget](./billing-budget)

35
modules/__experimental/net-dns-policy-address/README.md Normal file
View File

@ -0,0 +1,35 @@
# Google Cloud DNS Inbound Policy Addresses
This module allows discovering the addresses reserved in subnets when [DNS Inbound Policies](https://cloud.google.com/dns/docs/policies) are configured.
Since it's currently impossible to fetch those addresses using a GCP data source (see [this issue](https://github.com/hashicorp/terraform-provider-google/issues/3753) for more details), the workaround used here is to derive the authorization token from the Google provider, and do a direct HTTP call to the Compute API.
## Examples
```hcl
module "dns-policy-addresses" {
source = "./modules/__experimental/net-dns-policy-addresses"
project_id = "myproject"
regions = ["europe-west1", "europe-west3"]
}
# tftest skip
```
The output is a map with lists of addresses of type `DNS_RESOLVER` for each region specified in variables.
<!-- BEGIN TFDOC -->
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [project_id](variables.tf#L17) | Project id. | <code>string</code> | ✓ | |
| [regions](variables.tf#L22) | Regions to fetch addresses from. | <code>list&#40;string&#41;</code> | | <code>&#91;&#34;europe-west1&#34;&#93;</code> |
## Outputs
| name | description | sensitive |
|---|---|:---:|
| [addresses](outputs.tf#L24) | DNS inbound policy addresses per region. | |
<!-- END TFDOC -->

32
modules/__experimental/net-dns-policy-address/main.tf Normal file
View File

@ -0,0 +1,32 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
locals {
url = format(
"https://content-compute.googleapis.com/compute/v1/projects/%s",
var.project_id
)
}
data "google_client_config" "current" {}
data "http" "addresses" {
for_each = toset(var.regions)
url = "${local.url}/regions/${each.key}/addresses?filter=purpose%20%3D%20%22DNS_RESOLVER%22"
request_headers = {
Authorization = "Bearer ${data.google_client_config.current.access_token}"
}
}

31
modules/__experimental/net-dns-policy-address/outputs.tf Normal file
View File

@ -0,0 +1,31 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
locals {
region_addresses = {
for k, v in data.http.addresses : k => try(jsondecode(v.body), {})
}
}
output "addresses" {
description = "DNS inbound policy addresses per region."
value = {
for k, v in local.region_addresses : k => [
for i in try(v.items, []) : i.address
]
}
}

27
modules/__experimental/net-dns-policy-address/variables.tf Normal file
View File

@ -0,0 +1,27 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
variable "project_id" {
description = "Project id."
type = string
}
variable "regions" {
description = "Regions to fetch addresses from."
nullable = false
type = list(string)
default = ["europe-west1"]
}

13
tests/examples/cloud_operations/vm_migration/host_target_projects/__init__.py Normal file
View File

@ -0,0 +1,13 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

43
tests/examples/cloud_operations/vm_migration/host_target_projects/fixture/main.tf Normal file
View File

@ -0,0 +1,43 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
module "host-target-projects-test" {
source = "../../../../../../examples/cloud-operations/vm-migration/host-target-projects"
project_create = var.project_create
migration_admin_users = ["user:admin@example.com"]
migration_viewer_users = ["user:viewer@example.com"]
migration_target_projects = ["${module.test-target-project.name}"]
depends_on = [
module.test-target-project
]
}
variable "project_create" {
type = object({
billing_account_id = string
parent = string
})
default = {
billing_account_id = "1234-ABCD-1234"
parent = "folders/1234563"
}
}
#This is a dummy project created to run this test. The example, here tested, is expected to run on top of existing foundations.
module "test-target-project" {
source = "../../../../../../modules/project"
billing_account = "1234-ABCD-1234"
name = "test-target-project"
project_create = true
}

26
tests/examples/cloud_operations/vm_migration/host_target_projects/test_plan.py Normal file
View File

@ -0,0 +1,26 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
FIXTURES_DIR = os.path.join(os.path.dirname(__file__), 'fixture')
def test_resources(e2e_plan_runner):
"Test that plan works and the numbers of resources is as expected."
modules, resources = e2e_plan_runner(FIXTURES_DIR)
assert len(modules) == 3
assert len(resources) == 23

13
tests/examples/cloud_operations/vm_migration/host_target_sharedvpc/__init__.py Normal file
View File

@ -0,0 +1,13 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

51
tests/examples/cloud_operations/vm_migration/host_target_sharedvpc/fixture/main.tf Normal file
View File

@ -0,0 +1,51 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
module "host-target-sharedvpc-test" {
source = "../../../../../../examples/cloud-operations/vm-migration/host-target-sharedvpc"
project_create = var.project_create
migration_admin_users = ["user:admin@example.com"]
migration_viewer_users = ["user:viewer@example.com"]
migration_target_projects = [module.test-target-project.name]
sharedvpc_host_projects = [module.test-sharedvpc-host-project.name]
depends_on = [
module.test-target-project,
module.test-sharedvpc-host-project,
]
}
variable "project_create" {
type = object({
billing_account_id = string
parent = string
})
default = {
billing_account_id = "1234-ABCD-1234"
parent = "folders/1234563"
}
}
#These are a dummy projects created to run this test. The example, here tested, is expected to run on top of existing foundations.
module "test-target-project" {
source = "../../../../../../modules/project"
billing_account = "1234-ABCD-1234"
name = "test-target-project"
project_create = true
}
module "test-sharedvpc-host-project" {
source = "../../../../../../modules/project"
billing_account = "1234-ABCD-1234"
name = "test-sharedvpc-host-project"
project_create = true
}

26
tests/examples/cloud_operations/vm_migration/host_target_sharedvpc/test_plan.py Normal file
View File

@ -0,0 +1,26 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
FIXTURES_DIR = os.path.join(os.path.dirname(__file__), 'fixture')
def test_resources(e2e_plan_runner):
"Test that plan works and the numbers of resources is as expected."
modules, resources = e2e_plan_runner(FIXTURES_DIR)
assert len(modules) == 4
assert len(resources) == 23

13
tests/examples/cloud_operations/vm_migration/single_project/__init__.py Normal file
View File

@ -0,0 +1,13 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

31
tests/examples/cloud_operations/vm_migration/single_project/fixture/main.tf Normal file
View File

@ -0,0 +1,31 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
module "single-project-test" {
source = "../../../../../../examples/cloud-operations/vm-migration/single-project"
project_create = var.project_create
migration_admin_users = ["user:admin@example.com"]
migration_viewer_users = ["user:viewer@example.com"]
}
variable "project_create" {
type = object({
billing_account_id = string
parent = string
})
default = {
billing_account_id = "1234-ABCD-1234"
parent = "folders/1234563"
}
}

25
tests/examples/cloud_operations/vm_migration/single_project/test_plan.py Normal file
View File

@ -0,0 +1,25 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
FIXTURES_DIR = os.path.join(os.path.dirname(__file__), 'fixture')
def test_resources(e2e_plan_runner):
"Test that plan works and the numbers of resources is as expected."
modules, resources = e2e_plan_runner(FIXTURES_DIR)
assert len(modules) == 4
assert len(resources) == 18

2
tests/examples/data_solutions/gcs_to_bq_with_least_privileges/test_plan.py
View File

@ -24,4 +24,4 @@ def test_resources(e2e_plan_runner):
"Test that plan works and the numbers of resources is as expected."
modules, resources = e2e_plan_runner(FIXTURES_DIR)
assert len(modules) == 11
assert len(resources) == 43
assert len(resources) == 44