diff --git a/examples/cloud-operations/README.md b/examples/cloud-operations/README.md index 1a491e94..62f86db9 100644 --- a/examples/cloud-operations/README.md +++ b/examples/cloud-operations/README.md @@ -50,5 +50,10 @@ The example's feed tracks changes to Google Compute instances, and the Cloud Fun This [example](./onprem-sa-key-management) shows how to manage IAM Service Account Keys by manually generating a key pair and uploading the public part of the key to GCP. -s + +
+ +## Migrate for Compute Engine (v5) + This set of [examples](./vm-migration) shows how to deploy Migrate for Compute Engine (v5) on top of existing Cloud Foundations on different scenarios. An example on how to deploy the M4CE connector on VMWare ESXi is also part of the examples. +
\ No newline at end of file diff --git a/examples/cloud-operations/vm-migration/README.md b/examples/cloud-operations/vm-migration/README.md new file mode 100644 index 00000000..0c75af09 --- /dev/null +++ b/examples/cloud-operations/vm-migration/README.md @@ -0,0 +1,34 @@ +# Migrate for Compute Engine (v5) examples + +The examples in this folder implement **Migrate for Compute Engine (v5)** environments for the main migration scenarios like the ones with host and target project, or with shared VPC. + +They are meant to be used as minimal but complete starting points to create migration environment **on top of existing cloud foundations**, and as playgrounds to experiment with specific Google Cloud features. + +## Examples + +### M4CE on a single project + + This [example](./single-project/) implements a simple environment for Migrate for Compute Engine (v5) where both the API backend and the migration target environment are deployed on a single GCP project. + +This example represents the easiest sceario to implement a Migrate for Compute Engine (v5) enviroment suitable for small migrations on simple enviroments or for product showcases. +
+ +### M4CE with host and target projects + + This [example](./host-target-projects/) implements a Migrate for Compute Engine (v5) host and target projects topology where the API backend and access grants are implemented on the host project while workloads are migrated on a different target project. + +This example shows a complex scenario where Migrate for Compute Engine (v5) can be deployed on top of and existing HUB and SPOKE topology and the migration target projects are deployed with platform foundations. +
+ +### M4CE with Host and Target Projects and Shared VPC + + This [example](./host-target-sharedvpc/) implements a Migrate for Compute Engine (v5) host and target projects topology as described above with the support of shared VPC. + +The example shows how to implement a Migrate for Compute Engine (v5) environment on top of an existing shared VPC topology where the shared VPC service projects are the target projects for the migration. +
+ +### ESXi Connector + + This [example](./esxi/) implements a Migrate for Compute Engine (v5) environment on a VMWare ESXi cluster as source for VM migrations. + +The example shows how to deploy the Migrate for Compute Engine (v5) connector and implement all the security prerequisites for the migration to GCP. \ No newline at end of file diff --git a/examples/cloud-operations/vm-migration/esxi/README.md b/examples/cloud-operations/vm-migration/esxi/README.md new file mode 100644 index 00000000..8e77f37d --- /dev/null +++ b/examples/cloud-operations/vm-migration/esxi/README.md @@ -0,0 +1,43 @@ +# M4CE(v5) - ESXi Connector + +This example deploys a virtual machine from an OVA image and the security prerequisites to run the Migrate for Compute Engine (v5) [connector](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/migrate-connector) on VMWare ESXi. + +The example is designed to deploy the M4CE (v5) connector on and existing VMWare environment. The [network configuration](https://cloud.google.com/migrate/compute-engine/docs/5.0/concepts/architecture#migration_architecture) required to allow the communication of the migrate connetor to the GCP API is not included in this example. + +This is the high level diagram: + +![High-level diagram](diagram.png "High-level diagram") + +## Managed resources and services + +This sample creates several distinct groups of resources: + +- virtual machine + - [M4CE migrate connector](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/migrate-connector#installing_the_migrate_connector) +- IAM + - [vCenter user role](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/migrate-connector#step-1) + + +## Variables + +| name | description | type | required | default | +|---|---|:---:|:---:|:---:| +| [m4ce_ssh_public_key](variables.tf#L43) | Filesystem path to the public key for the SSH login | string | ✓ | | +| [vcenter_password](variables.tf#L48) | VCenter user password. | string | ✓ | | +| [vsphere_environment](variables.tf#L53) | VMVware VSphere connection parameters | object({…}) | ✓ | | +| [m4ce_appliance_properties](variables.tf#L15) | M4CE connector OVA image configuration parameters | object({…}) | | {…} | +| [m4ce_connector_ovf_url](variables.tf#L37) | http URL to the public M4CE connector OVA image | string | | "https://storage.googleapis.com/vmmigration-public-artifacts/migrate-connector-2-0-1663.ova" | + + +## Manual Steps +Once this example is deployed a VCenter user has to be created and binded to the M4CE role in order to allow the connector access the VMWare resources. +The user can be created manually through the VCenter web interface or througt GOV commandline if it is available: +```bash +export GOVC_URL= (eg. https://192.168.1.100/sdk) +export GOVC_USERNAME= (eg. administrator@example.local) +export GOVC_PASSWORD= +export GOVC_INSECURE=true + +govc sso.user.create -p -R gcp-m4ce-role gcp-m4ce-user +govc permissions.set -principal gcp-m4ce-user@example.local -propagate=true -role gcp-m4ce-role +``` diff --git a/examples/cloud-operations/vm-migration/esxi/backend.tf.sample b/examples/cloud-operations/vm-migration/esxi/backend.tf.sample new file mode 100644 index 00000000..99f84b17 --- /dev/null +++ b/examples/cloud-operations/vm-migration/esxi/backend.tf.sample @@ -0,0 +1,20 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +terraform { + backend "gcs" { + bucket = "" + } +} diff --git a/examples/cloud-operations/vm-migration/esxi/diagram.png b/examples/cloud-operations/vm-migration/esxi/diagram.png new file mode 100644 index 00000000..45ccb361 Binary files /dev/null and b/examples/cloud-operations/vm-migration/esxi/diagram.png differ diff --git a/examples/cloud-operations/vm-migration/esxi/main.tf b/examples/cloud-operations/vm-migration/esxi/main.tf new file mode 100644 index 00000000..ead0fd97 --- /dev/null +++ b/examples/cloud-operations/vm-migration/esxi/main.tf @@ -0,0 +1,58 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +locals { + rsa-public-key = file(var.m4ce_ssh_public_key) +} + +resource "vsphere_role" "gcp-m4ce-role" { + name = "gcp-m4ce-role" + role_privileges = [ + "Global.DisableMethods", + "Global.EnableMethods", + "VirtualMachine.Config.ChangeTracking", + "VirtualMachine.Interact.PowerOff", + "VirtualMachine.Provisioning.DiskRandomRead", + "VirtualMachine.Provisioning.GetVmFiles", + "VirtualMachine.State.CreateSnapshot", + "VirtualMachine.State.RemoveSnapshot" + ] +} + +resource "vsphere_virtual_machine" "gcp-m4ce-connector" { + name = var.m4ce_appliance_properties.hostname + resource_pool_id = data.vsphere_resource_pool.vsphere_pool.id + datastore_id = data.vsphere_datastore.vsphere_datastore.id + host_system_id = data.vsphere_host.vsphere_host.id + datacenter_id = data.vsphere_datacenter.vsphere_dc.id + num_cpus = 4 + memory = 16384 + + network_interface { + network_id = data.vsphere_network.vsphere_network.id + } + + wait_for_guest_net_timeout = 0 + wait_for_guest_ip_timeout = 0 + + scsi_type = "lsilogic-sas" + + ovf_deploy { + remote_ovf_url = var.m4ce_connector_ovf_url + } + + vapp { + properties = merge({ "public-keys" = local.rsa-public-key }, var.m4ce_appliance_properties) + } +} diff --git a/examples/cloud-operations/vm-migration/esxi/provider.tf b/examples/cloud-operations/vm-migration/esxi/provider.tf new file mode 100644 index 00000000..4361013a --- /dev/null +++ b/examples/cloud-operations/vm-migration/esxi/provider.tf @@ -0,0 +1,20 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +provider "vsphere" { + user = var.vsphere_environment.vcenter_user + password = var.vcenter_password + vsphere_server = var.vsphere_environment.vcenter_ip + allow_unverified_ssl = true +} diff --git a/examples/cloud-operations/vm-migration/esxi/variables.tf b/examples/cloud-operations/vm-migration/esxi/variables.tf new file mode 100644 index 00000000..ba886d43 --- /dev/null +++ b/examples/cloud-operations/vm-migration/esxi/variables.tf @@ -0,0 +1,66 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +variable "m4ce_appliance_properties" { + description = "M4CE connector OVA image configuration parameters" + type = object({ + hostname = string + ip0 = string + netmask0 = string + gateway = string + DNS = string + proxy = string + route0 = string + }) + default = { + "hostname" = "gcp-m4ce-connector" + "ip0" = "0.0.0.0" + "netmask0" = "0.0.0.0" + "gateway" = "0.0.0.0" + "DNS" = "" + "proxy" = "" + "route0" = "" + } +} + +variable "m4ce_connector_ovf_url" { + description = "http URL to the public M4CE connector OVA image" + type = string + default = "https://storage.googleapis.com/vmmigration-public-artifacts/migrate-connector-2-0-1663.ova" +} + +variable "m4ce_ssh_public_key" { + description = "Filesystem path to the public key for the SSH login" + type = string +} + +variable "vcenter_password" { + description = "VCenter user password." + type = string +} + +variable "vsphere_environment" { + description = "VMVware VSphere connection parameters" + type = object({ + vcenter_ip = string + vcenter_user = string + data_center = string + resource_pool = string + host_ip = string + datastore = string + virtual_net = string + }) +} + + diff --git a/examples/cloud-operations/vm-migration/esxi/vsphere.tf b/examples/cloud-operations/vm-migration/esxi/vsphere.tf new file mode 100644 index 00000000..3a8376b3 --- /dev/null +++ b/examples/cloud-operations/vm-migration/esxi/vsphere.tf @@ -0,0 +1,37 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +data "vsphere_datacenter" "vsphere_dc" { + name = var.vsphere_environment.data_center +} + +data "vsphere_resource_pool" "vsphere_pool" { + name = var.vsphere_environment.resource_pool + datacenter_id = data.vsphere_datacenter.vsphere_dc.id +} + +data "vsphere_host" "vsphere_host" { + name = var.vsphere_environment.host_ip + datacenter_id = data.vsphere_datacenter.vsphere_dc.id +} + +data "vsphere_datastore" "vsphere_datastore" { + name = var.vsphere_environment.datastore + datacenter_id = data.vsphere_datacenter.vsphere_dc.id +} + +data "vsphere_network" "vsphere_network" { + name = var.vsphere_environment.virtual_net + datacenter_id = data.vsphere_datacenter.vsphere_dc.id +} diff --git a/examples/cloud-operations/vm-migration/host-target-projects/README.md b/examples/cloud-operations/vm-migration/host-target-projects/README.md new file mode 100644 index 00000000..9580f898 --- /dev/null +++ b/examples/cloud-operations/vm-migration/host-target-projects/README.md @@ -0,0 +1,40 @@ +# M4CE(v5) - Host and Target Projects + +This example creates a Migrate for Compute Engine (v5) environment deployed on an host project with multiple [target projects](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#identifying_your_host_project). + +The example is designed to implement a M4CE (v5) environment on-top of complex migration landing environments where VMs have to be migrated to multiple target projects. It also includes the IAM wiring needed to make such scenarios work. + +This is the high level diagram: + +![High-level diagram](diagram.png "High-level diagram") + +## Managed resources and services + +This sample creates\updates several distinct groups of resources: + +- projects + - Deploy M4CE host project with [required services](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#enabling_required_services_on_the_host_project) on a new or existing project. + - M4CE target project prerequisites deployed on existing projects. +- IAM + - Create a [service account](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/migrate-connector#step-3) used at runtime by the M4CE connector for data replication + - Grant [migration admin roles](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#using_predefined_roles) to provided user accounts + - Grant [migration viewer role](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#using_predefined_roles) to provided user accounts + + +## Variables + +| name | description | type | required | default | +|---|---|:---:|:---:|:---:| +| [migration_admin_users](variables.tf#L15) | List of users authorized to create a new M4CE sources and perform all other migration operations, in IAM format | list(string) | ✓ | | +| [migration_target_projects](variables.tf#L20) | List of target projects for m4ce workload migrations | list(string) | ✓ | | +| [migration_viewer_users](variables.tf#L25) | List of users authorized to retrive information about M4CE in the Google Cloud Console, in IAM format | list(string) | | [] | +| [project_create](variables.tf#L31) | Parameters for the creation of the new project to host the M4CE backend | object({…}) | | null | +| [project_name](variables.tf#L40) | Name of an existing project or of the new project assigned as M4CE host project | string | | "m4ce-host-project-000" | + +## Outputs + +| name | description | sensitive | +|---|---|:---:| +| [m4ce_gmanaged_service_account](outputs.tf#L15) | Google managed service account created automatically during the migrate connector registration.. It is used by M4CE to perform activities on target projects | | + + diff --git a/examples/cloud-operations/vm-migration/host-target-projects/backend.tf.sample b/examples/cloud-operations/vm-migration/host-target-projects/backend.tf.sample new file mode 100644 index 00000000..4f2bb336 --- /dev/null +++ b/examples/cloud-operations/vm-migration/host-target-projects/backend.tf.sample @@ -0,0 +1,20 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +terraform { + backend "gcs" { + bucket = "" + } +} diff --git a/examples/cloud-operations/vm-migration/host-target-projects/diagram.png b/examples/cloud-operations/vm-migration/host-target-projects/diagram.png new file mode 100644 index 00000000..ae0eaf34 Binary files /dev/null and b/examples/cloud-operations/vm-migration/host-target-projects/diagram.png differ diff --git a/examples/cloud-operations/vm-migration/host-target-projects/main.tf b/examples/cloud-operations/vm-migration/host-target-projects/main.tf new file mode 100644 index 00000000..a0e899d5 --- /dev/null +++ b/examples/cloud-operations/vm-migration/host-target-projects/main.tf @@ -0,0 +1,73 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +module "host-project" { + source = "../../../../modules/project" + billing_account = (var.project_create != null + ? var.project_create.billing_account_id + : null + ) + name = var.project_name + parent = (var.project_create != null + ? var.project_create.parent + : null + ) + + services = [ + "cloudresourcemanager.googleapis.com", + "compute.googleapis.com", + "iam.googleapis.com", + "logging.googleapis.com", + "servicemanagement.googleapis.com", + "servicecontrol.googleapis.com", + "vmmigration.googleapis.com", + ] + + project_create = var.project_create != null + + iam_additive = { + "roles/iam.serviceAccountKeyAdmin" = var.migration_admin_users, + "roles/iam.serviceAccountCreator" = var.migration_admin_users, + "roles/vmmigration.admin" = var.migration_admin_users, + "roles/vmmigration.viewer" = var.migration_viewer_users, + } +} + +module "m4ce-service-account" { + source = "../../../../modules/iam-service-account" + project_id = module.host-project.project_id + name = "m4ce-sa" + generate_key = true +} + +module "target-projects" { + for_each = toset(var.migration_target_projects) + source = "../../../../modules/project" + name = each.key + project_create = false + + services = [ + "servicemanagement.googleapis.com", + "servicecontrol.googleapis.com", + "iam.googleapis.com", + "cloudresourcemanager.googleapis.com", + "compute.googleapis.com" + ] + + iam_additive = { + "roles/resourcemanager.projectIamAdmin" = var.migration_admin_users, + "roles/compute.viewer" = var.migration_admin_users, + "roles/iam.serviceAccountUser" = var.migration_admin_users + } +} diff --git a/examples/cloud-operations/vm-migration/host-target-projects/outputs.tf b/examples/cloud-operations/vm-migration/host-target-projects/outputs.tf new file mode 100644 index 00000000..ef78d4c7 --- /dev/null +++ b/examples/cloud-operations/vm-migration/host-target-projects/outputs.tf @@ -0,0 +1,18 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +output "m4ce_gmanaged_service_account" { + description = "Google managed service account created automatically during the migrate connector registration.. It is used by M4CE to perform activities on target projects" + value = "serviceAccount:service-${module.host-project.number}@gcp-sa-vmmigration.iam.gserviceaccount.com" +} diff --git a/examples/cloud-operations/vm-migration/host-target-projects/variables.tf b/examples/cloud-operations/vm-migration/host-target-projects/variables.tf new file mode 100644 index 00000000..f6e3345f --- /dev/null +++ b/examples/cloud-operations/vm-migration/host-target-projects/variables.tf @@ -0,0 +1,44 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +variable "migration_admin_users" { + description = "List of users authorized to create a new M4CE sources and perform all other migration operations, in IAM format" + type = list(string) +} + +variable "migration_target_projects" { + description = "List of target projects for m4ce workload migrations" + type = list(string) +} + +variable "migration_viewer_users" { + description = "List of users authorized to retrive information about M4CE in the Google Cloud Console, in IAM format" + type = list(string) + default = [] +} + +variable "project_create" { + description = "Parameters for the creation of the new project to host the M4CE backend" + type = object({ + billing_account_id = string + parent = string + }) + default = null +} + +variable "project_name" { + description = "Name of an existing project or of the new project assigned as M4CE host project" + type = string + default = "m4ce-host-project-000" +} diff --git a/examples/cloud-operations/vm-migration/host-target-sharedvpc/README.md b/examples/cloud-operations/vm-migration/host-target-sharedvpc/README.md new file mode 100644 index 00000000..bf82f236 --- /dev/null +++ b/examples/cloud-operations/vm-migration/host-target-sharedvpc/README.md @@ -0,0 +1,44 @@ +# M4CE(v5) - Host and Target Projects with Shared VPC + +This example creates a Migrate for Compute Engine (v5) environment deployed on an host project with multiple [target projects](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#identifying_your_host_project) and shared VPCs. + +The example is designed to implement a M4CE (v5) environment on-top of complex migration landing environment where VMs have to be migrated to multiple target projects. In this example targets are alse service projects for a shared VPC. It also includes the IAM wiring needed to make such scenarios work. + +This is the high level diagram: + +![High-level diagram](diagram.png "High-level diagram") + +## Managed resources and services + +This sample creates\update several distinct groups of resources: + +- projects + - M4CE host project with [required services](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#enabling_required_services_on_the_host_project) deployed on a new or existing project. + - M4CE target project prerequisites deployed on existing projects. +- IAM + - Create a [service account](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/migrate-connector#step-3) used at runtime by the M4CE connector for data replication + - Grant [migration admin roles](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#using_predefined_roles) to provided user accounts. + - Grant [migration viewer role](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#using_predefined_roles) to provided user accounts. + - Grant [roles on shared VPC](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/target-project#configure-permissions) to migration admins + + +## Variables + +| name | description | type | required | default | +|---|---|:---:|:---:|:---:| +| [migration_admin_users](variables.tf#L15) | List of users authorized to create a new M4CE sources and perform all other migration operations, in IAM format | list(string) | ✓ | | +| [migration_target_projects](variables.tf#L20) | List of target projects for m4ce workload migrations | list(string) | ✓ | | +| [sharedvpc_host_projects](variables.tf#L45) | List of host projects that share a VPC with the selected target projects | list(string) | ✓ | | +| [migration_viewer_users](variables.tf#L25) | List of users authorized to retrive information about M4CE in the Google Cloud Console, in IAM format | list(string) | | [] | +| [project_create](variables.tf#L30) | Parameters for the creation of the new project to host the M4CE backend | object({…}) | | null | +| [project_name](variables.tf#L39) | Name of an existing project or of the new project assigned as M4CE host project | string | | "m4ce-host-project-000" | + +## Outputs + +| name | description | sensitive | +|---|---|:---:| +| [m4ce_gmanaged_service_account](outputs.tf#L15) | Google managed service account created automatically during the migrate connector registration. It is used by M4CE to perform activities on target projects | | + + +## Manual Steps +Once this example is deployed the M4CE [m4ce_gmanaged_service_account](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/target-sa-compute-engine#configuring_the_default_service_account) has to be configured to grant the access to the shared VPC and allow the deploy of Compute Engine instances as the result of the migration. \ No newline at end of file diff --git a/examples/cloud-operations/vm-migration/host-target-sharedvpc/backend.tf.sample b/examples/cloud-operations/vm-migration/host-target-sharedvpc/backend.tf.sample new file mode 100644 index 00000000..4f2bb336 --- /dev/null +++ b/examples/cloud-operations/vm-migration/host-target-sharedvpc/backend.tf.sample @@ -0,0 +1,20 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +terraform { + backend "gcs" { + bucket = "" + } +} diff --git a/examples/cloud-operations/vm-migration/host-target-sharedvpc/diagram.png b/examples/cloud-operations/vm-migration/host-target-sharedvpc/diagram.png new file mode 100644 index 00000000..0746e8de Binary files /dev/null and b/examples/cloud-operations/vm-migration/host-target-sharedvpc/diagram.png differ diff --git a/examples/cloud-operations/vm-migration/host-target-sharedvpc/main.tf b/examples/cloud-operations/vm-migration/host-target-sharedvpc/main.tf new file mode 100644 index 00000000..803d1dd2 --- /dev/null +++ b/examples/cloud-operations/vm-migration/host-target-sharedvpc/main.tf @@ -0,0 +1,84 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +module "host-project" { + source = "../../../../modules/project" + billing_account = (var.project_create != null + ? var.project_create.billing_account_id + : null + ) + name = var.project_name + parent = (var.project_create != null + ? var.project_create.parent + : null + ) + + services = [ + "cloudresourcemanager.googleapis.com", + "compute.googleapis.com", + "iam.googleapis.com", + "logging.googleapis.com", + "servicemanagement.googleapis.com", + "servicecontrol.googleapis.com", + "vmmigration.googleapis.com", + ] + + project_create = var.project_create != null + + iam_additive = { + "roles/iam.serviceAccountKeyAdmin" = var.migration_admin_users, + "roles/iam.serviceAccountCreator" = var.migration_admin_users, + "roles/vmmigration.admin" = var.migration_admin_users, + "roles/vmmigration.viewer" = var.migration_viewer_users, + } +} + +module "m4ce-service-account" { + source = "../../../../modules/iam-service-account" + project_id = module.host-project.project_id + name = "m4ce-sa" +} + +module "target-projects" { + + for_each = toset(var.migration_target_projects) + source = "../../../../modules/project" + name = each.key + project_create = false + + services = [ + "cloudresourcemanager.googleapis.com", + "compute.googleapis.com", + "iam.googleapis.com", + "servicemanagement.googleapis.com", + "servicecontrol.googleapis.com", + ] + + iam_additive = { + "roles/resourcemanager.projectIamAdmin" = var.migration_admin_users, + "roles/iam.serviceAccountUser" = var.migration_admin_users, + } +} + +module "sharedvpc_host_project" { + + for_each = toset(var.sharedvpc_host_projects) + source = "../../../../modules/project" + name = each.key + project_create = false + + iam_additive = { + "roles/compute.viewer" = var.migration_admin_users, + } +} diff --git a/examples/cloud-operations/vm-migration/host-target-sharedvpc/outputs.tf b/examples/cloud-operations/vm-migration/host-target-sharedvpc/outputs.tf new file mode 100644 index 00000000..3e6d553d --- /dev/null +++ b/examples/cloud-operations/vm-migration/host-target-sharedvpc/outputs.tf @@ -0,0 +1,18 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +output "m4ce_gmanaged_service_account" { + description = "Google managed service account created automatically during the migrate connector registration. It is used by M4CE to perform activities on target projects" + value = "serviceAccount:service-${module.host-project.number}@gcp-sa-vmmigration.iam.gserviceaccount.com" +} diff --git a/examples/cloud-operations/vm-migration/host-target-sharedvpc/variables.tf b/examples/cloud-operations/vm-migration/host-target-sharedvpc/variables.tf new file mode 100644 index 00000000..85f333ce --- /dev/null +++ b/examples/cloud-operations/vm-migration/host-target-sharedvpc/variables.tf @@ -0,0 +1,48 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +variable "migration_admin_users" { + description = "List of users authorized to create a new M4CE sources and perform all other migration operations, in IAM format" + type = list(string) +} + +variable "migration_target_projects" { + description = "List of target projects for m4ce workload migrations" + type = list(string) +} + +variable "migration_viewer_users" { + description = "List of users authorized to retrive information about M4CE in the Google Cloud Console, in IAM format" + type = list(string) + default = [] +} +variable "project_create" { + description = "Parameters for the creation of the new project to host the M4CE backend" + type = object({ + billing_account_id = string + parent = string + }) + default = null +} + +variable "project_name" { + description = "Name of an existing project or of the new project assigned as M4CE host project" + type = string + default = "m4ce-host-project-000" +} + +variable "sharedvpc_host_projects" { + description = "List of host projects that share a VPC with the selected target projects" + type = list(string) +} diff --git a/examples/cloud-operations/vm-migration/single-project/README.md b/examples/cloud-operations/vm-migration/single-project/README.md new file mode 100644 index 00000000..d196ed8b --- /dev/null +++ b/examples/cloud-operations/vm-migration/single-project/README.md @@ -0,0 +1,41 @@ +# M4CE(v5) - Single Project + +This sample creates a simple M4CE (v5) environment deployed on a signle [host project](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#identifying_your_host_project). + +The example is designed for quick tests or product demos where it is required to setup a simple and minimal M4CE (v5) environment. It also includes the IAM wiring needed to make such scenarios work. + +This is the high level diagram: + +![High-level diagram](diagram.png "High-level diagram") + +## Managed resources and services + +This sample creates several distinct groups of resources: + +- projects + - M4CE host project with [required services](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#enabling_required_services_on_the_host_project) deployed on a new or existing project. +- networking + - Default VPC network +- IAM + - One [service account](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/migrate-connector#step-3) used at runtime by the M4CE connector for data replication + - Grant [migration admin roles](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#using_predefined_roles) to admin user accounts + - Grant [migration viewer role](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#using_predefined_roles) to viewer user accounts + + +## Variables + +| name | description | type | required | default | +|---|---|:---:|:---:|:---:| +| [migration_admin_users](variables.tf#L15) | List of users authorized to create a new M4CE sources and perform all other migration operations, in IAM format | list(string) | ✓ | | +| [migration_viewer_users](variables.tf#L20) | List of users authorized to retrive information about M4CE in the Google Cloud Console, in IAM format | list(string) | | [] | +| [project_create](variables.tf#L26) | Parameters for the creation of the new project to host the M4CE backend | object({…}) | | null | +| [project_name](variables.tf#L35) | Name of an existing project or of the new project assigned as M4CE host an target project | string | | "m4ce-host-project-000" | +| [vpc_config](variables.tf#L41) | Parameters to create a simple VPC on the M4CE project | object({…}) | | {…} | + +## Outputs + +| name | description | sensitive | +|---|---|:---:| +| [m4ce_gmanaged_service_account](outputs.tf#L15) | Google managed service account created automatically during the migrate connector registration. It is used by M4CE to perform activities on target projects | | + + diff --git a/examples/cloud-operations/vm-migration/single-project/backend.tf.sample b/examples/cloud-operations/vm-migration/single-project/backend.tf.sample new file mode 100644 index 00000000..4f2bb336 --- /dev/null +++ b/examples/cloud-operations/vm-migration/single-project/backend.tf.sample @@ -0,0 +1,20 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +terraform { + backend "gcs" { + bucket = "" + } +} diff --git a/examples/cloud-operations/vm-migration/single-project/diagram.png b/examples/cloud-operations/vm-migration/single-project/diagram.png new file mode 100644 index 00000000..b47bb295 Binary files /dev/null and b/examples/cloud-operations/vm-migration/single-project/diagram.png differ diff --git a/examples/cloud-operations/vm-migration/single-project/main.tf b/examples/cloud-operations/vm-migration/single-project/main.tf new file mode 100644 index 00000000..6adf9fbd --- /dev/null +++ b/examples/cloud-operations/vm-migration/single-project/main.tf @@ -0,0 +1,90 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +module "landing-project" { + source = "../../../../modules/project" + billing_account = (var.project_create != null + ? var.project_create.billing_account_id + : null + ) + name = var.project_name + parent = (var.project_create != null + ? var.project_create.parent + : null + ) + + services = [ + "cloudresourcemanager.googleapis.com", + "compute.googleapis.com", + "iam.googleapis.com", + "logging.googleapis.com", + "networkconnectivity.googleapis.com", + "servicemanagement.googleapis.com", + "servicecontrol.googleapis.com", + "vmmigration.googleapis.com" + ] + + project_create = var.project_create != null + + iam_additive = { + "roles/iam.serviceAccountKeyAdmin" = var.migration_admin_users, + "roles/iam.serviceAccountCreator" = var.migration_admin_users, + "roles/vmmigration.admin" = var.migration_admin_users, + "roles/vmmigration.viewer" = var.migration_viewer_users + } +} + +module "m4ce-service-account" { + source = "../../../../modules/iam-service-account" + project_id = module.landing-project.project_id + name = "m4ce-sa" + generate_key = true +} + +module "landing-vpc" { + source = "../../../../modules/net-vpc" + project_id = module.landing-project.project_id + name = "landing-vpc" + subnets = [ + { + ip_cidr_range = var.vpc_config.ip_cidr_range + name = "landing-vpc-${var.vpc_config.region}" + region = var.vpc_config.region + secondary_ip_range = {} + } + ] +} + +module "landing-vpc-firewall" { + source = "../../../../modules/net-vpc-firewall" + project_id = module.landing-project.project_id + network = module.landing-vpc.name + admin_ranges = [] + http_source_ranges = [] + https_source_ranges = [] + ssh_source_ranges = [] + custom_rules = { + allow-ssh = { + description = "Allow SSH from IAP" + direction = "INGRESS" + action = "allow" + sources = [] + ranges = ["35.235.240.0/20"] + targets = [] + use_service_accounts = false + rules = [{ protocol = "tcp", ports = ["22"] }] + extra_attributes = {} + } + } +} diff --git a/examples/cloud-operations/vm-migration/single-project/outputs.tf b/examples/cloud-operations/vm-migration/single-project/outputs.tf new file mode 100644 index 00000000..347eb54f --- /dev/null +++ b/examples/cloud-operations/vm-migration/single-project/outputs.tf @@ -0,0 +1,18 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +output "m4ce_gmanaged_service_account" { + description = "Google managed service account created automatically during the migrate connector registration. It is used by M4CE to perform activities on target projects" + value = "serviceAccount:service-${module.landing-project.number}@gcp-sa-vmmigration.iam.gserviceaccount.com" +} diff --git a/examples/cloud-operations/vm-migration/single-project/variables.tf b/examples/cloud-operations/vm-migration/single-project/variables.tf new file mode 100644 index 00000000..2d7214f4 --- /dev/null +++ b/examples/cloud-operations/vm-migration/single-project/variables.tf @@ -0,0 +1,51 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +variable "migration_admin_users" { + description = "List of users authorized to create a new M4CE sources and perform all other migration operations, in IAM format" + type = list(string) +} + +variable "migration_viewer_users" { + description = "List of users authorized to retrive information about M4CE in the Google Cloud Console, in IAM format" + type = list(string) + default = [] +} + +variable "project_create" { + description = "Parameters for the creation of the new project to host the M4CE backend" + type = object({ + billing_account_id = string + parent = string + }) + default = null +} + +variable "project_name" { + description = "Name of an existing project or of the new project assigned as M4CE host an target project" + type = string + default = "m4ce-host-project-000" +} + +variable "vpc_config" { + description = "Parameters to create a simple VPC on the M4CE project" + type = object({ + ip_cidr_range = string, + region = string + }) + default = { + ip_cidr_range = "10.200.0.0/20", + region = "us-west2" + } +} diff --git a/examples/data-solutions/gcs-to-bq-with-least-privileges/README.md b/examples/data-solutions/gcs-to-bq-with-least-privileges/README.md index 078ba264..8804fcc9 100644 --- a/examples/data-solutions/gcs-to-bq-with-least-privileges/README.md +++ b/examples/data-solutions/gcs-to-bq-with-least-privileges/README.md @@ -3,17 +3,18 @@ This example creates the infrastructure needed to run a [Cloud Dataflow](https://cloud.google.com/dataflow) pipeline to import data from [GCS](https://cloud.google.com/storage) to [Bigquery](https://cloud.google.com/bigquery). The example will create different service accounts with least privileges on resources. To run the pipeline, users listed in `data_eng_principals` can impersonate all those service accounts. The solution will use: - - internal IPs for GCE and Cloud Dataflow instances - - Cloud NAT to let resources egress to the Internet, to run system updates and install packages - - rely on [Service Account Impersonation](https://cloud.google.com/iam/docs/impersonating-service-accounts) to avoid the use of service account keys - - Service Accounts with least privilege on each resource - - (Optional) CMEK encription for GCS bucket, DataFlow instances and BigQuery tables - -The example is designed to match real-world use cases with a minimum amount of resources and some compromise listed below. It can be used as a starting point for more complex scenarios. +- internal IPs for GCE and Cloud Dataflow instances +- Cloud NAT to let resources egress to the Internet, to run system updates and install packages +- rely on [Service Account Impersonation](https://cloud.google.com/iam/docs/impersonating-service-accounts) to avoid the use of service account keys +- Service Accounts with least privilege on each resource +- (Optional) CMEK encription for GCS bucket, DataFlow instances and BigQuery tables + +The example is designed to match real-world use cases with a minimum amount of resources and some compromises listed below. It can be used as a starting point for more complex scenarios. This is the high level diagram: ![GCS to Biquery High-level diagram](diagram.png "GCS to Biquery High-level diagram") + ## Move to real use case consideration In the example we implemented some compromise to keep the example minimal and easy to read. On a real word use case, you may evaluate the option to: - Configure a Shared-VPC @@ -93,13 +94,13 @@ We need to create 3 file: - A `person_udf.js` containing the UDF javascript file used by the Dataflow template. - A `person_schema.json` file containing the table schema used to import the CSV. -You can find an example of those file in the folder `./data-demo`. You can copy the example files in the GCS bucket using the command returned in the terraform output as `command-01-gcs`. Below an example: +You can find an example of those file in the folder `./data-demo`. You can copy the example files in the GCS bucket using the command returned in the terraform output as `command_01_gcs`. Below an example: ```bash gsutil -i gcs-landing@PROJECT.iam.gserviceaccount.com cp data-demo/* gs://LANDING_BUCKET ``` -We can now run the Dataflow pipeline using the `gcloud` returned in the terraform output as `command-02-dataflow`. Below an example: +We can now run the Dataflow pipeline using the `gcloud` returned in the terraform output as `command_02_dataflow`. Below an example: ```bash gcloud --impersonate-service-account=orch-test@PROJECT.iam.gserviceaccount.com dataflow jobs run test_batch_01 \ @@ -119,7 +120,7 @@ outputTable=PROJECT:datalake.person,\ bigQueryLoadingTemporaryDirectory=gs://PREFIX-df-tmp ``` -You can check data imported into Google BigQuery using the command returned in the terraform output as `command-03-bq`. Below an example: +You can check data imported into Google BigQuery using the command returned in the terraform output as `command_03_bq`. Below an example: ``` bq query --use_legacy_sql=false 'SELECT * FROM `PROJECT.datalake.person` LIMIT 1000' @@ -144,10 +145,10 @@ bq query --use_legacy_sql=false 'SELECT * FROM `PROJECT.datalake.person` LIMIT 1 |---|---|:---:| | [bq_tables](outputs.tf#L15) | Bigquery Tables. | | | [buckets](outputs.tf#L20) | GCS bucket Cloud KMS crypto keys. | | -| [command-01-gcs](outputs.tf#L43) | gcloud command to copy data into the created bucket impersonating the service account. | | -| [command-02-dataflow](outputs.tf#L48) | Command to run Dataflow template impersonating the service account. | | -| [command-03-bq](outputs.tf#L70) | BigQuery command to query imported data. | | +| [command_01_gcs](outputs.tf#L43) | gcloud command to copy data into the created bucket impersonating the service account. | | +| [command_02_dataflow](outputs.tf#L48) | Command to run Dataflow template impersonating the service account. | | +| [command_03_bq](outputs.tf#L69) | BigQuery command to query imported data. | | | [project_id](outputs.tf#L28) | Project id. | | -| [serviceaccount](outputs.tf#L33) | Service account. | | +| [service_accounts](outputs.tf#L33) | Service account. | | diff --git a/examples/data-solutions/gcs-to-bq-with-least-privileges/bigquery.tftpl b/examples/data-solutions/gcs-to-bq-with-least-privileges/bigquery.tftpl new file mode 100644 index 00000000..3be2492e --- /dev/null +++ b/examples/data-solutions/gcs-to-bq-with-least-privileges/bigquery.tftpl @@ -0,0 +1,2 @@ +bq query --project_id=${project_id} --use_legacy_sql=false \ + 'SELECT * FROM `${project_id}.${bigquery_dataset}.${bigquery_table}` LIMIT ${sql_limit}' \ No newline at end of file diff --git a/examples/data-solutions/gcs-to-bq-with-least-privileges/dataflow.tftpl b/examples/data-solutions/gcs-to-bq-with-least-privileges/dataflow.tftpl new file mode 100644 index 00000000..820770fb --- /dev/null +++ b/examples/data-solutions/gcs-to-bq-with-least-privileges/dataflow.tftpl @@ -0,0 +1,18 @@ +gcloud \ + --impersonate-service-account=${sa_orch_email} \ + dataflow jobs run test_batch_01 \ + --gcs-location gs://dataflow-templates/latest/GCS_Text_to_BigQuery \ + --project ${project_id} \ + --region ${region} \ + --disable-public-ips \ + --subnetwork ${subnet} \ + --staging-location ${gcs_df_stg} \ + --service-account-email ${sa_df_email} \ + %{ if cmek_encryption }--dataflow-kms-key=${kms_key_df} %{ endif } \ + --parameters \ +javascriptTextTransformFunctionName=transform,\ +JSONPath=${data_schema_file},\ +javascriptTextTransformGcsPath=${data_udf_file},\ +inputFilePattern=${data_file},\ +outputTable=${project_id}:${bigquery_dataset}.${bigquery_table},\ +bigQueryLoadingTemporaryDirectory=${gcs_df_tmp} \ No newline at end of file diff --git a/examples/data-solutions/gcs-to-bq-with-least-privileges/datastorage.tf b/examples/data-solutions/gcs-to-bq-with-least-privileges/datastorage.tf index 35711125..98b15bb2 100644 --- a/examples/data-solutions/gcs-to-bq-with-least-privileges/datastorage.tf +++ b/examples/data-solutions/gcs-to-bq-with-least-privileges/datastorage.tf @@ -12,10 +12,6 @@ # See the License for the specific language governing permissions and # limitations under the License. -############################################################################### -# GCS # -############################################################################### - module "gcs-data" { source = "../../../modules/gcs" project_id = module.project.project_id @@ -23,7 +19,7 @@ module "gcs-data" { name = "data" location = var.region storage_class = "REGIONAL" - encryption_key = var.cmek_encryption ? try(module.kms[0].keys.key-gcs.id, null) : null + encryption_key = var.cmek_encryption ? module.kms[0].keys.key-gcs.id : null force_destroy = true } @@ -34,22 +30,20 @@ module "gcs-df-tmp" { name = "df-tmp" location = var.region storage_class = "REGIONAL" - encryption_key = var.cmek_encryption ? try(module.kms[0].keys.key-gcs.id, null) : null + encryption_key = var.cmek_encryption ? module.kms[0].keys.key-gcs.id : null force_destroy = true } -############################################################################### -# BQ # -############################################################################### - module "bigquery-dataset" { source = "../../../modules/bigquery-dataset" project_id = module.project.project_id id = "datalake" location = var.region - # Define Tables in Terraform for the porpuse of the example. - # Probably in a production environment you would handle Tables creation in a - # separate Terraform State or using a different tool/pipeline (for example: Dataform). + + # Note: we define tables in Terraform for the purpose of this + # example. A production environment would probably handle table + # creation in a separate terraform pipeline or using a different + # tool (for example: Dataform) tables = { person = { friendly_name = "Person. Dataflow import." @@ -64,10 +58,10 @@ module "bigquery-dataset" { deletion_protection = false options = { clustering = null - encryption_key = var.cmek_encryption ? try(module.kms[0].keys.key-bq.id, null) : null + encryption_key = var.cmek_encryption ? module.kms[0].keys.key-bq.id : null expiration_time = null } } } - encryption_key = var.cmek_encryption ? try(module.kms[0].keys.key-bq.id, null) : null + encryption_key = var.cmek_encryption ? module.kms[0].keys.key-bq.id : null } diff --git a/examples/data-solutions/gcs-to-bq-with-least-privileges/kms.tf b/examples/data-solutions/gcs-to-bq-with-least-privileges/kms.tf index d7133542..5e616630 100644 --- a/examples/data-solutions/gcs-to-bq-with-least-privileges/kms.tf +++ b/examples/data-solutions/gcs-to-bq-with-least-privileges/kms.tf @@ -17,7 +17,7 @@ module "kms" { source = "../../../modules/kms" project_id = module.project.project_id keyring = { - name = "${var.prefix}-keyring", + name = "${var.prefix}-keyring" location = var.region } keys = { diff --git a/examples/data-solutions/gcs-to-bq-with-least-privileges/main.tf b/examples/data-solutions/gcs-to-bq-with-least-privileges/main.tf index e2a50e4f..8c848644 100644 --- a/examples/data-solutions/gcs-to-bq-with-least-privileges/main.tf +++ b/examples/data-solutions/gcs-to-bq-with-least-privileges/main.tf @@ -58,19 +58,18 @@ locals { module.service-account-orch.iam_email, ] "roles/iam.serviceAccountTokenCreator" = concat( - var.data_eng_principals, - ) - "roles/viewer" = concat( var.data_eng_principals ) # Dataflow roles - "roles/dataflow.admin" = concat([ - module.service-account-orch.iam_email, - ], var.data_eng_principals + "roles/dataflow.admin" = concat( + [module.service-account-orch.iam_email], + var.data_eng_principals ) "roles/dataflow.worker" = [ module.service-account-df.iam_email, ] + "roles/dataflow.developer" = var.data_eng_principals + "roles/compute.viewer" = var.data_eng_principals # network roles "roles/compute.networkUser" = [ module.service-account-df.iam_email, @@ -79,10 +78,6 @@ locals { } } -############################################################################### -# Projects # -############################################################################### - module "project" { source = "../../../modules/project" name = var.project_id @@ -101,6 +96,7 @@ module "project" { "storage.googleapis.com", "storage-component.googleapis.com", ] + # additive IAM bindings avoid disrupting bindings in existing project iam = var.project_create != null ? local.iam : {} iam_additive = var.project_create == null ? local.iam : {} diff --git a/examples/data-solutions/gcs-to-bq-with-least-privileges/outputs.tf b/examples/data-solutions/gcs-to-bq-with-least-privileges/outputs.tf index 2114d2f9..7ffd0634 100644 --- a/examples/data-solutions/gcs-to-bq-with-least-privileges/outputs.tf +++ b/examples/data-solutions/gcs-to-bq-with-least-privileges/outputs.tf @@ -30,7 +30,7 @@ output "project_id" { value = module.project.project_id } -output "serviceaccount" { +output "service_accounts" { description = "Service account." value = { bq = module.service-account-bq.email @@ -40,36 +40,38 @@ output "serviceaccount" { } } -output "command-01-gcs" { +output "command_01_gcs" { description = "gcloud command to copy data into the created bucket impersonating the service account." value = "gsutil -i ${module.service-account-landing.email} cp data-demo/* ${module.gcs-data.url}" } -output "command-02-dataflow" { +output "command_02_dataflow" { description = "Command to run Dataflow template impersonating the service account." - value = < providers.tf # migrate state to GCS bucket configured in providers file terraform init -migrate-state +# run terraform apply to remo user iam binding +terraform apply ``` ## Customizations diff --git a/fast/stages/01-resman/README.md b/fast/stages/01-resman/README.md index 098c6506..50f8edaf 100644 --- a/fast/stages/01-resman/README.md +++ b/fast/stages/01-resman/README.md @@ -53,8 +53,8 @@ To simplify setup, the previous stage pre-configures a valid providers file in i If you have set a valid value for `outputs_location` in the bootstrap stage, simply link the relevant `providers.tf` file from this stage's folder in the path you specified: ```bash -# `outputs_location` is set to `../../configs/example` -ln -s ../../configs/example/01-resman/providers.tf +# `outputs_location` is set to `../../config` +ln -s ../../config/01-resman/providers.tf ``` If you have not configured `outputs_location` in bootstrap, you can derive the providers file from that stage's outputs: @@ -65,6 +65,8 @@ terraform output -json providers | jq -r '.["01-resman"]' \ > ../01-resman/providers.tf ``` +If you want to continue to rely on `outputs_location` logic, create a `terraform.tfvars` file and configure it as deacribed [here](../00-bootstrap/#output-files-and-cross-stage-variables). + ### Variable configuration There are two broad sets of variables you will need to fill in: @@ -77,8 +79,8 @@ To avoid the tedious job of filling in the first group of variable with values d If you configured a valid path for `outputs_location` in the bootstrap stage, simply link the relevant `terraform-*.auto.tfvars.json` files from this stage's outputs folder (under the path you specified), where the `*` above is set to the name of the stage that produced it. For this stage, a single `.tfvars` file is avalaible: ```bash -# `outputs_location` is set to `../../configs/example` -ln -s ../../configs/example/01-resman/terraform-bootstrap.auto.tfvars.json +# `outputs_location` is set to `../../config` +ln -s ../../config/01-resman/terraform-bootstrap.auto.tfvars.json ``` A second set of variables is specific to this stage, they are all optional so if you need to customize them, create an extra `terraform.tfvars` file. diff --git a/fast/stages/02-networking/README.md b/fast/stages/02-networking/README.md index 3f985475..f655a174 100644 --- a/fast/stages/02-networking/README.md +++ b/fast/stages/02-networking/README.md @@ -136,8 +136,8 @@ To simplify setup, the previous stage pre-configures a valid providers file in i If you have set a valid value for `outputs_location` in the bootstrap stage, simply link the relevant `providers.tf` file from this stage's folder in the path you specified: ```bash -# `outputs_location` is set to `../../configs/example` -ln -s ../../configs/example/02-networking/providers.tf +# `outputs_location` is set to `../../config` +ln -s ../../config/02-networking/providers.tf ``` If you have not configured `outputs_location` in bootstrap, you can derive the providers file from that stage's outputs: @@ -160,11 +160,13 @@ To avoid the tedious job of filling in the first group of variables with values If you have set a valid value for `outputs_location` in the bootstrap and in the resman stage, simply link the relevant `terraform-*.auto.tfvars.json` files from this stage's folder in the path you specified, where the `*` above is set to the name of the stage that produced it. For this stage, a single `.tfvars` file is available: ```bash -# `outputs_location` is set to `../../configs/example` -ln -s ../../configs/example/02-networking/terraform-bootstrap.auto.tfvars.json -ln -s ../../configs/example/02-networking/terraform-resman.auto.tfvars.json +# `outputs_location` is set to `../../config` +ln -s ../../config/02-networking/terraform-bootstrap.auto.tfvars.json +ln -s ../../config/02-networking/terraform-resman.auto.tfvars.json ``` +If you want to continue to rely on `outputs_location` logic, create a `terraform.tfvars` file and configure it as deacribed [here](../00-bootstrap/#output-files-and-cross-stage-variables). + Please refer to the [Variables](#variables) table below for a map of the variable origins, and to the sections below on how to adapt this stage to your networking configuration. ### VPCs diff --git a/fast/stages/02-security/README.md b/fast/stages/02-security/README.md index 36797f2f..621db3e9 100644 --- a/fast/stages/02-security/README.md +++ b/fast/stages/02-security/README.md @@ -57,8 +57,8 @@ To simplify setup, the previous stage pre-configures a valid providers file in i If you have set a valid value for `outputs_location` in the resource management stage, simply link the relevant `providers.tf` file from this stage's folder in the path you specified: ```bash -# `outputs_location` is set to `../../configs/example` -ln -s ../../configs/example/02-security/providers.tf +# `outputs_location` is set to `../../config` +ln -s ../../config/02-security/providers.tf ``` If you have not configured `outputs_location` in resource management, you can derive the providers file from that stage's outputs: @@ -69,6 +69,8 @@ terraform output -json providers | jq -r '.["02-security"]' \ > ../02-security/providers.tf ``` +If you want to continue to rely on `outputs_location` logic, create a `terraform.tfvars` file and configure it as deacribed [here](../00-bootstrap/#output-files-and-cross-stage-variables). + ### Variable configuration There are two broad sets of variables you will need to fill in: @@ -81,9 +83,9 @@ To avoid the tedious job of filling in the first group of variables with values If you configured a valid path for `outputs_location` in the previous stages, simply link the relevant `terraform-*.auto.tfvars.json` files from this stage's output folder (under the path you specified), where the `*` above is set to the name of the stage that produced it. For this stage, two `.tfvars` files are available: ```bash -# `outputs_location` is set to `../../configs/example` -ln -s ../../configs/example/02-security/terraform-bootstrap.auto.tfvars.json -ln -s ../../configs/example/02-security/terraform-resman.auto.tfvars.json +# `outputs_location` is set to `../../config` +ln -s ../../config/02-security/terraform-bootstrap.auto.tfvars.json +ln -s ../../config/02-security/terraform-resman.auto.tfvars.json ``` A second set of optional variables is specific to this stage. If you need to customize them, create an extra `terraform.tfvars` file. diff --git a/fast/stages/03-project-factory/prod/README.md b/fast/stages/03-project-factory/prod/README.md index 34f9be88..70b6cfce 100644 --- a/fast/stages/03-project-factory/prod/README.md +++ b/fast/stages/03-project-factory/prod/README.md @@ -15,51 +15,50 @@ Projects for each environment across different teams are created by dedicated se The project factory takes care of the following activities: -* Project creation -* API/Services enablement -* Service accounts creation -* IAM roles assignment for groups and service accounts -* KMS keys roles assignment -* Shared VPC attachment and subnets IAM binding -* DNS zones creation and visibility configuration -* Project-level org policies definition -* Billing setup (billing account attachment and budget configuration) -* Essential contacts definition (for [budget alerts](https://cloud.google.com/billing/docs/how-to/budgets) and [important notifications](https://cloud.google.com/resource-manager/docs/managing-notification-contacts?hl=en)) +- Project creation +- API/Services enablement +- Service accounts creation +- IAM roles assignment for groups and service accounts +- KMS keys roles assignment +- Shared VPC attachment and subnets IAM binding +- DNS zones creation and visibility configuration +- Project-level org policies definition +- Billing setup (billing account attachment and budget configuration) +- Essential contacts definition (for [budget alerts](https://cloud.google.com/billing/docs/how-to/budgets) and [important notifications](https://cloud.google.com/resource-manager/docs/managing-notification-contacts?hl=en)) - ## How to run this stage This stage is meant to be executed after "foundational stages" (i.e., stages [`00-bootstrap`](../../00-bootstrap), [`01-resman`](../../01-resman), [`02-networking`](../../02-networking) and [`02-security`](../../02-security)) have been run. It's of course possible to run this stage in isolation, by making sure the architectural prerequisites are satisfied (e.g., networking), and that the Service Account running the stage is granted the roles/permissions below: -* One service account per environment, each with appropriate permissions - * at the organization level a custom role for networking operations including the following permissions - * `"compute.organizations.enableXpnResource"`, - * `"compute.organizations.disableXpnResource"`, - * `"compute.subnetworks.setIamPolicy"`, - * `"dns.networks.bindPrivateDNSZone"` - * and role `"roles/orgpolicy.policyAdmin"` - * on each folder where projects are created - * `"roles/logging.admin"` - * `"roles/owner"` - * `"roles/resourcemanager.folderAdmin"` - * `"roles/resourcemanager.projectCreator"` - * on the host project for the Shared VPC - * `"roles/browser"` - * `"roles/compute.viewer"` - * `"roles/dns.admin"` -* If networking is used (e.g., for VMs, GKE Clusters or AppEngine flex), VPC Host projects and their subnets should exist when creating projects -* If per-environment DNS sub-zones are required, one "root" zone per environment should exist when creating projects (e.g., prod.gcp.example.com.) +- One service account per environment, each with appropriate permissions + - at the organization level a custom role for networking operations including the following permissions + - `"compute.organizations.enableXpnResource"`, + - `"compute.organizations.disableXpnResource"`, + - `"compute.subnetworks.setIamPolicy"`, + - `"dns.networks.bindPrivateDNSZone"` + - and role `"roles/orgpolicy.policyAdmin"` + - on each folder where projects are created + - `"roles/logging.admin"` + - `"roles/owner"` + - `"roles/resourcemanager.folderAdmin"` + - `"roles/resourcemanager.projectCreator"` + - on the host project for the Shared VPC + - `"roles/browser"` + - `"roles/compute.viewer"` + - `"roles/dns.admin"` +- If networking is used (e.g., for VMs, GKE Clusters or AppEngine flex), VPC Host projects and their subnets should exist when creating projects +- If per-environment DNS sub-zones are required, one "root" zone per environment should exist when creating projects (e.g., prod.gcp.example.com.) ### Providers configuration If you're running this on top of Fast, you should run the following commands to create the providers file, and populate the required variables from the previous stage. ```bash -# Variable `outputs_location` is set to `../../configs/example` in stage 01-resman +# Variable `outputs_location` is set to `../../config` in stage 01-resman $ cd fabric-fast/stages/03-project-factory/prod -ln -s ../../../configs/example/03-project-factory-prod/providers.tf +ln -s ../../../config/03-project-factory-prod/providers.tf ``` ### Variable configuration @@ -74,18 +73,17 @@ To avoid the tedious job of filling in the first group of variables with values If you configured a valid path for `outputs_location` in the bootstrap and networking stage, simply link the relevant `terraform-*.auto.tfvars.json` files from this stage's outputs folder (under the path you specified), where the `*` above is set to the name of the stage that produced it. For this stage, a single `.tfvars` file is available: ```bash -# Variable `outputs_location` is set to `../../configs/example` in stages 01-bootstrap and 02-networking -ln -s ../../../configs/example/03-project-factory-prod/terraform-bootstrap.auto.tfvars.json -ln -s ../../../configs/example/03-project-factory-prod/terraform-networking.auto.tfvars.json +# Variable `outputs_location` is set to `../../config` in stages 01-bootstrap and 02-networking +ln -s ../../../config/03-project-factory-prod/terraform-bootstrap.auto.tfvars.json +ln -s ../../../config/03-project-factory-prod/terraform-networking.auto.tfvars.json ``` If you're not using Fast, refer to the [Variables](#variables) table at the bottom of this document for a full list of variables, their origin (e.g., a stage or specific to this one), and descriptions explaining their meaning. -Besides the values above, a project factory takes 2 additional inputs: +Besides the values above, a project factory takes 2 additional inputs: -* `data/defaults.yaml`, manually configured by adapting the [`prod/data/defaults.yaml.sample`](./prod/data/defaults.yaml.sample), which defines per-environment default values e.g., for billing alerts and labels. - -* `data/projects/*.yaml`, one file per project (optionally grouped in folders), which configures each project. A [`prod/data/projects/project.yaml.sample`](./prod/data/projects/project.yaml.sample) is provided as reference and documentation for the schema. Projects will be named after the filename, e.g., `fast-prod-lab0.yaml` will create project `fast-prod-lab0`. +- `data/defaults.yaml`, manually configured by adapting the [`prod/data/defaults.yaml.sample`](./prod/data/defaults.yaml.sample), which defines per-environment default values e.g., for billing alerts and labels. +- `data/projects/*.yaml`, one file per project (optionally grouped in folders), which configures each project. A [`prod/data/projects/project.yaml.sample`](./prod/data/projects/project.yaml.sample) is provided as reference and documentation for the schema. Projects will be named after the filename, e.g., `fast-prod-lab0.yaml` will create project `fast-prod-lab0`. Once the configuration is complete, run the project factory by running diff --git a/modules/README.md b/modules/README.md index 31770ec5..9b1913c7 100644 --- a/modules/README.md +++ b/modules/README.md @@ -8,6 +8,25 @@ Authoritative IAM bindings are primarily used (e.g. `google_storage_bucket_iam_b Specific modules also offer support for non-authoritative bindings (e.g. `google_storage_bucket_iam_member` for service accounts), to allow granular permission management on resources that they don't manage directly. +These modules are not necessarily backward compatible. Changes breaking compatibility in modules are marked by major releases (but not all major releases contain breaking changes). Please be mindful when upgrading Fabric modules in existing Terraform setups, and always try to use versioned references in module sources so you can easily revert back to a previous version. Since the introduction of the `moved` block in Terraform we try to use it whenever possible to make updates non-breaking, but that does not cover all changes we might need to make. + +These modules are used in the examples included in this repository. If you are using any of those examples in your own Terraform configuration, make sure that you are using the same version for all the modules, and switch module sources to GitHub format using references. The recommended approach to working with Fabric modules is the following: + +- Fork the repository and own the fork. This will allow you to: + - Evolve the existing modules. + - Create your own modules. + - Sync from the upstream repository to get all the updates. +- Use GitHub sources with refs to reference the modules in your fork. See an example below: + + ``` + module "project" { + source = "github.com/my-fork/cloud-foundation-fabric.git//modules/project?ref=v12.0.0" + name = "my-project" + billing_account = "123456-123456-123456" + parent = "organizations/123456" + } + ``` + ## Foundational modules - [billing budget](./billing-budget) diff --git a/modules/__experimental/net-dns-policy-address/README.md b/modules/__experimental/net-dns-policy-address/README.md new file mode 100644 index 00000000..36f9e4e1 --- /dev/null +++ b/modules/__experimental/net-dns-policy-address/README.md @@ -0,0 +1,35 @@ +# Google Cloud DNS Inbound Policy Addresses + +This module allows discovering the addresses reserved in subnets when [DNS Inbound Policies](https://cloud.google.com/dns/docs/policies) are configured. + +Since it's currently impossible to fetch those addresses using a GCP data source (see [this issue](https://github.com/hashicorp/terraform-provider-google/issues/3753) for more details), the workaround used here is to derive the authorization token from the Google provider, and do a direct HTTP call to the Compute API. + +## Examples + +```hcl +module "dns-policy-addresses" { + source = "./modules/__experimental/net-dns-policy-addresses" + project_id = "myproject" + regions = ["europe-west1", "europe-west3"] +} +# tftest skip +``` + +The output is a map with lists of addresses of type `DNS_RESOLVER` for each region specified in variables. + + + +## Variables + +| name | description | type | required | default | +|---|---|:---:|:---:|:---:| +| [project_id](variables.tf#L17) | Project id. | string | ✓ | | +| [regions](variables.tf#L22) | Regions to fetch addresses from. | list(string) | | ["europe-west1"] | + +## Outputs + +| name | description | sensitive | +|---|---|:---:| +| [addresses](outputs.tf#L24) | DNS inbound policy addresses per region. | | + + diff --git a/modules/__experimental/net-dns-policy-address/main.tf b/modules/__experimental/net-dns-policy-address/main.tf new file mode 100644 index 00000000..cb3144de --- /dev/null +++ b/modules/__experimental/net-dns-policy-address/main.tf @@ -0,0 +1,32 @@ +/** + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + url = format( + "https://content-compute.googleapis.com/compute/v1/projects/%s", + var.project_id + ) +} + +data "google_client_config" "current" {} + +data "http" "addresses" { + for_each = toset(var.regions) + url = "${local.url}/regions/${each.key}/addresses?filter=purpose%20%3D%20%22DNS_RESOLVER%22" + request_headers = { + Authorization = "Bearer ${data.google_client_config.current.access_token}" + } +} diff --git a/modules/__experimental/net-dns-policy-address/outputs.tf b/modules/__experimental/net-dns-policy-address/outputs.tf new file mode 100644 index 00000000..d379f268 --- /dev/null +++ b/modules/__experimental/net-dns-policy-address/outputs.tf @@ -0,0 +1,31 @@ +/** + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + region_addresses = { + for k, v in data.http.addresses : k => try(jsondecode(v.body), {}) + } +} + + +output "addresses" { + description = "DNS inbound policy addresses per region." + value = { + for k, v in local.region_addresses : k => [ + for i in try(v.items, []) : i.address + ] + } +} diff --git a/modules/__experimental/net-dns-policy-address/variables.tf b/modules/__experimental/net-dns-policy-address/variables.tf new file mode 100644 index 00000000..1b80d160 --- /dev/null +++ b/modules/__experimental/net-dns-policy-address/variables.tf @@ -0,0 +1,27 @@ +/** + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "project_id" { + description = "Project id." + type = string +} + +variable "regions" { + description = "Regions to fetch addresses from." + nullable = false + type = list(string) + default = ["europe-west1"] +} diff --git a/tests/examples/cloud_operations/vm_migration/host_target_projects/__init__.py b/tests/examples/cloud_operations/vm_migration/host_target_projects/__init__.py new file mode 100644 index 00000000..6d6d1266 --- /dev/null +++ b/tests/examples/cloud_operations/vm_migration/host_target_projects/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tests/examples/cloud_operations/vm_migration/host_target_projects/fixture/main.tf b/tests/examples/cloud_operations/vm_migration/host_target_projects/fixture/main.tf new file mode 100644 index 00000000..94fb07f7 --- /dev/null +++ b/tests/examples/cloud_operations/vm_migration/host_target_projects/fixture/main.tf @@ -0,0 +1,43 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +module "host-target-projects-test" { + source = "../../../../../../examples/cloud-operations/vm-migration/host-target-projects" + project_create = var.project_create + migration_admin_users = ["user:admin@example.com"] + migration_viewer_users = ["user:viewer@example.com"] + migration_target_projects = ["${module.test-target-project.name}"] + depends_on = [ + module.test-target-project + ] +} + +variable "project_create" { + type = object({ + billing_account_id = string + parent = string + }) + default = { + billing_account_id = "1234-ABCD-1234" + parent = "folders/1234563" + } +} + +#This is a dummy project created to run this test. The example, here tested, is expected to run on top of existing foundations. +module "test-target-project" { + source = "../../../../../../modules/project" + billing_account = "1234-ABCD-1234" + name = "test-target-project" + project_create = true +} diff --git a/tests/examples/cloud_operations/vm_migration/host_target_projects/test_plan.py b/tests/examples/cloud_operations/vm_migration/host_target_projects/test_plan.py new file mode 100644 index 00000000..33f493ba --- /dev/null +++ b/tests/examples/cloud_operations/vm_migration/host_target_projects/test_plan.py @@ -0,0 +1,26 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import os + + +FIXTURES_DIR = os.path.join(os.path.dirname(__file__), 'fixture') + + +def test_resources(e2e_plan_runner): + "Test that plan works and the numbers of resources is as expected." + modules, resources = e2e_plan_runner(FIXTURES_DIR) + assert len(modules) == 3 + assert len(resources) == 23 diff --git a/tests/examples/cloud_operations/vm_migration/host_target_sharedvpc/__init__.py b/tests/examples/cloud_operations/vm_migration/host_target_sharedvpc/__init__.py new file mode 100644 index 00000000..6d6d1266 --- /dev/null +++ b/tests/examples/cloud_operations/vm_migration/host_target_sharedvpc/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tests/examples/cloud_operations/vm_migration/host_target_sharedvpc/fixture/main.tf b/tests/examples/cloud_operations/vm_migration/host_target_sharedvpc/fixture/main.tf new file mode 100644 index 00000000..dac8f5af --- /dev/null +++ b/tests/examples/cloud_operations/vm_migration/host_target_sharedvpc/fixture/main.tf @@ -0,0 +1,51 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +module "host-target-sharedvpc-test" { + source = "../../../../../../examples/cloud-operations/vm-migration/host-target-sharedvpc" + project_create = var.project_create + migration_admin_users = ["user:admin@example.com"] + migration_viewer_users = ["user:viewer@example.com"] + migration_target_projects = [module.test-target-project.name] + sharedvpc_host_projects = [module.test-sharedvpc-host-project.name] + depends_on = [ + module.test-target-project, + module.test-sharedvpc-host-project, + ] +} + +variable "project_create" { + type = object({ + billing_account_id = string + parent = string + }) + default = { + billing_account_id = "1234-ABCD-1234" + parent = "folders/1234563" + } +} + +#These are a dummy projects created to run this test. The example, here tested, is expected to run on top of existing foundations. +module "test-target-project" { + source = "../../../../../../modules/project" + billing_account = "1234-ABCD-1234" + name = "test-target-project" + project_create = true +} +module "test-sharedvpc-host-project" { + source = "../../../../../../modules/project" + billing_account = "1234-ABCD-1234" + name = "test-sharedvpc-host-project" + project_create = true +} diff --git a/tests/examples/cloud_operations/vm_migration/host_target_sharedvpc/test_plan.py b/tests/examples/cloud_operations/vm_migration/host_target_sharedvpc/test_plan.py new file mode 100644 index 00000000..83935711 --- /dev/null +++ b/tests/examples/cloud_operations/vm_migration/host_target_sharedvpc/test_plan.py @@ -0,0 +1,26 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import os + + +FIXTURES_DIR = os.path.join(os.path.dirname(__file__), 'fixture') + + +def test_resources(e2e_plan_runner): + "Test that plan works and the numbers of resources is as expected." + modules, resources = e2e_plan_runner(FIXTURES_DIR) + assert len(modules) == 4 + assert len(resources) == 23 diff --git a/tests/examples/cloud_operations/vm_migration/single_project/__init__.py b/tests/examples/cloud_operations/vm_migration/single_project/__init__.py new file mode 100644 index 00000000..6d6d1266 --- /dev/null +++ b/tests/examples/cloud_operations/vm_migration/single_project/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tests/examples/cloud_operations/vm_migration/single_project/fixture/main.tf b/tests/examples/cloud_operations/vm_migration/single_project/fixture/main.tf new file mode 100644 index 00000000..c2750298 --- /dev/null +++ b/tests/examples/cloud_operations/vm_migration/single_project/fixture/main.tf @@ -0,0 +1,31 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +module "single-project-test" { + source = "../../../../../../examples/cloud-operations/vm-migration/single-project" + project_create = var.project_create + migration_admin_users = ["user:admin@example.com"] + migration_viewer_users = ["user:viewer@example.com"] +} + +variable "project_create" { + type = object({ + billing_account_id = string + parent = string + }) + default = { + billing_account_id = "1234-ABCD-1234" + parent = "folders/1234563" + } +} diff --git a/tests/examples/cloud_operations/vm_migration/single_project/test_plan.py b/tests/examples/cloud_operations/vm_migration/single_project/test_plan.py new file mode 100644 index 00000000..7d8a47b1 --- /dev/null +++ b/tests/examples/cloud_operations/vm_migration/single_project/test_plan.py @@ -0,0 +1,25 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os + + +FIXTURES_DIR = os.path.join(os.path.dirname(__file__), 'fixture') + + +def test_resources(e2e_plan_runner): + "Test that plan works and the numbers of resources is as expected." + modules, resources = e2e_plan_runner(FIXTURES_DIR) + assert len(modules) == 4 + assert len(resources) == 18 diff --git a/tests/examples/data_solutions/gcs_to_bq_with_least_privileges/test_plan.py b/tests/examples/data_solutions/gcs_to_bq_with_least_privileges/test_plan.py index dffa25d2..37ea4e8a 100644 --- a/tests/examples/data_solutions/gcs_to_bq_with_least_privileges/test_plan.py +++ b/tests/examples/data_solutions/gcs_to_bq_with_least_privileges/test_plan.py @@ -24,4 +24,4 @@ def test_resources(e2e_plan_runner): "Test that plan works and the numbers of resources is as expected." modules, resources = e2e_plan_runner(FIXTURES_DIR) assert len(modules) == 11 - assert len(resources) == 43 + assert len(resources) == 44