Initial replacement for CI/CD stage (#903)
* github extra stage * remove original cicd stage * allow setting commit attributes via variabes * remove reference to deleted stage * optional repo creation, documentation
This commit is contained in:
Ludovico Magnocavallo
GitHub
parent
be1375e95e
commit
fc7bf40e69
|
|
@ -0,0 +1,92 @@
|
|||
# FAST GitHub repository management
|
||||
|
||||
This small extra stage allows creation and management of GitHub repositories used to host FAST stage code, including initial population of files and rewriting of module sources.
|
||||
|
||||
This stage is designed for quick repository creation in a GitHub organization, and is not suited for medium or long-term repository management especially if you enable initial population of files.
|
||||
|
||||
## Initial population caveats
|
||||
|
||||
Initial file population of repositories is controlled via the `populate_from` attribute, and needs a bit of care:
|
||||
|
||||
- never run this stage gain with the same variables used for population once the repository starts being used, as **Terraform will manage file state and revert any changes at each apply**, which is probably not what you want.
|
||||
- be mindful when enabling initial population of the modules repository, as the number of resulting files to manage is very close to the GitHub hourly limit for their API
|
||||
|
||||
The scenario for which this stage has been designed is one-shot creation and/or population of stage repositories, running it multiple times with different variables and Terraform states if incremental creation is needed for subsequent FAST stages (e.g. GKE, data platform, etc.).
|
||||
|
||||
## GitHub provider credentials
|
||||
|
||||
A [GitHub token](https://github.com/settings/tokens) is needed to authenticate against their API. The token needs organization-level permissions, like shown in this screenshot:
|
||||
|
||||
<p align="center">
|
||||
<img src="github_token.png" alt="GitHub token scopes.">
|
||||
</p>
|
||||
|
||||
## Variable configuration
|
||||
|
||||
The `organization` required variable sets the GitHub organization where repositories will be created, and is used to configure the Terraform provider.
|
||||
|
||||
The `repositories` variable is where you configure which repositories to create, whether initial population of files is desired, and which repository is used to host modules.
|
||||
|
||||
This is an example that creates repositories for stages 00 and 01, defines an existing repositories as the source for modules, and populates initial files for stages 00, 01, and 02:
|
||||
|
||||
```hcl
|
||||
organization = "ludomagno"
|
||||
repositories = {
|
||||
fast_00_bootstrap = {
|
||||
create_options = {
|
||||
description = "FAST bootstrap."
|
||||
features = {
|
||||
issues = true
|
||||
}
|
||||
}
|
||||
populate_from = "../../stages/00-bootstrap"
|
||||
}
|
||||
fast_01_resman = {
|
||||
create_options = {
|
||||
description = "FAST resource management."
|
||||
features = {
|
||||
issues = true
|
||||
}
|
||||
}
|
||||
populate_from = "../../stages/01-resman"
|
||||
}
|
||||
fast_02_networking = {
|
||||
populate_from = "../../stages/02-networking-peering"
|
||||
}
|
||||
fast_modules = {
|
||||
has_modules = true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `create_options` repository attribute controls creation: if the attribute is not present, the repository is assumed to be already existing.
|
||||
|
||||
Initial population depends on a modules repository being configured, identified by the `has_modules` attribute, and on `populate_from` attributes in each repository where population is required, pointing to the folder holding the files to be committed.
|
||||
|
||||
Finally, a `commit_config` variable is optional: it can be used to configure author, email and message used in commits for initial population of files, its defaults are probably fine for most use cases.
|
||||
|
||||
## Modules secret
|
||||
|
||||
When initial population is configured for a repository, this stage also adds a secret with the private key used to authenticate against the modules repository. This matches the configuration of the GitHub workflow files created for each FAST stage when CI/CD is enabled.
|
||||
|
||||
<!-- TFDOC OPTS files:1 -->
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Files
|
||||
|
||||
| name | description | resources |
|
||||
|---|---|---|
|
||||
| [cicd-versions.tf](./cicd-versions.tf) | Provider version. | |
|
||||
| [main.tf](./main.tf) | Module-level locals and resources. | <code>github_actions_secret</code> · <code>github_repository</code> · <code>github_repository_file</code> · <code>tls_private_key</code> |
|
||||
| [providers.tf](./providers.tf) | Provider configuration. | |
|
||||
| [variables.tf](./variables.tf) | Module variables. | |
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| [organization](variables.tf#L28) | GitHub organization. | <code>string</code> | ✓ | |
|
||||
| [commmit_config](variables.tf#L17) | Configure commit metadata. | <code title="object({ author = optional(string, "FAST loader") email = optional(string, "fast-loader@fast.gcp.tf") message = optional(string, "FAST initial loading") })">object({…})</code> | | <code>{}</code> |
|
||||
| [repositories](variables.tf#L33) | Repositories to create. | <code title="map(object({ create_options = optional(object({ allow = optional(object({ auto_merge = optional(bool) merge_commit = optional(bool) rebase_merge = optional(bool) squash_merge = optional(bool) })) auto_init = optional(bool) description = optional(string) features = optional(object({ issues = optional(bool) projects = optional(bool) wiki = optional(bool) })) templates = optional(object({ gitignore = optional(string, "Terraform") license = optional(string) repository = optional(object({ name = string owner = string })) }), {}) visibility = optional(string, "private") })) has_modules = optional(bool, false) populate_from = optional(string) }))">map(object({…}))</code> | | <code>{}</code> |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
|
@ -14,10 +14,16 @@
|
|||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description Output files persistence to automation GCS bucket.
|
||||
# tfdoc:file:description Provider version.
|
||||
|
||||
resource "google_storage_bucket_object" "tfvars" {
|
||||
bucket = var.automation.outputs_bucket
|
||||
name = "tfvars/00-bootstrap.auto.tfvars.json"
|
||||
content = jsonencode(local.tfvars)
|
||||
terraform {
|
||||
required_version = ">= 1.3.1"
|
||||
required_providers {
|
||||
github = {
|
||||
source = "integrations/github"
|
||||
version = "~> 4.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
|
|
@ -0,0 +1,124 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
locals {
|
||||
_modules_repository = [
|
||||
for k, v in var.repositories : local.repositories[k] if v.has_modules
|
||||
]
|
||||
_repository_files = flatten([
|
||||
for k, v in var.repositories : [
|
||||
for f in concat(
|
||||
[for f in fileset(path.module, "${v.populate_from}/*.md") : f],
|
||||
[for f in fileset(path.module, "${v.populate_from}/*.tf") : f]
|
||||
) : {
|
||||
repository = k
|
||||
file = f
|
||||
name = replace(f, "${v.populate_from}/", "")
|
||||
}
|
||||
] if v.populate_from != null
|
||||
])
|
||||
modules_repository = (
|
||||
length(local._modules_repository) > 0
|
||||
? local._modules_repository.0
|
||||
: null
|
||||
)
|
||||
repositories = {
|
||||
for k, v in var.repositories :
|
||||
k => v.create_options == null ? k : github_repository.default[k].name
|
||||
}
|
||||
repository_files = {
|
||||
for k in local._repository_files :
|
||||
"${k.repository}/${k.name}" => k
|
||||
if !endswith(k.name, ".tf") || (
|
||||
!startswith(k.name, "0") && k.name != "globals.tf"
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
resource "github_repository" "default" {
|
||||
for_each = {
|
||||
for k, v in var.repositories : k => v if v.create_options != null
|
||||
}
|
||||
name = each.key
|
||||
description = (
|
||||
each.value.create_options.description != null
|
||||
? each.value.create_options.description
|
||||
: "FAST stage ${each.key}."
|
||||
)
|
||||
visibility = each.value.create_options.visibility
|
||||
auto_init = each.value.create_options.auto_init
|
||||
allow_auto_merge = try(each.value.create_options.allow.auto_merge, null)
|
||||
allow_merge_commit = try(each.value.create_options.allow.merge_commit, null)
|
||||
allow_rebase_merge = try(each.value.create_options.allow.rebase_merge, null)
|
||||
allow_squash_merge = try(each.value.create_options.allow.squash_merge, null)
|
||||
has_issues = try(each.value.create_options.features.issues, null)
|
||||
has_projects = try(each.value.create_options.features.projects, null)
|
||||
has_wiki = try(each.value.create_options.features.wiki, null)
|
||||
gitignore_template = try(each.value.create_options.templates.gitignore, null)
|
||||
license_template = try(each.value.create_options.templates.license, null)
|
||||
|
||||
dynamic "template" {
|
||||
for_each = (
|
||||
try(each.value.create_options.templates.repository, null) != null
|
||||
? [""]
|
||||
: []
|
||||
)
|
||||
content {
|
||||
owner = each.value.create_options.templates.repository.owner
|
||||
repository = each.value.create_options.templates.repository.name
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "tls_private_key" "default" {
|
||||
count = local.modules_repository != null ? 1 : 0
|
||||
algorithm = "ED25519"
|
||||
}
|
||||
|
||||
resource "github_actions_secret" "default" {
|
||||
for_each = local.modules_repository == null ? {} : {
|
||||
for k, v in local.repositories :
|
||||
k => v if(
|
||||
k != local.modules_repository &&
|
||||
var.repositories[k].populate_from != null
|
||||
)
|
||||
}
|
||||
repository = local.repositories[local.modules_repository]
|
||||
secret_name = "CICD_MODULES_KEY"
|
||||
plaintext_value = tls_private_key.default.0.private_key_openssh
|
||||
}
|
||||
|
||||
resource "github_repository_file" "default" {
|
||||
for_each = (
|
||||
local.modules_repository == null ? {} : local.repository_files
|
||||
)
|
||||
repository = local.repositories[each.value.repository]
|
||||
branch = "main"
|
||||
file = each.value.name
|
||||
content = (
|
||||
endswith(each.value.name, ".tf") && local.modules_repository != null
|
||||
? replace(
|
||||
file(each.value.file),
|
||||
"/source\\s*=\\s*\"../../../",
|
||||
"source = \"git@github.com:${var.organization}/${local.modules_repository}.git/"
|
||||
)
|
||||
: file(each.value.file)
|
||||
)
|
||||
commit_message = "${var.commmit_config.message} (${each.value.name})"
|
||||
commit_author = var.commmit_config.author
|
||||
commit_email = var.commmit_config.email
|
||||
overwrite_on_create = true
|
||||
}
|
||||
|
|
@ -14,3 +14,8 @@
|
|||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description Provider configuration.
|
||||
|
||||
provider "github" {
|
||||
owner = var.organization
|
||||
}
|
||||
|
|
@ -0,0 +1,72 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
variable "commmit_config" {
|
||||
description = "Configure commit metadata."
|
||||
type = object({
|
||||
author = optional(string, "FAST loader")
|
||||
email = optional(string, "fast-loader@fast.gcp.tf")
|
||||
message = optional(string, "FAST initial loading")
|
||||
})
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "organization" {
|
||||
description = "GitHub organization."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "repositories" {
|
||||
description = "Repositories to create."
|
||||
type = map(object({
|
||||
create_options = optional(object({
|
||||
allow = optional(object({
|
||||
auto_merge = optional(bool)
|
||||
merge_commit = optional(bool)
|
||||
rebase_merge = optional(bool)
|
||||
squash_merge = optional(bool)
|
||||
}))
|
||||
auto_init = optional(bool)
|
||||
description = optional(string)
|
||||
features = optional(object({
|
||||
issues = optional(bool)
|
||||
projects = optional(bool)
|
||||
wiki = optional(bool)
|
||||
}))
|
||||
templates = optional(object({
|
||||
gitignore = optional(string, "Terraform")
|
||||
license = optional(string)
|
||||
repository = optional(object({
|
||||
name = string
|
||||
owner = string
|
||||
}))
|
||||
}), {})
|
||||
visibility = optional(string, "private")
|
||||
}))
|
||||
has_modules = optional(bool, false)
|
||||
populate_from = optional(string)
|
||||
}))
|
||||
default = {}
|
||||
nullable = true
|
||||
validation {
|
||||
condition = alltrue([
|
||||
for k, v in var.repositories :
|
||||
try(regex("^[a-zA-Z0-9_.]+$", k), null) != null
|
||||
])
|
||||
error_message = "Repository names must match '^[a-zA-Z0-9_.]+$'."
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
# FAST extra stages
|
||||
|
||||
This folder contains additional helper stages for FAST, which can be used to simplify specific operational tasks:
|
||||
|
||||
- [GitHub repository management](./00-cicd-github/)
|
||||
|
|
@ -1,206 +0,0 @@
|
|||
# CI/CD bootstrap
|
||||
|
||||
The primary purpose of this stage is to set up your CI/CD project structure automatically, with most of the necessary configuration to run the pipelines out of the box.
|
||||
|
||||
## How to run this stage
|
||||
|
||||
This stage is meant to be executed after the [bootstrap](../00-bootstrap) stage has run, as it leverages the automation service account and bucket created there.
|
||||
The entire stage is optional, you may also choose to create your repositories manually.
|
||||
|
||||
### Providers configuration
|
||||
|
||||
The default way of making sure you have the right permissions, is to use the identity of the service account pre-created for this stage during bootstrap, and that you are a member of the group that can impersonate it via provider-level configuration (`gcp-devops` or `organization-admins`).
|
||||
|
||||
To simplify setup, the previous stage pre-configures a valid providers file in its output, and optionally writes it to a local file if the `outputs_location` variable is set to a valid path.
|
||||
|
||||
If you have set a valid value for `outputs_location` in the bootstrap stage (see the [bootstrap stage README](../00-bootstrap/#output-files-and-cross-stage-variables) for more details), simply link the relevant `providers.tf` file from this stage's folder in the path you specified:
|
||||
|
||||
```bash
|
||||
# `outputs_location` is set to `~/fast-config`
|
||||
ln -s ~/fast-config/providers/00-cicd-providers.tf .
|
||||
```
|
||||
|
||||
If you have not configured `outputs_location` in bootstrap, you can derive the providers file from that stage's outputs:
|
||||
|
||||
```bash
|
||||
cd ../00-bootstrap
|
||||
terraform output -json providers | jq -r '.["00-cicd"]' \
|
||||
> ../00-cicd/providers.tf
|
||||
```
|
||||
|
||||
If you want to continue to rely on `outputs_location` logic, create a `terraform.tfvars` file and configure it as described [here](../00-bootstrap/#output-files-and-cross-stage-variables).
|
||||
|
||||
### Variable configuration
|
||||
|
||||
There are two broad sets of variables you will need to fill in:
|
||||
|
||||
- variables shared by other stages (org id, billing account id, etc.), or derived from a resource managed by a different stage (folder id, automation project id, etc.)
|
||||
- variables specific to resources managed by this stage
|
||||
|
||||
To avoid the tedious job of filling in the first group of variable with values derived from other stages' outputs, the same mechanism used above for the provider configuration can be used to leverage pre-configured `.tfvars` files.
|
||||
|
||||
If you configured a valid path for `outputs_location` in the bootstrap stage, simply link the relevant `terraform-*.auto.tfvars.json` files from the outputs folder. For this stage, you need the `.tfvars` file compiled manually for the bootstrap stage, and the one generated by it:
|
||||
|
||||
```bash
|
||||
# `outputs_location` is set to `~/fast-config`
|
||||
ln -s ~/fast-config/tfvars/00-bootstrap.auto.tfvars.json .
|
||||
# also copy the tfvars file used for the bootstrap stage
|
||||
cp ../00-bootstrap/terraform.tfvars .
|
||||
```
|
||||
|
||||
A second set of variables is specific to this stage, they are all optional so if you need to customize them, create an extra `terraform.tfvars` file or add them to the file copied from bootstrap.
|
||||
|
||||
Refer to the [Variables](#variables) table at the bottom of this document, for a full list of variables, their origin (e.g. a stage or specific to this one), and descriptions explaining their meaning. The sections below also describe some of the possible customizations.
|
||||
|
||||
### CI/CD systems
|
||||
|
||||
#### Gitlab
|
||||
|
||||
To configure Gitlab, add the following variable:
|
||||
|
||||
```hcl
|
||||
gitlab = {
|
||||
url = "https://gitlab.com" # Or self-hosted URL
|
||||
project_visibility = "private"
|
||||
shared_runners_enabled = true
|
||||
}
|
||||
```
|
||||
|
||||
Also set `GITLAB_TOKEN` to a token that has appropriate permissions.
|
||||
|
||||
#### GitHub
|
||||
|
||||
To configure GitHub, add the following variable:
|
||||
|
||||
```hcl
|
||||
github = {
|
||||
url = null # Or GitHub Enterprise base URL
|
||||
visibility = "private"
|
||||
}
|
||||
```
|
||||
|
||||
Also set `GITHUB_TOKEN` to a token that has appropriate permissions.
|
||||
|
||||
### CI/CD repositories
|
||||
|
||||
While the other stages create the necessary supporting structure for their CI/CD pipelines, like service accounts
|
||||
and such, the `00-cicd` stage creates all the repositories in your CI/CD system through automation. Its
|
||||
configuration is essentially a combination of all the `cicd_repositories` variables of the other stages
|
||||
plus additional CI/CD system specific configuration information.
|
||||
|
||||
This is an example of configuring the repositories in this stage.
|
||||
|
||||
```hcl
|
||||
cicd_repositories = {
|
||||
bootstrap = {
|
||||
branch = null
|
||||
identity_provider = "github-sample"
|
||||
name = "my-gh-org/fast-bootstrap"
|
||||
description = "Google Cloud bootstrapping"
|
||||
type = "github"
|
||||
create = true
|
||||
create_group = true
|
||||
}
|
||||
cicd = {
|
||||
branch = null
|
||||
identity_provider = "github-sample"
|
||||
name = "my-gh-org/fast-cicd"
|
||||
description = "Fabric FAST CI/CD setup"
|
||||
type = "github"
|
||||
create = true
|
||||
create_group = true
|
||||
}
|
||||
resman = {
|
||||
branch = "main"
|
||||
identity_provider = "github-sample"
|
||||
name = "my-gh-org/fast-resman"
|
||||
description = "Google Cloud resource management"
|
||||
type = "github"
|
||||
create = true
|
||||
create_group = true
|
||||
}
|
||||
networking = {
|
||||
branch = "main"
|
||||
identity_provider = "github-sample"
|
||||
name = "my-gh-org/fast-networking"
|
||||
description = "Google Cloud networking setup"
|
||||
type = "github"
|
||||
create = true
|
||||
create_group = true
|
||||
}
|
||||
security = {
|
||||
branch = "main"
|
||||
identity_provider = "github-sample"
|
||||
description = "Google Cloud security settings"
|
||||
name = "my-gh-org/fast-security"
|
||||
type = "github"
|
||||
create = true
|
||||
create_group = true
|
||||
}
|
||||
data-platform = {
|
||||
branch = "main"
|
||||
identity_provider = "github-sample"
|
||||
name = "my-gh-org/fast-data-platform"
|
||||
description = "Google Cloud data platform"
|
||||
type = "github"
|
||||
create = true
|
||||
create_group = true
|
||||
}
|
||||
project-factory = {
|
||||
branch = "main"
|
||||
identity_provider = "github-sample"
|
||||
name = "my-gh-org/fast-project-factory"
|
||||
description = "Google Cloud project factory"
|
||||
type = "github"
|
||||
create = true
|
||||
create_group = true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `type` attribute can be set to one of the supported repository types: `github` or `gitlab`.
|
||||
|
||||
Once the stage is applied the generated output files will contain pre-configured workflow files for each repository, that will use Workload Identity Federation via a dedicated service account for each repository to impersonate the automation service account for the stage.
|
||||
|
||||
Once done, you can run this stage:
|
||||
|
||||
```bash
|
||||
terraform init
|
||||
terraform apply
|
||||
```
|
||||
|
||||
<!-- TFDOC OPTS files:1 show_extra:1 -->
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Files
|
||||
|
||||
| name | description | resources |
|
||||
|---|---|---|
|
||||
| [cicd.tf](./cicd.tf) | None | <code>tls_private_key</code> |
|
||||
| [github.tf](./github.tf) | None | <code>github_actions_secret</code> · <code>github_repository</code> |
|
||||
| [gitlab.tf](./gitlab.tf) | None | <code>gitlab_group</code> · <code>gitlab_project</code> · <code>gitlab_project_variable</code> |
|
||||
| [main.tf](./main.tf) | Module-level locals and resources. | |
|
||||
| [outputs-files.tf](./outputs-files.tf) | Output files persistence to local filesystem. | <code>local_file</code> |
|
||||
| [outputs-gcs.tf](./outputs-gcs.tf) | Output files persistence to automation GCS bucket. | <code>google_storage_bucket_object</code> |
|
||||
| [outputs.tf](./outputs.tf) | Module outputs. | |
|
||||
| [variables.tf](./variables.tf) | Module variables. | |
|
||||
| [versions.tf](./versions.tf) | Version pins. | |
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default | producer |
|
||||
|---|---|:---:|:---:|:---:|:---:|
|
||||
| [automation](variables.tf#L17) | Automation resources created by the bootstrap stage. | <code title="object({ outputs_bucket = string project_id = string project_number = string federated_identity_pool = string federated_identity_providers = map(object({ issuer = string issuer_uri = string name = string principal_tpl = string principalset_tpl = string })) })">object({…})</code> | ✓ | | <code>00-bootstrap</code> |
|
||||
| [cicd_repositories](variables.tf#L35) | CI/CD repository configuration. Identity providers reference keys in the `federated_identity_providers` variable. Set to null to disable, or set individual repositories to null if not needed. | <code title="object({ bootstrap = object({ branch = string name = string description = string type = string create = bool create_group = bool }) resman = object({ branch = string name = string description = string type = string create = bool create_group = bool }) networking = object({ branch = string name = string description = string type = string create = bool create_group = bool }) security = object({ branch = string name = string description = string type = string create = bool create_group = bool }) data-platform = object({ branch = string name = string description = string type = string create = bool create_group = bool }) project-factory = object({ branch = string name = string description = string type = string create = bool create_group = bool }) })">object({…})</code> | | <code>null</code> | |
|
||||
| [custom_roles](variables.tf#L132) | Custom roles defined at the org level, in key => id format. | <code title="object({ service_project_network_admin = string })">object({…})</code> | | <code>null</code> | <code>00-bootstrap</code> |
|
||||
| [github](variables.tf#L120) | GitHub settings | <code title="object({ url = string visibility = string })">object({…})</code> | | <code title="{ url = null visibility = "private" }">{…}</code> | |
|
||||
| [gitlab](variables.tf#L106) | Gitlab settings | <code title="object({ url = string project_visibility = string shared_runners_enabled = bool })">object({…})</code> | | <code title="{ url = "https://gitlab.com" project_visibility = "private" shared_runners_enabled = true }">{…}</code> | |
|
||||
| [outputs_location](variables.tf#L141) | Enable writing provider, tfvars and CI/CD workflow files to local filesystem. Leave null to disable | <code>string</code> | | <code>null</code> | |
|
||||
|
||||
## Outputs
|
||||
|
||||
| name | description | sensitive | consumers |
|
||||
|---|---|:---:|---|
|
||||
| [tfvars](outputs.tf#L30) | Terraform variable files for the following stages. | ✓ | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
|
|
@ -1,38 +0,0 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
locals {
|
||||
supported_cicd_systems = ["gitlab", "github", "sourcerepo"]
|
||||
cicd_repositories = {
|
||||
for k, v in coalesce(var.cicd_repositories, {}) : k => merge(v,
|
||||
{ group = join("/", slice(split("/", v.name), 0, length(split("/", v.name)) - 1)) },
|
||||
{ name = element(split("/", v.name), length(split("/", v.name)) - 1) },
|
||||
{ create_group = try(v.create_group, true) })
|
||||
if(
|
||||
v != null
|
||||
&&
|
||||
contains(local.supported_cicd_systems, try(v.type, ""))
|
||||
)
|
||||
}
|
||||
cicd_repositories_by_system = { for system in local.supported_cicd_systems : system => {
|
||||
for k, v in local.cicd_repositories : k => v if v.type == system
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "tls_private_key" "cicd-modules-key" {
|
||||
algorithm = "ED25519"
|
||||
}
|
||||
|
|
@ -1,51 +0,0 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
locals {
|
||||
github_groups = distinct([for k, v in local.cicd_repositories_by_system["github"] : v.group])
|
||||
}
|
||||
|
||||
provider "github" {
|
||||
base_url = var.github.url
|
||||
owner = local.github_groups[0]
|
||||
}
|
||||
|
||||
data "github_organization" "organization" {
|
||||
for_each = toset(local.github_groups)
|
||||
name = each.value
|
||||
}
|
||||
|
||||
data "github_repository" "repositories" {
|
||||
for_each = { for name, repo in local.cicd_repositories_by_system["github"] : name => repo if !try(repo.create, true) }
|
||||
full_name = format("%s/%s", each.value.group, each.value.name)
|
||||
}
|
||||
|
||||
resource "github_repository" "repositories" {
|
||||
for_each = { for name, repo in local.cicd_repositories_by_system["github"] : name => repo if try(repo.create, true) }
|
||||
|
||||
name = each.value.name
|
||||
description = each.value.description
|
||||
|
||||
visibility = var.github.visibility
|
||||
}
|
||||
|
||||
resource "github_actions_secret" "actions-modules-key" {
|
||||
for_each = { for name, repo in local.cicd_repositories_by_system["github"] : name => repo }
|
||||
|
||||
repository = try(each.value.create, true) ? github_repository.repositories[each.key].name : data.github_repository.repositories[each.key].name
|
||||
secret_name = "CICD_MODULES_KEY"
|
||||
plaintext_value = tls_private_key.cicd-modules-key.private_key_openssh
|
||||
}
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
locals {
|
||||
gitlab_create_groups = distinct([for k, v in local.cicd_repositories_by_system["gitlab"] : v.group if try(v.create_group, false)])
|
||||
gitlab_existing_groups = distinct([for k, v in local.cicd_repositories_by_system["gitlab"] : v.group if !try(v.create_group, false)])
|
||||
}
|
||||
|
||||
provider "gitlab" {
|
||||
base_url = var.gitlab.url
|
||||
}
|
||||
|
||||
data "gitlab_group" "group" {
|
||||
for_each = toset(local.gitlab_existing_groups)
|
||||
full_path = each.value
|
||||
}
|
||||
|
||||
data "gitlab_project" "projects" {
|
||||
for_each = { for name, repo in local.cicd_repositories_by_system["gitlab"] : name => repo if !try(repo.create, true) }
|
||||
id = format("%s/%s", each.value.group, each.value.name)
|
||||
}
|
||||
|
||||
resource "gitlab_group" "group" {
|
||||
for_each = toset(local.gitlab_create_groups)
|
||||
|
||||
name = each.value
|
||||
path = each.value
|
||||
description = "Cloud Foundation Fabric FAST: github.com/GoogleCloudPlatform/cloud-foundation-fabric/tree/master/fast/"
|
||||
}
|
||||
|
||||
resource "gitlab_project" "projects" {
|
||||
for_each = { for name, repo in local.cicd_repositories_by_system["gitlab"] : name => repo if try(repo.create, true) }
|
||||
|
||||
name = each.value.name
|
||||
namespace_id = each.value.create_group ? gitlab_group.group[each.value.group].id : data.gitlab_group.group[each.value.group].id
|
||||
description = each.value.description
|
||||
|
||||
visibility_level = var.gitlab.project_visibility
|
||||
shared_runners_enabled = var.gitlab.shared_runners_enabled
|
||||
auto_devops_enabled = false
|
||||
}
|
||||
|
||||
resource "gitlab_project_variable" "project-modules-key" {
|
||||
for_each = { for name, repo in local.cicd_repositories_by_system["gitlab"] : name => repo }
|
||||
|
||||
project = try(each.value.create, true) ? gitlab_project.projects[each.key].id : data.gitlab_project.projects[each.key].id
|
||||
|
||||
key = "CICD_MODULES_KEY"
|
||||
value = base64encode(tls_private_key.cicd-modules-key.private_key_openssh)
|
||||
|
||||
protected = false
|
||||
masked = true
|
||||
variable_type = "env_var"
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
|
@ -1,24 +0,0 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description Output files persistence to local filesystem.
|
||||
|
||||
resource "local_file" "tfvars" {
|
||||
for_each = var.outputs_location == null ? {} : { 1 = 1 }
|
||||
file_permission = "0644"
|
||||
filename = "${pathexpand(var.outputs_location)}/tfvars/00-cicd.auto.tfvars.json"
|
||||
content = jsonencode(local.tfvars)
|
||||
}
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
locals {
|
||||
gitlab_cicd_https = { for k, v in local.cicd_repositories_by_system["gitlab"] : k => v.create ? gitlab_project.projects[k].http_url_to_repo : data.gitlab_project.projects[k].http_url_to_repo }
|
||||
gitlab_cicd_ssh = { for k, v in local.cicd_repositories_by_system["gitlab"] : k => v.create ? gitlab_project.projects[k].ssh_url_to_repo : data.gitlab_project.projects[k].ssh_url_to_repo }
|
||||
github_cicd_https = { for k, v in local.cicd_repositories_by_system["github"] : k => v.create ? github_repository.repositories[k].http_clone_url : data.github_repository.repositories[k].http_clone_url }
|
||||
github_cicd_ssh = { for k, v in local.cicd_repositories_by_system["github"] : k => v.create ? github_repository.repositories[k].git_clone_url : data.github_repository.repositories[k].git_clone_url }
|
||||
|
||||
tfvars = {
|
||||
cicd_repositories = merge(local.cicd_repositories_by_system["gitlab"], local.cicd_repositories_by_system["github"])
|
||||
cicd_ssh_urls = merge(local.gitlab_cicd_ssh, local.github_cicd_ssh)
|
||||
cicd_https_urls = merge(local.gitlab_cicd_https, local.github_cicd_https)
|
||||
}
|
||||
}
|
||||
|
||||
output "tfvars" {
|
||||
description = "Terraform variable files for the following stages."
|
||||
sensitive = true
|
||||
value = local.tfvars
|
||||
}
|
||||
|
|
@ -1,145 +0,0 @@
|
|||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
variable "automation" {
|
||||
# tfdoc:variable:source 00-bootstrap
|
||||
description = "Automation resources created by the bootstrap stage."
|
||||
type = object({
|
||||
outputs_bucket = string
|
||||
project_id = string
|
||||
project_number = string
|
||||
federated_identity_pool = string
|
||||
federated_identity_providers = map(object({
|
||||
issuer = string
|
||||
issuer_uri = string
|
||||
name = string
|
||||
principal_tpl = string
|
||||
principalset_tpl = string
|
||||
}))
|
||||
})
|
||||
}
|
||||
|
||||
variable "cicd_repositories" {
|
||||
description = "CI/CD repository configuration. Identity providers reference keys in the `federated_identity_providers` variable. Set to null to disable, or set individual repositories to null if not needed."
|
||||
type = object({
|
||||
bootstrap = object({
|
||||
branch = string
|
||||
name = string
|
||||
description = string
|
||||
type = string
|
||||
create = bool
|
||||
create_group = bool
|
||||
})
|
||||
resman = object({
|
||||
branch = string
|
||||
name = string
|
||||
description = string
|
||||
type = string
|
||||
create = bool
|
||||
create_group = bool
|
||||
})
|
||||
networking = object({
|
||||
branch = string
|
||||
name = string
|
||||
description = string
|
||||
type = string
|
||||
create = bool
|
||||
create_group = bool
|
||||
})
|
||||
security = object({
|
||||
branch = string
|
||||
name = string
|
||||
description = string
|
||||
type = string
|
||||
create = bool
|
||||
create_group = bool
|
||||
})
|
||||
data-platform = object({
|
||||
branch = string
|
||||
name = string
|
||||
description = string
|
||||
type = string
|
||||
create = bool
|
||||
create_group = bool
|
||||
})
|
||||
project-factory = object({
|
||||
branch = string
|
||||
name = string
|
||||
description = string
|
||||
type = string
|
||||
create = bool
|
||||
create_group = bool
|
||||
})
|
||||
})
|
||||
default = null
|
||||
validation {
|
||||
condition = alltrue([
|
||||
for k, v in coalesce(var.cicd_repositories, {}) :
|
||||
v == null || try(v.name, null) != null
|
||||
])
|
||||
error_message = "Non-null repositories need a non-null name."
|
||||
}
|
||||
validation {
|
||||
condition = alltrue([
|
||||
for k, v in coalesce(var.cicd_repositories, {}) :
|
||||
v == null || (
|
||||
contains(["github", "gitlab", "sourcerepo"], coalesce(try(v.type, null), "null"))
|
||||
)
|
||||
])
|
||||
error_message = "Invalid repository type, supported types: 'github' 'gitlab' or 'sourcerepo'."
|
||||
}
|
||||
}
|
||||
|
||||
variable "gitlab" {
|
||||
description = "Gitlab settings"
|
||||
type = object({
|
||||
url = string
|
||||
project_visibility = string
|
||||
shared_runners_enabled = bool
|
||||
})
|
||||
default = {
|
||||
url = "https://gitlab.com"
|
||||
project_visibility = "private"
|
||||
shared_runners_enabled = true
|
||||
}
|
||||
}
|
||||
|
||||
variable "github" {
|
||||
description = "GitHub settings"
|
||||
type = object({
|
||||
url = string
|
||||
visibility = string
|
||||
})
|
||||
default = {
|
||||
url = null
|
||||
visibility = "private"
|
||||
}
|
||||
}
|
||||
|
||||
variable "custom_roles" {
|
||||
# tfdoc:variable:source 00-bootstrap
|
||||
description = "Custom roles defined at the org level, in key => id format."
|
||||
type = object({
|
||||
service_project_network_admin = string
|
||||
})
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "outputs_location" {
|
||||
description = "Enable writing provider, tfvars and CI/CD workflow files to local filesystem. Leave null to disable"
|
||||
type = string
|
||||
default = null
|
||||
}
|
||||
|
|
@ -1,38 +0,0 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
terraform {
|
||||
required_version = ">= 1.3.1"
|
||||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = ">= 4.36.0" # tftest
|
||||
}
|
||||
google-beta = {
|
||||
source = "hashicorp/google-beta"
|
||||
version = ">= 4.36.0" # tftest
|
||||
}
|
||||
github = {
|
||||
source = "integrations/github"
|
||||
version = "~> 4.0"
|
||||
}
|
||||
gitlab = {
|
||||
source = "gitlabhq/gitlab"
|
||||
version = ">= 3.16.1"
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -24,8 +24,6 @@ To destroy a previous FAST deployment follow the instructions detailed in [clean
|
|||
- [Bootstrap](00-bootstrap/README.md)
|
||||
Enables critical organization-level functionality that depends on broad permissions. It has two primary purposes. The first is to bootstrap the resources needed for automation of this and the following stages (service accounts, GCS buckets). And secondly, it applies the minimum amount of configuration needed at the organization level, to avoid the need of broad permissions later on, and to implement a minimum of security features like sinks and exports from the start.\
|
||||
Exports: automation variables, organization-level custom roles
|
||||
- [CI/CD Bootstrap](00-cicd/README.md)
|
||||
Optionally set up CI/CD repositories and project structure automatically for GitHub and Gitlab. This stage is not needed if repository are created manually.
|
||||
- [Resource Management](01-resman/README.md)
|
||||
Creates the base resource hierarchy (folders) and the automation resources required later to delegate deployment of each part of the hierarchy to separate stages. This stage also configures organization-level policies and any exceptions needed by different branches of the resource hierarchy.\
|
||||
Exports: folder ids, automation service account emails
|
||||
|
|
|
|||
|
|
@ -1,13 +0,0 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import os
|
||||
'''
|
||||
github = {
|
||||
source = "integrations/github"
|
||||
version = "~> 4.0"
|
||||
}
|
||||
gitlab = {
|
||||
source = "gitlabhq/gitlab"
|
||||
version = ">= 3.16.1"
|
||||
}
|
||||
'''
|
||||
|
||||
|
||||
def test_providers(basedir):
|
||||
"Test providers file."
|
||||
p = os.path.join(basedir, 'fast/stages/00-cicd/versions.tf')
|
||||
with open(p) as f:
|
||||
data = f.read()
|
||||
assert 'integrations/github' in data
|
||||
assert 'gitlabhq/gitlab' in data
|
||||
Loading…
Reference in New Issue