Commit Graph

66 Commits

Author SHA1 Message Date
Julio Castillo 30496068e2
Implement GKE patterns naming conventions (#2120)
* GKE Pattern Naming

* Add MySQL pattern estimated cost

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2024-02-29 06:57:22 +00:00
Ludovico Magnocavallo 6941313c7d
Factories refactor (#1843)
* factories refactor doc

* Adds file schema and filesystem organization

* Update 20231106-factories.md

* move factories out of blueprints and create new factories  README

* align factory in billing-account module

* align factory in dataplex-datascan module

* align factory in billing-account module

* align factory in net-firewall-policy module

* align factory in dns-response-policy module

* align factory in net-vpc-firewall module

* align factory in net-vpc module

* align factory variable names in FAST

* remove decentralized firewall blueprint

* bump terraform version

* bump module versions

* update top-level READMEs

* move project factory to modules

* fix variable names and tests

* tfdoc

* remove changelog link

* add project factory to top-level README

* fix cludrun eventarc diff

* fix README

* fix cludrun eventarc diff

---------

Co-authored-by: Simone Ruffilli <sruffilli@google.com>
2024-02-26 10:16:52 +00:00
Ludovico Magnocavallo 220ab76e40
enable shielded nodes by default on GKE mt blueprint and FAST stage (#2105) 2024-02-22 07:35:27 +00:00
Julio Castillo 8102803d2e
Update tutorial.md 2024-02-21 15:01:23 +01:00
Ludovico Magnocavallo b408494a74
Update main.tf (#2082) 2024-02-16 15:15:21 +01:00
simonebruzzechesse d11c380aec
Format python files in blueprints (#2079)
* format python files in blueprints
* update check on blueprints python code
* update python linter in CI workflow
2024-02-15 09:37:49 +01:00
Ludovico Magnocavallo 71a64487d5
Extend FAST to support different principal types (#2064)
* add doc draft

* typos

* typo

* typo

* typos

* rewording

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* move iam variables to a separate file

* move billing-account module to iam_principals

* move data-catalog-policy-tag module to iam_principals

* move dataplex-datascan module to iam_principals

* move dataproc module to iam_principals

* move folder module to iam_principals

* copyright

* move organization module to iam_principals

* move project module to iam_principals

* move source-repository module to iam_principals

* update blueprints for iam_principals interface

* FAST bootstrap

* module READMEs fixes

* FAST bootstrap

* FAST networking stages

* FAST security stage

* FAST gke stage

* FAST multitenant bootstrap stage

* FAST multitenant resman stage

* tfdoc

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* fix module test

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Rename iam_principals to iam_by_principals

* Update IAM template to include iam_by_principals

* Update Resman README

* Fix ADR link format

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-12 14:35:30 +01:00
Wiktor Niesiobędzki 597579fa2b
HA MySQL cluster deployment on GKE (#2061)
* MySQL pattern on GKE

* Use terraform managed password

* Use hardcoded network references

* Explain why Cloud NAT

* Rename versions_override.tf

* Fix subnet reference

* Fix password

* Fix MysQL connect commands

* Remove self-link

* Update README.md

* Add TOC and Variables table

* Fix outputs

* Fix linter

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-09 11:23:35 +01:00
Julio Castillo c42c4c141f
GKE stateful blueprints (#2059)
* var definitions

* skeleton, untested

* fix errors, test with existing cluster

* test vpc creation, todo notes

* initial variables for AR and image

* initial variables for AR and image

* Add support for remote repositories to artifact-registry

* Add support for virtual repositories to artifact-registry

* Add support for extra config options to artifact-registry

* artifact registry module: add validation and precondition, fix tests

* ar module id/name

* registry

* service accoutn and roles

* fetch pods, remove image prefix

* small changes

* use additive IAM at project level

* use additive IAM at project level

* configmaps

* manifests

* fix statefulset manifest

* service manifest

* fix configmap mode

* add todo

* job (broken)

* job

* wait on manifest, endpoints datasource

* fix job

* Fix local

* sa

* Update README.md

* Restructure gke bp

* refactor tree and infra variables

* no create test

* simplify cluster SA

* test cluster and vpc creation

* project creation fixes

* use iam_members variable

* nits

* readme with examples

* readme with examples

* outputs

* variables, provider configuration

* variables, manifests

* start cluster job

* fix redis cluster creation

Co-authored-by: Julio Castillo <juliocc@users.noreply.github.com>

* Revert changes in autopilot cluster

* Default templates path, use namespace for node names

* Update readmes

* Fix IAM bindings

* Make STABLE the default release channel

* Use Cloud DNS as default DNS provider

* Allow optional Cloud NAT creation

* Allow backup agent and proxy only subnet

* Work around terraform not short-circuiting logical operators

* Rename create variables to be more consistent with other blueprints

* Add basic features

* Update variable names

* Initial kafka JS

* Move providers to a new file

* Kafka / Strimzi

* First possibily working version for MySQL (with a lot of todo's left)

* Explicitly use proxy repo + some other fixes

* Strimzi draft

* Refactor variables, use CluterIP as pointer for mysql-router for bootstraping

* Validate number of replicas, autoscale required number of running nodes to n/2+1

* Use seaprate service for bootstrap, do not recreate all resources on change of replicas count as the config is preserved in PV

* Test dual chart kafka

* Update chart for kafka

* Expose basic kafka configuration options

* Remove unused manifest

* Added batch blueprint

* Added README

* switch to kubectl_manifest

* Add README and support for static IP address

* Move namespace creation to helm

* Interpolate kafka variables

* Rename kafka-strimzi to kafka

* Added TUTORIAL for cloudshell for batch blueprint

* deleted tutorial

* Remove commented replace trigger

* Move to helm chart

* WIP of Cloud Shell tutorial for MySQL

* Rename folders

* Fix rename

* Update paths

* Unify styles

* Update paths

* Add Readme links

* Update mysql tutorial

* Fix path according to self-link

* Use relative path to cwd

* Fix service_account variable location

* Fix tfvars creation

* Restore some fixes for helm deployment

* Add cluster deletion_prevention

* Fixes for tutorial

* Update cluster docs

* Fixes to batch tutorial

* Bare bones readme for batch

* Update batch readme

* README fixes

* Fix README title for redis

* Fix Typos

* Make it easy to pass variables from autopilot-cluster to other modules

* Add connectivity test and bastion host

* updates to readme, and gpu fix

* Add versions.tf and README updates

* Fix typo

* Kafka and Redis README updates

* Update versions.tf

* Fixes

* Add boilerplate

* Fix linting

* Move mysql to separate branch

* Update cloud shell links

* Fix broken link

---------

Co-authored-by: Ludo <ludomagno@google.com>
Co-authored-by: Daniel Marzini <44803752+danielmarzini@users.noreply.github.com>
Co-authored-by: Wiktor Niesiobędzki <wiktorn@google.com>
Co-authored-by: Miren Esnaola <mirene@google.com>
2024-02-08 18:28:41 +00:00
Wiktor Niesiobędzki 29232c10c4 Fix failing tests after Terraform upgrade 2024-02-07 17:25:11 +01:00
Ludovico Magnocavallo d127c25ad0
Shielded nodes and custom service account in FAST GKE stage and blueprint (CSPR-related) (#2036)
* default to shielded nodes in FAST gke stage

* use custom service account in GKE multitenant blueprint
2024-02-01 15:16:00 +00:00
Ludovico Magnocavallo 11d7edac64
Add example to FAST GKE stage, streamline GKE Hub module variables and usage (#1977)
* implement optionals in gke-hub module

* simplify gke hub module call in mc mesh blueprint

* simplify gke hub module call and variables in multitenant blueprint

* gke hub inventory

* provide cluster and fleet examples in stage
2024-01-20 10:06:38 +00:00
apichick 66bd9d5160 Added workstation-cluster module 2023-11-30 07:02:28 +01:00
Luca Prete 7c6726e79b
[net-address] enable ipv6 (#1821)
---------

Co-authored-by: Luca Prete <lucaprete@google.com>
2023-10-28 15:36:30 +02:00
Simone Ruffilli 6d89b88149
versions.tf maintenance + copyright notice bump (#1782)
* Bump copyright notice to 2023

* Delete versions.tf on blueprints

* Pin provider to major version 5

* Remove comment

* Fix lint

* fix bq-ml blueprint readme

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
Co-authored-by: Julio Castillo <jccb@google.com>
2023-10-20 18:17:47 +02:00
Julio Castillo dfc5023e0b
Make deletion protection consistent across all modules (#1735)
* Expose deletion_protection in GKE modules

* Make deletion protection consistent across all modules

* Add deletion_protection option to blueprints

* Fix blueprints tests

* Fix types

* Update READMEs

* Fix dp readme

* Fix cmek blueprint default deletion_protection

* Fix blueprints tests
2023-10-05 17:31:07 +02:00
Ludovico Magnocavallo 789328ff5a
Bump provider versions to v5.0.0 (#1724)
* bump provider versions to 5.0.0

* fix cloud run, logging and vpc-sc

* Fix secret manager

* fix gke nodepool

* fix gke multitenant stage and blueprint

* Moving alloydb module to experimental.

* Add project to bare resources in examples

* tfdoc

* fix svpc blueprint test

* Revert "fix svpc blueprint test"

This reverts commit 14f02659098070136e64ead600580dd52c23c339.

* Fix GKE peering project

* Disable tests in alloydb module

* Bring back secret ids in secret manager tests

* Remove duplicate key

* last push

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2023-10-03 12:15:36 +00:00
Ludovico Magnocavallo df5daab6cc
Allow using no service account in compute-vm (#1692)
* module and tests

* align blueprints

* tfdoc

* remove stale variables

* fix blueprint

* variable description
2023-09-19 16:56:51 +00:00
Julio Castillo 9c878dc9cf Fix tests for new KMS IAM interface 2023-09-17 00:21:36 +02:00
Oliver Frolovs 6eb862a775
GKE cluster modules: add optional kube state metrics (#1682)
* `gke-cluster-standard`: add optional kube state metrics
* `gke-cluster-autopilot`: add optional kube state metrics

* FAST: add kube state metrics support for GKE
* blueprints/gke: add kube state metrics support

* Bump up the provider version to `v4.82.0`
2023-09-15 12:18:45 +01:00
Oliver Frolovs b3dc91b5cd
Upgrades to `monitoring_config` in `gke-cluster-*`, docs update, and cosmetics fixes to GKE cluster modules (#1680)
* gke-cluster-standard: upgrade `monitoring_config` to use object style. Add tests.
* gke-cluster-standard: update docs

* gke-cluster-autopilot: move gateway_api_config block (cosmetic change)
* gke-cluster-autopilot: update docs and fix typos

* Update blueprints due to `monitoring_config` changes in `gke-cluster-standard`.
* Update FAST due to `monitoring_config` changes in `gke-cluster-standard`.
* Update docs for affected blueprints and FAST stages
2023-09-14 23:25:57 +01:00
Julio Castillo c1be435b09 Fix range names definition of GKE clusters
Fixes #1677
2023-09-14 12:51:43 +02:00
Oliver Frolovs 988fd2ee05
gke-cluster-standard: change logging configuration (#1638)
* Update logging configuration of this module to use object interface in harmony with `gke-cluster-autopilot` module.
* Update blueprints that use this module.
* Add "WORKLOADS" log source to logging configuration of the blueprints where the README files say so.
* Update FAST stage 3 because it uses this module.
2023-08-31 12:49:15 +01:00
Alejandro Leal ea0de3adbb Fixing some typos 2023-08-18 05:51:00 +00:00
Ludovico Magnocavallo 5a534edf5c
update and refactor artifact registry module (#1536) 2023-07-28 09:54:36 +00:00
dependabot[bot] e64c043aaa
Bump semver and @google-cloud/storage in /blueprints/gke/binauthz/image (#1469)
Removes [semver](https://github.com/npm/node-semver). It's no longer used after updating ancestor dependency [@google-cloud/storage](https://github.com/googleapis/nodejs-storage). These dependencies need to be updated together.


Removes `semver`

Updates `@google-cloud/storage` from 5.18.3 to 6.11.0
- [Release notes](https://github.com/googleapis/nodejs-storage/releases)
- [Changelog](https://github.com/googleapis/nodejs-storage/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/nodejs-storage/compare/v5.18.3...v6.11.0)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
- dependency-name: "@google-cloud/storage"
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-26 15:03:47 +02:00
Julio Castillo cecbd2072c Fix modules and blueprints tests 2023-05-26 16:38:41 +02:00
Alejandro Leal 87cd83f5c0 Several updates
Several updates
2023-05-13 23:51:46 -04:00
Jack P 491b52f023
update variables files for gke nodepool taints (#1358)
* update variables files for gke node config taints to allow passing of node objects

* forgot to run terraform fmt..

* update module docs
2023-05-05 19:42:00 +02:00
Julio Castillo d6ee1b6551
Merge branch 'master' into autopilot-fix-requests 2023-04-21 14:26:13 +02:00
Avinash Jha e881537f87
Separating GKE Standard and Autopilot Modules (#1330)
* separating GKE Standard and Autopilot Modules

* Changes for Updating the terraform and provide versions

* Changes for Autopilot Readme

* Changes for Autopilot Variable

* Changes for Autopilot Readme

* Changes for Autopilot Readme

* Changes for Blueprint

* Changes for Blueprint ReadMe

* Changes for gke-standard-cluster dependency

* Changes for gke-standard-cluster in gke-fleet

* Changes for gke-standard-cluster in cluster-mesh-gke-fleet-api

* python formatting

* python formatting

* python formatting

* GKE module naming convention

* Readme Changes

* test module

* Removing comment code from Autopilot
2023-04-21 12:08:13 +00:00
Miren Esnaola 93ceab5be9 Set all resource requests to the autopilot minimum as the existing values was causing problems with the locust workers 2023-04-21 12:58:43 +02:00
Miren Esnaola 8b4f29ac24 Fixed wait_time in locust script 2023-04-11 21:42:06 +02:00
Jack Powell 024801dc74 add missing enable_addons reference in gke blueprint for multitenant-fleet 2023-04-11 11:27:07 +01:00
Miren Esnaola 8cbaa932ed Fixed permissions of files created 2023-03-27 14:49:09 +02:00
Miren Esnaola 21fa6d1f13 Fixes related to boot_disk in compute-vm module 2023-03-16 15:58:39 +01:00
Miren Esnaola 57282d5dd3 Autopilot fixes 2023-03-13 12:55:45 +01:00
Miren Esnaola a39fa7ca64 Added autopilot blueprint 2023-03-07 15:37:20 +01:00
Julio Castillo 3c68780537 Update gke blueprints tests 2023-02-25 19:26:11 +01:00
Julio Castillo a5e905cb80 Update remaining org policies 2023-02-21 15:49:16 +01:00
Ludovico Magnocavallo 5453c585e0
FAST multitenant bootstrap and resource management, rename org-level FAST stages (#1052)
* rename stages

* remove support for external org billing, rename output files

* resman: make groups optional, align on new billing account variable

* bootstrap: multitenant outputs

* tenant bootstrap stage, untested

* fix folder name

* fix stage 0 output names

* optional creation for tag keys in organization module

* single tenant bootstrap minus tag

* rename output files, add tenant tag key

* fix organization module tag values output

* test skipping creation for tags in organization module

* single tenant bootstrap plan working

* multitenant bootstrap

* tfdoc

* fix check links error messages

* fix links

* tfdoc

* fix links

* rename fast tests, fix bootstrap tests

* multitenant stages have their own folder, simplify stage numbering

* stage renumbering

* wip

* rename tests

* exclude fast providers in fixture

* stage 0 tests

* stage 1 tests

* network stages tests

* stage tests

* tfdoc

* fix links

* tfdoc

* multitenant tests

* remove local files

* stage links command

* fix links script, TODO

* wip

* wip single tenant bootstrap

* working tenant bootstrap

* update gitignore

* remove local files

* tfdoc

* remove local files

* allow tests for tenant bootstrap stage

* tenant bootstrap proxies stage 1 tfvars

* stage 2 and 3 service accounts and IAM in tenant bootstrap

* wip

* wip

* wip

* drop multitenant bootstrap

* tfdoc

* add missing stage 2 SAs, fix org-level IAM condition

* wip

* wip

* optional tag value creation in organization module

* stage 1 working

* linting

* linting

* READMEs

* wip

* Make stage-links script work in old macos bash

* stage links command help

* fix output file names

* diagrams

* fix svg

* stage 0 skeleton and diagram

* test svg

* test svg

* test diagram

* diagram

* readme

* fix stage links script

* stage 0 readme

* README changes

* stage readmes

* fix outputs order

* fix link

* fix tests

* stage 1 test

* skip stage example

* boilerplate

* fix tftest skip

* default bootstrap stage log sinks to log buckets

* add logging to tenant bootstrap

* move iam variables out of tenant config

* fix cicd, reintroduce missing variable

* use optional in stage 1 cicd variable

* rename extras stage

* rename and move identity providers local, use optional for cicd variable

* tfdoc

* add support for wif pool and providers, ci/cd

* tfdoc

* fix links

* better handling of modules repository

* add missing role on logging project

* fix cicd pools in locals, test cicd

* fix workflow extension

* fix module source replacement

* allow tenant bootstrap cicd sa to impersonate resman sa

* tenant workflow templates fix for no providers file

* fix output files, push github workflow template to new repository

* remove try from outpout files

* align stage 1 cicd internals to stage 0

* tfdoc

* tests

* fix tests

* tests

* improve variable descriptions

* use optional in fast features

* actually create tenant log sinks, and allow the resman sa to do it

* test

* tests

* aaaand tests again

* fast features tenant override

* fast features tenant override

* fix wording

* add missing comment

* configure pf service accounts

* add missing comment

* tfdoc

* tests

* IAM docs

* update copyright

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2023-02-04 15:00:45 +01:00
Wiktor Niesiobędzki 2976df596e Fix tests (add JIT-ed SA for multiclusteringress.googleapis.com) 2023-01-29 15:50:24 +01:00
Wiktor Niesiobędzki b38ef22572 Even more test fixes 2023-01-19 09:37:48 +01:00
Wiktor Niesiobędzki 720213593e Use google_gkehub_feature_membership
Use google_gkehub_feature_membership to provision ASM on clusters.

Ensure that the cluster membership is refreshed on cluster recreation.
2023-01-19 09:37:48 +01:00
Julio Castillo e700a27079 Enforce terraform fmt in examples 2022-12-18 14:00:19 +01:00
apichick ba17e10ebd
Merge branch 'master' into binauthz-fixes 2022-11-28 12:33:34 +01:00
Julio Castillo 612d7283e8 Enforce nonempty descriptions ending in a colon 2022-11-24 18:56:01 +01:00
Sebastian Kunze e4fc47a063
Normalize prefix handling in blueprints (#1003) 2022-11-23 11:09:00 +01:00
Miren Esnaola 5ba65b99ed Binauthz blueprint fixes related to project creation 2022-11-22 12:45:43 +01:00
Julio Castillo 826ff14ef9 Sort variables and outputs 2022-11-21 13:17:55 +01:00