Commit Graph

805 Commits

Author SHA1 Message Date
Ben Swenka 1b44af595c
Updated diagram to better reflect PSC terminology (#2208) 2024-04-09 15:18:43 +00:00
Tone 2831af09fa
feat(gke-cluster-standard): Add optional `CiliumClusterWideNetworkPolicy` (#2207)
* feat(gke-cluster-standard): Add optionnal `CiliumClusterWideNetworkPolicy`

Add `CiliumClusterWideNetworkPolicy` option on cluster.

Ref:
 - https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/container_cluster#enable_cilium_clusterwide_network_policy
 - https://cloud.google.com/kubernetes-engine/docs/how-to/configure-cilium-network-policy

* feat(gke-cluster-standard): Update Google provider to manage new feature

* feat(gke-cluster-standard): Move `cilium_clusterwide_network_policy` to `enable_features` field

* fix(scheduled-asset-inventory-export-bq): Set `dataset_id` with underscores

* fix(bigquery-dataset): Set `dataset_id` with underscores
2024-04-09 17:08:36 +02:00
bluPhy c420f17636
Updating cloud-run-v2 terraform and some typos (#2201)
* Updating cloud-run-v2 terraform and some typos

Updating modules/cloud-run-v2/main.tf to determine the tcp_socket value

Fixin typos in:
tests/fixtures.py
modules/workstation-cluster/README.md
modules/net-lb-int/README.md
modules/dataplex/README.md
modules/dataform-repository/README.md
modules/__docs/20230816-iam-refactor.md
CONTRIBUTING.md
blueprints/third-party-solutions/f5-bigip/f5-bigip-ha-active/startup-script.tpl
blueprints/third-party-solutions/f5-bigip/f5-bigip-ha-active/README.md

* Update README.md

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
Co-authored-by: Ludovico Magnocavallo <ludo@qix.it>
2024-04-07 09:49:07 +00:00
Elia b80132a618
FAST GCVE stage (#2191)
* GCVE stage prerequisites

* fix gcve prereq

* gcve stage first deploy

* Updated readme filex

* docs updated

* some fixes after testing

* updated readme files

* elia fix

* gcve net admin custom role

* gcve net admin custom role

* elia fix

* ven peering deploy

* elia fix

* added blueprint and stage tests

* Edits to Readme files

* typo in outputs

* clean-up

* gcve stage tests fix

* readme fix

* fix sorting

* fix copyrights and readme file

* fix test

* fix copyright

* fixed gcve feature flag cond.

* removed validation

* fixed typo

* fixed typo

* fixed gcve tests

* fixed typo

* fixed typo

* fixed sorting

* fixed sorting

---------

Co-authored-by: Konrad Schieban <kschieban@google.com>
2024-04-03 17:25:12 +02:00
dependabot[bot] 4e8f60e8e2
Bump express from 4.17.3 to 4.19.2 in /blueprints/gke/binauthz/image (#2181)
Bumps [express](https://github.com/expressjs/express) from 4.17.3 to 4.19.2.
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/master/History.md)
- [Commits](https://github.com/expressjs/express/compare/4.17.3...4.19.2)

---
updated-dependencies:
- dependency-name: express
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-28 07:14:50 +01:00
Wiktor Niesiobędzki f2806f347b Bump provider version to 5.18, so fix non-empty plan for google_notebooks_instance 2024-03-25 19:57:14 +01:00
Ludovico Magnocavallo a590deb58b
Fix subnet configuration in cloud nat module (#2171)
* support optional secondary ranges in net-cloudnat module

* fix subnet configuration

* fix packer blueprint
2024-03-22 15:59:02 +01:00
Ludovico Magnocavallo 5ac86ecae7
Support advanced_datapath_observability in gke cluster standard module (#2168)
* support advanced_datapath_observability in gke cluster standard module

* bump provider to 5.17.0
2024-03-22 08:25:43 +01:00
Ludovico Magnocavallo e051cf42a1
stub READMEs for the removed hub and spoke blueprints (#2169) 2024-03-22 06:48:46 +00:00
Julio Castillo 93e9909166 Fix typo in kafka pattern 2024-03-13 17:59:07 +03:00
Julio Castillo f16da6099d Fix typos in GKE patterns 2024-03-13 17:57:07 +03:00
Wiktor Niesiobędzki 67382f53b7 Fix data-platform-minimal blueprint 2024-03-11 11:05:33 +01:00
Tianyou3 c21ec4ebf3
Update README.md (#2143)
Fixed typo in /cloud-foundation-fabric/tree/master/blueprints/cloud-operations/README.md
2024-03-10 14:25:12 +00:00
Julio Castillo 993bef71aa
Introduce mandatory OWNERS file for blueprint maintainership (#2131)
* Delete deprecated/broken blueprints

* Adding OWNERS to all blueprints

* Fix links

* Update OWNERS

---------

Co-authored-by: javiergp <javiergp@users.noreply.github.com>
2024-03-08 09:40:46 +01:00
Ben Swenka dcd54246cc
Updated diagram to better reflect code naming. (#2133)
* Updated diagram to better reflect code naming.
2024-03-06 19:23:36 +00:00
Julio Castillo dfed7cc5bd
Rename `modules/cloudsql-instance` deletion protection variables (#2135)
* Rename Cloud SQL deletion protection variables

* Fix tests
2024-03-06 10:44:54 +00:00
Ludovico Magnocavallo e12cf83188
add links to factories doc (#2134) 2024-03-06 08:25:43 +01:00
simonebruzzechesse a5f8c501a0
Fix phpipam blueprint (#2119)
* fix phpipam blueprint
2024-02-29 09:33:07 +01:00
Julio Castillo 30496068e2
Implement GKE patterns naming conventions (#2120)
* GKE Pattern Naming

* Add MySQL pattern estimated cost

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2024-02-29 06:57:22 +00:00
simonebruzzechesse a34d93fb43
Gitlab blueprint (#2110)
* add gitlab blueprint
* add TODO.md

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-27 18:36:46 +01:00
Ludovico Magnocavallo 6941313c7d
Factories refactor (#1843)
* factories refactor doc

* Adds file schema and filesystem organization

* Update 20231106-factories.md

* move factories out of blueprints and create new factories  README

* align factory in billing-account module

* align factory in dataplex-datascan module

* align factory in billing-account module

* align factory in net-firewall-policy module

* align factory in dns-response-policy module

* align factory in net-vpc-firewall module

* align factory in net-vpc module

* align factory variable names in FAST

* remove decentralized firewall blueprint

* bump terraform version

* bump module versions

* update top-level READMEs

* move project factory to modules

* fix variable names and tests

* tfdoc

* remove changelog link

* add project factory to top-level README

* fix cludrun eventarc diff

* fix README

* fix cludrun eventarc diff

---------

Co-authored-by: Simone Ruffilli <sruffilli@google.com>
2024-02-26 10:16:52 +00:00
Ludovico Magnocavallo 220ab76e40
enable shielded nodes by default on GKE mt blueprint and FAST stage (#2105) 2024-02-22 07:35:27 +00:00
Julio Castillo 8102803d2e
Update tutorial.md 2024-02-21 15:01:23 +01:00
Ludovico Magnocavallo b408494a74
Update main.tf (#2082) 2024-02-16 15:15:21 +01:00
Wiktor Niesiobędzki e02d871d58 fix handling default values 2024-02-15 21:57:47 +01:00
Wiktor Niesiobędzki b1956403d0 Fix missing region 2024-02-15 21:57:47 +01:00
Wiktor Niesiobędzki 19be54a72f Use Fabric modules when possibile 2024-02-15 21:57:47 +01:00
Ben Swenka d158aecba1 Clarifying that the default path --> Producer A 2024-02-15 18:27:26 +01:00
Ben Swenka 4513d7de0a Updated path matchers to be more user friendly, added better test examples in README.md 2024-02-15 18:27:26 +01:00
simonebruzzechesse d11c380aec
Format python files in blueprints (#2079)
* format python files in blueprints
* update check on blueprints python code
* update python linter in CI workflow
2024-02-15 09:37:49 +01:00
Ben Swenka 87a350db93
Bswenka/psc glb and armor 2 producers (#2071)
* Enhanced this blueprint to add a second producer, and modularized the producer.

* Fixed terraform formatting

* Updating README.md with tfdoc

* Fixed test case conditions & module variable passing
2024-02-14 16:40:51 +01:00
Wiktor Niesiobędzki b902b1dab9 Fix non-empty plan after apply 2024-02-13 07:40:31 +01:00
Ludovico Magnocavallo 71a64487d5
Extend FAST to support different principal types (#2064)
* add doc draft

* typos

* typo

* typo

* typos

* rewording

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* move iam variables to a separate file

* move billing-account module to iam_principals

* move data-catalog-policy-tag module to iam_principals

* move dataplex-datascan module to iam_principals

* move dataproc module to iam_principals

* move folder module to iam_principals

* copyright

* move organization module to iam_principals

* move project module to iam_principals

* move source-repository module to iam_principals

* update blueprints for iam_principals interface

* FAST bootstrap

* module READMEs fixes

* FAST bootstrap

* FAST networking stages

* FAST security stage

* FAST gke stage

* FAST multitenant bootstrap stage

* FAST multitenant resman stage

* tfdoc

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* fix module test

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Rename iam_principals to iam_by_principals

* Update IAM template to include iam_by_principals

* Update Resman README

* Fix ADR link format

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-12 14:35:30 +01:00
Ben Swenka 89779c0d0e
Bswenka/psc glb and armor subnet fix (#2058)
* Updated ilb-subnetwork issue

* Fixed Terraform formatting issue

* Fixing merge issue

* Fixing merge issue in producer.tf

* Fixed linting issues

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-09 10:41:14 +00:00
Wiktor Niesiobędzki 597579fa2b
HA MySQL cluster deployment on GKE (#2061)
* MySQL pattern on GKE

* Use terraform managed password

* Use hardcoded network references

* Explain why Cloud NAT

* Rename versions_override.tf

* Fix subnet reference

* Fix password

* Fix MysQL connect commands

* Remove self-link

* Update README.md

* Add TOC and Variables table

* Fix outputs

* Fix linter

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-09 11:23:35 +01:00
Julio Castillo c42c4c141f
GKE stateful blueprints (#2059)
* var definitions

* skeleton, untested

* fix errors, test with existing cluster

* test vpc creation, todo notes

* initial variables for AR and image

* initial variables for AR and image

* Add support for remote repositories to artifact-registry

* Add support for virtual repositories to artifact-registry

* Add support for extra config options to artifact-registry

* artifact registry module: add validation and precondition, fix tests

* ar module id/name

* registry

* service accoutn and roles

* fetch pods, remove image prefix

* small changes

* use additive IAM at project level

* use additive IAM at project level

* configmaps

* manifests

* fix statefulset manifest

* service manifest

* fix configmap mode

* add todo

* job (broken)

* job

* wait on manifest, endpoints datasource

* fix job

* Fix local

* sa

* Update README.md

* Restructure gke bp

* refactor tree and infra variables

* no create test

* simplify cluster SA

* test cluster and vpc creation

* project creation fixes

* use iam_members variable

* nits

* readme with examples

* readme with examples

* outputs

* variables, provider configuration

* variables, manifests

* start cluster job

* fix redis cluster creation

Co-authored-by: Julio Castillo <juliocc@users.noreply.github.com>

* Revert changes in autopilot cluster

* Default templates path, use namespace for node names

* Update readmes

* Fix IAM bindings

* Make STABLE the default release channel

* Use Cloud DNS as default DNS provider

* Allow optional Cloud NAT creation

* Allow backup agent and proxy only subnet

* Work around terraform not short-circuiting logical operators

* Rename create variables to be more consistent with other blueprints

* Add basic features

* Update variable names

* Initial kafka JS

* Move providers to a new file

* Kafka / Strimzi

* First possibily working version for MySQL (with a lot of todo's left)

* Explicitly use proxy repo + some other fixes

* Strimzi draft

* Refactor variables, use CluterIP as pointer for mysql-router for bootstraping

* Validate number of replicas, autoscale required number of running nodes to n/2+1

* Use seaprate service for bootstrap, do not recreate all resources on change of replicas count as the config is preserved in PV

* Test dual chart kafka

* Update chart for kafka

* Expose basic kafka configuration options

* Remove unused manifest

* Added batch blueprint

* Added README

* switch to kubectl_manifest

* Add README and support for static IP address

* Move namespace creation to helm

* Interpolate kafka variables

* Rename kafka-strimzi to kafka

* Added TUTORIAL for cloudshell for batch blueprint

* deleted tutorial

* Remove commented replace trigger

* Move to helm chart

* WIP of Cloud Shell tutorial for MySQL

* Rename folders

* Fix rename

* Update paths

* Unify styles

* Update paths

* Add Readme links

* Update mysql tutorial

* Fix path according to self-link

* Use relative path to cwd

* Fix service_account variable location

* Fix tfvars creation

* Restore some fixes for helm deployment

* Add cluster deletion_prevention

* Fixes for tutorial

* Update cluster docs

* Fixes to batch tutorial

* Bare bones readme for batch

* Update batch readme

* README fixes

* Fix README title for redis

* Fix Typos

* Make it easy to pass variables from autopilot-cluster to other modules

* Add connectivity test and bastion host

* updates to readme, and gpu fix

* Add versions.tf and README updates

* Fix typo

* Kafka and Redis README updates

* Update versions.tf

* Fixes

* Add boilerplate

* Fix linting

* Move mysql to separate branch

* Update cloud shell links

* Fix broken link

---------

Co-authored-by: Ludo <ludomagno@google.com>
Co-authored-by: Daniel Marzini <44803752+danielmarzini@users.noreply.github.com>
Co-authored-by: Wiktor Niesiobędzki <wiktorn@google.com>
Co-authored-by: Miren Esnaola <mirene@google.com>
2024-02-08 18:28:41 +00:00
Wiktor Niesiobędzki 29232c10c4 Fix failing tests after Terraform upgrade 2024-02-07 17:25:11 +01:00
Ludovico Magnocavallo d127c25ad0
Shielded nodes and custom service account in FAST GKE stage and blueprint (CSPR-related) (#2036)
* default to shielded nodes in FAST gke stage

* use custom service account in GKE multitenant blueprint
2024-02-01 15:16:00 +00:00
Ludovico Magnocavallo 3b4b355800
Ensure data platform service accounts meet FAST requirements (#2016)
* use 9-chars prefix in data platform example test

* use 9-chars prefix in data platform example test
2024-01-28 14:00:32 +01:00
Wiktor Niesiobędzki 526185fd1f
Remove default region for Cloud Function and Cloud Run (#2004)
Remove default region for Cloud Function and Cloud Run
2024-01-24 10:23:40 +00:00
Ludovico Magnocavallo 11d7edac64
Add example to FAST GKE stage, streamline GKE Hub module variables and usage (#1977)
* implement optionals in gke-hub module

* simplify gke hub module call in mc mesh blueprint

* simplify gke hub module call and variables in multitenant blueprint

* gke hub inventory

* provide cluster and fleet examples in stage
2024-01-20 10:06:38 +00:00
lcaggio 208902c8da
Fix Data platform foundation (#1992)
* FAST + Minimal DP

* Fix tests

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-01-20 08:49:46 +01:00
Aurélien Legrand eee6aaff2c
Network dashboard - fixing 2 bugs: overriden variable and page token … (#1976)
* Network dashboard - fixing 2 bugs: overriden variable and page token not handled properly

* changing pageToken in the regexp
2024-01-15 14:28:17 +01:00
Julio Diez e623fa4a66
Merge branch 'master' into jd/serverless-program 2024-01-05 21:49:12 +01:00
Julio Diez 8435ad85f3 Address some style comments 2024-01-05 21:48:02 +01:00
Wiktor Niesiobędzki 6c258e6562 Fix description 2023-12-29 12:09:16 +01:00
Wiktor Niesiobędzki 1eea077460 Add service account email to outputs to manage its permissions 2023-12-29 12:09:16 +01:00
Wiktor Niesiobędzki 724dfda09c Provider doesn't set defaults on config
Can't provide just one size (like `web_server` or `triggerrer`) because
of no defaults are taken:
module.composer.google_composer_environment.env: Modifying... [id=***]
╷
│ Error: googleapi: Error 400: Found 6 problems:
│       1) You have to specify Scheduler CPUs not lower than 0.5.
│       2) You have to specify number of schedulers larger than 0.
│       3) You have to specify Web Server CPUs not lower than 0.5.
│       4) You have to specify Worker CPUs not lower than 0.5.
│       5) You have to specify minimum number of workers larger than 0.
│       6) Triggerer memory must be between 1.00GB and 6.50GB for given vCpu

So provide the defaults as set workloads_config == null
2023-12-29 12:09:16 +01:00
Julio Diez 6bad85d758 Update related serverless READMEs 2023-12-29 11:14:16 +01:00
Julio Diez 3685eb8493 Fix tftest values 2023-12-28 20:30:10 +01:00