Commit Graph

791 Commits

Author SHA1 Message Date
Ben Swenka dcd54246cc
Updated diagram to better reflect code naming. (#2133)
* Updated diagram to better reflect code naming.
2024-03-06 19:23:36 +00:00
Julio Castillo dfed7cc5bd
Rename `modules/cloudsql-instance` deletion protection variables (#2135)
* Rename Cloud SQL deletion protection variables

* Fix tests
2024-03-06 10:44:54 +00:00
Ludovico Magnocavallo e12cf83188
add links to factories doc (#2134) 2024-03-06 08:25:43 +01:00
simonebruzzechesse a5f8c501a0
Fix phpipam blueprint (#2119)
* fix phpipam blueprint
2024-02-29 09:33:07 +01:00
Julio Castillo 30496068e2
Implement GKE patterns naming conventions (#2120)
* GKE Pattern Naming

* Add MySQL pattern estimated cost

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2024-02-29 06:57:22 +00:00
simonebruzzechesse a34d93fb43
Gitlab blueprint (#2110)
* add gitlab blueprint
* add TODO.md

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-27 18:36:46 +01:00
Ludovico Magnocavallo 6941313c7d
Factories refactor (#1843)
* factories refactor doc

* Adds file schema and filesystem organization

* Update 20231106-factories.md

* move factories out of blueprints and create new factories  README

* align factory in billing-account module

* align factory in dataplex-datascan module

* align factory in billing-account module

* align factory in net-firewall-policy module

* align factory in dns-response-policy module

* align factory in net-vpc-firewall module

* align factory in net-vpc module

* align factory variable names in FAST

* remove decentralized firewall blueprint

* bump terraform version

* bump module versions

* update top-level READMEs

* move project factory to modules

* fix variable names and tests

* tfdoc

* remove changelog link

* add project factory to top-level README

* fix cludrun eventarc diff

* fix README

* fix cludrun eventarc diff

---------

Co-authored-by: Simone Ruffilli <sruffilli@google.com>
2024-02-26 10:16:52 +00:00
Ludovico Magnocavallo 220ab76e40
enable shielded nodes by default on GKE mt blueprint and FAST stage (#2105) 2024-02-22 07:35:27 +00:00
Julio Castillo 8102803d2e
Update tutorial.md 2024-02-21 15:01:23 +01:00
Ludovico Magnocavallo b408494a74
Update main.tf (#2082) 2024-02-16 15:15:21 +01:00
Wiktor Niesiobędzki e02d871d58 fix handling default values 2024-02-15 21:57:47 +01:00
Wiktor Niesiobędzki b1956403d0 Fix missing region 2024-02-15 21:57:47 +01:00
Wiktor Niesiobędzki 19be54a72f Use Fabric modules when possibile 2024-02-15 21:57:47 +01:00
Ben Swenka d158aecba1 Clarifying that the default path --> Producer A 2024-02-15 18:27:26 +01:00
Ben Swenka 4513d7de0a Updated path matchers to be more user friendly, added better test examples in README.md 2024-02-15 18:27:26 +01:00
simonebruzzechesse d11c380aec
Format python files in blueprints (#2079)
* format python files in blueprints
* update check on blueprints python code
* update python linter in CI workflow
2024-02-15 09:37:49 +01:00
Ben Swenka 87a350db93
Bswenka/psc glb and armor 2 producers (#2071)
* Enhanced this blueprint to add a second producer, and modularized the producer.

* Fixed terraform formatting

* Updating README.md with tfdoc

* Fixed test case conditions & module variable passing
2024-02-14 16:40:51 +01:00
Wiktor Niesiobędzki b902b1dab9 Fix non-empty plan after apply 2024-02-13 07:40:31 +01:00
Ludovico Magnocavallo 71a64487d5
Extend FAST to support different principal types (#2064)
* add doc draft

* typos

* typo

* typo

* typos

* rewording

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* move iam variables to a separate file

* move billing-account module to iam_principals

* move data-catalog-policy-tag module to iam_principals

* move dataplex-datascan module to iam_principals

* move dataproc module to iam_principals

* move folder module to iam_principals

* copyright

* move organization module to iam_principals

* move project module to iam_principals

* move source-repository module to iam_principals

* update blueprints for iam_principals interface

* FAST bootstrap

* module READMEs fixes

* FAST bootstrap

* FAST networking stages

* FAST security stage

* FAST gke stage

* FAST multitenant bootstrap stage

* FAST multitenant resman stage

* tfdoc

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* fix module test

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Rename iam_principals to iam_by_principals

* Update IAM template to include iam_by_principals

* Update Resman README

* Fix ADR link format

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-12 14:35:30 +01:00
Ben Swenka 89779c0d0e
Bswenka/psc glb and armor subnet fix (#2058)
* Updated ilb-subnetwork issue

* Fixed Terraform formatting issue

* Fixing merge issue

* Fixing merge issue in producer.tf

* Fixed linting issues

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-09 10:41:14 +00:00
Wiktor Niesiobędzki 597579fa2b
HA MySQL cluster deployment on GKE (#2061)
* MySQL pattern on GKE

* Use terraform managed password

* Use hardcoded network references

* Explain why Cloud NAT

* Rename versions_override.tf

* Fix subnet reference

* Fix password

* Fix MysQL connect commands

* Remove self-link

* Update README.md

* Add TOC and Variables table

* Fix outputs

* Fix linter

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-09 11:23:35 +01:00
Julio Castillo c42c4c141f
GKE stateful blueprints (#2059)
* var definitions

* skeleton, untested

* fix errors, test with existing cluster

* test vpc creation, todo notes

* initial variables for AR and image

* initial variables for AR and image

* Add support for remote repositories to artifact-registry

* Add support for virtual repositories to artifact-registry

* Add support for extra config options to artifact-registry

* artifact registry module: add validation and precondition, fix tests

* ar module id/name

* registry

* service accoutn and roles

* fetch pods, remove image prefix

* small changes

* use additive IAM at project level

* use additive IAM at project level

* configmaps

* manifests

* fix statefulset manifest

* service manifest

* fix configmap mode

* add todo

* job (broken)

* job

* wait on manifest, endpoints datasource

* fix job

* Fix local

* sa

* Update README.md

* Restructure gke bp

* refactor tree and infra variables

* no create test

* simplify cluster SA

* test cluster and vpc creation

* project creation fixes

* use iam_members variable

* nits

* readme with examples

* readme with examples

* outputs

* variables, provider configuration

* variables, manifests

* start cluster job

* fix redis cluster creation

Co-authored-by: Julio Castillo <juliocc@users.noreply.github.com>

* Revert changes in autopilot cluster

* Default templates path, use namespace for node names

* Update readmes

* Fix IAM bindings

* Make STABLE the default release channel

* Use Cloud DNS as default DNS provider

* Allow optional Cloud NAT creation

* Allow backup agent and proxy only subnet

* Work around terraform not short-circuiting logical operators

* Rename create variables to be more consistent with other blueprints

* Add basic features

* Update variable names

* Initial kafka JS

* Move providers to a new file

* Kafka / Strimzi

* First possibily working version for MySQL (with a lot of todo's left)

* Explicitly use proxy repo + some other fixes

* Strimzi draft

* Refactor variables, use CluterIP as pointer for mysql-router for bootstraping

* Validate number of replicas, autoscale required number of running nodes to n/2+1

* Use seaprate service for bootstrap, do not recreate all resources on change of replicas count as the config is preserved in PV

* Test dual chart kafka

* Update chart for kafka

* Expose basic kafka configuration options

* Remove unused manifest

* Added batch blueprint

* Added README

* switch to kubectl_manifest

* Add README and support for static IP address

* Move namespace creation to helm

* Interpolate kafka variables

* Rename kafka-strimzi to kafka

* Added TUTORIAL for cloudshell for batch blueprint

* deleted tutorial

* Remove commented replace trigger

* Move to helm chart

* WIP of Cloud Shell tutorial for MySQL

* Rename folders

* Fix rename

* Update paths

* Unify styles

* Update paths

* Add Readme links

* Update mysql tutorial

* Fix path according to self-link

* Use relative path to cwd

* Fix service_account variable location

* Fix tfvars creation

* Restore some fixes for helm deployment

* Add cluster deletion_prevention

* Fixes for tutorial

* Update cluster docs

* Fixes to batch tutorial

* Bare bones readme for batch

* Update batch readme

* README fixes

* Fix README title for redis

* Fix Typos

* Make it easy to pass variables from autopilot-cluster to other modules

* Add connectivity test and bastion host

* updates to readme, and gpu fix

* Add versions.tf and README updates

* Fix typo

* Kafka and Redis README updates

* Update versions.tf

* Fixes

* Add boilerplate

* Fix linting

* Move mysql to separate branch

* Update cloud shell links

* Fix broken link

---------

Co-authored-by: Ludo <ludomagno@google.com>
Co-authored-by: Daniel Marzini <44803752+danielmarzini@users.noreply.github.com>
Co-authored-by: Wiktor Niesiobędzki <wiktorn@google.com>
Co-authored-by: Miren Esnaola <mirene@google.com>
2024-02-08 18:28:41 +00:00
Wiktor Niesiobędzki 29232c10c4 Fix failing tests after Terraform upgrade 2024-02-07 17:25:11 +01:00
Ludovico Magnocavallo d127c25ad0
Shielded nodes and custom service account in FAST GKE stage and blueprint (CSPR-related) (#2036)
* default to shielded nodes in FAST gke stage

* use custom service account in GKE multitenant blueprint
2024-02-01 15:16:00 +00:00
Ludovico Magnocavallo 3b4b355800
Ensure data platform service accounts meet FAST requirements (#2016)
* use 9-chars prefix in data platform example test

* use 9-chars prefix in data platform example test
2024-01-28 14:00:32 +01:00
Wiktor Niesiobędzki 526185fd1f
Remove default region for Cloud Function and Cloud Run (#2004)
Remove default region for Cloud Function and Cloud Run
2024-01-24 10:23:40 +00:00
Ludovico Magnocavallo 11d7edac64
Add example to FAST GKE stage, streamline GKE Hub module variables and usage (#1977)
* implement optionals in gke-hub module

* simplify gke hub module call in mc mesh blueprint

* simplify gke hub module call and variables in multitenant blueprint

* gke hub inventory

* provide cluster and fleet examples in stage
2024-01-20 10:06:38 +00:00
lcaggio 208902c8da
Fix Data platform foundation (#1992)
* FAST + Minimal DP

* Fix tests

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-01-20 08:49:46 +01:00
Aurélien Legrand eee6aaff2c
Network dashboard - fixing 2 bugs: overriden variable and page token … (#1976)
* Network dashboard - fixing 2 bugs: overriden variable and page token not handled properly

* changing pageToken in the regexp
2024-01-15 14:28:17 +01:00
Julio Diez e623fa4a66
Merge branch 'master' into jd/serverless-program 2024-01-05 21:49:12 +01:00
Julio Diez 8435ad85f3 Address some style comments 2024-01-05 21:48:02 +01:00
Wiktor Niesiobędzki 6c258e6562 Fix description 2023-12-29 12:09:16 +01:00
Wiktor Niesiobędzki 1eea077460 Add service account email to outputs to manage its permissions 2023-12-29 12:09:16 +01:00
Wiktor Niesiobędzki 724dfda09c Provider doesn't set defaults on config
Can't provide just one size (like `web_server` or `triggerrer`) because
of no defaults are taken:
module.composer.google_composer_environment.env: Modifying... [id=***]
╷
│ Error: googleapi: Error 400: Found 6 problems:
│       1) You have to specify Scheduler CPUs not lower than 0.5.
│       2) You have to specify number of schedulers larger than 0.
│       3) You have to specify Web Server CPUs not lower than 0.5.
│       4) You have to specify Worker CPUs not lower than 0.5.
│       5) You have to specify minimum number of workers larger than 0.
│       6) Triggerer memory must be between 1.00GB and 6.50GB for given vCpu

So provide the defaults as set workloads_config == null
2023-12-29 12:09:16 +01:00
Julio Diez 6bad85d758 Update related serverless READMEs 2023-12-29 11:14:16 +01:00
Julio Diez 3685eb8493 Fix tftest values 2023-12-28 20:30:10 +01:00
Julio Diez bae9845f4b Update README 2023-12-28 20:06:11 +01:00
Julio Diez 2e500c69a7 Avoid permadiff in launch_stage 2023-12-28 17:57:15 +01:00
Julio Diez 2ca24d320e Use of new module cloud-run-v2 2023-12-28 17:30:41 +01:00
Julio Diez 8889c18690
Merge branch 'master' into jd/serverless-program 2023-12-27 12:31:09 +01:00
Simone Ruffilli 87548f9739
Networking Sandbox Blueprint (#1939)
This blueprint creates a networking playground showing a number of different VPC connectivity options:

Hub and spoke via HA VPN
Hub and spoke via VPC peering
Interconnecting two networks via a network virtual appliance (aka NVA)
On top of that, this blueprint implements Policy Based Routing (aka PBR) to show how to force all traffic within a VPC to be funneled through an internal network passthrough load balancer, to implement an Intrusion Prevention System (IPS). PBR is enabled in the hub VPC, matching all traffic originating from within that VPC.
2023-12-21 17:50:38 +01:00
simonebruzzechesse 717f7ecad1
Blueprints naming convention update (#1942)
* update network-dashboard and quota-monitoring naming convention
2023-12-21 17:02:25 +01:00
Simone Ruffilli 0255c80e90
Move squid to __need_fixing (#1936)
* Moved `modules/cloud-config-container/squid` to __need_fixing
* Moved `blueprints/networking/filtering-proxy{,-psc}` to __need_fixing
2023-12-19 14:27:37 +00:00
Ludovico Magnocavallo 21562eefbe
don't fail quota fetch on deleted project (#1931) 2023-12-15 20:20:49 +01:00
simonebruzzechesse c50b732c79
Allow granting network user role on host project from project module and factory (#1930)
* Update shared vpc config for project factory and project module for more granular Shared VPC configuration

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2023-12-15 14:39:21 +01:00
maunope 1dc6965694
Update quota monitor blueprint to support project discovery (#1924)
* fist test

* dev complete

* update tf with permissions, enabled APIs and discovery root management

* updated readme

* moved projects discovery to a separate method

* reviewed Mauri's changes

* add missing lines from last change

* - fixed discovery page size to 100
- removed last_asset_page_reached var from discover_projects
- added cast to list for projects var in _main, to make the script work both using CLI and pub/sub

* fixed discovery_root default value to work when no value is passed

* fixed tfdoc

* fixed tftest resources #

---------

Co-authored-by: Ludo <ludomagno@google.com>
2023-12-12 19:17:01 +01:00
Ludovico Magnocavallo bba814c091
Custom role factories for organization and project modules (#1912)
* backport custom role factories

* backport from fast ci/cd branch

* indent

* tfdoc

* fix module tests
2023-12-11 14:16:39 +00:00
Wiktor Niesiobędzki 886734e1e9
Add trigger configuration for Composer (#1916) 2023-12-11 12:54:49 +01:00
Ludovico Magnocavallo f548b65b1c
Add support for subnet-level service network user grants to project module, improve docs (#1907)
* improve project factory example

* light refactor of project modules shared vpc internals and docs

* add support for subnet-level grants on host project
2023-12-07 09:07:48 +00:00
apichick 66bd9d5160 Added workstation-cluster module 2023-11-30 07:02:28 +01:00