Commit Graph

5209 Commits

Author SHA1 Message Date
simonebruzzechesse d11c380aec
Format python files in blueprints (#2079)
* format python files in blueprints
* update check on blueprints python code
* update python linter in CI workflow
2024-02-15 09:37:49 +01:00
simonebruzzechesse 946ae148f7
Add workforce_identity_federation in 0-bootstrap (#2077)
* add workforce_identity_federation in 0-bootstrap
* update tests
2024-02-15 00:10:24 +01:00
Ben Swenka 87a350db93
Bswenka/psc glb and armor 2 producers (#2071)
* Enhanced this blueprint to add a second producer, and modularized the producer.

* Fixed terraform formatting

* Updating README.md with tfdoc

* Fixed test case conditions & module variable passing
2024-02-14 16:40:51 +01:00
lcaggio aa9b1479b7
Data catalog Tag module (#2060)
* First commit

* Update README

* Add todo

* Fix required_version
2024-02-13 17:24:17 +01:00
Ludovico Magnocavallo 47c3d0cb64
Update organization.tf
fix typos
2024-02-13 08:50:05 +01:00
Wiktor Niesiobędzki 72183be254 Use less conflicting IP 2024-02-13 07:40:31 +01:00
Wiktor Niesiobędzki b902b1dab9 Fix non-empty plan after apply 2024-02-13 07:40:31 +01:00
Ludovico Magnocavallo 20c5bc8444
Update 0-domainless-iam.md 2024-02-12 14:39:21 +01:00
Ludovico Magnocavallo 5c3507aa72
Update 0-domainless-iam.md 2024-02-12 14:39:03 +01:00
Ludovico Magnocavallo 4e77193c6a
Update 0-domainless-iam.md 2024-02-12 14:38:35 +01:00
Ludovico Magnocavallo 71a64487d5
Extend FAST to support different principal types (#2064)
* add doc draft

* typos

* typo

* typo

* typos

* rewording

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* move iam variables to a separate file

* move billing-account module to iam_principals

* move data-catalog-policy-tag module to iam_principals

* move dataplex-datascan module to iam_principals

* move dataproc module to iam_principals

* move folder module to iam_principals

* copyright

* move organization module to iam_principals

* move project module to iam_principals

* move source-repository module to iam_principals

* update blueprints for iam_principals interface

* FAST bootstrap

* module READMEs fixes

* FAST bootstrap

* FAST networking stages

* FAST security stage

* FAST gke stage

* FAST multitenant bootstrap stage

* FAST multitenant resman stage

* tfdoc

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* fix module test

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Rename iam_principals to iam_by_principals

* Update IAM template to include iam_by_principals

* Update Resman README

* Fix ADR link format

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-12 14:35:30 +01:00
Wiktor Niesiobędzki 3397d4cd52
Fix imports of org policies (#2065) 2024-02-11 07:22:11 +01:00
Ben Swenka 89779c0d0e
Bswenka/psc glb and armor subnet fix (#2058)
* Updated ilb-subnetwork issue

* Fixed Terraform formatting issue

* Fixing merge issue

* Fixing merge issue in producer.tf

* Fixed linting issues

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-09 10:41:14 +00:00
Wiktor Niesiobędzki 597579fa2b
HA MySQL cluster deployment on GKE (#2061)
* MySQL pattern on GKE

* Use terraform managed password

* Use hardcoded network references

* Explain why Cloud NAT

* Rename versions_override.tf

* Fix subnet reference

* Fix password

* Fix MysQL connect commands

* Remove self-link

* Update README.md

* Add TOC and Variables table

* Fix outputs

* Fix linter

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-09 11:23:35 +01:00
lcaggio 50c7d3c0e9
Add Tags in project output. (#2062) 2024-02-09 10:42:18 +01:00
Julio Castillo c42c4c141f
GKE stateful blueprints (#2059)
* var definitions

* skeleton, untested

* fix errors, test with existing cluster

* test vpc creation, todo notes

* initial variables for AR and image

* initial variables for AR and image

* Add support for remote repositories to artifact-registry

* Add support for virtual repositories to artifact-registry

* Add support for extra config options to artifact-registry

* artifact registry module: add validation and precondition, fix tests

* ar module id/name

* registry

* service accoutn and roles

* fetch pods, remove image prefix

* small changes

* use additive IAM at project level

* use additive IAM at project level

* configmaps

* manifests

* fix statefulset manifest

* service manifest

* fix configmap mode

* add todo

* job (broken)

* job

* wait on manifest, endpoints datasource

* fix job

* Fix local

* sa

* Update README.md

* Restructure gke bp

* refactor tree and infra variables

* no create test

* simplify cluster SA

* test cluster and vpc creation

* project creation fixes

* use iam_members variable

* nits

* readme with examples

* readme with examples

* outputs

* variables, provider configuration

* variables, manifests

* start cluster job

* fix redis cluster creation

Co-authored-by: Julio Castillo <juliocc@users.noreply.github.com>

* Revert changes in autopilot cluster

* Default templates path, use namespace for node names

* Update readmes

* Fix IAM bindings

* Make STABLE the default release channel

* Use Cloud DNS as default DNS provider

* Allow optional Cloud NAT creation

* Allow backup agent and proxy only subnet

* Work around terraform not short-circuiting logical operators

* Rename create variables to be more consistent with other blueprints

* Add basic features

* Update variable names

* Initial kafka JS

* Move providers to a new file

* Kafka / Strimzi

* First possibily working version for MySQL (with a lot of todo's left)

* Explicitly use proxy repo + some other fixes

* Strimzi draft

* Refactor variables, use CluterIP as pointer for mysql-router for bootstraping

* Validate number of replicas, autoscale required number of running nodes to n/2+1

* Use seaprate service for bootstrap, do not recreate all resources on change of replicas count as the config is preserved in PV

* Test dual chart kafka

* Update chart for kafka

* Expose basic kafka configuration options

* Remove unused manifest

* Added batch blueprint

* Added README

* switch to kubectl_manifest

* Add README and support for static IP address

* Move namespace creation to helm

* Interpolate kafka variables

* Rename kafka-strimzi to kafka

* Added TUTORIAL for cloudshell for batch blueprint

* deleted tutorial

* Remove commented replace trigger

* Move to helm chart

* WIP of Cloud Shell tutorial for MySQL

* Rename folders

* Fix rename

* Update paths

* Unify styles

* Update paths

* Add Readme links

* Update mysql tutorial

* Fix path according to self-link

* Use relative path to cwd

* Fix service_account variable location

* Fix tfvars creation

* Restore some fixes for helm deployment

* Add cluster deletion_prevention

* Fixes for tutorial

* Update cluster docs

* Fixes to batch tutorial

* Bare bones readme for batch

* Update batch readme

* README fixes

* Fix README title for redis

* Fix Typos

* Make it easy to pass variables from autopilot-cluster to other modules

* Add connectivity test and bastion host

* updates to readme, and gpu fix

* Add versions.tf and README updates

* Fix typo

* Kafka and Redis README updates

* Update versions.tf

* Fixes

* Add boilerplate

* Fix linting

* Move mysql to separate branch

* Update cloud shell links

* Fix broken link

---------

Co-authored-by: Ludo <ludomagno@google.com>
Co-authored-by: Daniel Marzini <44803752+danielmarzini@users.noreply.github.com>
Co-authored-by: Wiktor Niesiobędzki <wiktorn@google.com>
Co-authored-by: Miren Esnaola <mirene@google.com>
2024-02-08 18:28:41 +00:00
Wiktor Niesiobędzki da11396e3a Postpone setting essential contacts until it is done through SA 2024-02-07 20:08:44 +01:00
Wiktor Niesiobędzki 29232c10c4 Fix failing tests after Terraform upgrade 2024-02-07 17:25:11 +01:00
Wiktor Niesiobędzki c58e61e98e Introduce variable to disable imports, fix tests 2024-02-07 17:25:11 +01:00
Wiktor Niesiobędzki 2c9eb5795b Bum terraform to version 1.7.0 2024-02-07 17:25:11 +01:00
Wiktor Niesiobędzki ec3f314c08 Import default policies 2024-02-07 17:25:11 +01:00
luigi-bitonti cb9a9ab373
Added destroy_scheduled_duration variable (#2053)
* Added destroy_scheduled_duration variable

* Fix doc

* Removed default value
2024-02-07 15:47:49 +00:00
Deepak Kumar 27e503a3ac
fix: auto_provisioning_defaults is not really optional (#2051)
Due to the disk_type validation for auto provision node pool,
this module always forced to create a GKE standard cluster
with a auto provisioned node pool. This is not desirable if
you manage pools separately like using the `gke-nodepool`.
2024-02-06 07:09:13 +01:00
Julio Castillo e219d92217
Enable additional recommended org policies (#2050)
* Enable additional recommended org policies

Fixes #2047
Fixes #2048
Fixes #2049

* Fix tests
2024-02-05 10:46:37 +01:00
Ludovico Magnocavallo 5448ab64c4
Leverage net-vpc module for DNS logging in FAST (#2041)
* revert #2023

* leverage net vpc module for dns logging in fast
2024-02-03 08:16:00 +01:00
Julio Castillo efb76b93e4
Remove unused tfeditor (#2039) 2024-02-02 10:14:14 +00:00
Julio Castillo 13636ba07b
Make Cloud NAT creation optional in FAST net stages. (#2038)
* Make Cloud NAT creation optional in FAST net stages.

Fixes #2021

* Update READMEs
2024-02-02 10:58:16 +01:00
Ludo 5ae2f6987d
update changelog 2024-02-01 16:24:03 +01:00
Ludovico Magnocavallo d127c25ad0
Shielded nodes and custom service account in FAST GKE stage and blueprint (CSPR-related) (#2036)
* default to shielded nodes in FAST gke stage

* use custom service account in GKE multitenant blueprint
2024-02-01 15:16:00 +00:00
Harald Haas dda0250c4c
Fix dnssec_config issue on state off (#2035) 2024-02-01 06:53:32 +00:00
Julio Castillo 4c68c016a9
Add DNS query logging to FAST net stages (#2033)
* Add DNS query logging to FAST net stages

Fixes #2020

* Update readmes

* Add variable to toggle DNS logging

* Extend DNS logging toggle to other net stages
2024-01-31 13:44:51 +01:00
Ludovico Magnocavallo 01c7f806ce
Selectively enable logging in FAST and firewall policy module rules (#2032)
* use logging in firewall policy module examples

* enable logging for selected hierarchical firewall rules
2024-01-31 09:50:35 +01:00
Ludo 1e06c35a1f
fix typos 2024-01-31 09:02:55 +01:00
Ludovico Magnocavallo c9db1fde20
clarify relationship with checklist groups (#2031) 2024-01-31 08:51:20 +01:00
Julio Castillo da95434308
logging for default ingress rules in FAST (#2030)
* Add default ingress deny rule with logging to FAST net stages.

Fixes #2024

* Allow firewall factory to omit rules key

* Fix tests

* Fix fast tests

* fix fast tests
2024-01-30 16:53:01 +00:00
shourya116 7b58114d65
Updated the DataQualitySpec for Dataplex Datascan (#2008)
* Updated the DataQualitySpec for Dataplex Datascan

* Fix linting

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
Co-authored-by: Julio Castillo <jccb@google.com>
2024-01-30 15:14:49 +00:00
lcaggio 37fc16ab42
Fix tests in README (#2027) 2024-01-30 11:04:47 +00:00
Julio Castillo cdf65300f0
Fix sourcerepo templates and concat call (#2019)
* Fix sourcerepo templates and concat call

Fixes #2018

* Fix iam

* Fix another sourcerepo template
2024-01-30 11:46:33 +01:00
Ludovico Magnocavallo 3b4b355800
Ensure data platform service accounts meet FAST requirements (#2016)
* use 9-chars prefix in data platform example test

* use 9-chars prefix in data platform example test
2024-01-28 14:00:32 +01:00
Ludovico Magnocavallo bf93b6fb4e
fix typo in logging sinks interface (#2015) 2024-01-28 10:27:28 +01:00
Ludo f69dfbe0c2
update changelog 2024-01-28 08:30:56 +01:00
lcaggio 19dc6090fc
Add Tag Template module (#2013)
* Tag policy module

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2024-01-27 12:30:21 +01:00
Ludovico Magnocavallo 99228363b2
enforce trusted image projects constraint in stage 0 (#2014) 2024-01-26 10:14:44 +00:00
Blake Corbitt 9c1afa6261
Issue #2011 - add support for target_resources in hierarchical policy for net-firewall-policy module. (#2012)
* Issue #2011 - Add support for target_resources in hierarchical firewall policy

* Removing errant character in factory.tf
2024-01-25 17:56:17 +00:00
Amela Spica 1a8400e60e
Fixes and additional support for ssl_mode for CloudSQL module (#2002)
* added ssl mode, ignore password change and maintenance fix

* validation - ssl mode can be null

* removed ignore password

* readme update

* added moved, shortened ssl_mode desc

* order fix

* fmt

* created ssl variable

* handle exceptions

* removed null from optional

* change ssl from null to empty object

* output ordered alphabetically

* readme update

* default value of require_ssl
2024-01-25 16:29:07 +01:00
Ludovico Magnocavallo 6d9b6403dd
add support for essential contacts to FAST (#2010) 2024-01-25 12:20:14 +01:00
Ludo 853a9c23b5
prepare changelog for release 2024-01-24 20:14:34 +01:00
Ludovico Magnocavallo c5416f3af1
Tighten up security of automation project (CSPR-related) (#2009)
* enforce compute/iam policies on the automation project

* tests
2024-01-24 18:40:36 +00:00
marcjwo 6b4dad01d6
Marcwo/dataform module (#2001)
* pre PR commit

* added IAM functionality

* clean up of readme

* versions.tf fix

* added separate iam.tf, facilitated existing secret manager module

* corrected optional variable defaults

* adjusted readme to new changes

* adjusted tftest line in readme for changed example

* reverted the module back to a state where it only manages one instance

* minor fix for main readme.md

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-01-24 17:13:21 +01:00
Miro Michalicka 15439c3f5b
Fix named ranges behaviour if cidr_tpl_file variable not provided. (#2005)
* Fix named ranges behaviour if cidr_tpl_file variable not provided.

* Fix and extend tests.

* fix map syntax

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2024-01-24 11:18:11 +00:00