Commit Graph

5194 Commits

Author SHA1 Message Date
Julio Castillo c42c4c141f
GKE stateful blueprints (#2059)
* var definitions

* skeleton, untested

* fix errors, test with existing cluster

* test vpc creation, todo notes

* initial variables for AR and image

* initial variables for AR and image

* Add support for remote repositories to artifact-registry

* Add support for virtual repositories to artifact-registry

* Add support for extra config options to artifact-registry

* artifact registry module: add validation and precondition, fix tests

* ar module id/name

* registry

* service accoutn and roles

* fetch pods, remove image prefix

* small changes

* use additive IAM at project level

* use additive IAM at project level

* configmaps

* manifests

* fix statefulset manifest

* service manifest

* fix configmap mode

* add todo

* job (broken)

* job

* wait on manifest, endpoints datasource

* fix job

* Fix local

* sa

* Update README.md

* Restructure gke bp

* refactor tree and infra variables

* no create test

* simplify cluster SA

* test cluster and vpc creation

* project creation fixes

* use iam_members variable

* nits

* readme with examples

* readme with examples

* outputs

* variables, provider configuration

* variables, manifests

* start cluster job

* fix redis cluster creation

Co-authored-by: Julio Castillo <juliocc@users.noreply.github.com>

* Revert changes in autopilot cluster

* Default templates path, use namespace for node names

* Update readmes

* Fix IAM bindings

* Make STABLE the default release channel

* Use Cloud DNS as default DNS provider

* Allow optional Cloud NAT creation

* Allow backup agent and proxy only subnet

* Work around terraform not short-circuiting logical operators

* Rename create variables to be more consistent with other blueprints

* Add basic features

* Update variable names

* Initial kafka JS

* Move providers to a new file

* Kafka / Strimzi

* First possibily working version for MySQL (with a lot of todo's left)

* Explicitly use proxy repo + some other fixes

* Strimzi draft

* Refactor variables, use CluterIP as pointer for mysql-router for bootstraping

* Validate number of replicas, autoscale required number of running nodes to n/2+1

* Use seaprate service for bootstrap, do not recreate all resources on change of replicas count as the config is preserved in PV

* Test dual chart kafka

* Update chart for kafka

* Expose basic kafka configuration options

* Remove unused manifest

* Added batch blueprint

* Added README

* switch to kubectl_manifest

* Add README and support for static IP address

* Move namespace creation to helm

* Interpolate kafka variables

* Rename kafka-strimzi to kafka

* Added TUTORIAL for cloudshell for batch blueprint

* deleted tutorial

* Remove commented replace trigger

* Move to helm chart

* WIP of Cloud Shell tutorial for MySQL

* Rename folders

* Fix rename

* Update paths

* Unify styles

* Update paths

* Add Readme links

* Update mysql tutorial

* Fix path according to self-link

* Use relative path to cwd

* Fix service_account variable location

* Fix tfvars creation

* Restore some fixes for helm deployment

* Add cluster deletion_prevention

* Fixes for tutorial

* Update cluster docs

* Fixes to batch tutorial

* Bare bones readme for batch

* Update batch readme

* README fixes

* Fix README title for redis

* Fix Typos

* Make it easy to pass variables from autopilot-cluster to other modules

* Add connectivity test and bastion host

* updates to readme, and gpu fix

* Add versions.tf and README updates

* Fix typo

* Kafka and Redis README updates

* Update versions.tf

* Fixes

* Add boilerplate

* Fix linting

* Move mysql to separate branch

* Update cloud shell links

* Fix broken link

---------

Co-authored-by: Ludo <ludomagno@google.com>
Co-authored-by: Daniel Marzini <44803752+danielmarzini@users.noreply.github.com>
Co-authored-by: Wiktor Niesiobędzki <wiktorn@google.com>
Co-authored-by: Miren Esnaola <mirene@google.com>
2024-02-08 18:28:41 +00:00
Wiktor Niesiobędzki da11396e3a Postpone setting essential contacts until it is done through SA 2024-02-07 20:08:44 +01:00
Wiktor Niesiobędzki 29232c10c4 Fix failing tests after Terraform upgrade 2024-02-07 17:25:11 +01:00
Wiktor Niesiobędzki c58e61e98e Introduce variable to disable imports, fix tests 2024-02-07 17:25:11 +01:00
Wiktor Niesiobędzki 2c9eb5795b Bum terraform to version 1.7.0 2024-02-07 17:25:11 +01:00
Wiktor Niesiobędzki ec3f314c08 Import default policies 2024-02-07 17:25:11 +01:00
luigi-bitonti cb9a9ab373
Added destroy_scheduled_duration variable (#2053)
* Added destroy_scheduled_duration variable

* Fix doc

* Removed default value
2024-02-07 15:47:49 +00:00
Deepak Kumar 27e503a3ac
fix: auto_provisioning_defaults is not really optional (#2051)
Due to the disk_type validation for auto provision node pool,
this module always forced to create a GKE standard cluster
with a auto provisioned node pool. This is not desirable if
you manage pools separately like using the `gke-nodepool`.
2024-02-06 07:09:13 +01:00
Julio Castillo e219d92217
Enable additional recommended org policies (#2050)
* Enable additional recommended org policies

Fixes #2047
Fixes #2048
Fixes #2049

* Fix tests
2024-02-05 10:46:37 +01:00
Ludovico Magnocavallo 5448ab64c4
Leverage net-vpc module for DNS logging in FAST (#2041)
* revert #2023

* leverage net vpc module for dns logging in fast
2024-02-03 08:16:00 +01:00
Julio Castillo efb76b93e4
Remove unused tfeditor (#2039) 2024-02-02 10:14:14 +00:00
Julio Castillo 13636ba07b
Make Cloud NAT creation optional in FAST net stages. (#2038)
* Make Cloud NAT creation optional in FAST net stages.

Fixes #2021

* Update READMEs
2024-02-02 10:58:16 +01:00
Ludo 5ae2f6987d
update changelog 2024-02-01 16:24:03 +01:00
Ludovico Magnocavallo d127c25ad0
Shielded nodes and custom service account in FAST GKE stage and blueprint (CSPR-related) (#2036)
* default to shielded nodes in FAST gke stage

* use custom service account in GKE multitenant blueprint
2024-02-01 15:16:00 +00:00
Harald Haas dda0250c4c
Fix dnssec_config issue on state off (#2035) 2024-02-01 06:53:32 +00:00
Julio Castillo 4c68c016a9
Add DNS query logging to FAST net stages (#2033)
* Add DNS query logging to FAST net stages

Fixes #2020

* Update readmes

* Add variable to toggle DNS logging

* Extend DNS logging toggle to other net stages
2024-01-31 13:44:51 +01:00
Ludovico Magnocavallo 01c7f806ce
Selectively enable logging in FAST and firewall policy module rules (#2032)
* use logging in firewall policy module examples

* enable logging for selected hierarchical firewall rules
2024-01-31 09:50:35 +01:00
Ludo 1e06c35a1f
fix typos 2024-01-31 09:02:55 +01:00
Ludovico Magnocavallo c9db1fde20
clarify relationship with checklist groups (#2031) 2024-01-31 08:51:20 +01:00
Julio Castillo da95434308
logging for default ingress rules in FAST (#2030)
* Add default ingress deny rule with logging to FAST net stages.

Fixes #2024

* Allow firewall factory to omit rules key

* Fix tests

* Fix fast tests

* fix fast tests
2024-01-30 16:53:01 +00:00
shourya116 7b58114d65
Updated the DataQualitySpec for Dataplex Datascan (#2008)
* Updated the DataQualitySpec for Dataplex Datascan

* Fix linting

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
Co-authored-by: Julio Castillo <jccb@google.com>
2024-01-30 15:14:49 +00:00
lcaggio 37fc16ab42
Fix tests in README (#2027) 2024-01-30 11:04:47 +00:00
Julio Castillo cdf65300f0
Fix sourcerepo templates and concat call (#2019)
* Fix sourcerepo templates and concat call

Fixes #2018

* Fix iam

* Fix another sourcerepo template
2024-01-30 11:46:33 +01:00
Ludovico Magnocavallo 3b4b355800
Ensure data platform service accounts meet FAST requirements (#2016)
* use 9-chars prefix in data platform example test

* use 9-chars prefix in data platform example test
2024-01-28 14:00:32 +01:00
Ludovico Magnocavallo bf93b6fb4e
fix typo in logging sinks interface (#2015) 2024-01-28 10:27:28 +01:00
Ludo f69dfbe0c2
update changelog 2024-01-28 08:30:56 +01:00
lcaggio 19dc6090fc
Add Tag Template module (#2013)
* Tag policy module

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2024-01-27 12:30:21 +01:00
Ludovico Magnocavallo 99228363b2
enforce trusted image projects constraint in stage 0 (#2014) 2024-01-26 10:14:44 +00:00
Blake Corbitt 9c1afa6261
Issue #2011 - add support for target_resources in hierarchical policy for net-firewall-policy module. (#2012)
* Issue #2011 - Add support for target_resources in hierarchical firewall policy

* Removing errant character in factory.tf
2024-01-25 17:56:17 +00:00
Amela Spica 1a8400e60e
Fixes and additional support for ssl_mode for CloudSQL module (#2002)
* added ssl mode, ignore password change and maintenance fix

* validation - ssl mode can be null

* removed ignore password

* readme update

* added moved, shortened ssl_mode desc

* order fix

* fmt

* created ssl variable

* handle exceptions

* removed null from optional

* change ssl from null to empty object

* output ordered alphabetically

* readme update

* default value of require_ssl
2024-01-25 16:29:07 +01:00
Ludovico Magnocavallo 6d9b6403dd
add support for essential contacts to FAST (#2010) 2024-01-25 12:20:14 +01:00
Ludo 853a9c23b5
prepare changelog for release 2024-01-24 20:14:34 +01:00
Ludovico Magnocavallo c5416f3af1
Tighten up security of automation project (CSPR-related) (#2009)
* enforce compute/iam policies on the automation project

* tests
2024-01-24 18:40:36 +00:00
marcjwo 6b4dad01d6
Marcwo/dataform module (#2001)
* pre PR commit

* added IAM functionality

* clean up of readme

* versions.tf fix

* added separate iam.tf, facilitated existing secret manager module

* corrected optional variable defaults

* adjusted readme to new changes

* adjusted tftest line in readme for changed example

* reverted the module back to a state where it only manages one instance

* minor fix for main readme.md

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-01-24 17:13:21 +01:00
Miro Michalicka 15439c3f5b
Fix named ranges behaviour if cidr_tpl_file variable not provided. (#2005)
* Fix named ranges behaviour if cidr_tpl_file variable not provided.

* Fix and extend tests.

* fix map syntax

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2024-01-24 11:18:11 +00:00
Wiktor Niesiobędzki 526185fd1f
Remove default region for Cloud Function and Cloud Run (#2004)
Remove default region for Cloud Function and Cloud Run
2024-01-24 10:23:40 +00:00
Wiktor Niesiobędzki 277777d1c7
Fix DNS E2E test + add one to net-lb-app-int-cross-region (#1993)
* Fix DNS E2E test + add one to net-lb-app-int-cross-region

* Update README.md

* Fix inventory for tests

* Fix tests

* Fix number of resources

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-01-23 16:34:45 +01:00
Ludovico Magnocavallo 070584ae74
Checklist attribution bucket (#2000) 2024-01-23 11:32:14 +00:00
luigi-bitonti 0ef7ee4670
Added beta apis feature (#1999) 2024-01-23 11:09:22 +00:00
Ludovico Magnocavallo 4b911a6047
update checklist parsing for top-level key (#1997) 2024-01-23 07:34:03 +01:00
Ludovico Magnocavallo 37dc48bca4
fix factory type for firewall rule ports (#1996) 2024-01-21 12:38:24 +01:00
Ludo 0276a9d365
update changelog 2024-01-21 10:07:29 +01:00
dibaskar-google 4ed738688a
dns reponse policy e2e changes (#1994)
dns reponse policy e2e changes
2024-01-20 19:47:02 +01:00
Ludovico Magnocavallo 11d7edac64
Add example to FAST GKE stage, streamline GKE Hub module variables and usage (#1977)
* implement optionals in gke-hub module

* simplify gke hub module call in mc mesh blueprint

* simplify gke hub module call and variables in multitenant blueprint

* gke hub inventory

* provide cluster and fleet examples in stage
2024-01-20 10:06:38 +00:00
Deepak Kumar d62012cebc
Specify `docker_repository` field for google_cloudfunctions2_function (#1987)
* fix: allow configuring `docker_repository` for cloud-function2

When docker repository is not specified by default docker repository
`projects/PROJECT_ID/locations/REGION/repositories/gcf-artifacts` is used.
In such a case, terraform plan always generates a difference for `docker_repository`
field as the module passes null value but the tfstate file has the above specified
default value. This fix allows one to prevent unnecessary infrastructure change when
using the default repository as well as any user created repository.

* doc: updated README for cloud-function-v2

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2024-01-20 09:40:26 +00:00
lcaggio 208902c8da
Fix Data platform foundation (#1992)
* FAST + Minimal DP

* Fix tests

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-01-20 08:49:46 +01:00
apichick 1ba8298b0e
Merge pull request #1990 from apichick/dns-geo-health-checked-targets
Fixed README and test for dns module
2024-01-19 10:12:20 +01:00
Wiktor Niesiobędzki 09176feaec
Fix typo in example 2024-01-18 19:46:53 +01:00
apichick 228a4a82c3 Fixed README and test for dns module 2024-01-18 19:04:03 +01:00
apichick 4a12a33e3c
Merge pull request #1988 from apichick/dns-geo-health-checked-targets
Added health checked targets for geo routing policy in dns module
2024-01-18 18:46:45 +01:00