Commit Graph

17 Commits

Author SHA1 Message Date
Julio Castillo dfc5023e0b
Make deletion protection consistent across all modules (#1735)
* Expose deletion_protection in GKE modules

* Make deletion protection consistent across all modules

* Add deletion_protection option to blueprints

* Fix blueprints tests

* Fix types

* Update READMEs

* Fix dp readme

* Fix cmek blueprint default deletion_protection

* Fix blueprints tests
2023-10-05 17:31:07 +02:00
lcaggio 6889f02954
Fix data platform roles (#1725)
* Fix Data Platform roles

* Fix README

* Fix blueprint tests

* Update cleanup dp steps

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2023-10-04 07:31:40 +02:00
Ludovico Magnocavallo 819894d2ba
IAM interface refactor (#1595)
* IAM modules refactor proposal

* policy

* subheading

* Update 20230816-iam-refactor.md

* log Julio's +1

* data-catalog-policy-tag

* dataproc

* dataproc

* folder

* folder

* folder

* folder

* project

* better filtering in test examples

* project

* folder

* folder

* organization

* fix variable descriptions

* kms

* net-vpc

* dataplex-datascan

* modules/iam-service-account

* modules/source-repository/

* blueprints/cloud-operations/vm-migration/

* blueprints/third-party-solutions/wordpress

* dataplex-datascan

* blueprints/cloud-operations/workload-identity-federation

* blueprints/data-solutions/cloudsql-multiregion/

* blueprints/data-solutions/composer-2

* Update 20230816-iam-refactor.md

* Update 20230816-iam-refactor.md

* capture discussion in architectural doc

* update variable names and refactor proposal

* project

* blueprints first round

* folder

* organization

* data-catalog-policy-tag

* re-enable folder inventory

* project module style fix

* dataproc

* source-repository

* source-repository tests

* dataplex-datascan

* dataplex-datascan tests

* net-vpc

* net-vpc test examples

* iam-service-account

* iam-service-account test examples

* kms

* boilerplate

* tfdoc

* fix module tests

* more blueprint fixes

* fix typo in data blueprints

* incomplete refactor of data platform foundations

* tfdoc

* data platform foundation

* refactor data platform foundation iam locals

* remove redundant example test

* shielded folder fix

* fix typo

* project factory

* project factory outputs

* tfdoc

* test workflow: less verbose tests, fix tf version

* re-enable -vv, shorter traceback, fix action version

* ignore github extension warning, re-enable action version

* fast bootstrap IAM, untested

* bootstrap stage IAM fixes

* stage 0 tests

* fast stage 1

* tenant stage 1

* minor changes to fast stage 0 and 1

* fast security stage

* fast mt stage 0

* fast mt stage 0

* fast pf
2023-08-20 09:44:20 +02:00
Ludovico Magnocavallo 5a534edf5c
update and refactor artifact registry module (#1536) 2023-07-28 09:54:36 +00:00
Ludovico Magnocavallo 75cc2f3d7a
FAST: shorten stage 3 prefixes, enforce prefix length in stage 3s (#1346)
* shorten stage 3 prefixes, enforce prefix length in stage 3s

* tfdoc

* tfdoc
2023-05-03 07:39:41 +02:00
lcaggio bb2abb8822 Fix data-eng role on orchestration project. 2023-04-06 13:41:29 +02:00
lcaggio 2108b4650d Fix Tests, rely on iam additive. 2023-02-22 01:36:01 +01:00
lcaggio c8f25512eb Fix variabler to support existing projects. 2023-02-20 00:39:37 +01:00
lcaggio ad6667a8fa First commit 2023-02-19 22:37:32 +01:00
Ayman Farhat 02d8d8367a
[Feature] Update data platform blue print with Dataflow Flex template (#1105)
* Add initial dataflow template code + TF infra

* Refactor the datapipeline DAG to use flex template operator, cleanup code

* Remove unneeded bash scripts, update README with manual examples

* Refactor datapipeline_dc_tags.py and include new Flex template

* Update docs to reflect changes

* Remove sub-dependencies and keep apache beam

* Add missing license headers and update tests

* Set resouces to 291 in tests

* Update outputs via tfdoc

* Update with outputs order and tfdoc

* Correct number of resources

* Fix to add region into command from var

* Enable service account impersonation for running builds

* Update example dataflow run command to use orchestrator SA

* Remove hard coded values in example

* Keep original airflow files, add new which use Flex template as example

* Update tests and doc

* Fix number of resources in plan

* Run tfdoc remove files section in README

* Fix number of modules in tfdoc

* Update number of resources

* Add missin service account

* Update DF demo README

* Quick rename

---------

Co-authored-by: lcaggio <lorenzo.caggioni@gmail.com>
Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2023-02-06 07:35:40 +01:00
lcaggio 488b6d4dc0 Remove plg project, composer 2 2023-01-11 00:31:52 +01:00
Ludovico Magnocavallo fae5654e33
Refactor VPC firewall module for Terraform 1.3 (#949)
* module and module tests/examples

* align blueprints and fast

* fix null ranges

* make ports optional

* tfdoc

* make rules optional defaulting to all protocols

* review comments

* last round of comments

* invert precedence of template variables

* add option to disable all default rules

* add option to disable all default rules

* split egress/ingress

* tests

* fix tests
2022-11-04 13:56:07 +01:00
Ludovico Magnocavallo 8c51a2a386
Enable org policy service and add README notice to modules (#936)
* enable org policy service and add README notice to modules

* fix tests
2022-11-01 14:25:07 +01:00
Julio Castillo ad420138ad Fix readme examples 2022-10-28 17:49:44 +02:00
Ludovico Magnocavallo 4fa1dc431d
Refactor net-vpc module for Terraform 1.3 (#880)
* module tests pass

* doc examples

* refactor blueprints

* fast stages

* fix comment typo

* fix module factory test
2022-10-14 11:02:33 +02:00
Julio Castillo 894647ff9e Leverage new shared VPC project config defaults across the repo 2022-10-07 08:55:47 +02:00
Julio Castillo a248602827 Rename examples folder to blueprints 2022-09-09 16:38:43 +02:00