Commit Graph

191 Commits

Author SHA1 Message Date
lcaggio dc034d74f7 Variables. 2023-03-05 22:24:16 +01:00
lcaggio 9e19f89608 Implement PR comments. 2023-03-05 22:02:41 +01:00
lcaggio 2b8ba16a9a Fix typos 2023-03-04 14:32:54 +01:00
lcaggio 652495e530 Update versions. 2023-03-04 14:12:50 +01:00
lcaggio f8a7aa865a Fix test. 2023-03-04 08:25:29 +01:00
lcaggio 8d70e1d900
Merge branch 'master' into lcaggio/bqml 2023-03-04 08:22:22 +01:00
lcaggio ccd68b2fa6 Fix linting. 2023-03-04 08:19:47 +01:00
lcaggio 0d4b599e99 Fix README 2023-03-04 08:13:53 +01:00
lcaggio 98e17bb997 Fix readme. 2023-03-04 08:09:29 +01:00
Giorgio Conte 6526dda8c7 sql linting 2023-03-03 14:52:35 +00:00
lcaggio 594a615e1e Update 2023-03-03 15:08:57 +01:00
lcaggio 32808f93ea Update README. 2023-03-03 14:52:33 +01:00
Dan Farmer 52eb83758f Fix compute-vm:CloudKMS test for provider>=4.54.0
* TF provider >= 4.54.0 now returns `rsa_encrypted_key` for
  `google_compute_disk.disks["attached-disk"]` (see
  hashicorp/terraform-provider-google#4448)
* Add this field to expected model to fix test assertion failure
* Update required TF provider to 4.55.0 (latest) since the assertion
  will now fail with <4.54.0, which do not return `rsa_encrypted_key`
  * Updated the whole repo on advice from @ludoo
2023-02-28 15:10:22 +00:00
Giorgio Conte 17b8a461f0 fixed notebook with dynamic model name
cleared output from cells
added creation of view instead of table
2023-02-27 15:28:49 +00:00
Giorgio Conte 3271acd2f2 Added sql and jupyter notebook to run the demo 2023-02-27 10:56:47 +00:00
Julio Castillo be60753246 Fix failing tests 2023-02-25 19:26:11 +01:00
Julio Castillo edce6edd28 Update factories and apigee tests 2023-02-25 19:26:11 +01:00
Julio Castillo b186365cab Update data solutions tests 2023-02-25 19:26:11 +01:00
Julio Castillo 99e69e319f Update data platform tests 2023-02-25 19:26:11 +01:00
Giorgio Conte a51c682005 Updated tf file to add the following features:
- default location of dataset to US
- changed name of vertex metastore to "default"
- add ai user and service account us to notebook SA
- add ai user to vertex sa
2023-02-24 13:27:44 +00:00
lcaggio 50856e6951 First commit 2023-02-23 18:36:03 +01:00
lcaggio 47855cb682
Merge branch 'master' into lcaggio/dp-projectcreate 2023-02-23 11:54:48 +01:00
lcaggio b279c083a0 Fix README and IAM files 2023-02-23 11:54:16 +01:00
lcaggio 2108b4650d Fix Tests, rely on iam additive. 2023-02-22 01:36:01 +01:00
Julio Castillo a5e905cb80 Update remaining org policies 2023-02-21 15:49:16 +01:00
Julio Castillo d3bcf625f9 Update yaml org policies 2023-02-21 15:49:16 +01:00
lcaggio eda9597f0b Fix linting 2023-02-20 01:03:38 +01:00
lcaggio c523dce3a6 Fix 2023-02-20 01:00:57 +01:00
lcaggio f4c1fa6c20 Fix tests. 2023-02-20 00:56:32 +01:00
lcaggio db6a4f9ac7 Remove variables 2023-02-20 00:45:40 +01:00
lcaggio c8f25512eb Fix variabler to support existing projects. 2023-02-20 00:39:37 +01:00
lcaggio ad6667a8fa First commit 2023-02-19 22:37:32 +01:00
Ayman Farhat eac2065ae2 Update on docs wording 2023-02-16 17:04:05 +01:00
Ayman Farhat fcdadf521d Fix spelling mistake 2023-02-16 15:48:12 +01:00
Ayman Farhat a853dc4fe2 update Data Platform blueprint README with more example Dataflow commands 2023-02-16 15:45:20 +01:00
lcaggio ebc4bc51a5 Workaround to mitigate issue 9164 2023-02-13 15:25:24 +01:00
Julio Castillo 08f2540872 Bump provider version 2023-02-07 15:59:29 +01:00
lcaggio d78815d3d7 Fix README 2023-02-07 13:57:26 +01:00
lcaggio 71469679e3 Add variables validation 2023-02-07 13:53:54 +01:00
lcaggio dd77c2fe79
Merge branch 'master' into lcaggio/shieldedfolder 2023-02-07 07:58:00 +01:00
lcaggio 307cf32cf9 Fix 2023-02-06 21:34:29 +01:00
lcaggio d68bb4b320 Fixes 2023-02-06 21:33:04 +01:00
lcaggio 1e4499c8ab Fixes 2023-02-06 14:58:40 +01:00
Ayman Farhat 02d8d8367a
[Feature] Update data platform blue print with Dataflow Flex template (#1105)
* Add initial dataflow template code + TF infra

* Refactor the datapipeline DAG to use flex template operator, cleanup code

* Remove unneeded bash scripts, update README with manual examples

* Refactor datapipeline_dc_tags.py and include new Flex template

* Update docs to reflect changes

* Remove sub-dependencies and keep apache beam

* Add missing license headers and update tests

* Set resouces to 291 in tests

* Update outputs via tfdoc

* Update with outputs order and tfdoc

* Correct number of resources

* Fix to add region into command from var

* Enable service account impersonation for running builds

* Update example dataflow run command to use orchestrator SA

* Remove hard coded values in example

* Keep original airflow files, add new which use Flex template as example

* Update tests and doc

* Fix number of resources in plan

* Run tfdoc remove files section in README

* Fix number of modules in tfdoc

* Update number of resources

* Add missin service account

* Update DF demo README

* Quick rename

---------

Co-authored-by: lcaggio <lorenzo.caggioni@gmail.com>
Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2023-02-06 07:35:40 +01:00
lcaggio 39ffdcf184
Merge branch 'master' into lcaggio/shieldedfolder 2023-02-05 22:45:14 +01:00
lcaggio f49e5ccec7 Fix readmes and variables. 2023-02-05 22:44:31 +01:00
lcaggio 47c75f8862 Implement project_config variable 2023-02-05 21:12:46 +01:00
lcaggio 38fc9e8b9b First commit 2023-02-05 07:52:33 +01:00
lcaggio ce1cc5f75a
Merge branch 'master' into lcaggio/shieldedfolder 2023-02-04 19:42:19 +01:00
Ludovico Magnocavallo 5453c585e0
FAST multitenant bootstrap and resource management, rename org-level FAST stages (#1052)
* rename stages

* remove support for external org billing, rename output files

* resman: make groups optional, align on new billing account variable

* bootstrap: multitenant outputs

* tenant bootstrap stage, untested

* fix folder name

* fix stage 0 output names

* optional creation for tag keys in organization module

* single tenant bootstrap minus tag

* rename output files, add tenant tag key

* fix organization module tag values output

* test skipping creation for tags in organization module

* single tenant bootstrap plan working

* multitenant bootstrap

* tfdoc

* fix check links error messages

* fix links

* tfdoc

* fix links

* rename fast tests, fix bootstrap tests

* multitenant stages have their own folder, simplify stage numbering

* stage renumbering

* wip

* rename tests

* exclude fast providers in fixture

* stage 0 tests

* stage 1 tests

* network stages tests

* stage tests

* tfdoc

* fix links

* tfdoc

* multitenant tests

* remove local files

* stage links command

* fix links script, TODO

* wip

* wip single tenant bootstrap

* working tenant bootstrap

* update gitignore

* remove local files

* tfdoc

* remove local files

* allow tests for tenant bootstrap stage

* tenant bootstrap proxies stage 1 tfvars

* stage 2 and 3 service accounts and IAM in tenant bootstrap

* wip

* wip

* wip

* drop multitenant bootstrap

* tfdoc

* add missing stage 2 SAs, fix org-level IAM condition

* wip

* wip

* optional tag value creation in organization module

* stage 1 working

* linting

* linting

* READMEs

* wip

* Make stage-links script work in old macos bash

* stage links command help

* fix output file names

* diagrams

* fix svg

* stage 0 skeleton and diagram

* test svg

* test svg

* test diagram

* diagram

* readme

* fix stage links script

* stage 0 readme

* README changes

* stage readmes

* fix outputs order

* fix link

* fix tests

* stage 1 test

* skip stage example

* boilerplate

* fix tftest skip

* default bootstrap stage log sinks to log buckets

* add logging to tenant bootstrap

* move iam variables out of tenant config

* fix cicd, reintroduce missing variable

* use optional in stage 1 cicd variable

* rename extras stage

* rename and move identity providers local, use optional for cicd variable

* tfdoc

* add support for wif pool and providers, ci/cd

* tfdoc

* fix links

* better handling of modules repository

* add missing role on logging project

* fix cicd pools in locals, test cicd

* fix workflow extension

* fix module source replacement

* allow tenant bootstrap cicd sa to impersonate resman sa

* tenant workflow templates fix for no providers file

* fix output files, push github workflow template to new repository

* remove try from outpout files

* align stage 1 cicd internals to stage 0

* tfdoc

* tests

* fix tests

* tests

* improve variable descriptions

* use optional in fast features

* actually create tenant log sinks, and allow the resman sa to do it

* test

* tests

* aaaand tests again

* fast features tenant override

* fast features tenant override

* fix wording

* add missing comment

* configure pf service accounts

* add missing comment

* tfdoc

* tests

* IAM docs

* update copyright

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2023-02-04 15:00:45 +01:00
lcaggio 45a74f37f9 Mention policies in the costomization section. 2023-02-04 06:24:35 +01:00
lcaggio f00d4d3edd
Merge branch 'master' into lcaggio/shieldedfolder 2023-02-03 22:17:30 +01:00
javiergp ce1f86d20b
Vertex Pipelines MLOps framework blueprint (#1038)
* First release of the MLOps blueprint
2023-02-02 19:13:13 +01:00
lcaggio 30f2b49c9c README Typos 2023-02-01 17:34:31 +01:00
lcaggio e3daaf2503 Update READMEs 2023-02-01 13:02:07 +01:00
lcaggio f9e3eb7d5a Update copyright 2023-02-01 09:30:28 +01:00
lcaggio 840fc86b3e Fix groups. Fix tests. 2023-02-01 08:55:33 +01:00
lcaggio 053d4151de Chane feature name, improve README 2023-02-01 00:50:22 +01:00
lcaggio 3995ec1550
Merge branch 'master' into lcaggio/shieldedfolder 2023-02-01 00:28:01 +01:00
lcaggio f36b004664 Make VPCSC optional; ingress policy 2023-02-01 00:26:57 +01:00
lcaggio fb04e78829 Fix variables 2023-01-31 22:35:48 +01:00
lcaggio 0e5f2f71a2 Fix Linting 2023-01-30 23:57:20 +01:00
lcaggio a166c612d5 Merge branch 'master' into lcaggio/shieldedfolder 2023-01-30 23:52:46 +01:00
lcaggio 6a39d80487 Fix README 2023-01-30 23:49:47 +01:00
lcaggio 66a10d76c3 Add test 2023-01-30 23:47:33 +01:00
lcaggio a808ea3293 Draft README 2023-01-30 23:32:13 +01:00
Wiktor Niesiobędzki 1a657b31d3 Bump beta provider to 4.48
This is the first version that supports `gateway_api_config` block
2023-01-29 15:50:24 +01:00
Ayman Farhat edd3a82453
Include cloudbuild API in project module (#1116)
* Include cloudbuild API in project module

* Increase number of resources
2023-01-27 21:38:01 +01:00
lcaggio 9a46f06dd0 update readme 2023-01-27 10:05:19 +01:00
lcaggio 1189e38788 Fix 2023-01-25 18:30:21 +01:00
lcaggio 0ce110f170 Make KMS and Log sink optionals 2023-01-25 18:22:43 +01:00
lcaggio 440e9c59b9 Update 2023-01-25 12:32:10 +01:00
lcaggio 4007d42705 Add KMS and Log. 2023-01-21 01:08:51 +01:00
lcaggio dcbfdd9c91 Merge branch 'master' into lcaggio/shieldedfolder 2023-01-19 10:59:33 +01:00
lcaggio 3c88aace1a Fix dataflow shared-vpc support 2023-01-19 01:04:11 +01:00
lcaggio 6477d0790d Remove temp file. 2023-01-19 00:39:55 +01:00
lcaggio 07a7be29e3 Add shared-vpc support on Data Playgroud blueprint 2023-01-19 00:33:31 +01:00
lcaggio b9a4c27a03 VPC-SC 2023-01-18 15:50:49 +01:00
lcaggio 191e56fa80 Merge branch 'master' into lcaggio/shieldedfolder 2023-01-17 15:51:55 +01:00
lcaggio 84be665172 First commit. 2023-01-17 08:49:04 +01:00
lcaggio 02938bdfa1 Fix Table in readme. 2023-01-16 15:58:11 +01:00
lcaggio 1672cfe9c1 Fix README. 2023-01-12 12:45:39 +01:00
lcaggio 50a52ae01a Implement PR comments. 2023-01-12 12:41:00 +01:00
lcaggio 07a975f84e Fix example test. 2023-01-11 12:37:08 +01:00
lcaggio 72abd47520 Fix documentation. 2023-01-11 08:47:42 +01:00
lcaggio c172c7bb2e Remove plg project and update to composer 2 2023-01-11 00:31:58 +01:00
lcaggio 488b6d4dc0 Remove plg project, composer 2 2023-01-11 00:31:52 +01:00
Ludovico Magnocavallo 66a402083b
Allow setting no ranges in firewall module custom rules (#1073)
* allow setting no ranges in custom firewall rules

* fix blueprint

* fix example

* fix example
2022-12-23 09:03:31 +01:00
Israel Herraiz 7df527e1ef Bump versions EVERYWHERE 2022-12-23 00:06:42 +01:00
lcaggio e56762b4d3
Merge branch 'master' into lcaggio/fix-20221123 2022-11-25 14:54:47 +01:00
Julio Castillo 612d7283e8 Enforce nonempty descriptions ending in a colon 2022-11-24 18:56:01 +01:00
lcaggio d1cc0519d8
Merge branch 'master' into lcaggio/fix-20221123 2022-11-24 16:07:33 +01:00
lcaggio 6b01b8641c Fix encryption. 2022-11-24 15:25:29 +01:00
Sebastian Kunze e4fc47a063
Normalize prefix handling in blueprints (#1003) 2022-11-23 11:09:00 +01:00
Julio Castillo 826ff14ef9 Sort variables and outputs 2022-11-21 13:17:55 +01:00
Julio Castillo 7be3288cc7 Add tests for factory examples 2022-11-18 17:48:52 +01:00
Ludovico Magnocavallo fae5654e33
Refactor VPC firewall module for Terraform 1.3 (#949)
* module and module tests/examples

* align blueprints and fast

* fix null ranges

* make ports optional

* tfdoc

* make rules optional defaulting to all protocols

* review comments

* last round of comments

* invert precedence of template variables

* add option to disable all default rules

* add option to disable all default rules

* split egress/ingress

* tests

* fix tests
2022-11-04 13:56:07 +01:00
Ludovico Magnocavallo 0563260678
Refactor ILB module for Terraform 1.3 (#941)
* wip

* wip

* wip

* backport health check to compute-mig

* align blueprints and fast

* fix blueprints

* tfdoc
2022-11-02 18:05:20 +01:00
Ludovico Magnocavallo 8c51a2a386
Enable org policy service and add README notice to modules (#936)
* enable org policy service and add README notice to modules

* fix tests
2022-11-01 14:25:07 +01:00
Julio Castillo ad420138ad Fix readme examples 2022-10-28 17:49:44 +02:00