Commit Graph

33 Commits

Author SHA1 Message Date
Julio Castillo b186365cab Update data solutions tests 2023-02-25 19:26:11 +01:00
Julio Castillo 99e69e319f Update data platform tests 2023-02-25 19:26:11 +01:00
lcaggio 2108b4650d Fix Tests, rely on iam additive. 2023-02-22 01:36:01 +01:00
lcaggio f4c1fa6c20 Fix tests. 2023-02-20 00:56:32 +01:00
lcaggio dd77c2fe79
Merge branch 'master' into lcaggio/shieldedfolder 2023-02-07 07:58:00 +01:00
lcaggio d68bb4b320 Fixes 2023-02-06 21:33:04 +01:00
lcaggio 1e4499c8ab Fixes 2023-02-06 14:58:40 +01:00
Ayman Farhat 02d8d8367a
[Feature] Update data platform blue print with Dataflow Flex template (#1105)
* Add initial dataflow template code + TF infra

* Refactor the datapipeline DAG to use flex template operator, cleanup code

* Remove unneeded bash scripts, update README with manual examples

* Refactor datapipeline_dc_tags.py and include new Flex template

* Update docs to reflect changes

* Remove sub-dependencies and keep apache beam

* Add missing license headers and update tests

* Set resouces to 291 in tests

* Update outputs via tfdoc

* Update with outputs order and tfdoc

* Correct number of resources

* Fix to add region into command from var

* Enable service account impersonation for running builds

* Update example dataflow run command to use orchestrator SA

* Remove hard coded values in example

* Keep original airflow files, add new which use Flex template as example

* Update tests and doc

* Fix number of resources in plan

* Run tfdoc remove files section in README

* Fix number of modules in tfdoc

* Update number of resources

* Add missin service account

* Update DF demo README

* Quick rename

---------

Co-authored-by: lcaggio <lorenzo.caggioni@gmail.com>
Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2023-02-06 07:35:40 +01:00
lcaggio 39ffdcf184
Merge branch 'master' into lcaggio/shieldedfolder 2023-02-05 22:45:14 +01:00
lcaggio 47c75f8862 Implement project_config variable 2023-02-05 21:12:46 +01:00
lcaggio 38fc9e8b9b First commit 2023-02-05 07:52:33 +01:00
lcaggio f00d4d3edd
Merge branch 'master' into lcaggio/shieldedfolder 2023-02-03 22:17:30 +01:00
javiergp ce1f86d20b
Vertex Pipelines MLOps framework blueprint (#1038)
* First release of the MLOps blueprint
2023-02-02 19:13:13 +01:00
lcaggio f9e3eb7d5a Update copyright 2023-02-01 09:30:28 +01:00
Julio Castillo 351698873b Fix test names and update to yaml-based tests 2023-02-01 09:03:53 +01:00
lcaggio 840fc86b3e Fix groups. Fix tests. 2023-02-01 08:55:33 +01:00
lcaggio f36b004664 Make VPCSC optional; ingress policy 2023-02-01 00:26:57 +01:00
lcaggio fb04e78829 Fix variables 2023-01-31 22:35:48 +01:00
lcaggio 2584eb149c Fix test 2023-01-31 00:02:30 +01:00
lcaggio a166c612d5 Merge branch 'master' into lcaggio/shieldedfolder 2023-01-30 23:52:46 +01:00
lcaggio 66a10d76c3 Add test 2023-01-30 23:47:33 +01:00
Ayman Farhat edd3a82453
Include cloudbuild API in project module (#1116)
* Include cloudbuild API in project module

* Increase number of resources
2023-01-27 21:38:01 +01:00
lcaggio 07a7be29e3 Add shared-vpc support on Data Playgroud blueprint 2023-01-19 00:33:31 +01:00
lcaggio 488b6d4dc0 Remove plg project, composer 2 2023-01-11 00:31:52 +01:00
lcaggio b1430232b8 Fix tests 2022-11-24 15:50:26 +01:00
Ludovico Magnocavallo 8c51a2a386
Enable org policy service and add README notice to modules (#936)
* enable org policy service and add README notice to modules

* fix tests
2022-11-01 14:25:07 +01:00
lcaggio bf5f30e0bd Fix test. 2022-09-27 12:55:26 +02:00
lcaggio 0704a0ad30 Fixes based on comments. 2022-09-20 12:26:57 +02:00
lcaggio a88447d10f Merge branch 'master' into lcaggio/composer-2 2022-09-20 11:49:20 +02:00
lcaggio f28dc5cb04 Fix tests 2022-09-20 10:51:11 +02:00
lcaggio 5968661e8d First commit 2022-09-19 10:34:46 +02:00
Julio Castillo 269894e92b Update all internal links examples -> blueprints 2022-09-09 16:39:01 +02:00
Julio Castillo a248602827 Rename examples folder to blueprints 2022-09-09 16:38:43 +02:00