87cd83f5c0
Several updates
...
Several updates
2023-05-13 23:51:46 -04:00
3cc6c71e96
Add Minimal Data Platform blueprint ( #1362 )
...
Minimal Data Platform blueprint
2023-05-08 10:25:06 +02:00
05ed1bf12f
Add blueprint metadata validation tool
2023-05-05 16:57:48 +02:00
4d66623de1
Fix Shielded Folder - VertexML interoperability ( #1355 )
...
* Fix Shielded Folder - VertexML interoperability
* Fix README outputs
* Adding prefix for keyrings and update output
2023-05-05 07:54:57 +00:00
75cc2f3d7a
FAST: shorten stage 3 prefixes, enforce prefix length in stage 3s ( #1346 )
...
* shorten stage 3 prefixes, enforce prefix length in stage 3s
* tfdoc
* tfdoc
2023-05-03 07:39:41 +02:00
94589efc34
Fix external documentation links ( #1347 )
2023-05-02 05:26:58 +00:00
1547a55e1f
chore: update metadata schema ( #1345 )
2023-04-28 17:14:20 -05:00
1e0b1da0d2
Add directory to vertex-mlops blueprint metadata
2023-04-27 08:52:24 +02:00
bca8a33f1e
Fix typo and blueprint module name
2023-04-24 20:36:50 +02:00
8488e866bc
Update readme
2023-04-20 16:53:09 +02:00
b896ccb9e5
Update metadata types and defaults.
2023-04-19 15:20:29 +02:00
9fdf80738f
Fix linting.
2023-04-19 14:09:40 +02:00
30bef8546f
Create metadata yaml file
2023-04-19 12:58:00 +02:00
f4490fcaea
Fix comments
2023-04-19 11:22:50 +02:00
1f4fac2f1d
fix lint
2023-04-18 17:51:24 +02:00
edf67fc5d0
First commit
2023-04-18 17:32:15 +02:00
91a1168d1d
rename blueprint providers file ( #1334 )
2023-04-18 11:44:09 +02:00
4b2e2d2ef7
Add providers to vertex-mlops blueprint
2023-04-18 09:42:46 +02:00
5e9b7091c2
Merge branch 'master' into lcaggioni/data-eng-fix
2023-04-12 15:57:39 +02:00
c6e79ddfc4
fix: create log-export-dataset on shielded-folder when no ecryption keys are defined
2023-04-12 15:31:08 +03:00
4722efea40
bump provider versions
2023-04-11 22:18:31 +10:00
c8d5b85ac9
Fix test in README
2023-04-06 15:55:34 +02:00
1bd5c66b42
Fix test
2023-04-06 14:21:57 +02:00
83e5ef180c
Remove state
2023-04-06 13:54:47 +02:00
badec52d11
Update IAM.md
2023-04-06 13:53:46 +02:00
bb2abb8822
Fix data-eng role on orchestration project.
2023-04-06 13:41:29 +02:00
865ef463bb
Support new fields in bigquery module, bump provider versions, unpin local provider ( #1306 )
...
* add support for new attributes to bigquery module
* bump provider versions
2023-04-05 14:22:52 +00:00
230cbe4903
Fix variable terraform.tfvars.sample ( #1261 )
2023-03-17 11:13:10 +01:00
21fa6d1f13
Fixes related to boot_disk in compute-vm module
2023-03-16 15:58:39 +01:00
79a6e9b191
pin local provider ( #1256 )
2023-03-16 10:59:06 +00:00
368472c9a0
Fix 1236
2023-03-13 21:24:27 +01:00
3f9bbc2e5c
Add cmek support on google_vertex_ai_metadata_store.
2023-03-09 09:13:21 +01:00
1671c5b4f3
Add cmek support on google_vertex_ai_metadata_store
2023-03-09 09:11:47 +01:00
ca9898395d
Merge branch 'master' into conteg/bqml-fix
2023-03-08 17:52:03 +01:00
e213f156ad
Fix Jit notebook service account.
2023-03-08 16:06:27 +01:00
a9bb717554
minor fix on bqml demo
2023-03-07 14:20:48 +00:00
3123d67ddf
moved sql to notebook
2023-03-06 12:28:58 +00:00
ee55127ede
Fix notebook
2023-03-06 12:51:46 +01:00
0ac6dd65cf
sql fix and more comments on demo notebook
2023-03-06 11:21:30 +00:00
c82e7cca7b
added demo README file
2023-03-06 11:12:36 +00:00
0852ae3778
demo: added batch prediction example
2023-03-06 10:16:21 +00:00
dc034d74f7
Variables.
2023-03-05 22:24:16 +01:00
9e19f89608
Implement PR comments.
2023-03-05 22:02:41 +01:00
2b8ba16a9a
Fix typos
2023-03-04 14:32:54 +01:00
652495e530
Update versions.
2023-03-04 14:12:50 +01:00
f8a7aa865a
Fix test.
2023-03-04 08:25:29 +01:00
8d70e1d900
Merge branch 'master' into lcaggio/bqml
2023-03-04 08:22:22 +01:00
ccd68b2fa6
Fix linting.
2023-03-04 08:19:47 +01:00
0d4b599e99
Fix README
2023-03-04 08:13:53 +01:00
98e17bb997
Fix readme.
2023-03-04 08:09:29 +01:00
6526dda8c7
sql linting
2023-03-03 14:52:35 +00:00
594a615e1e
Update
2023-03-03 15:08:57 +01:00
32808f93ea
Update README.
2023-03-03 14:52:33 +01:00
52eb83758f
Fix compute-vm:CloudKMS test for provider>=4.54.0
...
* TF provider >= 4.54.0 now returns `rsa_encrypted_key` for
`google_compute_disk.disks["attached-disk"]` (see
hashicorp/terraform-provider-google#4448 )
* Add this field to expected model to fix test assertion failure
* Update required TF provider to 4.55.0 (latest) since the assertion
will now fail with <4.54.0, which do not return `rsa_encrypted_key`
* Updated the whole repo on advice from @ludoo
2023-02-28 15:10:22 +00:00
17b8a461f0
fixed notebook with dynamic model name
...
cleared output from cells
added creation of view instead of table
2023-02-27 15:28:49 +00:00
3271acd2f2
Added sql and jupyter notebook to run the demo
2023-02-27 10:56:47 +00:00
be60753246
Fix failing tests
2023-02-25 19:26:11 +01:00
edce6edd28
Update factories and apigee tests
2023-02-25 19:26:11 +01:00
b186365cab
Update data solutions tests
2023-02-25 19:26:11 +01:00
99e69e319f
Update data platform tests
2023-02-25 19:26:11 +01:00
a51c682005
Updated tf file to add the following features:
...
- default location of dataset to US
- changed name of vertex metastore to "default"
- add ai user and service account us to notebook SA
- add ai user to vertex sa
2023-02-24 13:27:44 +00:00
50856e6951
First commit
2023-02-23 18:36:03 +01:00
47855cb682
Merge branch 'master' into lcaggio/dp-projectcreate
2023-02-23 11:54:48 +01:00
b279c083a0
Fix README and IAM files
2023-02-23 11:54:16 +01:00
2108b4650d
Fix Tests, rely on iam additive.
2023-02-22 01:36:01 +01:00
a5e905cb80
Update remaining org policies
2023-02-21 15:49:16 +01:00
d3bcf625f9
Update yaml org policies
2023-02-21 15:49:16 +01:00
eda9597f0b
Fix linting
2023-02-20 01:03:38 +01:00
c523dce3a6
Fix
2023-02-20 01:00:57 +01:00
f4c1fa6c20
Fix tests.
2023-02-20 00:56:32 +01:00
db6a4f9ac7
Remove variables
2023-02-20 00:45:40 +01:00
c8f25512eb
Fix variabler to support existing projects.
2023-02-20 00:39:37 +01:00
ad6667a8fa
First commit
2023-02-19 22:37:32 +01:00
eac2065ae2
Update on docs wording
2023-02-16 17:04:05 +01:00
fcdadf521d
Fix spelling mistake
2023-02-16 15:48:12 +01:00
a853dc4fe2
update Data Platform blueprint README with more example Dataflow commands
2023-02-16 15:45:20 +01:00
ebc4bc51a5
Workaround to mitigate issue 9164
2023-02-13 15:25:24 +01:00
08f2540872
Bump provider version
2023-02-07 15:59:29 +01:00
d78815d3d7
Fix README
2023-02-07 13:57:26 +01:00
71469679e3
Add variables validation
2023-02-07 13:53:54 +01:00
dd77c2fe79
Merge branch 'master' into lcaggio/shieldedfolder
2023-02-07 07:58:00 +01:00
307cf32cf9
Fix
2023-02-06 21:34:29 +01:00
d68bb4b320
Fixes
2023-02-06 21:33:04 +01:00
1e4499c8ab
Fixes
2023-02-06 14:58:40 +01:00
02d8d8367a
[Feature] Update data platform blue print with Dataflow Flex template ( #1105 )
...
* Add initial dataflow template code + TF infra
* Refactor the datapipeline DAG to use flex template operator, cleanup code
* Remove unneeded bash scripts, update README with manual examples
* Refactor datapipeline_dc_tags.py and include new Flex template
* Update docs to reflect changes
* Remove sub-dependencies and keep apache beam
* Add missing license headers and update tests
* Set resouces to 291 in tests
* Update outputs via tfdoc
* Update with outputs order and tfdoc
* Correct number of resources
* Fix to add region into command from var
* Enable service account impersonation for running builds
* Update example dataflow run command to use orchestrator SA
* Remove hard coded values in example
* Keep original airflow files, add new which use Flex template as example
* Update tests and doc
* Fix number of resources in plan
* Run tfdoc remove files section in README
* Fix number of modules in tfdoc
* Update number of resources
* Add missin service account
* Update DF demo README
* Quick rename
---------
Co-authored-by: lcaggio <lorenzo.caggioni@gmail.com>
Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2023-02-06 07:35:40 +01:00
39ffdcf184
Merge branch 'master' into lcaggio/shieldedfolder
2023-02-05 22:45:14 +01:00
f49e5ccec7
Fix readmes and variables.
2023-02-05 22:44:31 +01:00
47c75f8862
Implement project_config variable
2023-02-05 21:12:46 +01:00
38fc9e8b9b
First commit
2023-02-05 07:52:33 +01:00
ce1cc5f75a
Merge branch 'master' into lcaggio/shieldedfolder
2023-02-04 19:42:19 +01:00
5453c585e0
FAST multitenant bootstrap and resource management, rename org-level FAST stages ( #1052 )
...
* rename stages
* remove support for external org billing, rename output files
* resman: make groups optional, align on new billing account variable
* bootstrap: multitenant outputs
* tenant bootstrap stage, untested
* fix folder name
* fix stage 0 output names
* optional creation for tag keys in organization module
* single tenant bootstrap minus tag
* rename output files, add tenant tag key
* fix organization module tag values output
* test skipping creation for tags in organization module
* single tenant bootstrap plan working
* multitenant bootstrap
* tfdoc
* fix check links error messages
* fix links
* tfdoc
* fix links
* rename fast tests, fix bootstrap tests
* multitenant stages have their own folder, simplify stage numbering
* stage renumbering
* wip
* rename tests
* exclude fast providers in fixture
* stage 0 tests
* stage 1 tests
* network stages tests
* stage tests
* tfdoc
* fix links
* tfdoc
* multitenant tests
* remove local files
* stage links command
* fix links script, TODO
* wip
* wip single tenant bootstrap
* working tenant bootstrap
* update gitignore
* remove local files
* tfdoc
* remove local files
* allow tests for tenant bootstrap stage
* tenant bootstrap proxies stage 1 tfvars
* stage 2 and 3 service accounts and IAM in tenant bootstrap
* wip
* wip
* wip
* drop multitenant bootstrap
* tfdoc
* add missing stage 2 SAs, fix org-level IAM condition
* wip
* wip
* optional tag value creation in organization module
* stage 1 working
* linting
* linting
* READMEs
* wip
* Make stage-links script work in old macos bash
* stage links command help
* fix output file names
* diagrams
* fix svg
* stage 0 skeleton and diagram
* test svg
* test svg
* test diagram
* diagram
* readme
* fix stage links script
* stage 0 readme
* README changes
* stage readmes
* fix outputs order
* fix link
* fix tests
* stage 1 test
* skip stage example
* boilerplate
* fix tftest skip
* default bootstrap stage log sinks to log buckets
* add logging to tenant bootstrap
* move iam variables out of tenant config
* fix cicd, reintroduce missing variable
* use optional in stage 1 cicd variable
* rename extras stage
* rename and move identity providers local, use optional for cicd variable
* tfdoc
* add support for wif pool and providers, ci/cd
* tfdoc
* fix links
* better handling of modules repository
* add missing role on logging project
* fix cicd pools in locals, test cicd
* fix workflow extension
* fix module source replacement
* allow tenant bootstrap cicd sa to impersonate resman sa
* tenant workflow templates fix for no providers file
* fix output files, push github workflow template to new repository
* remove try from outpout files
* align stage 1 cicd internals to stage 0
* tfdoc
* tests
* fix tests
* tests
* improve variable descriptions
* use optional in fast features
* actually create tenant log sinks, and allow the resman sa to do it
* test
* tests
* aaaand tests again
* fast features tenant override
* fast features tenant override
* fix wording
* add missing comment
* configure pf service accounts
* add missing comment
* tfdoc
* tests
* IAM docs
* update copyright
---------
Co-authored-by: Julio Castillo <jccb@google.com>
2023-02-04 15:00:45 +01:00
45a74f37f9
Mention policies in the costomization section.
2023-02-04 06:24:35 +01:00
f00d4d3edd
Merge branch 'master' into lcaggio/shieldedfolder
2023-02-03 22:17:30 +01:00
ce1f86d20b
Vertex Pipelines MLOps framework blueprint ( #1038 )
...
* First release of the MLOps blueprint
2023-02-02 19:13:13 +01:00
30f2b49c9c
README Typos
2023-02-01 17:34:31 +01:00
e3daaf2503
Update READMEs
2023-02-01 13:02:07 +01:00
f9e3eb7d5a
Update copyright
2023-02-01 09:30:28 +01:00
840fc86b3e
Fix groups. Fix tests.
2023-02-01 08:55:33 +01:00
053d4151de
Chane feature name, improve README
2023-02-01 00:50:22 +01:00
3995ec1550
Merge branch 'master' into lcaggio/shieldedfolder
2023-02-01 00:28:01 +01:00
Alejandro Leal
lcaggio
Julio Castillo
Ludovico Magnocavallo
Bob Idle
Bharath KKB
bgdanix
Fawzi
Dedeco
Miren Esnaola
Giorgio Conte
Giorgio Conte
Dan Farmer
Ayman Farhat
Ayman Farhat
javiergp