d11c380aec
Format python files in blueprints ( #2079 )
...
* format python files in blueprints
* update check on blueprints python code
* update python linter in CI workflow
2024-02-15 09:37:49 +01:00
9ff67fdaf0
Add DLP Service Agent role
2023-11-20 15:34:27 +01:00
543ea6e7f3
Fix/dlpagent ( #1868 )
...
Create DLP Service Account on service activation.
2023-11-20 14:11:01 +01:00
a0ae43fc6f
[Minimal Data Platform] Fix Landing and curated IAM ( #1832 )
...
* Fix IAM on Minimal DP
* fix
2023-11-01 17:53:06 +01:00
6d89b88149
versions.tf maintenance + copyright notice bump ( #1782 )
...
* Bump copyright notice to 2023
* Delete versions.tf on blueprints
* Pin provider to major version 5
* Remove comment
* Fix lint
* fix bq-ml blueprint readme
---------
Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
Co-authored-by: Julio Castillo <jccb@google.com>
2023-10-20 18:17:47 +02:00
dfc5023e0b
Make deletion protection consistent across all modules ( #1735 )
...
* Expose deletion_protection in GKE modules
* Make deletion protection consistent across all modules
* Add deletion_protection option to blueprints
* Fix blueprints tests
* Fix types
* Update READMEs
* Fix dp readme
* Fix cmek blueprint default deletion_protection
* Fix blueprints tests
2023-10-05 17:31:07 +02:00
de883957fe
Minimal Data Platform - Fix ( #1730 )
...
* Fix
* Fix README
* Fix data platform minimal tests
---------
Co-authored-by: Julio Castillo <jccb@google.com>
2023-10-04 12:15:51 +02:00
79723f9ce1
Add lineage on Minimal Data Platform blueprint ( #1679 )
...
Add lineage on Minimal Data Platform blueprint
2023-09-14 17:52:19 +02:00
0e7cfc8b29
add AIRFLOW_VAR_ prefix to environment variables in data-platform blueprints ( #1651 )
...
* add AIRFLOW_VAR_ prefix to env vars in minimal blueprint
* update DAGs for env vars in minimal blueprint
* add AIRFLOW_VAR_ prefix to env vars in foundation blueprint
* update DAGs for env vars in foundation blueprint
* apply tf linting
---------
Co-authored-by: lcaggio <lorenzo.caggioni@gmail.com>
Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2023-09-08 07:38:28 +00:00
819894d2ba
IAM interface refactor ( #1595 )
...
* IAM modules refactor proposal
* policy
* subheading
* Update 20230816-iam-refactor.md
* log Julio's +1
* data-catalog-policy-tag
* dataproc
* dataproc
* folder
* folder
* folder
* folder
* project
* better filtering in test examples
* project
* folder
* folder
* organization
* fix variable descriptions
* kms
* net-vpc
* dataplex-datascan
* modules/iam-service-account
* modules/source-repository/
* blueprints/cloud-operations/vm-migration/
* blueprints/third-party-solutions/wordpress
* dataplex-datascan
* blueprints/cloud-operations/workload-identity-federation
* blueprints/data-solutions/cloudsql-multiregion/
* blueprints/data-solutions/composer-2
* Update 20230816-iam-refactor.md
* Update 20230816-iam-refactor.md
* capture discussion in architectural doc
* update variable names and refactor proposal
* project
* blueprints first round
* folder
* organization
* data-catalog-policy-tag
* re-enable folder inventory
* project module style fix
* dataproc
* source-repository
* source-repository tests
* dataplex-datascan
* dataplex-datascan tests
* net-vpc
* net-vpc test examples
* iam-service-account
* iam-service-account test examples
* kms
* boilerplate
* tfdoc
* fix module tests
* more blueprint fixes
* fix typo in data blueprints
* incomplete refactor of data platform foundations
* tfdoc
* data platform foundation
* refactor data platform foundation iam locals
* remove redundant example test
* shielded folder fix
* fix typo
* project factory
* project factory outputs
* tfdoc
* test workflow: less verbose tests, fix tf version
* re-enable -vv, shorter traceback, fix action version
* ignore github extension warning, re-enable action version
* fast bootstrap IAM, untested
* bootstrap stage IAM fixes
* stage 0 tests
* fast stage 1
* tenant stage 1
* minor changes to fast stage 0 and 1
* fast security stage
* fast mt stage 0
* fast mt stage 0
* fast pf
2023-08-20 09:44:20 +02:00
9c75aa469c
More module descriptions ( #1572 )
...
* bigquery dataset
* data catalog policy tag
* net-address
* fix data catalog callers
* bigquery dataset views
* fix data catalog callers
* logging bucket
* net vpn ha
2023-08-06 09:25:45 +00:00
0b224a7f55
Return only bucket name of composer, not full url to dags folder
2023-07-10 11:20:51 +02:00
43b3490ef1
Updating a few files for typos
...
fast/stages/3-data-platform/dev/README.md
fast/stages/3-data-platform/dev/outputs.tf
CHANGELOG.md
blueprints/data-solutions/data-platform-minimal/README.md
blueprints/data-solutions/data-platform-minimal/outputs.tf
blueprints/data-solutions/data-platform-foundations/README.md
2023-06-29 21:47:17 -04:00
b203198441
Merge branch 'master' into kthhrv-patch-1
2023-06-29 10:10:14 +01:00
912497a158
use key/value and update docs
2023-06-29 09:57:23 +01:00
f15e2f3509
default to empty map
2023-06-29 09:55:10 +01:00
e3e05540de
change var name, drop default, update docs
2023-06-29 09:45:29 +01:00
026071209c
Minimal Data Platform - Shared VPC ( #1475 )
...
* Fix
* Fix dataproc vpc links
* Add missing networkUser role.
* Fix README.
2023-06-28 21:58:03 +02:00
52ad1fee34
more linting
2023-06-28 14:07:36 +01:00
f9742c5405
lint
2023-06-28 14:06:05 +01:00
9e8323bd7d
Update 02-composer.tf
2023-06-28 13:56:35 +01:00
b30efa8bfe
Update variables.tf
2023-06-28 13:56:00 +01:00
7cd9083fd7
data-platform-minimal - support web_server_network_access_control
2023-06-28 13:28:28 +01:00
099ad03910
Improve Minimal Data Platform Blueprint ( #1473 )
...
* Add SA to access to Curated resources
* Add BQ dataset in the landing project
* Provide example to move data from landing to curated using BQ engine
* Improve diagram
2023-06-28 09:05:48 +02:00
261ad646a8
Improve Minimal Data Platform blueprint ( #1451 )
2023-06-20 16:47:15 +00:00
dc964411e0
data-platform-minimal - 02-processing.tf typo ( #1454 )
...
error is
```
│ Error: Unsupported attribute
│
│ on ../../../../blueprints/data-solutions/data-platform-minimal/02-processing.tf line 46, in locals:
│ 46: ? var.network_config.subnet_self_links.processingestration
│ ├────────────────
│ │ var.network_config.subnet_self_links is object with 2 attributes
│
│ This object does not have an attribute named "processingestration".
╵
```
I'm guessing this should be `processing_transformation` the other option is `processing_composer`
2023-06-20 15:26:10 +02:00
a37b99501e
fix typo data-platform-minimal ( #1453 )
...
Fix `container-engine` service agent typo.
2023-06-20 13:12:00 +02:00
26e5662e84
Some fixes
2023-06-16 00:01:07 +00:00
cecbd2072c
Fix modules and blueprints tests
2023-05-26 16:38:41 +02:00
40656a23de
Minimal Data Platform - Make components optional ( #1380 )
...
make some components optional: Composer and Data Proc history server.
2023-05-16 14:08:03 +02:00
3cc6c71e96
Add Minimal Data Platform blueprint ( #1362 )
...
Minimal Data Platform blueprint
2023-05-08 10:25:06 +02:00
simonebruzzechesse
Wiktor Niesiobędzki
ddaluka
lcaggio
Simone Ruffilli
Julio Castillo
Pierre Formont
Ludovico Magnocavallo
Alejandro Leal
Keith Harvey
bluPhy