d11c380aec
Format python files in blueprints ( #2079 )
...
* format python files in blueprints
* update check on blueprints python code
* update python linter in CI workflow
2024-02-15 09:37:49 +01:00
71a64487d5
Extend FAST to support different principal types ( #2064 )
...
* add doc draft
* typos
* typo
* typo
* typos
* rewording
* Update 0-domainless-iam.md
* Update 0-domainless-iam.md
* Update 0-domainless-iam.md
* Update 0-domainless-iam.md
* Update 0-domainless-iam.md
* Update 0-domainless-iam.md
* Update 0-domainless-iam.md
* Update 0-domainless-iam.md
* Update 0-domainless-iam.md
* Update 0-domainless-iam.md
* move iam variables to a separate file
* move billing-account module to iam_principals
* move data-catalog-policy-tag module to iam_principals
* move dataplex-datascan module to iam_principals
* move dataproc module to iam_principals
* move folder module to iam_principals
* copyright
* move organization module to iam_principals
* move project module to iam_principals
* move source-repository module to iam_principals
* update blueprints for iam_principals interface
* FAST bootstrap
* module READMEs fixes
* FAST bootstrap
* FAST networking stages
* FAST security stage
* FAST gke stage
* FAST multitenant bootstrap stage
* FAST multitenant resman stage
* tfdoc
* Update 0-domainless-iam.md
* Update 0-domainless-iam.md
* Update 0-domainless-iam.md
* Update 0-domainless-iam.md
* fix module test
* Update 0-domainless-iam.md
* Update 0-domainless-iam.md
* Rename iam_principals to iam_by_principals
* Update IAM template to include iam_by_principals
* Update Resman README
* Fix ADR link format
---------
Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-12 14:35:30 +01:00
3b4b355800
Ensure data platform service accounts meet FAST requirements ( #2016 )
...
* use 9-chars prefix in data platform example test
* use 9-chars prefix in data platform example test
2024-01-28 14:00:32 +01:00
208902c8da
Fix Data platform foundation ( #1992 )
...
* FAST + Minimal DP
* Fix tests
---------
Co-authored-by: Julio Castillo <jccb@google.com>
2024-01-20 08:49:46 +01:00
543ea6e7f3
Fix/dlpagent ( #1868 )
...
Create DLP Service Account on service activation.
2023-11-20 14:11:01 +01:00
6d89b88149
versions.tf maintenance + copyright notice bump ( #1782 )
...
* Bump copyright notice to 2023
* Delete versions.tf on blueprints
* Pin provider to major version 5
* Remove comment
* Fix lint
* fix bq-ml blueprint readme
---------
Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
Co-authored-by: Julio Castillo <jccb@google.com>
2023-10-20 18:17:47 +02:00
dfc5023e0b
Make deletion protection consistent across all modules ( #1735 )
...
* Expose deletion_protection in GKE modules
* Make deletion protection consistent across all modules
* Add deletion_protection option to blueprints
* Fix blueprints tests
* Fix types
* Update READMEs
* Fix dp readme
* Fix cmek blueprint default deletion_protection
* Fix blueprints tests
2023-10-05 17:31:07 +02:00
6889f02954
Fix data platform roles ( #1725 )
...
* Fix Data Platform roles
* Fix README
* Fix blueprint tests
* Update cleanup dp steps
---------
Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2023-10-04 07:31:40 +02:00
0e7cfc8b29
add AIRFLOW_VAR_ prefix to environment variables in data-platform blueprints ( #1651 )
...
* add AIRFLOW_VAR_ prefix to env vars in minimal blueprint
* update DAGs for env vars in minimal blueprint
* add AIRFLOW_VAR_ prefix to env vars in foundation blueprint
* update DAGs for env vars in foundation blueprint
* apply tf linting
---------
Co-authored-by: lcaggio <lorenzo.caggioni@gmail.com>
Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2023-09-08 07:38:28 +00:00
819894d2ba
IAM interface refactor ( #1595 )
...
* IAM modules refactor proposal
* policy
* subheading
* Update 20230816-iam-refactor.md
* log Julio's +1
* data-catalog-policy-tag
* dataproc
* dataproc
* folder
* folder
* folder
* folder
* project
* better filtering in test examples
* project
* folder
* folder
* organization
* fix variable descriptions
* kms
* net-vpc
* dataplex-datascan
* modules/iam-service-account
* modules/source-repository/
* blueprints/cloud-operations/vm-migration/
* blueprints/third-party-solutions/wordpress
* dataplex-datascan
* blueprints/cloud-operations/workload-identity-federation
* blueprints/data-solutions/cloudsql-multiregion/
* blueprints/data-solutions/composer-2
* Update 20230816-iam-refactor.md
* Update 20230816-iam-refactor.md
* capture discussion in architectural doc
* update variable names and refactor proposal
* project
* blueprints first round
* folder
* organization
* data-catalog-policy-tag
* re-enable folder inventory
* project module style fix
* dataproc
* source-repository
* source-repository tests
* dataplex-datascan
* dataplex-datascan tests
* net-vpc
* net-vpc test examples
* iam-service-account
* iam-service-account test examples
* kms
* boilerplate
* tfdoc
* fix module tests
* more blueprint fixes
* fix typo in data blueprints
* incomplete refactor of data platform foundations
* tfdoc
* data platform foundation
* refactor data platform foundation iam locals
* remove redundant example test
* shielded folder fix
* fix typo
* project factory
* project factory outputs
* tfdoc
* test workflow: less verbose tests, fix tf version
* re-enable -vv, shorter traceback, fix action version
* ignore github extension warning, re-enable action version
* fast bootstrap IAM, untested
* bootstrap stage IAM fixes
* stage 0 tests
* fast stage 1
* tenant stage 1
* minor changes to fast stage 0 and 1
* fast security stage
* fast mt stage 0
* fast mt stage 0
* fast pf
2023-08-20 09:44:20 +02:00
182ce27143
Add information about required groups
2023-08-06 20:27:59 +02:00
9c75aa469c
More module descriptions ( #1572 )
...
* bigquery dataset
* data catalog policy tag
* net-address
* fix data catalog callers
* bigquery dataset views
* fix data catalog callers
* logging bucket
* net vpn ha
2023-08-06 09:25:45 +00:00
5a534edf5c
update and refactor artifact registry module ( #1536 )
2023-07-28 09:54:36 +00:00
009c6d0f93
Make demo pipeline append into BQ tables ( #1533 )
2023-07-27 11:38:00 -04:00
d3739cc0ff
Update outputs.tf
2023-06-29 22:36:27 -04:00
43b3490ef1
Updating a few files for typos
...
fast/stages/3-data-platform/dev/README.md
fast/stages/3-data-platform/dev/outputs.tf
CHANGELOG.md
blueprints/data-solutions/data-platform-minimal/README.md
blueprints/data-solutions/data-platform-minimal/outputs.tf
blueprints/data-solutions/data-platform-foundations/README.md
2023-06-29 21:47:17 -04:00
884cb8b4bf
Ensure all modules have an `id` output ( #1410 )
...
* net-vpc
* a-d
* complete modules
* fix error
2023-06-02 16:07:22 +02:00
cecbd2072c
Fix modules and blueprints tests
2023-05-26 16:38:41 +02:00
6a89d71e96
Update to multiple README.md ( #1379 )
...
- blueprints/data-solutions/data-platform-foundations/README.md
- blueprints/factories/project-factory/README.md
- modules/net-ilb-l7/README.md
- modules/project/README.md
2023-05-16 06:11:34 +00:00
87cd83f5c0
Several updates
...
Several updates
2023-05-13 23:51:46 -04:00
3cc6c71e96
Add Minimal Data Platform blueprint ( #1362 )
...
Minimal Data Platform blueprint
2023-05-08 10:25:06 +02:00
75cc2f3d7a
FAST: shorten stage 3 prefixes, enforce prefix length in stage 3s ( #1346 )
...
* shorten stage 3 prefixes, enforce prefix length in stage 3s
* tfdoc
* tfdoc
2023-05-03 07:39:41 +02:00
c8d5b85ac9
Fix test in README
2023-04-06 15:55:34 +02:00
1bd5c66b42
Fix test
2023-04-06 14:21:57 +02:00
83e5ef180c
Remove state
2023-04-06 13:54:47 +02:00
badec52d11
Update IAM.md
2023-04-06 13:53:46 +02:00
bb2abb8822
Fix data-eng role on orchestration project.
2023-04-06 13:41:29 +02:00
230cbe4903
Fix variable terraform.tfvars.sample ( #1261 )
2023-03-17 11:13:10 +01:00
be60753246
Fix failing tests
2023-02-25 19:26:11 +01:00
99e69e319f
Update data platform tests
2023-02-25 19:26:11 +01:00
b279c083a0
Fix README and IAM files
2023-02-23 11:54:16 +01:00
2108b4650d
Fix Tests, rely on iam additive.
2023-02-22 01:36:01 +01:00
eda9597f0b
Fix linting
2023-02-20 01:03:38 +01:00
c523dce3a6
Fix
2023-02-20 01:00:57 +01:00
f4c1fa6c20
Fix tests.
2023-02-20 00:56:32 +01:00
db6a4f9ac7
Remove variables
2023-02-20 00:45:40 +01:00
c8f25512eb
Fix variabler to support existing projects.
2023-02-20 00:39:37 +01:00
ad6667a8fa
First commit
2023-02-19 22:37:32 +01:00
eac2065ae2
Update on docs wording
2023-02-16 17:04:05 +01:00
fcdadf521d
Fix spelling mistake
2023-02-16 15:48:12 +01:00
a853dc4fe2
update Data Platform blueprint README with more example Dataflow commands
2023-02-16 15:45:20 +01:00
02d8d8367a
[Feature] Update data platform blue print with Dataflow Flex template ( #1105 )
...
* Add initial dataflow template code + TF infra
* Refactor the datapipeline DAG to use flex template operator, cleanup code
* Remove unneeded bash scripts, update README with manual examples
* Refactor datapipeline_dc_tags.py and include new Flex template
* Update docs to reflect changes
* Remove sub-dependencies and keep apache beam
* Add missing license headers and update tests
* Set resouces to 291 in tests
* Update outputs via tfdoc
* Update with outputs order and tfdoc
* Correct number of resources
* Fix to add region into command from var
* Enable service account impersonation for running builds
* Update example dataflow run command to use orchestrator SA
* Remove hard coded values in example
* Keep original airflow files, add new which use Flex template as example
* Update tests and doc
* Fix number of resources in plan
* Run tfdoc remove files section in README
* Fix number of modules in tfdoc
* Update number of resources
* Add missin service account
* Update DF demo README
* Quick rename
---------
Co-authored-by: lcaggio <lorenzo.caggioni@gmail.com>
Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2023-02-06 07:35:40 +01:00
5453c585e0
FAST multitenant bootstrap and resource management, rename org-level FAST stages ( #1052 )
...
* rename stages
* remove support for external org billing, rename output files
* resman: make groups optional, align on new billing account variable
* bootstrap: multitenant outputs
* tenant bootstrap stage, untested
* fix folder name
* fix stage 0 output names
* optional creation for tag keys in organization module
* single tenant bootstrap minus tag
* rename output files, add tenant tag key
* fix organization module tag values output
* test skipping creation for tags in organization module
* single tenant bootstrap plan working
* multitenant bootstrap
* tfdoc
* fix check links error messages
* fix links
* tfdoc
* fix links
* rename fast tests, fix bootstrap tests
* multitenant stages have their own folder, simplify stage numbering
* stage renumbering
* wip
* rename tests
* exclude fast providers in fixture
* stage 0 tests
* stage 1 tests
* network stages tests
* stage tests
* tfdoc
* fix links
* tfdoc
* multitenant tests
* remove local files
* stage links command
* fix links script, TODO
* wip
* wip single tenant bootstrap
* working tenant bootstrap
* update gitignore
* remove local files
* tfdoc
* remove local files
* allow tests for tenant bootstrap stage
* tenant bootstrap proxies stage 1 tfvars
* stage 2 and 3 service accounts and IAM in tenant bootstrap
* wip
* wip
* wip
* drop multitenant bootstrap
* tfdoc
* add missing stage 2 SAs, fix org-level IAM condition
* wip
* wip
* optional tag value creation in organization module
* stage 1 working
* linting
* linting
* READMEs
* wip
* Make stage-links script work in old macos bash
* stage links command help
* fix output file names
* diagrams
* fix svg
* stage 0 skeleton and diagram
* test svg
* test svg
* test diagram
* diagram
* readme
* fix stage links script
* stage 0 readme
* README changes
* stage readmes
* fix outputs order
* fix link
* fix tests
* stage 1 test
* skip stage example
* boilerplate
* fix tftest skip
* default bootstrap stage log sinks to log buckets
* add logging to tenant bootstrap
* move iam variables out of tenant config
* fix cicd, reintroduce missing variable
* use optional in stage 1 cicd variable
* rename extras stage
* rename and move identity providers local, use optional for cicd variable
* tfdoc
* add support for wif pool and providers, ci/cd
* tfdoc
* fix links
* better handling of modules repository
* add missing role on logging project
* fix cicd pools in locals, test cicd
* fix workflow extension
* fix module source replacement
* allow tenant bootstrap cicd sa to impersonate resman sa
* tenant workflow templates fix for no providers file
* fix output files, push github workflow template to new repository
* remove try from outpout files
* align stage 1 cicd internals to stage 0
* tfdoc
* tests
* fix tests
* tests
* improve variable descriptions
* use optional in fast features
* actually create tenant log sinks, and allow the resman sa to do it
* test
* tests
* aaaand tests again
* fast features tenant override
* fast features tenant override
* fix wording
* add missing comment
* configure pf service accounts
* add missing comment
* tfdoc
* tests
* IAM docs
* update copyright
---------
Co-authored-by: Julio Castillo <jccb@google.com>
2023-02-04 15:00:45 +01:00
edd3a82453
Include cloudbuild API in project module ( #1116 )
...
* Include cloudbuild API in project module
* Increase number of resources
2023-01-27 21:38:01 +01:00
02938bdfa1
Fix Table in readme.
2023-01-16 15:58:11 +01:00
1672cfe9c1
Fix README.
2023-01-12 12:45:39 +01:00
50a52ae01a
Implement PR comments.
2023-01-12 12:41:00 +01:00
07a975f84e
Fix example test.
2023-01-11 12:37:08 +01:00
72abd47520
Fix documentation.
2023-01-11 08:47:42 +01:00
c172c7bb2e
Remove plg project and update to composer 2
2023-01-11 00:31:58 +01:00
simonebruzzechesse
Ludovico Magnocavallo
lcaggio
ddaluka
Simone Ruffilli
Julio Castillo
Pierre Formont
Wiktor Niesiobędzki
Daniel De Leo
Alejandro Leal
Dedeco
Ayman Farhat
Ayman Farhat