Commit Graph

658 Commits

Author SHA1 Message Date
Ludovico Magnocavallo 91f71fef68
trap requests timeout error (#1607) 2023-08-21 18:37:54 +02:00
Ludovico Magnocavallo 819894d2ba
IAM interface refactor (#1595)
* IAM modules refactor proposal

* policy

* subheading

* Update 20230816-iam-refactor.md

* log Julio's +1

* data-catalog-policy-tag

* dataproc

* dataproc

* folder

* folder

* folder

* folder

* project

* better filtering in test examples

* project

* folder

* folder

* organization

* fix variable descriptions

* kms

* net-vpc

* dataplex-datascan

* modules/iam-service-account

* modules/source-repository/

* blueprints/cloud-operations/vm-migration/

* blueprints/third-party-solutions/wordpress

* dataplex-datascan

* blueprints/cloud-operations/workload-identity-federation

* blueprints/data-solutions/cloudsql-multiregion/

* blueprints/data-solutions/composer-2

* Update 20230816-iam-refactor.md

* Update 20230816-iam-refactor.md

* capture discussion in architectural doc

* update variable names and refactor proposal

* project

* blueprints first round

* folder

* organization

* data-catalog-policy-tag

* re-enable folder inventory

* project module style fix

* dataproc

* source-repository

* source-repository tests

* dataplex-datascan

* dataplex-datascan tests

* net-vpc

* net-vpc test examples

* iam-service-account

* iam-service-account test examples

* kms

* boilerplate

* tfdoc

* fix module tests

* more blueprint fixes

* fix typo in data blueprints

* incomplete refactor of data platform foundations

* tfdoc

* data platform foundation

* refactor data platform foundation iam locals

* remove redundant example test

* shielded folder fix

* fix typo

* project factory

* project factory outputs

* tfdoc

* test workflow: less verbose tests, fix tf version

* re-enable -vv, shorter traceback, fix action version

* ignore github extension warning, re-enable action version

* fast bootstrap IAM, untested

* bootstrap stage IAM fixes

* stage 0 tests

* fast stage 1

* tenant stage 1

* minor changes to fast stage 0 and 1

* fast security stage

* fast mt stage 0

* fast mt stage 0

* fast pf
2023-08-20 09:44:20 +02:00
Alejandro Leal ea0de3adbb Fixing some typos 2023-08-18 05:51:00 +00:00
Ludovico Magnocavallo 79373721df
Remove firewall policy management from resource management modules (#1581)
* rename firewall policy module, fix outputs

* add TOC to firewall policy module

* don't depend policy on parent id

* remove firewall policy from resource management modules

* remove factory conditionals

* fast net a and b

* fast stages

* fast tfdoc

* fast tfdoc

* remove unused test

* fix shielded folder blueprint

* fix shielded folder blueprint
2023-08-09 11:23:07 +00:00
Miren Esnaola 6a13742e3c Apigee addons 2023-08-09 08:12:06 +02:00
Eunan Hardy dd3a298892 Moved allow_net_admin to enable_features flag. Bumped provider version to 4.76 2023-08-07 14:27:20 +01:00
Wiktor Niesiobędzki 182ce27143 Add information about required groups 2023-08-06 20:27:59 +02:00
Ludovico Magnocavallo 9c75aa469c
More module descriptions (#1572)
* bigquery dataset

* data catalog policy tag

* net-address

* fix data catalog callers

* bigquery dataset views

* fix data catalog callers

* logging bucket

* net vpn ha
2023-08-06 09:25:45 +00:00
Miren Esnaola fa34499767 Removed unused attribute in variable of ha-vpn-over-blueprint blueprint 2023-08-02 13:02:17 +02:00
javiergp a2eddd0be0
Fixed Cloud Build default bucket name and vpc/subnet names (#1548) 2023-07-31 12:52:36 +02:00
Ludovico Magnocavallo b524aa137c
Peering module refactor (#1547)
* refactor net-vpc-peering module

* hub and spoke peering blueprint

* fast stages

* boilerplate

* fast tfdoc

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2023-07-29 21:33:57 +02:00
Wiktor Niesiobędzki 4998f1d376
Grant IAM rights to service identities in host project (#1542)
* [module/project] Grant IAM rights to service identities based on used services in host project
* [blueprints/factories/project-factory] enable granting IAM permissions in host VPC for service identities directly or by specifying services in use
2023-07-29 20:07:21 +02:00
Ludovico Magnocavallo 5a534edf5c
update and refactor artifact registry module (#1536) 2023-07-28 09:54:36 +00:00
Daniel De Leo 009c6d0f93
Make demo pipeline append into BQ tables (#1533) 2023-07-27 11:38:00 -04:00
Miren Esnaola cacb0c02e2 Refactoring of dns module 2023-07-19 12:57:44 +02:00
dependabot[bot] 69028a801e
Bump semver in /blueprints/serverless/api-gateway/function (#1504)
Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md)
- [Commits](https://github.com/npm/node-semver/compare/v5.7.1...v5.7.2)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-13 08:05:52 +02:00
Miren Esnaola b79261dfa5 Fix in nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg blueprint 2023-07-11 11:46:53 +02:00
Wiktor Niesiobędzki 0b224a7f55 Return only bucket name of composer, not full url to dags folder 2023-07-10 11:20:51 +02:00
Ludovico Magnocavallo d9f7cef1e7
allow using a separate resource for boot disk (#1496) 2023-07-07 15:40:13 +00:00
Ludovico Magnocavallo 0bc6dffce0
Fix and improve quota monitor blueprint (#1488)
* quota monitoring blueprint fixes

* wip

* wip

* quota

* improvements

* improve variables

* refactor http code

* fix http post

* improve logging

* fix project creation, improve readme

* fix test

* Update main.py

* remove unneeded constant

* exit with http error message instead of json when failing to decode api response

* actually do what previous commit wanted :)

* nits
2023-07-03 09:23:49 +02:00
Alejandro Leal d3739cc0ff Update outputs.tf 2023-06-29 22:36:27 -04:00
Alejandro Leal 43b3490ef1 Updating a few files for typos
fast/stages/3-data-platform/dev/README.md
fast/stages/3-data-platform/dev/outputs.tf
CHANGELOG.md
blueprints/data-solutions/data-platform-minimal/README.md
blueprints/data-solutions/data-platform-minimal/outputs.tf
blueprints/data-solutions/data-platform-foundations/README.md
2023-06-29 21:47:17 -04:00
Julio Castillo 0bd3f5ab0e
Merge branch 'master' into kthhrv-patch-1 2023-06-29 18:18:39 +02:00
Ludovico Magnocavallo 981068e442
add region to cf (#1482) 2023-06-29 13:02:56 +02:00
Keith Harvey b203198441
Merge branch 'master' into kthhrv-patch-1 2023-06-29 10:10:14 +01:00
Keith Harvey 912497a158 use key/value and update docs 2023-06-29 09:57:23 +01:00
Keith Harvey f15e2f3509 default to empty map 2023-06-29 09:55:10 +01:00
Keith Harvey e3e05540de change var name, drop default, update docs 2023-06-29 09:45:29 +01:00
lcaggio 026071209c
Minimal Data Platform - Shared VPC (#1475)
* Fix

* Fix dataproc vpc links

* Add missing networkUser role.

* Fix README.
2023-06-28 21:58:03 +02:00
Keith Harvey 52ad1fee34
more linting 2023-06-28 14:07:36 +01:00
Keith Harvey f9742c5405
lint 2023-06-28 14:06:05 +01:00
Keith Harvey 9e8323bd7d
Update 02-composer.tf 2023-06-28 13:56:35 +01:00
Keith Harvey b30efa8bfe
Update variables.tf 2023-06-28 13:56:00 +01:00
Keith Harvey 7cd9083fd7
data-platform-minimal - support web_server_network_access_control 2023-06-28 13:28:28 +01:00
lcaggio 099ad03910
Improve Minimal Data Platform Blueprint (#1473)
* Add SA to access to Curated resources

* Add BQ dataset in the landing project

* Provide example to move data from landing to curated using BQ engine

* Improve diagram
2023-06-28 09:05:48 +02:00
Ludovico Magnocavallo eace97cf4b
Merge branch 'master' into apigateway-blueprint-dependency-update 2023-06-27 08:13:15 +02:00
dependabot[bot] e64c043aaa
Bump semver and @google-cloud/storage in /blueprints/gke/binauthz/image (#1469)
Removes [semver](https://github.com/npm/node-semver). It's no longer used after updating ancestor dependency [@google-cloud/storage](https://github.com/googleapis/nodejs-storage). These dependencies need to be updated together.


Removes `semver`

Updates `@google-cloud/storage` from 5.18.3 to 6.11.0
- [Release notes](https://github.com/googleapis/nodejs-storage/releases)
- [Changelog](https://github.com/googleapis/nodejs-storage/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/nodejs-storage/compare/v5.18.3...v6.11.0)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
- dependency-name: "@google-cloud/storage"
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-26 15:03:47 +02:00
Ludovico Magnocavallo e11d1a8ea8
Merge branch 'master' into apigateway-blueprint-dependency-update 2023-06-26 09:56:52 +02:00
Miren Esnaola 2c7c41b25f Dependencies update for API Gateway blueprint 2023-06-26 09:51:28 +02:00
Ludovico Magnocavallo 638841c8d1
Rename network load balancer modules (#1466)
* update LB modules to new names

* update LB modules names

* update test paths
2023-06-26 07:50:10 +00:00
Wiktor Niesiobędzki 4b6552a6f6 Update README 2023-06-26 09:22:08 +02:00
Wiktor Niesiobędzki 173a00d795 Use EAD per enum https://cloud.google.com/compute/docs/reference/rest/v1/interconnectAttachments#InterconnectAttachment.FIELDS.edge_availability_domain 2023-06-26 09:22:08 +02:00
Wiktor Niesiobędzki 8a6555c581 Add preliminary support for partner interconnect 2023-06-26 09:22:08 +02:00
lcaggio 45adcf1187 First commit. 2023-06-23 15:52:08 +02:00
Alejandro Leal 15ae95df90 Fixing typos
- modules/organization/variables.tf
- modules/organization/README.md
- blueprints/data-solutions/shielded-folder/main.tf
2023-06-23 00:14:02 -04:00
Canburak Tümer adc1ff3fca
match readme groups with variables file (#1455)
I faced error during my deployment even though I created data-engineers group. Then I found out that variables file uses gcp-data-engineers as default. The same is valid for data-security.
2023-06-21 11:51:32 +02:00
lcaggio 261ad646a8
Improve Minimal Data Platform blueprint (#1451) 2023-06-20 16:47:15 +00:00
Keith Harvey dc964411e0
data-platform-minimal - 02-processing.tf typo (#1454)
error is 
```
│ Error: Unsupported attribute
│ 
│   on ../../../../blueprints/data-solutions/data-platform-minimal/02-processing.tf line 46, in locals:
│   46:     ? var.network_config.subnet_self_links.processingestration
│     ├────────────────
│     │ var.network_config.subnet_self_links is object with 2 attributes
│ 
│ This object does not have an attribute named "processingestration".
╵
```

I'm guessing this should be `processing_transformation` the other option is `processing_composer`
2023-06-20 15:26:10 +02:00
Keith Harvey a37b99501e
fix typo data-platform-minimal (#1453)
Fix `container-engine` service agent typo.
2023-06-20 13:12:00 +02:00
Albert Lloveras 7cacc46b4b
fixup(project-factory): Use the correct KMS Service Agents attribute … (#1446)
* fixup(project-factory): Use the correct KMS Service Agents attribute name

* Add new KMS bindings to tests

* Update test resource counts

* Update README.md resource count
2023-06-19 23:53:08 +00:00