Commit Graph

712 Commits

Author SHA1 Message Date
Julio Diez b3e1e8e1ba
Delete blueprints/serverless/api-gateway directory 2023-08-29 11:48:13 +02:00
Julio Diez a7ec00acba
Delete API Gateway from serverless' README 2023-08-29 11:45:15 +02:00
Julio Diez 6ce8fd6942
Delete API Gateway from blueprint's README 2023-08-29 11:40:31 +02:00
Julio Castillo 44c1cec041 Fix tests 2023-08-24 20:09:23 +02:00
Ludovico Magnocavallo 91f71fef68
trap requests timeout error (#1607) 2023-08-21 18:37:54 +02:00
Ludovico Magnocavallo 819894d2ba
IAM interface refactor (#1595)
* IAM modules refactor proposal

* policy

* subheading

* Update 20230816-iam-refactor.md

* log Julio's +1

* data-catalog-policy-tag

* dataproc

* dataproc

* folder

* folder

* folder

* folder

* project

* better filtering in test examples

* project

* folder

* folder

* organization

* fix variable descriptions

* kms

* net-vpc

* dataplex-datascan

* modules/iam-service-account

* modules/source-repository/

* blueprints/cloud-operations/vm-migration/

* blueprints/third-party-solutions/wordpress

* dataplex-datascan

* blueprints/cloud-operations/workload-identity-federation

* blueprints/data-solutions/cloudsql-multiregion/

* blueprints/data-solutions/composer-2

* Update 20230816-iam-refactor.md

* Update 20230816-iam-refactor.md

* capture discussion in architectural doc

* update variable names and refactor proposal

* project

* blueprints first round

* folder

* organization

* data-catalog-policy-tag

* re-enable folder inventory

* project module style fix

* dataproc

* source-repository

* source-repository tests

* dataplex-datascan

* dataplex-datascan tests

* net-vpc

* net-vpc test examples

* iam-service-account

* iam-service-account test examples

* kms

* boilerplate

* tfdoc

* fix module tests

* more blueprint fixes

* fix typo in data blueprints

* incomplete refactor of data platform foundations

* tfdoc

* data platform foundation

* refactor data platform foundation iam locals

* remove redundant example test

* shielded folder fix

* fix typo

* project factory

* project factory outputs

* tfdoc

* test workflow: less verbose tests, fix tf version

* re-enable -vv, shorter traceback, fix action version

* ignore github extension warning, re-enable action version

* fast bootstrap IAM, untested

* bootstrap stage IAM fixes

* stage 0 tests

* fast stage 1

* tenant stage 1

* minor changes to fast stage 0 and 1

* fast security stage

* fast mt stage 0

* fast mt stage 0

* fast pf
2023-08-20 09:44:20 +02:00
Alejandro Leal ea0de3adbb Fixing some typos 2023-08-18 05:51:00 +00:00
Ludovico Magnocavallo 79373721df
Remove firewall policy management from resource management modules (#1581)
* rename firewall policy module, fix outputs

* add TOC to firewall policy module

* don't depend policy on parent id

* remove firewall policy from resource management modules

* remove factory conditionals

* fast net a and b

* fast stages

* fast tfdoc

* fast tfdoc

* remove unused test

* fix shielded folder blueprint

* fix shielded folder blueprint
2023-08-09 11:23:07 +00:00
Miren Esnaola 6a13742e3c Apigee addons 2023-08-09 08:12:06 +02:00
Eunan Hardy dd3a298892 Moved allow_net_admin to enable_features flag. Bumped provider version to 4.76 2023-08-07 14:27:20 +01:00
Wiktor Niesiobędzki 182ce27143 Add information about required groups 2023-08-06 20:27:59 +02:00
Ludovico Magnocavallo 9c75aa469c
More module descriptions (#1572)
* bigquery dataset

* data catalog policy tag

* net-address

* fix data catalog callers

* bigquery dataset views

* fix data catalog callers

* logging bucket

* net vpn ha
2023-08-06 09:25:45 +00:00
Miren Esnaola fa34499767 Removed unused attribute in variable of ha-vpn-over-blueprint blueprint 2023-08-02 13:02:17 +02:00
javiergp a2eddd0be0
Fixed Cloud Build default bucket name and vpc/subnet names (#1548) 2023-07-31 12:52:36 +02:00
Ludovico Magnocavallo b524aa137c
Peering module refactor (#1547)
* refactor net-vpc-peering module

* hub and spoke peering blueprint

* fast stages

* boilerplate

* fast tfdoc

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2023-07-29 21:33:57 +02:00
Wiktor Niesiobędzki 4998f1d376
Grant IAM rights to service identities in host project (#1542)
* [module/project] Grant IAM rights to service identities based on used services in host project
* [blueprints/factories/project-factory] enable granting IAM permissions in host VPC for service identities directly or by specifying services in use
2023-07-29 20:07:21 +02:00
Ludovico Magnocavallo 5a534edf5c
update and refactor artifact registry module (#1536) 2023-07-28 09:54:36 +00:00
Daniel De Leo 009c6d0f93
Make demo pipeline append into BQ tables (#1533) 2023-07-27 11:38:00 -04:00
Miren Esnaola cacb0c02e2 Refactoring of dns module 2023-07-19 12:57:44 +02:00
dependabot[bot] 69028a801e
Bump semver in /blueprints/serverless/api-gateway/function (#1504)
Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md)
- [Commits](https://github.com/npm/node-semver/compare/v5.7.1...v5.7.2)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-13 08:05:52 +02:00
Miren Esnaola b79261dfa5 Fix in nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg blueprint 2023-07-11 11:46:53 +02:00
Wiktor Niesiobędzki 0b224a7f55 Return only bucket name of composer, not full url to dags folder 2023-07-10 11:20:51 +02:00
Ludovico Magnocavallo d9f7cef1e7
allow using a separate resource for boot disk (#1496) 2023-07-07 15:40:13 +00:00
Ludovico Magnocavallo 0bc6dffce0
Fix and improve quota monitor blueprint (#1488)
* quota monitoring blueprint fixes

* wip

* wip

* quota

* improvements

* improve variables

* refactor http code

* fix http post

* improve logging

* fix project creation, improve readme

* fix test

* Update main.py

* remove unneeded constant

* exit with http error message instead of json when failing to decode api response

* actually do what previous commit wanted :)

* nits
2023-07-03 09:23:49 +02:00
Alejandro Leal d3739cc0ff Update outputs.tf 2023-06-29 22:36:27 -04:00
Alejandro Leal 43b3490ef1 Updating a few files for typos
fast/stages/3-data-platform/dev/README.md
fast/stages/3-data-platform/dev/outputs.tf
CHANGELOG.md
blueprints/data-solutions/data-platform-minimal/README.md
blueprints/data-solutions/data-platform-minimal/outputs.tf
blueprints/data-solutions/data-platform-foundations/README.md
2023-06-29 21:47:17 -04:00
Julio Castillo 0bd3f5ab0e
Merge branch 'master' into kthhrv-patch-1 2023-06-29 18:18:39 +02:00
Ludovico Magnocavallo 981068e442
add region to cf (#1482) 2023-06-29 13:02:56 +02:00
Keith Harvey b203198441
Merge branch 'master' into kthhrv-patch-1 2023-06-29 10:10:14 +01:00
Keith Harvey 912497a158 use key/value and update docs 2023-06-29 09:57:23 +01:00
Keith Harvey f15e2f3509 default to empty map 2023-06-29 09:55:10 +01:00
Keith Harvey e3e05540de change var name, drop default, update docs 2023-06-29 09:45:29 +01:00
lcaggio 026071209c
Minimal Data Platform - Shared VPC (#1475)
* Fix

* Fix dataproc vpc links

* Add missing networkUser role.

* Fix README.
2023-06-28 21:58:03 +02:00
Keith Harvey 52ad1fee34
more linting 2023-06-28 14:07:36 +01:00
Keith Harvey f9742c5405
lint 2023-06-28 14:06:05 +01:00
Keith Harvey 9e8323bd7d
Update 02-composer.tf 2023-06-28 13:56:35 +01:00
Keith Harvey b30efa8bfe
Update variables.tf 2023-06-28 13:56:00 +01:00
Keith Harvey 7cd9083fd7
data-platform-minimal - support web_server_network_access_control 2023-06-28 13:28:28 +01:00
lcaggio 099ad03910
Improve Minimal Data Platform Blueprint (#1473)
* Add SA to access to Curated resources

* Add BQ dataset in the landing project

* Provide example to move data from landing to curated using BQ engine

* Improve diagram
2023-06-28 09:05:48 +02:00
Ludovico Magnocavallo eace97cf4b
Merge branch 'master' into apigateway-blueprint-dependency-update 2023-06-27 08:13:15 +02:00
dependabot[bot] e64c043aaa
Bump semver and @google-cloud/storage in /blueprints/gke/binauthz/image (#1469)
Removes [semver](https://github.com/npm/node-semver). It's no longer used after updating ancestor dependency [@google-cloud/storage](https://github.com/googleapis/nodejs-storage). These dependencies need to be updated together.


Removes `semver`

Updates `@google-cloud/storage` from 5.18.3 to 6.11.0
- [Release notes](https://github.com/googleapis/nodejs-storage/releases)
- [Changelog](https://github.com/googleapis/nodejs-storage/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/nodejs-storage/compare/v5.18.3...v6.11.0)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
- dependency-name: "@google-cloud/storage"
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-26 15:03:47 +02:00
Ludovico Magnocavallo e11d1a8ea8
Merge branch 'master' into apigateway-blueprint-dependency-update 2023-06-26 09:56:52 +02:00
Miren Esnaola 2c7c41b25f Dependencies update for API Gateway blueprint 2023-06-26 09:51:28 +02:00
Ludovico Magnocavallo 638841c8d1
Rename network load balancer modules (#1466)
* update LB modules to new names

* update LB modules names

* update test paths
2023-06-26 07:50:10 +00:00
Wiktor Niesiobędzki 4b6552a6f6 Update README 2023-06-26 09:22:08 +02:00
Wiktor Niesiobędzki 173a00d795 Use EAD per enum https://cloud.google.com/compute/docs/reference/rest/v1/interconnectAttachments#InterconnectAttachment.FIELDS.edge_availability_domain 2023-06-26 09:22:08 +02:00
Wiktor Niesiobędzki 8a6555c581 Add preliminary support for partner interconnect 2023-06-26 09:22:08 +02:00
lcaggio 45adcf1187 First commit. 2023-06-23 15:52:08 +02:00
Alejandro Leal 15ae95df90 Fixing typos
- modules/organization/variables.tf
- modules/organization/README.md
- blueprints/data-solutions/shielded-folder/main.tf
2023-06-23 00:14:02 -04:00
Canburak Tümer adc1ff3fca
match readme groups with variables file (#1455)
I faced error during my deployment even though I created data-engineers group. Then I found out that variables file uses gcp-data-engineers as default. The same is valid for data-security.
2023-06-21 11:51:32 +02:00
lcaggio 261ad646a8
Improve Minimal Data Platform blueprint (#1451) 2023-06-20 16:47:15 +00:00
Keith Harvey dc964411e0
data-platform-minimal - 02-processing.tf typo (#1454)
error is 
```
│ Error: Unsupported attribute
│ 
│   on ../../../../blueprints/data-solutions/data-platform-minimal/02-processing.tf line 46, in locals:
│   46:     ? var.network_config.subnet_self_links.processingestration
│     ├────────────────
│     │ var.network_config.subnet_self_links is object with 2 attributes
│ 
│ This object does not have an attribute named "processingestration".
╵
```

I'm guessing this should be `processing_transformation` the other option is `processing_composer`
2023-06-20 15:26:10 +02:00
Keith Harvey a37b99501e
fix typo data-platform-minimal (#1453)
Fix `container-engine` service agent typo.
2023-06-20 13:12:00 +02:00
Albert Lloveras 7cacc46b4b
fixup(project-factory): Use the correct KMS Service Agents attribute … (#1446)
* fixup(project-factory): Use the correct KMS Service Agents attribute name

* Add new KMS bindings to tests

* Update test resource counts

* Update README.md resource count
2023-06-19 23:53:08 +00:00
Ludovico Magnocavallo bd3296bc46
Split Cloud Function module in separate v1 and v2 modules (#1450)
* split v1

* v2

* blueprints

* remove _http
2023-06-19 12:50:36 +02:00
Miren Esnaola a424635221 Refactored apigee module and adjusted the blueprints accordingly 2023-06-19 09:01:32 +02:00
bluPhy 26e5662e84 Some fixes 2023-06-16 00:01:07 +00:00
Taneli Leppä 81285065ca Added module for Secure Web Proxy. 2023-06-13 08:49:58 +02:00
apichick 9db280bc28
Merge branch 'master' into net-vlan-attachment 2023-06-13 08:20:26 +02:00
Julio Castillo fca74c2258 Fix test 2023-06-12 21:35:35 +02:00
Julio Castillo bd9aa1514e Fix linting errors 2023-06-12 21:25:15 +02:00
Miren Esnaola 500179f02f Move net-dedicated-vlan-attachment module to net-vlan-attachment and added support for Partner interconnect 2023-06-12 14:21:01 +02:00
mark1000 7bdb69f8f8
Add newline to bottom of main.tf 2023-06-08 11:12:01 -07:00
mark1000 ef3d988da6
Merge branch 'master' into vanguard/peering-config 2023-06-08 11:08:53 -07:00
Julio Castillo e900e9c951 Make internal/external addresses optional in compute-vm
Fixes 1431
2023-06-08 14:00:10 +02:00
Mark Schlagenhauf 359b30c141 Updating hub-and-spoke peering blueprint to use HA VPN. 2023-06-07 22:53:45 +00:00
lcaggio 39b27ac25e
Add support for Log Analytics on logging-bucket module and bump provider version (#1423)
* first commit

* Bump provider versions

* Fix tests
2023-06-07 23:23:28 +02:00
lcaggio 7ed197aff4
Fix and improve GCS2BQ blueprint (#1416)
* Fix roles

* Implement Shared VPC roles relying on project service_identity_iam variable

* Group project variable into project_config variable

* Rely on optional variables to avoid try()
2023-06-06 09:06:58 +02:00
Ludovico Magnocavallo 884cb8b4bf
Ensure all modules have an `id` output (#1410)
* net-vpc

* a-d

* complete modules

* fix error
2023-06-02 16:07:22 +02:00
Alejandro Leal 6c11527762 Multiple Updates
modules/net-dedicated-vlan-attachment/variables.tf
modules/net-dedicated-vlan-attachment/README.md
CHANGELOG.md
blueprints/networking/ha-vpn-over-interconnect/README.md
2023-05-31 13:26:22 -04:00
Simone Ruffilli 7f561565e7
HA VPN over Interconnect modules and blueprint (#1390)
Two new modules, net-ipsec-over-interconnect to establish HA VPN over existing VLAN attachments, and net-dedicated-vlan-attachment to create (optionally encrypted) vlan attachments, as well as the ha-vpn-over-interconnect blueprint that shows how to compose such modules to create a 99.9% encrypted interconnect.
2023-05-31 10:53:38 +00:00
Julio Castillo cecbd2072c Fix modules and blueprints tests 2023-05-26 16:38:41 +02:00
Julio Castillo 563b5fa0cb Update net-vpc README 2023-05-26 12:46:16 +02:00
dependabot[bot] d245088fa1
Bump requests in /blueprints/cloud-operations/network-dashboard/src (#1389)
Bumps [requests](https://github.com/psf/requests) from 2.28.1 to 2.31.0.
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](https://github.com/psf/requests/compare/v2.28.1...v2.31.0)

---
updated-dependencies:
- dependency-name: requests
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-23 05:37:15 +00:00
Ana Fernandez del Alamo 4129eb11ae Firewall Validator fix target_service_accounts ref
The Firewall Validator schema configuration contains a field
`target_service_account`. This should be updated to
`target_service_accounts` to match the `google_compute_firewall`
Terraform resource argument:

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall#target_service_accounts
2023-05-22 15:28:48 +01:00
Bharath KKB 47ae6dc7c3
chore: update mlops blueprint metadata (#1382) 2023-05-17 07:41:57 +00:00
lcaggio 40656a23de
Minimal Data Platform - Make components optional (#1380)
make some components optional: Composer and Data Proc history server.
2023-05-16 14:08:03 +02:00
Alejandro Leal 6a89d71e96
Update to multiple README.md (#1379)
- blueprints/data-solutions/data-platform-foundations/README.md
- blueprints/factories/project-factory/README.md
- modules/net-ilb-l7/README.md
- modules/project/README.md
2023-05-16 06:11:34 +00:00
Alejandro Leal 31c1c8534a Updates to blueprints/data-solutions/vertex-mlops
Updates to blueprints/data-solutions/vertex-mlops
2023-05-15 19:41:55 -04:00
Alejandro Leal a0beec747e README to blueprints/data-solutions/shielded-folder 2023-05-15 17:46:48 -04:00
Alejandro Leal 11f7f3efed Reverted variables.tf metadata.yaml README.md to commit 85407109c8 2023-05-15 16:45:13 -04:00
Alejandro Leal 65e6ef98fa Reverted README.md to commit 85407109c8 2023-05-15 16:34:18 -04:00
Alejandro Leal a2a2942e2c Update README.md 2023-05-15 14:50:47 -04:00
Alejandro Leal 87cd83f5c0 Several updates
Several updates
2023-05-13 23:51:46 -04:00
Jay Schwerberg 20b8002602
feat(net-cloudnat): add config_port_allocation configuration block 2023-05-11 12:20:43 -07:00
lcaggio 3cc6c71e96
Add Minimal Data Platform blueprint (#1362)
Minimal Data Platform blueprint
2023-05-08 10:25:06 +02:00
Julio Diez 404920658c Adapt image width 2023-05-06 14:05:47 +02:00
Julio Diez 5454817919 Update tftest values in README 2023-05-06 14:01:28 +02:00
Julio Diez b9e7bf7aaa Add Shared VPC use case to README 2023-05-06 13:55:25 +02:00
Julio Diez eadc1c22a4 New use case: Cloud Run services in service projects
Shared VPC Ingress feature (public preview) allows to call a Cloud
Run service revision from the Shared VPC network that it is connected
to.
2023-05-06 13:03:20 +02:00
Jack P 491b52f023
update variables files for gke nodepool taints (#1358)
* update variables files for gke node config taints to allow passing of node objects

* forgot to run terraform fmt..

* update module docs
2023-05-05 19:42:00 +02:00
Julio Castillo 05ed1bf12f Add blueprint metadata validation tool 2023-05-05 16:57:48 +02:00
lcaggio 4d66623de1
Fix Shielded Folder - VertexML interoperability (#1355)
* Fix Shielded Folder - VertexML interoperability

* Fix README outputs

* Adding prefix for keyrings and update output
2023-05-05 07:54:57 +00:00
Miren Esnaola 75e0a092aa fix in IAM binding of Apigee BigQuery analytics blueprint 2023-05-03 18:12:13 +02:00
Ludovico Magnocavallo 75cc2f3d7a
FAST: shorten stage 3 prefixes, enforce prefix length in stage 3s (#1346)
* shorten stage 3 prefixes, enforce prefix length in stage 3s

* tfdoc

* tfdoc
2023-05-03 07:39:41 +02:00
Bob Idle 94589efc34
Fix external documentation links (#1347) 2023-05-02 05:26:58 +00:00
Bharath KKB 1547a55e1f
chore: update metadata schema (#1345) 2023-04-28 17:14:20 -05:00
Miren Esnaola 2637d1e9aa Fix because of changes in the cloud functions module and the Apigee analytics export schema 2023-04-27 10:26:45 +02:00
Julio Castillo 1e0b1da0d2 Add directory to vertex-mlops blueprint metadata 2023-04-27 08:52:24 +02:00
Julio Castillo bca8a33f1e Fix typo and blueprint module name 2023-04-24 20:36:50 +02:00