Commit Graph

706 Commits

Author SHA1 Message Date
Alejandro Leal 87cd83f5c0 Several updates
Several updates
2023-05-13 23:51:46 -04:00
Julio Castillo d20a078134 Cloud NAT rules support 2023-05-12 16:20:38 +02:00
Julio Diez 252be12bd4 Update README showing the Shared VPC use case 2023-05-05 19:38:34 +02:00
Julio Diez 9eea6e3bbc Update pytest inventory file to meet module change 2023-05-05 19:36:46 +02:00
Prabha Arya 23299f7bd7
Add cloud dataplex module (#1308)
* Add dataplex module

* fix dataplex test

* resolve comments

* python test removed

* Change variable desc

* refactor variables

* fix typos

* fix assets & zones resources

* fix linting error

* fix tests

* fix typo
2023-05-05 09:26:46 +02:00
Ludovico Magnocavallo efb0ebe689
Switch FAST networking stages to network policies for Google domains (#1352)
* peering stage implementation

* vpn stage implementation

* tfdoc

* tests

* add most supported google domains

* align all net stages

* add support for factory to DNS response policy module

* use dns policy factory in network stages

* boilerplate
2023-05-04 07:38:40 +02:00
Julio Castillo ee4a908b10 Remove last remaining legacy fixture 2023-04-25 15:12:32 +02:00
Julio Castillo 20ccf7416f Enable parallel tests 2023-04-25 15:12:11 +02:00
Julio Castillo b097d297ff Extend tests to use lockfile if available
This commints allows the test framework to use a lockfile generated
within the tools/lockfile directory. This allows reusing the provider
cache and also running tests in parallel.
2023-04-25 15:12:11 +02:00
Julio Castillo e61df0aa3b Deprecate plan runner fixture and all its variants 2023-04-21 17:52:30 +02:00
Julio Castillo df7cf3d278 Migrate compute-mig tests 2023-04-21 17:51:20 +02:00
Julio Castillo cb13d481d6 Migrate cloud-function tests 2023-04-21 17:51:20 +02:00
Julio Castillo eab9053e16 Migrate billing-budget tests 2023-04-21 17:51:20 +02:00
Julio Castillo 1649787905 Migrate kms tests 2023-04-21 17:51:20 +02:00
Julio Castillo 1c7f8f5907 Migrate bigquery-dataset tests 2023-04-21 17:51:20 +02:00
Julio Castillo 230a1034ec Migrate net-address tests 2023-04-21 17:51:19 +02:00
Julio Castillo 924230eb12 Migrate gke-hub tests 2023-04-21 17:51:19 +02:00
Julio Castillo 4abe5e2baf Migrate net-ilb tests 2023-04-21 17:51:19 +02:00
Julio Castillo c819305a42 Migrate apigee tests 2023-04-21 17:51:19 +02:00
Avinash Jha e881537f87
Separating GKE Standard and Autopilot Modules (#1330)
* separating GKE Standard and Autopilot Modules

* Changes for Updating the terraform and provide versions

* Changes for Autopilot Readme

* Changes for Autopilot Variable

* Changes for Autopilot Readme

* Changes for Autopilot Readme

* Changes for Blueprint

* Changes for Blueprint ReadMe

* Changes for gke-standard-cluster dependency

* Changes for gke-standard-cluster in gke-fleet

* Changes for gke-standard-cluster in cluster-mesh-gke-fleet-api

* python formatting

* python formatting

* python formatting

* GKE module naming convention

* Readme Changes

* test module

* Removing comment code from Autopilot
2023-04-21 12:08:13 +00:00
Julio Castillo 4770cb06fd Migrate cloudsql-instance examples 2023-04-17 09:17:54 +02:00
Julio Castillo 461f7780c2 Migrate bigtable-instance tests 2023-04-17 09:17:54 +02:00
Julio Castillo 5f82938739 Migrate cloud-identity-group tests 2023-04-17 09:17:54 +02:00
Julio Castillo 771dd02b18 Migrate logging-bucket tests 2023-04-17 09:17:54 +02:00
Julio Castillo 0150e97e51 Migrate endpoints tests 2023-04-17 09:17:54 +02:00
Julio Castillo 3dd27b6248 Migrate container-registry tests 2023-04-17 09:17:54 +02:00
Julio Castillo 01b9921f5c Migrate PubSub tests 2023-04-17 09:17:54 +02:00
Julio Castillo 1927b138d6 Migrate secret-manager tests 2023-04-17 09:17:54 +02:00
Julio Castillo a2bffc7302 Migrate service-directory tests 2023-04-17 09:17:54 +02:00
Julio Castillo ed49f3db39 Migrate source repository tests 2023-04-17 09:17:54 +02:00
Julio Castillo f5e1717319 Migrate VPC-SC tests to examples 2023-04-17 09:17:54 +02:00
Julio Castillo 2d4bd5a244 Remove net-interconnect-attachment-direct 2023-04-14 11:01:54 +02:00
Julio Castillo eca0a9583e Add inventories to net-vpc-firewall tests 2023-04-12 14:04:58 +02:00
Fawzi 2ff0b47218 create firewall_policy_enforcement_order.yaml test 2023-04-11 23:40:05 +10:00
Ludovico Magnocavallo 865ef463bb
Support new fields in bigquery module, bump provider versions, unpin local provider (#1306)
* add support for new attributes to bigquery module

* bump provider versions
2023-04-05 14:22:52 +00:00
Luca Prete a9cba47ce8
Add FAST stage 2-networking-e-nva-bgp (NVA+NCC)
Co-authored-by: Luca Prete <lucaprete@google.com>
Co-authored-by: Simone Bruzzechesse <bruzzechesse@google.com>
Co-authored-by: Simone Ruffilli <sruffilli@google.com>
2023-04-04 20:41:04 +02:00
Ludovico Magnocavallo bc2d9372aa
Fix urlmap in ILB L7 module (#1299)
* port tests to new format

* add test to catch error and fix urlmap

* boilerplate

* boilerplate
2023-04-03 15:47:37 +02:00
Ludovico Magnocavallo 92b71a5098
Refactor cloud run module to use optionals and support all features (#1293)
* refactor cloud run resource and variables

* vpc connector variables and resource

* refactor env variable, combine and fix env examples

* fix volume example

* fix traffic split example

* fix eventarc/pubsub example

* fix eventarc/audit log example

* fix SA examples, fix examples formatting

* refactor eventarc variable

* tfdoc

* add IAM to example

* add examples for revision annotations and serverless connector

* fix new examples

* remove legacy module tests

* blueprint tests

* Add tests to cloud-run module

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2023-04-01 14:06:29 +02:00
Valerio Ponza 1fb7e114fd fixing tests and refactoring default 2023-03-30 12:47:39 +02:00
Valerio Ponza 935e95de2a add Backup for GKE support 2023-03-30 12:47:39 +02:00
Wiktor Niesiobędzki 9005a51a95 Use example testing instead custom test 2023-03-27 14:13:38 +02:00
Wiktor Niesiobędzki 9a0137bcfc Add test veryfing multiple filenames are used by default 2023-03-27 14:13:38 +02:00
Ludovico Magnocavallo 28da881c3f
DNS Response Policy module (#1276)
* dns response policy module

* tfdoc
2023-03-26 17:42:57 +02:00
Ludovico Magnocavallo 3d41d01efc
FAST plugin system (#1266)
* plugin folder, gitignore, serverless connector example

* add support to fast plugin variables and outputs to tfdoc

* rename folder, READMEs

* add variable description

* show diffs

* check documentation, use multiple files

* debug check doc

* try a different glob

* debug tfdoc names

* more debug

* and even more debug

* fix gitignore

* fix links

* support extra files in tests

* fix fixture, switch stage 2 peering to new tests

* tfdoc

* Allow globs in extra files

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2023-03-24 12:28:32 +00:00
Ludovico Magnocavallo 5fb17cb3ac
Widen scope for prod project factory SA to dev (#1263)
* restrict storage role on outputs bucket for stage SAs

* grant prod project factory SA authority over prod and dev org policies

* network stages delegated grants on dev to prod pf SA

* security grants to prod pf SA on dev

* tfdoc

* tests
2023-03-17 16:24:55 +00:00
Ludovico Magnocavallo 8a8b7ea35f
Add support for `iam_additive` and simplify factory interface in net VPC module (#1259)
* initial implementation, no tests

* change interface, align tests

* add examples ToC

* fix variable type, test module-level variable
2023-03-17 10:12:34 +00:00
Ludovico Magnocavallo 112d9a8d9c
Allow using existing boot disk in compute-vm module (#1241)
* allow using existing boot disk in compute-vm module

* allow setting initialize params to null

* tests

* fast

* blueprints
2023-03-12 10:53:59 +01:00
Ludovico Magnocavallo 6ba0f8b0ba
allow overriding name in net-vpc subnet factory (#1239) 2023-03-11 09:30:42 +01:00
Ludovico Magnocavallo be06554bba
Simplify VPN implementation in FAST networking stages (#1228)
* peering stage

* fix link, toc

* vpn stage

* fix link

* nva stage

* fix examples and test

* separate envs stage

* tfdoc
2023-03-09 17:57:44 +01:00
Julio Castillo 165515f9fd Update contributing guide with new test framework 2023-03-09 14:41:18 +01:00
Julio Castillo e33caf0059 Fix tests 2023-03-07 17:52:00 +01:00
Ludovico Magnocavallo 8fc9549c58
add support for proxy and psc subnets to module factory (#1211) 2023-03-05 17:08:43 +01:00
Julio Castillo 67bc391b66 Add test for #1197 2023-03-01 09:58:50 +01:00
Dan Farmer 52eb83758f Fix compute-vm:CloudKMS test for provider>=4.54.0
* TF provider >= 4.54.0 now returns `rsa_encrypted_key` for
  `google_compute_disk.disks["attached-disk"]` (see
  hashicorp/terraform-provider-google#4448)
* Add this field to expected model to fix test assertion failure
* Update required TF provider to 4.55.0 (latest) since the assertion
  will now fail with <4.54.0, which do not return `rsa_encrypted_key`
  * Updated the whole repo on advice from @ludoo
2023-02-28 15:10:22 +00:00
Valerio Ponza 52c16e973b fixing workloadidentity for anthos service mesh - authority was not created 2023-02-27 14:35:49 +00:00
Julio Castillo 6be11dc013 Add empty test in blueprints folder 2023-02-25 19:26:11 +01:00
Julio Castillo be60753246 Fix failing tests 2023-02-25 19:26:11 +01:00
Julio Castillo edce6edd28 Update factories and apigee tests 2023-02-25 19:26:11 +01:00
Julio Castillo b186365cab Update data solutions tests 2023-02-25 19:26:11 +01:00
Julio Castillo 3c68780537 Update gke blueprints tests 2023-02-25 19:26:11 +01:00
Julio Castillo a25f83656f Update cloud operations tests 2023-02-25 19:26:11 +01:00
Julio Castillo d71d0c5c22 Update networking bluprints tests 2023-02-25 19:26:11 +01:00
Julio Castillo daccd803ed Update decentralized-firewall tests 2023-02-25 19:26:11 +01:00
Julio Castillo 3c88315f6e Update tests for hub and spoke blueprints 2023-02-25 19:26:11 +01:00
Julio Castillo 7c17af66aa Update asset-inventory-feed-remediation tests 2023-02-25 19:26:11 +01:00
Julio Castillo f777e97cef Update tests terraform_enterprise_wif 2023-02-25 19:26:11 +01:00
Julio Castillo 3e4b6c0639 Update iam-delegated-role-grants tests 2023-02-25 19:26:11 +01:00
Julio Castillo d114a81b08 Update packer_image_builder tests 2023-02-25 19:26:11 +01:00
Julio Castillo 23c564cb74 Update unmanaged-instances-healthcheck tests 2023-02-25 19:26:11 +01:00
Julio Castillo 99e69e319f Update data platform tests 2023-02-25 19:26:11 +01:00
Julio Castillo 3a0f69dc2d Revert examples/conftest.py 2023-02-25 19:26:11 +01:00
Julio Castillo 71e96dfdee Allow blueprints to be tested from README files without examples. 2023-02-25 19:26:11 +01:00
lcaggio 47855cb682
Merge branch 'master' into lcaggio/dp-projectcreate 2023-02-23 11:54:48 +01:00
lcaggio e39be7b01d Fix 2023-02-22 13:02:29 +01:00
lcaggio 2108b4650d Fix Tests, rely on iam additive. 2023-02-22 01:36:01 +01:00
Julio Castillo a5e905cb80 Update remaining org policies 2023-02-21 15:49:16 +01:00
Julio Castillo d3bcf625f9 Update yaml org policies 2023-02-21 15:49:16 +01:00
Julio Castillo 6b767c9035 Simplify org policies data model in resman modules. 2023-02-21 15:49:16 +01:00
Julio Castillo 3085922cea Fix tests 2023-02-20 12:19:36 +01:00
lcaggio f4c1fa6c20 Fix tests. 2023-02-20 00:56:32 +01:00
Aleksandr Averbukh 4ad30b812b Fix tests for tf-cloud-dynamic-credentials blueprint 2023-02-19 01:14:24 +01:00
Aleksandr Averbukh befc73ec12 feat: Update TFC+WIF blueprint with TFC Dynamic Credentials feature. 2023-02-19 00:51:49 +01:00
Wiktor Niesiobędzki e64e8db20d Allow additive IAM grants by robots name
Regreatablly member name will be known after apply, hence changes in the
tests
2023-02-16 14:39:21 +01:00
Julio Castillo d6e3716815 Fix gke-cluster dns config feature
Fixes #1141
2023-02-10 11:14:07 +01:00
Julio Castillo d7b88b7d0e Fix bq factory docs 2023-02-08 18:22:10 +01:00
Ludovico Magnocavallo 8708f490ce
Allow configuring regions from tfvars in FAST networking stages (#1137)
* configurable regions

* vpn, tests

* tfdoc

* separate envs

* nva

* test resources

* add new custom role for tenant network service accounts

* allow setting firewall policy name in networking stages

* fix stage links script

* set custom role to tenant networking service account

* rename tenant stage 1 provider files

* remove extra file

* fix peering and vpn

* tfdoc

* fix variable order

* tests
2023-02-08 09:59:43 +01:00
Julio Castillo bb565b72f1 Add Disk examples 2023-02-07 15:59:29 +01:00
Julio Castillo 1784dd8cad Remove unneeded __init__.py 2023-02-07 15:59:29 +01:00
Julio Castillo 04602353ce New compute-vm examples and tests 2023-02-07 15:59:29 +01:00
lcaggio dd77c2fe79
Merge branch 'master' into lcaggio/shieldedfolder 2023-02-07 07:58:00 +01:00
lcaggio d68bb4b320 Fixes 2023-02-06 21:33:04 +01:00
lcaggio 1e4499c8ab Fixes 2023-02-06 14:58:40 +01:00
Ayman Farhat 02d8d8367a
[Feature] Update data platform blue print with Dataflow Flex template (#1105)
* Add initial dataflow template code + TF infra

* Refactor the datapipeline DAG to use flex template operator, cleanup code

* Remove unneeded bash scripts, update README with manual examples

* Refactor datapipeline_dc_tags.py and include new Flex template

* Update docs to reflect changes

* Remove sub-dependencies and keep apache beam

* Add missing license headers and update tests

* Set resouces to 291 in tests

* Update outputs via tfdoc

* Update with outputs order and tfdoc

* Correct number of resources

* Fix to add region into command from var

* Enable service account impersonation for running builds

* Update example dataflow run command to use orchestrator SA

* Remove hard coded values in example

* Keep original airflow files, add new which use Flex template as example

* Update tests and doc

* Fix number of resources in plan

* Run tfdoc remove files section in README

* Fix number of modules in tfdoc

* Update number of resources

* Add missin service account

* Update DF demo README

* Quick rename

---------

Co-authored-by: lcaggio <lorenzo.caggioni@gmail.com>
Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2023-02-06 07:35:40 +01:00
lcaggio 39ffdcf184
Merge branch 'master' into lcaggio/shieldedfolder 2023-02-05 22:45:14 +01:00
lcaggio 47c75f8862 Implement project_config variable 2023-02-05 21:12:46 +01:00
lcaggio 38fc9e8b9b First commit 2023-02-05 07:52:33 +01:00
lcaggio ce1cc5f75a
Merge branch 'master' into lcaggio/shieldedfolder 2023-02-04 19:42:19 +01:00
Ludovico Magnocavallo 5453c585e0
FAST multitenant bootstrap and resource management, rename org-level FAST stages (#1052)
* rename stages

* remove support for external org billing, rename output files

* resman: make groups optional, align on new billing account variable

* bootstrap: multitenant outputs

* tenant bootstrap stage, untested

* fix folder name

* fix stage 0 output names

* optional creation for tag keys in organization module

* single tenant bootstrap minus tag

* rename output files, add tenant tag key

* fix organization module tag values output

* test skipping creation for tags in organization module

* single tenant bootstrap plan working

* multitenant bootstrap

* tfdoc

* fix check links error messages

* fix links

* tfdoc

* fix links

* rename fast tests, fix bootstrap tests

* multitenant stages have their own folder, simplify stage numbering

* stage renumbering

* wip

* rename tests

* exclude fast providers in fixture

* stage 0 tests

* stage 1 tests

* network stages tests

* stage tests

* tfdoc

* fix links

* tfdoc

* multitenant tests

* remove local files

* stage links command

* fix links script, TODO

* wip

* wip single tenant bootstrap

* working tenant bootstrap

* update gitignore

* remove local files

* tfdoc

* remove local files

* allow tests for tenant bootstrap stage

* tenant bootstrap proxies stage 1 tfvars

* stage 2 and 3 service accounts and IAM in tenant bootstrap

* wip

* wip

* wip

* drop multitenant bootstrap

* tfdoc

* add missing stage 2 SAs, fix org-level IAM condition

* wip

* wip

* optional tag value creation in organization module

* stage 1 working

* linting

* linting

* READMEs

* wip

* Make stage-links script work in old macos bash

* stage links command help

* fix output file names

* diagrams

* fix svg

* stage 0 skeleton and diagram

* test svg

* test svg

* test diagram

* diagram

* readme

* fix stage links script

* stage 0 readme

* README changes

* stage readmes

* fix outputs order

* fix link

* fix tests

* stage 1 test

* skip stage example

* boilerplate

* fix tftest skip

* default bootstrap stage log sinks to log buckets

* add logging to tenant bootstrap

* move iam variables out of tenant config

* fix cicd, reintroduce missing variable

* use optional in stage 1 cicd variable

* rename extras stage

* rename and move identity providers local, use optional for cicd variable

* tfdoc

* add support for wif pool and providers, ci/cd

* tfdoc

* fix links

* better handling of modules repository

* add missing role on logging project

* fix cicd pools in locals, test cicd

* fix workflow extension

* fix module source replacement

* allow tenant bootstrap cicd sa to impersonate resman sa

* tenant workflow templates fix for no providers file

* fix output files, push github workflow template to new repository

* remove try from outpout files

* align stage 1 cicd internals to stage 0

* tfdoc

* tests

* fix tests

* tests

* improve variable descriptions

* use optional in fast features

* actually create tenant log sinks, and allow the resman sa to do it

* test

* tests

* aaaand tests again

* fast features tenant override

* fast features tenant override

* fix wording

* add missing comment

* configure pf service accounts

* add missing comment

* tfdoc

* tests

* IAM docs

* update copyright

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2023-02-04 15:00:45 +01:00
lcaggio f00d4d3edd
Merge branch 'master' into lcaggio/shieldedfolder 2023-02-03 22:17:30 +01:00
javiergp ce1f86d20b
Vertex Pipelines MLOps framework blueprint (#1038)
* First release of the MLOps blueprint
2023-02-02 19:13:13 +01:00