Commit Graph

884 Commits

Author SHA1 Message Date
Julio Castillo fde7b76036
Allow per-module terraform fixtures (#1914)
* Allow terraform fixtures for examples

* Allow defining multiple fixtures, and named fixtures under tests/fixtures/

* Enable e2e for wiktorn

* Fix prepare_files call for e2e

* Move fixture to separate file, fix test

* Revert shallow-copying symlinks, performane penalty - 20%

* Update tfdoc.py to list used fixtures

---------

Co-authored-by: Wiktor Niesiobędzki <wiktorn@google.com>
2023-12-29 09:43:44 +00:00
andybubu 2ad109ae23
Fix variable region (#1953)
This PR changes variable region's default value in example tests to real region value.

Some of the modules parse the region name to decide whether to create regional or zonal resources.
2023-12-28 15:04:15 +01:00
Ludovico Magnocavallo 9d6e61428b
(WIP) Read-only service accounts for automation and CI/CD (#1899)
* add design doc for the new CI/CD sa

* describe the actual implementation

* specify which files will need to be changed

* Update 0-cicd-plan-sa.md

* Update 0-cicd-plan-sa.md

* Update 0-cicd-plan-sa.md

* Update 0-cicd-plan-sa.md

* Update 0-cicd-plan-sa.md

* Update 0-cicd-plan-sa.md

* Update 0-cicd-plan-sa.md

* Fix typo

* stage 0 read-only service accounts

* stage 0 IAM map

* linting

* cicd read-only service accounts

* tweak workflow templates

* roles and github workflow fixes

* tfdoc

* Ad-hoc custom role factory for FAST bootstrap

* use factory variable for custom roles data path

* custom roles factory in org/project modules

* tfdoc

* rename custom roles factory variable, fix gitlab template

* gitlab workflow fixes

* fix merge

* output plan results on failed assertion

* update stage 0 expected values

* data platform branch

* gke

* networking

* security

* project factory

* outputs

* workflow templates

* resman apply fixes

* tfdoc

* fix stage 1 test fixture

* fix gh workflow

* read-only resman sa roles

* fix test

* read-only resman sa roles

* read-only resman sa roles

* read-only resman sa roles

* read-only resman sa roles

* fix test variables

* rename wif principal attribute names

* rename wif principal variables

* multitenant stages

---------

Co-authored-by: Wiktor Niesiobędzki <wiktorn@google.com>
Co-authored-by: Julio Castillo <jccb@google.com>
2023-12-27 11:33:16 +00:00
dibaskar-google 969111f0cf
dns e2e tests (#1944) 2023-12-23 10:29:32 +00:00
Luca Prete 44b1115b9f
Fix GCVE network policy (#1948)
Co-authored-by: Luca Prete <lucaprete@google.com>
2023-12-22 10:29:43 +00:00
Luca Prete 06b2a97291
GCVE: add network policy configuration 2023-12-22 10:02:12 +00:00
Luca Prete c4123044b7
Use new resources in GCVE module, bump up provider versions (#1941) 2023-12-21 13:23:38 +00:00
simonebruzzechesse c9a8d777ba
Add kernels.googleusercontent.com zone in dns response policy (#1940)
* Add kernels.googleusercontent.com zone in dns response policy
* update fast tests
2023-12-20 11:18:11 +01:00
Wiktor Niesiobędzki 0d486fb34e E2E tests fixes 2023-12-19 11:01:03 +01:00
Julio Castillo b6e0557bbb
Simplify organization tags.tf locals (#1932)
* Simplify organization tags.tf locals

* Fix boilerplate

* Override github provider version for tests
2023-12-18 16:09:22 +00:00
simonebruzzechesse c50b732c79
Allow granting network user role on host project from project module and factory (#1930)
* Update shared vpc config for project factory and project module for more granular Shared VPC configuration

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2023-12-15 14:39:21 +01:00
Simone Ruffilli ca3c86cb5c
Add support for policy based routes to net-vpc (#1926)
* Add support for PBR to net-vpc
2023-12-13 15:19:40 +00:00
Wiktor Niesiobędzki c24d023c23 Sync tf version to version used by tests 2023-12-12 09:43:09 +01:00
Ludovico Magnocavallo b65c57bcc2
bump tf version (#1920) 2023-12-12 09:19:46 +01:00
Ludovico Magnocavallo bba814c091
Custom role factories for organization and project modules (#1912)
* backport custom role factories

* backport from fast ci/cd branch

* indent

* tfdoc

* fix module tests
2023-12-11 14:16:39 +00:00
dibaskar-google c65e242685
net_lb_ext module e2e and example testing changes (#1909)
E2E tests for net_lb_ext
2023-12-08 10:04:07 +01:00
Ludovico Magnocavallo f548b65b1c
Add support for subnet-level service network user grants to project module, improve docs (#1907)
* improve project factory example

* light refactor of project modules shared vpc internals and docs

* add support for subnet-level grants on host project
2023-12-07 09:07:48 +00:00
Luca Prete fa7664434b
Enable sole tenancy (`node_affinities`) on compute_vm (#1903) 2023-12-05 18:05:23 +01:00
Luca Prete 7916cd2081
Add IPv6 to HA VPN module + test inventories (#1901)
---------

Co-authored-by: Luca Prete <lucaprete@google.com>
2023-12-04 23:38:41 +01:00
Wiktor Niesiobędzki 4668b90e8a Fix inventory for project 2023-12-03 10:37:23 +00:00
Thangaraju R e2d170c1a6
e2e tests for net-vpc-firewall module (#1896)
e2e tests for net-vpc-firewall module
2023-12-01 13:50:56 +01:00
Thomas Colomb 3a2484843c
artifact-registry: Support cleanup policies (#1891) 2023-12-01 10:33:02 +00:00
Thangaraju R da5371b391
e2e test fix for iam-service-account module (#1894) 2023-12-01 09:23:37 +01:00
Wiktor Niesiobędzki e36b4fcec8
Merge branch 'master' into wiktorn-e2e-setup-fixes 2023-11-30 19:51:03 +01:00
Wiktor Niesiobędzki 6d68b50bc8 Review fixes 2023-11-30 17:12:12 +00:00
Thangaraju Rajasekaran 224b98c786 removed prefix and updated net-vpc shared-vpc for e2e test 2023-11-30 14:03:49 +00:00
Thangaraju Rajasekaran d9cd46d8a7 fixed e2e test for shared-vpc and subnet-iam 2023-11-30 14:03:49 +00:00
Thangaraju Rajasekaran a259d2cbdb formatted the file 2023-11-30 14:03:49 +00:00
Thangaraju Rajasekaran 0af5e31ca3 E2E tests for net-vpc module 2023-11-30 14:03:49 +00:00
Wiktor Niesiobędzki 3557ed8a95 Pin provider version also for setup module to prevent test failures 2023-11-30 07:44:23 +00:00
Wiktor Niesiobędzki c5c127b9df Use TFTEST_E2E_ instead of TF_VAR variables
Use of TF_VAR variables modified results of `tests/examples` and
required setting different environment to run `tests/examples` and
`tests/examples_e2e` tests. No both can be run using the same
environment.
2023-11-30 07:43:46 +00:00
apichick 66bd9d5160 Added workstation-cluster module 2023-11-30 07:02:28 +01:00
Wiktor Niesiobędzki 55f308cbea
Fix failing E2E tests for folders (#1884)
* Run tests requiring uniqueness on org level serially (organization tags, firewall policies)
* make gcs bucket name globally unique

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2023-11-24 08:09:13 +00:00
Ludovico Magnocavallo 637926fb68
Support boot disk KMS key in GKE cluster modules (#1881)
* gke cluster standard

* tfdoc

* gke cluster autopilot

* fix autopilot tags test
2023-11-23 11:52:13 +00:00
Luca Prete 642f65b841
F5 blueprint (#1787)
---------

Co-authored-by: Luca Prete <lucaprete@google.com>
2023-11-22 19:48:13 +01:00
dibaskar-google 2d70bb8db2
E2E tests for folder module (#1876)
E2E tests for folder module
2023-11-22 10:25:11 +01:00
Wiktor Niesiobędzki 28b8edced5 Add end-to-end tests to Cloud Run 2023-11-18 10:00:25 +00:00
Wiktor Niesiobędzki 7dd76c1003 Additional services for Cloud Run 2023-11-18 09:58:51 +00:00
Wiktor Niesiobędzki 86cf5dfc84 Fix reporting non-empty plan after apply 2023-11-17 11:47:44 +00:00
Wiktor Niesiobędzki 6ca688d694 terraform fmt 2023-11-16 11:16:26 +00:00
Wiktor Niesiobędzki 985569fa83 Allow running E2E tests that are skipped in examples 2023-11-15 09:33:03 +00:00
Wiktor Niesiobędzki 1498696b6b JIT service account for storage 2023-11-15 09:33:03 +00:00
Wiktor Niesiobędzki 1fbd018f5f E2E tests for GCS 2023-11-15 09:33:03 +00:00
Wiktor Niesiobędzki 03bf0b15b3
Organization module end-to-end tests (#1860)
* added tag serial to mark tests to be run serially
* always run tests using loadgroup distribution to make use of serial tag
* added end-to-end tests for organization, not adding to custom constraints as the name has to be unique
* fixed granting custom roles created in the same module call
2023-11-14 18:54:59 +01:00
Francesco Spinelli 82c74e4ab6
Dataproc module bug fix (#1848)
* bug fix

* bug fix

* fix dinamic for_each

---------

Co-authored-by: Francesco Spinelli <francesco.spinelli@nttdata.com>
2023-11-09 15:48:29 +00:00
Ludovico Magnocavallo 03937f2b6e
Support multilevel data and allow overriding project id in project factory (#1851) 2023-11-09 08:29:46 +00:00
Wiktor Niesiobędzki ecaa253594
Merge branch 'master' into wiktorn-provider-5.4.0 2023-11-04 08:31:56 +01:00
Wiktor Niesiobędzki aec7fc87c2 Trim down inventories for Cloud SQL 2023-11-04 07:31:33 +00:00
Wiktor Niesiobędzki fe485414e6
Add end-to-end tests for project module (#1823)
* Add end-to-end tests for project module
* Add inventory to data tests
* Add files to end-to-end test cases
* Review fixes - use named groups

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2023-11-03 18:04:19 +01:00
Ludovico Magnocavallo de0325b3a3
Avoid map-related casting errors in project factory (#1836)
* try to repro pf example error

* repro

* repro

* pf fix

* remove extra file

* FAST stage
2023-11-02 08:24:50 +01:00
cmalpe b8bb000073
Merge branch 'master' into cmalpe/kms-import-job 2023-10-30 20:32:50 +05:30
Luca Prete 7c6726e79b
[net-address] enable ipv6 (#1821)
---------

Co-authored-by: Luca Prete <lucaprete@google.com>
2023-10-28 15:36:30 +02:00
Ludovico Magnocavallo b29987bb61
Merge branch 'master' into cmalpe/kms-import-job 2023-10-28 01:29:28 +02:00
apichick 022b9f5060 Added iam_bindings and iam_bindings_additive to apigee module 2023-10-27 18:22:07 +02:00
Wiktor Niesiobędzki 5b17c11d01
Merge branch 'master' into fix/ai-models-support 2023-10-27 10:22:47 +02:00
cmalpe b0e5231f50 Merge branch 'master' into cmalpe/kms-import-job 2023-10-27 07:21:36 +00:00
Chaitanya Malpe 8a76b10161 added test for import job 2023-10-27 11:53:35 +05:30
Ludovico Magnocavallo d0b1ced280
fix logic for default source range in firewall ingress rules (#1815) 2023-10-26 15:25:36 +00:00
Ewa Wojtach 6fc960ea0b empty gpu sharing config fix 2023-10-26 07:23:13 +02:00
Ewa Wojtach 33ce0e1db5
AI models support (#1750)
* nodepool config

* added gpu driver configuration

* documentation update

* regenerated docs

* review comments

* review comments

* blocks structure

* documentation update

* test fix and doc update

* review comments

* doc

* Extend inventory

* Update README

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
Co-authored-by: Julio Castillo <jccb@google.com>
2023-10-25 09:42:36 +00:00
Luca Prete a23b3d62ae
net-lb-ext: add support for multiple forwarding rules (IPs) and dual-stack (IPv4/IPv6) 2023-10-21 18:19:18 +02:00
Simone Ruffilli 6d89b88149
versions.tf maintenance + copyright notice bump (#1782)
* Bump copyright notice to 2023

* Delete versions.tf on blueprints

* Pin provider to major version 5

* Remove comment

* Fix lint

* fix bq-ml blueprint readme

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
Co-authored-by: Julio Castillo <jccb@google.com>
2023-10-20 18:17:47 +02:00
Wiktor Niesiobędzki d07daf966a
End-to-end tests for terraform modules (#1751)
Add end-to-end tests (apply, plan, destroy) for examples.

When run, `tests/examples_e2e`:
1. Create an environment for tests to run (folder, project vpc network) 
2. For each marked example (with `e2e` tftest directive), run apply, plan, destroy
3. Verify:
* no failure in apply
* empty plan after apply
* no failure during destroy
4. When all tests are done, destroy test environment

More details in `tests/examples_e2e/README.md`
2023-10-20 09:59:52 +02:00
luigi-bitonti 4e439720aa
Added ProtectedApplication feature to GKE Backup (#1774)
* Added ProtectedApplication feature to GKE Backup

* Fixed location name and added example

* Modified test module

* Changed test

* Changed test

* Changed test

* Restore old "all_namespaces" logic

* Bug fix

* Ran fmt on README example

* Modified variable structure

* Fix test

* Fix
2023-10-19 19:54:22 +02:00
Ludovico Magnocavallo e0d84fb10b
add sink for workspace logs (#1780) 2023-10-19 14:51:01 +00:00
Ludovico Magnocavallo e7e188818a
Add service usage consumer role to IaC SAs, refactor delegated grants in FAST (#1773)
* add serviceusage role to iac sas, refactor delegated grants

* fix test

* tfdoc
2023-10-18 12:18:31 +00:00
Ludovico Magnocavallo 02ccc576f5
fix resource manager tag bindings in compute-vm module (#1771) 2023-10-18 09:24:00 +00:00
Wiktor Niesiobędzki c21fa4558f
Remove incompatible balancing_mode (#1769)
## net-lb-int
* Fix error on apply of example:
```
Error creating RegionBackendService: googleapi: Error 400: Invalid value for field 'resource.backends[0].balancingMode': 'UTILIZATION'. Balancing mode must be CONNECTION for an INTERNAL backend service., invalid
```
* remove unused `balancing_mode` variable, as only one value is possible anyhow

## net-lb-ext
* update in the `backends` description

## net-lb-proxy-int
* update in the `backends` description

## net-lb-app-int
* added validation of `balancing_mode`
* fixed other validations

## net-lb-app-ext
* added validation of `balancing_mode`
* fixed other validations
* removed validation for `locality_lb_policy` as this variable is not used in this module

Closes: #1767
2023-10-18 08:11:32 +02:00
Luca Prete 6c48512f7e
[#1764] net-lb-int: add support for dual stack and multiple forwarding rules 2023-10-17 09:30:34 +00:00
jeroenmonteban f464557525
Add autoclass to GCS (#1757)
* Add autoclass to GCS

* Fix linting

* Make autoclass block dynamic

* Fix syntax

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2023-10-16 07:45:10 +00:00
Ludovico Magnocavallo 252127bde5
Billing account module (#1743)
* initial untested draft

* readme and tests

* folder module tfdoc

* remove redundant billing cost manager role in fast stage 0

* fix FAST test
2023-10-15 15:02:50 +00:00
Julio Castillo 9ab3b49f69
Add PSA peered domains support to `net-vpc` (#1741)
* Add PSA peered domains support to `net-vpc`

* Fix tests
2023-10-06 15:31:32 +00:00
Julio Castillo dfc5023e0b
Make deletion protection consistent across all modules (#1735)
* Expose deletion_protection in GKE modules

* Make deletion protection consistent across all modules

* Add deletion_protection option to blueprints

* Fix blueprints tests

* Fix types

* Update READMEs

* Fix dp readme

* Fix cmek blueprint default deletion_protection

* Fix blueprints tests
2023-10-05 17:31:07 +02:00
Luca Prete 20dd3df27c
Support multiple protocols (L3_DEFAULT) through net-ilb-int 2023-10-04 08:30:11 +00:00
apichick e7c31cb83a
Merge branch 'master' into gcs-iam 2023-10-04 08:15:00 +02:00
Miren Esnaola c10eb72a8f Changes in IAM to GCS module 2023-10-04 08:13:45 +02:00
Julio Castillo 4b15605711
Fix dnssec keys lookup (#1728)
* Fix dnssec keys lookup

* Fix DNS examples

* Fix FAST and blueprints resource counts
2023-10-03 21:37:21 +02:00
Ludovico Magnocavallo 789328ff5a
Bump provider versions to v5.0.0 (#1724)
* bump provider versions to 5.0.0

* fix cloud run, logging and vpc-sc

* Fix secret manager

* fix gke nodepool

* fix gke multitenant stage and blueprint

* Moving alloydb module to experimental.

* Add project to bare resources in examples

* tfdoc

* fix svpc blueprint test

* Revert "fix svpc blueprint test"

This reverts commit 14f02659098070136e64ead600580dd52c23c339.

* Fix GKE peering project

* Disable tests in alloydb module

* Bring back secret ids in secret manager tests

* Remove duplicate key

* last push

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2023-10-03 12:15:36 +00:00
Ludovico Magnocavallo aa5d883deb
add support for org policies to project factory (#1722) 2023-10-02 14:13:56 +00:00
Anmol Sachdeva 06d4bf5788 fix: apigee unit tests 2023-09-29 16:40:48 +00:00
luigi-bitonti be5d9b8fe0
Add bug fix in bucket local variable (#1709)
Fix bucket name reference in Cloud Functions if bucket is created and prefix is used.
2023-09-28 10:17:53 +00:00
Wiktor Niesiobędzki 71def9e6dd Allow partial specification for deeply nested inventories
For objects that are not simple attributes of the resource (i.e.
objects, lists) allow specifing subset of values that needs to be
verified.

In case of lists, count of list objects must match, but if values are
objects / dicts, they can be specified as empty dicts which should pass
all the checks, as no keys are specified.
2023-09-26 11:22:35 +02:00
Wiktor Niesiobędzki 1c2f0c6b14 Tests for secrets for CF 2023-09-26 11:22:35 +02:00
Julio Castillo 1dfa72cadf
Define and adopt standard IP ranges for FAST networking (#1697)
* Define and adopt standard IP ranges for FAST networking

This PR documents and adopts a consistent IP address plan for FAST
networking stages

Fixes #1644

* Fix documented aggregated ranges for FAST

* Fix tests

* Fix ip ranges in documentation

* Fix NVA stages README
2023-09-21 14:27:53 +00:00
Jason Steenblik 79b0dc9751
Add deletion_protection_enabled attribute to cloudsql-instance to enable the GCP native deletion protection flag (#1696) 2023-09-20 15:09:38 +02:00
Ludovico Magnocavallo df5daab6cc
Allow using no service account in compute-vm (#1692)
* module and tests

* align blueprints

* tfdoc

* remove stale variables

* fix blueprint

* variable description
2023-09-19 16:56:51 +00:00
Ludovico Magnocavallo 96c28e605b
fix repd disk attachment in compute-vm module (#1688) 2023-09-18 13:02:28 +00:00
Julio Castillo 9d61c6e26d Update IAM for pubsub topics and subscriptions 2023-09-17 00:21:36 +02:00
Julio Castillo da883bab8c Update kms module key-level IAM 2023-09-17 00:21:36 +02:00
Julio Castillo 4d5df5aeb2 Fix tests 2023-09-15 14:57:32 +02:00
Ludovico Magnocavallo bd7b9b4dd1
Merge branch 'master' into jccb/fix-subnet-iam-bindings 2023-09-15 14:31:28 +02:00
Julio Castillo 616c6fbc48 Fix subnet iam_bindings to use arbitrary keys 2023-09-15 14:30:27 +02:00
Oliver Frolovs 6eb862a775
GKE cluster modules: add optional kube state metrics (#1682)
* `gke-cluster-standard`: add optional kube state metrics
* `gke-cluster-autopilot`: add optional kube state metrics

* FAST: add kube state metrics support for GKE
* blueprints/gke: add kube state metrics support

* Bump up the provider version to `v4.82.0`
2023-09-15 12:18:45 +01:00
Oliver Frolovs b3dc91b5cd
Upgrades to `monitoring_config` in `gke-cluster-*`, docs update, and cosmetics fixes to GKE cluster modules (#1680)
* gke-cluster-standard: upgrade `monitoring_config` to use object style. Add tests.
* gke-cluster-standard: update docs

* gke-cluster-autopilot: move gateway_api_config block (cosmetic change)
* gke-cluster-autopilot: update docs and fix typos

* Update blueprints due to `monitoring_config` changes in `gke-cluster-standard`.
* Update FAST due to `monitoring_config` changes in `gke-cluster-standard`.
* Update docs for affected blueprints and FAST stages
2023-09-14 23:25:57 +01:00
Oliver Frolovs 05c0195a06
GKE Autopilot module: add network tags (#1675)
* gke-cluster-autopilot: add support for network tags

* gke-cluster-autopilot: add validation for network tags

* gke-cluster-autopilot: expand README and fix some typos

* gke-cluster-autopilot: fix Cloud DNS section in README

Removed a reference to Standard clusters and updated the section to include a warning because the new versions of Autopilot clusters can only use Cloud DNS and it is pre-configured by default so the example in the README does not apply to them.

* gke-cluster-autopilot: cosmetic fixes in README

* gke-cluster-autopilot: rollback validation on network tags var

* gke-cluster-autopilot: rollback docs string update for network tags var

* gke-cluster-autopilot: rollback some updates to README

* gke-cluster-autopilot: remove dead code

* gke-cluster-autopilot: add a tftest for network tags

* gke-cluster-autopilot: fix a tftest
2023-09-14 09:34:51 +00:00
apichick 008abe4ddc
Merge branch 'master' into apigee-module-fix 2023-09-13 15:25:51 +02:00
Miren Esnaola fdb133f1de Fixed, added back environments to each instance, that way we can also create instances for already existing environments 2023-09-13 14:43:37 +02:00
Dave Gulli 09e6e8f4d2 created passing test, fixed logic 2023-09-13 16:49:14 +10:00
Dave Gulli f95e3ad9a2 fixed 2023-09-12 17:17:10 +10:00
Dave Gulli 3479c40735 fixing tests 2023-09-12 15:53:28 +10:00