Commit Graph

60 Commits

Author SHA1 Message Date
Ludovico Magnocavallo 3d41d01efc
FAST plugin system (#1266)
* plugin folder, gitignore, serverless connector example

* add support to fast plugin variables and outputs to tfdoc

* rename folder, READMEs

* add variable description

* show diffs

* check documentation, use multiple files

* debug check doc

* try a different glob

* debug tfdoc names

* more debug

* and even more debug

* fix gitignore

* fix links

* support extra files in tests

* fix fixture, switch stage 2 peering to new tests

* tfdoc

* Allow globs in extra files

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2023-03-24 12:28:32 +00:00
Ludovico Magnocavallo 5fb17cb3ac
Widen scope for prod project factory SA to dev (#1263)
* restrict storage role on outputs bucket for stage SAs

* grant prod project factory SA authority over prod and dev org policies

* network stages delegated grants on dev to prod pf SA

* security grants to prod pf SA on dev

* tfdoc

* tests
2023-03-17 16:24:55 +00:00
Ludovico Magnocavallo be06554bba
Simplify VPN implementation in FAST networking stages (#1228)
* peering stage

* fix link, toc

* vpn stage

* fix link

* nva stage

* fix examples and test

* separate envs stage

* tfdoc
2023-03-09 17:57:44 +01:00
Julio Castillo e33caf0059 Fix tests 2023-03-07 17:52:00 +01:00
Julio Castillo a5e905cb80 Update remaining org policies 2023-02-21 15:49:16 +01:00
Ludovico Magnocavallo 8708f490ce
Allow configuring regions from tfvars in FAST networking stages (#1137)
* configurable regions

* vpn, tests

* tfdoc

* separate envs

* nva

* test resources

* add new custom role for tenant network service accounts

* allow setting firewall policy name in networking stages

* fix stage links script

* set custom role to tenant networking service account

* rename tenant stage 1 provider files

* remove extra file

* fix peering and vpn

* tfdoc

* fix variable order

* tests
2023-02-08 09:59:43 +01:00
Ludovico Magnocavallo 5453c585e0
FAST multitenant bootstrap and resource management, rename org-level FAST stages (#1052)
* rename stages

* remove support for external org billing, rename output files

* resman: make groups optional, align on new billing account variable

* bootstrap: multitenant outputs

* tenant bootstrap stage, untested

* fix folder name

* fix stage 0 output names

* optional creation for tag keys in organization module

* single tenant bootstrap minus tag

* rename output files, add tenant tag key

* fix organization module tag values output

* test skipping creation for tags in organization module

* single tenant bootstrap plan working

* multitenant bootstrap

* tfdoc

* fix check links error messages

* fix links

* tfdoc

* fix links

* rename fast tests, fix bootstrap tests

* multitenant stages have their own folder, simplify stage numbering

* stage renumbering

* wip

* rename tests

* exclude fast providers in fixture

* stage 0 tests

* stage 1 tests

* network stages tests

* stage tests

* tfdoc

* fix links

* tfdoc

* multitenant tests

* remove local files

* stage links command

* fix links script, TODO

* wip

* wip single tenant bootstrap

* working tenant bootstrap

* update gitignore

* remove local files

* tfdoc

* remove local files

* allow tests for tenant bootstrap stage

* tenant bootstrap proxies stage 1 tfvars

* stage 2 and 3 service accounts and IAM in tenant bootstrap

* wip

* wip

* wip

* drop multitenant bootstrap

* tfdoc

* add missing stage 2 SAs, fix org-level IAM condition

* wip

* wip

* optional tag value creation in organization module

* stage 1 working

* linting

* linting

* READMEs

* wip

* Make stage-links script work in old macos bash

* stage links command help

* fix output file names

* diagrams

* fix svg

* stage 0 skeleton and diagram

* test svg

* test svg

* test diagram

* diagram

* readme

* fix stage links script

* stage 0 readme

* README changes

* stage readmes

* fix outputs order

* fix link

* fix tests

* stage 1 test

* skip stage example

* boilerplate

* fix tftest skip

* default bootstrap stage log sinks to log buckets

* add logging to tenant bootstrap

* move iam variables out of tenant config

* fix cicd, reintroduce missing variable

* use optional in stage 1 cicd variable

* rename extras stage

* rename and move identity providers local, use optional for cicd variable

* tfdoc

* add support for wif pool and providers, ci/cd

* tfdoc

* fix links

* better handling of modules repository

* add missing role on logging project

* fix cicd pools in locals, test cicd

* fix workflow extension

* fix module source replacement

* allow tenant bootstrap cicd sa to impersonate resman sa

* tenant workflow templates fix for no providers file

* fix output files, push github workflow template to new repository

* remove try from outpout files

* align stage 1 cicd internals to stage 0

* tfdoc

* tests

* fix tests

* tests

* improve variable descriptions

* use optional in fast features

* actually create tenant log sinks, and allow the resman sa to do it

* test

* tests

* aaaand tests again

* fast features tenant override

* fast features tenant override

* fix wording

* add missing comment

* configure pf service accounts

* add missing comment

* tfdoc

* tests

* IAM docs

* update copyright

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2023-02-04 15:00:45 +01:00
Ayman Farhat edd3a82453
Include cloudbuild API in project module (#1116)
* Include cloudbuild API in project module

* Increase number of resources
2023-01-27 21:38:01 +01:00
Julio Castillo 09ad53000e Remove recursive_e2e_plan_runner 2022-12-18 14:00:20 +01:00
Julio Castillo be0e807435 Bring back `tests` key in test yaml spec 2022-12-06 00:06:29 +01:00
Julio Castillo 589f7a5c2f Simplify yaml test spec 2022-12-06 00:06:29 +01:00
Julio Castillo 34f01762c3 Simplify fast bootstrap test 2022-12-06 00:06:29 +01:00
Julio Castillo 2af4a826fa Initial FAST bootstrap fixture 2022-12-06 00:06:29 +01:00
Julio Castillo b4d3aa2055 Migrate organizations tests 2022-12-06 00:06:29 +01:00
Julio Castillo 8631d698cb Reorder fixture parameters 2022-12-06 00:06:29 +01:00
Julio Castillo 354ab110f8 Simplify path handling 2022-12-06 00:06:29 +01:00
Julio Castillo 181533786b remove key from fast values inventory 2022-12-06 00:06:29 +01:00
Julio Castillo 0619b35ae6 Fix fast test 2022-12-06 00:06:29 +01:00
Julio Castillo dc1fda0fd8 First tests using fast 2022-12-06 00:06:29 +01:00
Julio Castillo 8c43b72dd4 Remove stale xmark from parellel testing attempt 2022-12-06 00:06:29 +01:00
Ludovico Magnocavallo a9c47681d8
Refactor vps-sc module for Terraform 1.3 (#963)
* wip

* example tests

* module tests

* streamline example

* fast

* tfdoc

* use collections.Counter in tests
2022-11-10 19:34:45 +01:00
Ludovico Magnocavallo fc7bf40e69
Initial replacement for CI/CD stage (#903)
* github extra stage

* remove original cicd stage

* allow setting commit attributes via variabes

* remove reference to deleted stage

* optional repo creation, documentation
2022-10-23 19:52:45 +02:00
Ludovico Magnocavallo 4b798fb34d
Run tests in parallel using `pytest-xdist` (#881)
* test terraform cache

* try pytest-xdist

* revert cache changes

* extend to other jobs

* change dist, bump processes to 4

* revert

* mark tests

* run init to prime providers cache

* prime providers cache

* prime providers cache for all jobs

* add local provider to versions

* remove leftover code
2022-10-14 14:56:16 +02:00
Ludovico Magnocavallo e8056577ce
Refactor GKE nodepool and blueprints (#875)
* first shot, untested

* example tests working

* module tests

* work on gke blueprints

* multitenant fleet doc examples

* fix gke hub doc examples

* blueprint tests

* move master range to vpc config

* fast stage 3 gke test

* tfdoc

* bump provider versions

* and bump provider again to latest
2022-10-12 12:59:36 +02:00
Simone Ruffilli 24c3ffe66b
FAST: Separate network environment (#566) 2022-10-10 11:50:07 +02:00
Ludovico Magnocavallo 78d1a09aeb
Enable FAST 00-cicd provider test (#865)
* enable fast 00-cicd provider test

* don't overwrite version files in CI

* change provider pinning for all tests in CI file
2022-10-07 13:20:56 +02:00
Ludovico Magnocavallo a0171b2c49
Bump terraform required version (#864)
* bump terraform required version

* fix test

* debug test

* debug test

* disable test

* fix CI file, bump terraform action version
2022-10-07 12:51:56 +02:00
Julio Castillo 7e9173d35d Create fully recursive e2e test runner 2022-09-12 13:58:11 +02:00
Julio Castillo 72da12e377 Rename gke-serverless to gke and add test for fast gke stage 2022-09-12 09:56:25 +02:00
Julio Castillo 52e7e0517a Merge branch 'master' into fast/gke2 2022-08-24 15:37:15 +02:00
Stefan Tomm 5b7f6ab91a
fix data-platform-dev folder in stage 03-data-platform (#774) 2022-08-16 09:36:24 +02:00
Ludovico Magnocavallo 0bec03b0a0 Merge remote-tracking branch 'origin/master' into fast/gke2 2022-08-03 20:42:09 +02:00
Ludovico Magnocavallo 9b6262de41
FAST: move region trigrams to a variable in network stages (#765)
* add region trigram variable

* debug check documentation

* fix linting
2022-08-03 11:36:27 +02:00
Julio Castillo 07c5da4283 Fix tests 2022-07-12 12:39:00 +02:00
Ludovico Magnocavallo dfb447a2c4
Merge branch 'master' into fast/gke2 2022-06-23 13:50:28 +02:00
Ludovico Magnocavallo ed4b301a17 align subnets between vpn and peering network stages 2022-06-23 09:05:20 +02:00
Ludovico Magnocavallo 666d8a469c
Merge branch 'master' into fast/gke2 2022-06-23 08:34:09 +02:00
Lorenzo Caggioni ceb611bb81 Remove custom role and rely on conditions. 2022-06-23 07:04:35 +02:00
lcaggio 5c653638bc
Merge branch 'master' into lcaggio/data-p-200622 2022-06-22 16:13:28 +02:00
Lorenzo Caggioni bcfa324825 Fix test 2022-06-20 22:39:29 +02:00
Lorenzo Caggioni 7625773dea Fix tests 2022-06-20 22:05:22 +02:00
Lorenzo Caggioni 41d8847a60 Fix test 2022-06-20 17:56:36 +02:00
Ludovico Magnocavallo 528219bbf3 fix stage1 tests 2022-06-16 22:03:08 +02:00
Julio Castillo f9b808b4bc Fix permissions and update NVA and peering net stages for gke 2022-06-08 11:42:04 +02:00
Julio Castillo 96b01cf68c Fix networking-vpn tests 2022-06-08 11:42:04 +02:00
Ludovico Magnocavallo ff25809020
Fix tfdoc parsing on newllines in outputs (#649)
* fix

* use the right (ugly) Python formatter

* boilerplate on new file

* and do the actual tfdoc run on the original outputs file
2022-05-05 14:43:38 +02:00
Ludovico Magnocavallo 725f7effce
Initial MVP for CI/CD (#608)
* preliminary support for wif in stage 0

* IAM wif role

* IAM wif role TODO

* add support for external SA IAM to SA module

* add name output to SA module

* separate cicd SA

* tfdoc

* GITLAB principal (untested)

* make GCS name output static

* outputs bucket

* fix stage 1 test

* tweak outputs

* tfdoc

* move wif_pool to automation variable

* add support for top-level and repository providers

* add missing boilerplate

* fix branchless principal

* initial workflow

* symlink provider template in stages

* remove service accounts from stage 0 cicd tfvars

* add cicd interface variable to resman stage

* fix cicd variable in resman stage

* better condition on outputs_location

* fix last change

* change outputs_location type

* revert outputs_location change

* split outputs in stage 0

* update ci/cd temporary notes

* rename additive IAM resource in SA module

* split outputs in stage 1

* remove unused locals

* fix stage 1 tests

* tfdoc

* Upload action files to outputs_bucket

* Fix tests and README

* rename template, streamline outputs

* local templates and gcs output for all stage 2

* add workflows to local output files

* Use lowercase WIF providers everywhere

* Bring back suffix for workflow files

* Remove unused files

* Update READMEs

* preliminary CI/CD implementation for stage 1

* fix stage 1

* stage 1 cicd

* tfdoc

* fix tests

* readme and links for cicd and wif

* refactor wif providers

* refactor cicd for stage 1

* fix stage 1

* wif org policies

* split identity provider configuration from cicd

* add type attribute to cicd repositories

* valid cicd repositories have a workflow template

* refactor stage 01

* fix stage 01 tests

* minimal CI/CD documentation

* better check_links error reporting

* fix links

* Added Gitlab specific configurations

Set the default issuer_uri for Gitlab. Added allowed audiences to OIDC configuration.

* Fixed TF formatting in identity providers.

* Changing identity provider audience to null

Changing identity provider audience to default to null.

* add instructions for renaming workflows

* address Julio's comments

Co-authored-by: Julio Castillo <jccb@google.com>
Co-authored-by: alexmeissner <alexmeissner@google.com>
2022-04-12 08:17:27 +02:00
Ludovico Magnocavallo a22cf80b60
Fix PSA (#584)
* Fix PSA

* fix typo

* fix vpn/peering diff

* output filename on failed checksum test
2022-03-17 11:09:02 +01:00
Julio Castillo a988107006 Compare md5 sums too 2022-03-01 14:36:40 +01:00
Julio Castillo a8dd5ae7cc Add test to ensure parity between 02-networking-[vpn/peering] 2022-03-01 14:36:40 +01:00