Commit Graph

33 Commits

Author SHA1 Message Date
Julio Castillo c42c4c141f
GKE stateful blueprints (#2059)
* var definitions

* skeleton, untested

* fix errors, test with existing cluster

* test vpc creation, todo notes

* initial variables for AR and image

* initial variables for AR and image

* Add support for remote repositories to artifact-registry

* Add support for virtual repositories to artifact-registry

* Add support for extra config options to artifact-registry

* artifact registry module: add validation and precondition, fix tests

* ar module id/name

* registry

* service accoutn and roles

* fetch pods, remove image prefix

* small changes

* use additive IAM at project level

* use additive IAM at project level

* configmaps

* manifests

* fix statefulset manifest

* service manifest

* fix configmap mode

* add todo

* job (broken)

* job

* wait on manifest, endpoints datasource

* fix job

* Fix local

* sa

* Update README.md

* Restructure gke bp

* refactor tree and infra variables

* no create test

* simplify cluster SA

* test cluster and vpc creation

* project creation fixes

* use iam_members variable

* nits

* readme with examples

* readme with examples

* outputs

* variables, provider configuration

* variables, manifests

* start cluster job

* fix redis cluster creation

Co-authored-by: Julio Castillo <juliocc@users.noreply.github.com>

* Revert changes in autopilot cluster

* Default templates path, use namespace for node names

* Update readmes

* Fix IAM bindings

* Make STABLE the default release channel

* Use Cloud DNS as default DNS provider

* Allow optional Cloud NAT creation

* Allow backup agent and proxy only subnet

* Work around terraform not short-circuiting logical operators

* Rename create variables to be more consistent with other blueprints

* Add basic features

* Update variable names

* Initial kafka JS

* Move providers to a new file

* Kafka / Strimzi

* First possibily working version for MySQL (with a lot of todo's left)

* Explicitly use proxy repo + some other fixes

* Strimzi draft

* Refactor variables, use CluterIP as pointer for mysql-router for bootstraping

* Validate number of replicas, autoscale required number of running nodes to n/2+1

* Use seaprate service for bootstrap, do not recreate all resources on change of replicas count as the config is preserved in PV

* Test dual chart kafka

* Update chart for kafka

* Expose basic kafka configuration options

* Remove unused manifest

* Added batch blueprint

* Added README

* switch to kubectl_manifest

* Add README and support for static IP address

* Move namespace creation to helm

* Interpolate kafka variables

* Rename kafka-strimzi to kafka

* Added TUTORIAL for cloudshell for batch blueprint

* deleted tutorial

* Remove commented replace trigger

* Move to helm chart

* WIP of Cloud Shell tutorial for MySQL

* Rename folders

* Fix rename

* Update paths

* Unify styles

* Update paths

* Add Readme links

* Update mysql tutorial

* Fix path according to self-link

* Use relative path to cwd

* Fix service_account variable location

* Fix tfvars creation

* Restore some fixes for helm deployment

* Add cluster deletion_prevention

* Fixes for tutorial

* Update cluster docs

* Fixes to batch tutorial

* Bare bones readme for batch

* Update batch readme

* README fixes

* Fix README title for redis

* Fix Typos

* Make it easy to pass variables from autopilot-cluster to other modules

* Add connectivity test and bastion host

* updates to readme, and gpu fix

* Add versions.tf and README updates

* Fix typo

* Kafka and Redis README updates

* Update versions.tf

* Fixes

* Add boilerplate

* Fix linting

* Move mysql to separate branch

* Update cloud shell links

* Fix broken link

---------

Co-authored-by: Ludo <ludomagno@google.com>
Co-authored-by: Daniel Marzini <44803752+danielmarzini@users.noreply.github.com>
Co-authored-by: Wiktor Niesiobędzki <wiktorn@google.com>
Co-authored-by: Miren Esnaola <mirene@google.com>
2024-02-08 18:28:41 +00:00
Tahar JEGHAM 1c99bae649
feat: enable mtls on external application application load balancer (#1979)
* feat: enable mtls on external application application load balancer

* refactor: move variable inside https_proxy_config block

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-01-17 06:24:54 +00:00
simonebruzzechesse 717f7ecad1
Blueprints naming convention update (#1942)
* update network-dashboard and quota-monitoring naming convention
2023-12-21 17:02:25 +01:00
Teodelas 0f502a8cfb
Fix modules to support new Apigee X environment types (#1841)
* Update main.tf

* Update variables.tf

* Update main.tf

Updated environment members to be alphabetical order

* fixed linting and terraform fmt

* removed venv

* removed venv directory

---------

Co-authored-by: Teo De Las Heras <teodlh@google.com>
2023-11-06 09:56:03 +01:00
Alejandro Leal c2c7ec4f1d Adding exception for idx 2023-08-18 06:35:09 +00:00
Miren Esnaola a39fa7ca64 Added autopilot blueprint 2023-03-07 15:37:20 +01:00
Ludovico Magnocavallo 5453c585e0
FAST multitenant bootstrap and resource management, rename org-level FAST stages (#1052)
* rename stages

* remove support for external org billing, rename output files

* resman: make groups optional, align on new billing account variable

* bootstrap: multitenant outputs

* tenant bootstrap stage, untested

* fix folder name

* fix stage 0 output names

* optional creation for tag keys in organization module

* single tenant bootstrap minus tag

* rename output files, add tenant tag key

* fix organization module tag values output

* test skipping creation for tags in organization module

* single tenant bootstrap plan working

* multitenant bootstrap

* tfdoc

* fix check links error messages

* fix links

* tfdoc

* fix links

* rename fast tests, fix bootstrap tests

* multitenant stages have their own folder, simplify stage numbering

* stage renumbering

* wip

* rename tests

* exclude fast providers in fixture

* stage 0 tests

* stage 1 tests

* network stages tests

* stage tests

* tfdoc

* fix links

* tfdoc

* multitenant tests

* remove local files

* stage links command

* fix links script, TODO

* wip

* wip single tenant bootstrap

* working tenant bootstrap

* update gitignore

* remove local files

* tfdoc

* remove local files

* allow tests for tenant bootstrap stage

* tenant bootstrap proxies stage 1 tfvars

* stage 2 and 3 service accounts and IAM in tenant bootstrap

* wip

* wip

* wip

* drop multitenant bootstrap

* tfdoc

* add missing stage 2 SAs, fix org-level IAM condition

* wip

* wip

* optional tag value creation in organization module

* stage 1 working

* linting

* linting

* READMEs

* wip

* Make stage-links script work in old macos bash

* stage links command help

* fix output file names

* diagrams

* fix svg

* stage 0 skeleton and diagram

* test svg

* test svg

* test diagram

* diagram

* readme

* fix stage links script

* stage 0 readme

* README changes

* stage readmes

* fix outputs order

* fix link

* fix tests

* stage 1 test

* skip stage example

* boilerplate

* fix tftest skip

* default bootstrap stage log sinks to log buckets

* add logging to tenant bootstrap

* move iam variables out of tenant config

* fix cicd, reintroduce missing variable

* use optional in stage 1 cicd variable

* rename extras stage

* rename and move identity providers local, use optional for cicd variable

* tfdoc

* add support for wif pool and providers, ci/cd

* tfdoc

* fix links

* better handling of modules repository

* add missing role on logging project

* fix cicd pools in locals, test cicd

* fix workflow extension

* fix module source replacement

* allow tenant bootstrap cicd sa to impersonate resman sa

* tenant workflow templates fix for no providers file

* fix output files, push github workflow template to new repository

* remove try from outpout files

* align stage 1 cicd internals to stage 0

* tfdoc

* tests

* fix tests

* tests

* improve variable descriptions

* use optional in fast features

* actually create tenant log sinks, and allow the resman sa to do it

* test

* tests

* aaaand tests again

* fast features tenant override

* fast features tenant override

* fix wording

* add missing comment

* configure pf service accounts

* add missing comment

* tfdoc

* tests

* IAM docs

* update copyright

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2023-02-04 15:00:45 +01:00
Miren Esnaola 33f5607ff1 Apigee hybrid on GKE 2023-01-04 10:53:49 +01:00
Miren Esnaola 0235690aaa Moved apigee bigquery analytics blueprint, added apigee network patterns 2022-12-23 14:38:15 +01:00
Miren Esnaola 3501d20785 Apigee module and blueprint 2022-11-16 19:49:26 +01:00
Maurizio Noseda Pedraglio cebc9360f8 fixes to dashboard and statc routes category metric name 2022-10-21 17:43:00 +02:00
Miren Esnaola 5c9f336441 Example of a multi-cluster mesh on GKE configuring managed control plane using the Fleet API 2022-08-08 16:32:52 +02:00
Miren Esnaola 68e56058ab AD FS example 2022-07-11 09:52:35 +02:00
Ayman Farhat 54d805dac0
New example for a data playground Terraform setup (#655)
* Initial commit for adding a sample data playground

* Update README

* Add license boilerplate to variables.tf

* Apply linting rules

* rename var to ptoject_id, create prefix var, remove extra zone var

* Adds the option for using an existing project by default

* Bundles all VPC related variables in a single vpc_config variable of type object

* Add encryption_key usage example + policy_boolean

* Add tests, apply linting and todos for upcoming PRs

* Update variables in readme

* Fix formatting via fmt

* Rename test dir to fix module conflict issue

* Add high level diagram and sort vars/outputs by alphabetical

* Modify diagram and update main README under data examples with link / summary

* Line break

* Use png in diagram

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2022-07-10 09:27:18 +02:00
Miren Esnaola 2e9fdea1a4 Binary authorization module and example 2022-06-17 16:09:12 +02:00
Lorenzo Caggioni 6eeda3da7a Add KMS support 2022-04-13 11:09:34 +02:00
Julio Castillo 2644627837 Remove broken link and ignore globals in fast stages 2022-04-12 21:33:03 +02:00
Ludovico Magnocavallo 725f7effce
Initial MVP for CI/CD (#608)
* preliminary support for wif in stage 0

* IAM wif role

* IAM wif role TODO

* add support for external SA IAM to SA module

* add name output to SA module

* separate cicd SA

* tfdoc

* GITLAB principal (untested)

* make GCS name output static

* outputs bucket

* fix stage 1 test

* tweak outputs

* tfdoc

* move wif_pool to automation variable

* add support for top-level and repository providers

* add missing boilerplate

* fix branchless principal

* initial workflow

* symlink provider template in stages

* remove service accounts from stage 0 cicd tfvars

* add cicd interface variable to resman stage

* fix cicd variable in resman stage

* better condition on outputs_location

* fix last change

* change outputs_location type

* revert outputs_location change

* split outputs in stage 0

* update ci/cd temporary notes

* rename additive IAM resource in SA module

* split outputs in stage 1

* remove unused locals

* fix stage 1 tests

* tfdoc

* Upload action files to outputs_bucket

* Fix tests and README

* rename template, streamline outputs

* local templates and gcs output for all stage 2

* add workflows to local output files

* Use lowercase WIF providers everywhere

* Bring back suffix for workflow files

* Remove unused files

* Update READMEs

* preliminary CI/CD implementation for stage 1

* fix stage 1

* stage 1 cicd

* tfdoc

* fix tests

* readme and links for cicd and wif

* refactor wif providers

* refactor cicd for stage 1

* fix stage 1

* wif org policies

* split identity provider configuration from cicd

* add type attribute to cicd repositories

* valid cicd repositories have a workflow template

* refactor stage 01

* fix stage 01 tests

* minimal CI/CD documentation

* better check_links error reporting

* fix links

* Added Gitlab specific configurations

Set the default issuer_uri for Gitlab. Added allowed audiences to OIDC configuration.

* Fixed TF formatting in identity providers.

* Changing identity provider audience to null

Changing identity provider audience to default to null.

* add instructions for renaming workflows

* address Julio's comments

Co-authored-by: Julio Castillo <jccb@google.com>
Co-authored-by: alexmeissner <alexmeissner@google.com>
2022-04-12 08:17:27 +02:00
apichick 00473fda64 API gateway module and example that creates a multi-region deployment for API gateway 2022-02-22 12:00:44 +01:00
Ludovico Magnocavallo 369253cc3a write providers to a common folder 2022-02-16 00:14:49 +01:00
Ludovico Magnocavallo 8d5116fbe8 gitignore 2022-02-16 00:11:14 +01:00
Simone Ruffilli c5ba74374b
Update subnets (#466) 2022-01-28 08:26:03 +01:00
Ludovico Magnocavallo e866db2c88 Merge Fabric FAST (#435)
Co-authored-by: Simone Ruffilli <sruffilli@google.com>
Co-authored-by: Julio Castillo <jccb@google.com>
2022-01-19 14:19:39 +01:00
Julio Castillo f0773d4883 Simplify tests by figuring out fixture dir automatically
We always use the same directory for terraform fixtures, so it's quite
easy to figure out its path from a pytest fixture by inspecting the
stack. This commit implements this functionality and decreases the
amount of boilerplate needed to write a test.

(Ported from fast)
2022-01-11 11:54:13 +01:00
Mikołaj Stefaniak b481d9baff
Packer example changelog (#316)
* Updated top-level README and CHANGELOG to reflect packer-builder example

* Added packer related files to gitignore
2021-10-04 17:35:31 +02:00
Julio Castillo 94b1a439ee Fix IAM bindings for logging sinks
- Move to non-authoritative bindings to allow multiple sinks to write to
  the same destination
- Allow automatically adding logging bucket IAM bindings
2021-03-31 09:59:28 +02:00
Ludovico Magnocavallo fe71be72d3
Aurelien's SVPC DNS example (#186)
* Cloud DNS and Shared VPC (#184)

* Cloud DNS and Shared VPC module to allow application teams to have their own Cloud DNS configuration.

* Cleaning up README.md

* Improving Formating.

* Adding license to all .tf files.

* Removing dead code.

* Moving this example into the Cloud Operations folder.

* Using fabric resources and refactoring. Only the 'test.example' file is not using the proper modules now.

* normalize README, use autogenerated vars table, add types to variables

* refactor

* simple tests

* add diagram, update READMEs

Co-authored-by: Aurélien Legrand <aurelien.legrand01@gmail.com>
2021-01-11 11:57:57 +01:00
Ludovico Magnocavallo 14fe796885
Add missing role to GKE nodepool service account, refactor test runners and parallelize tests (#164)
* add missing role to GKE nodepool service account

* refactor plan test runners

* remove spurious print statements from test

* use concurrency via locking the fixture dir

* add filelock to test requirements

* fix pytest arg in cloud build

* and yet another dep and args fix

* fix e2e runner, use correct runner in env e2e test

* revert parallel test changes, split modules and environments triggers

* I should stop experimenting in PRs
2020-11-09 21:32:09 +01:00
Ludovico Magnocavallo 1b5af2d923
Cloud Asset feed operations example (#110)
* first working example for asset inventory feeds

* move tf files out of the tf folder

* add input/outputs to README

* smaller diagram

* use narrow scoped service account for cf, account for gke tags in code

* Update README.md

* new top-level folder README

* Update README.md

* add TODO for DNS example in operations README

* fix README conflict

* Update README.md

* Update README.md

* update diagram

* cloud shell

* cloud shell

* Update README.md

* rename outputs, first complete README draft

* Update main.py

* Update README.md

* Update README.md

* better error handling in the cloud function

* remove branch from cloud shell link
2020-07-05 19:08:24 +02:00
Ludovico Magnocavallo ae2e4dc3ad interpolate vpc name in routes 2020-06-12 12:07:28 +02:00
Ludovico Magnocavallo d948668a6a add .terraform.tfstate.lock.info to git ignore file 2020-05-10 09:36:24 +02:00
Ludovico Magnocavallo e553da6464
Org environments module tests and tests CI configuration
* sample test

* bump tftest library version in requirements

* org env: initial tests

* Fix boilerplate

* add boilerplate to skel testing cloud build

* use package structure for tests

* use package structure for tests

* add bolerplate to package init files

* wip build config for tests

* initial test cloud build configuration

* dummy change

* add comment to test configuration file

* add missing test docstring

* remove debug logging config from tests fixture
2019-09-11 13:36:07 +02:00
Ludovico Magnocavallo e4fa25f22d
Organization sample: environments (Almanac bootstrap port)
* add gitignore file

* data and infra skeletons

* org skeleton

* org environments sample skeleton

* Organization teams sample skeleton.

* Organization env sample GCS for tf state.

* org env: service accounts and GCS roles

* org env: folders

* org env: audit export

* org env: shared project

* org env: switch to released 3.1.0 project module version

* rename organization top-level folder to organization-bootstrap

* org env: use folders module with better outputs, module outputs

* org env: switch the service accounts module to v2.0.0.

* Merge ludo's branch from forked repo (#2)

* org env: update gcs, sa, project modules

* Use correct folder ID in format  without  prefix

* org env: update folders module version, improve comments

* org env: initial work on README, diagram, add variable for xpn roles

* org env: update roles in README, backend file

* org env: README changes

* org env: README changes

* org env: README changes

* org env: README changes

* org env: add IAM variables for audit and shared projects

* org env: address README TODOs

* org env: minor README changes

* org env: minor README changes

* org env: minor README changes

* org env: simplify the sample's README by moving general considerations into the section README

* Org examples README changes

* org env: simplify initial state management, add comment for shared folder

* org env: change state instructions to copy and rename instead of renaming backend file

* org env: add a section in the README file to explain shared services, and detail options

* fix comment for shared services project
2019-09-07 05:44:24 +02:00