Commit Graph

40 Commits

Author SHA1 Message Date
Ludovico Magnocavallo ff25809020
Fix tfdoc parsing on newllines in outputs (#649)
* fix

* use the right (ugly) Python formatter

* boilerplate on new file

* and do the actual tfdoc run on the original outputs file
2022-05-05 14:43:38 +02:00
Lorenzo Caggioni ca655265f1 Add python module 2022-04-22 00:31:28 +02:00
Ludovico Magnocavallo 725f7effce
Initial MVP for CI/CD (#608)
* preliminary support for wif in stage 0

* IAM wif role

* IAM wif role TODO

* add support for external SA IAM to SA module

* add name output to SA module

* separate cicd SA

* tfdoc

* GITLAB principal (untested)

* make GCS name output static

* outputs bucket

* fix stage 1 test

* tweak outputs

* tfdoc

* move wif_pool to automation variable

* add support for top-level and repository providers

* add missing boilerplate

* fix branchless principal

* initial workflow

* symlink provider template in stages

* remove service accounts from stage 0 cicd tfvars

* add cicd interface variable to resman stage

* fix cicd variable in resman stage

* better condition on outputs_location

* fix last change

* change outputs_location type

* revert outputs_location change

* split outputs in stage 0

* update ci/cd temporary notes

* rename additive IAM resource in SA module

* split outputs in stage 1

* remove unused locals

* fix stage 1 tests

* tfdoc

* Upload action files to outputs_bucket

* Fix tests and README

* rename template, streamline outputs

* local templates and gcs output for all stage 2

* add workflows to local output files

* Use lowercase WIF providers everywhere

* Bring back suffix for workflow files

* Remove unused files

* Update READMEs

* preliminary CI/CD implementation for stage 1

* fix stage 1

* stage 1 cicd

* tfdoc

* fix tests

* readme and links for cicd and wif

* refactor wif providers

* refactor cicd for stage 1

* fix stage 1

* wif org policies

* split identity provider configuration from cicd

* add type attribute to cicd repositories

* valid cicd repositories have a workflow template

* refactor stage 01

* fix stage 01 tests

* minimal CI/CD documentation

* better check_links error reporting

* fix links

* Added Gitlab specific configurations

Set the default issuer_uri for Gitlab. Added allowed audiences to OIDC configuration.

* Fixed TF formatting in identity providers.

* Changing identity provider audience to null

Changing identity provider audience to default to null.

* add instructions for renaming workflows

* address Julio's comments

Co-authored-by: Julio Castillo <jccb@google.com>
Co-authored-by: alexmeissner <alexmeissner@google.com>
2022-04-12 08:17:27 +02:00
Lorenzo Caggioni 9011e370b8 Fix linting 2022-04-04 00:45:27 +02:00
Lorenzo Caggioni 6f00caaaab Handle Service Identity SA. Update IAM.md 2022-04-04 00:38:34 +02:00
Luca Prete 29b469133f Delete unused line 2022-03-28 23:14:16 +02:00
Luca Prete 4e8ef9d770 Simplify code 2022-03-28 20:36:51 +02:00
Luca Prete 3f3c7f9402 Providing default value in function arg for option external 2022-03-28 20:33:16 +02:00
Luca Prete 603732f7e8 Add option to test external links 2022-03-28 18:53:32 +02:00
Julio Castillo ba30ceb5c1
Simplify link checker and recurse into all elements with children (#599) 2022-03-28 16:54:06 +02:00
Luca Prete 0837d41bb1 Fix python syntax 2022-03-28 09:47:24 +02:00
Luca Prete 2bb9e3f64e [#597] Fix link_checker 2022-03-28 09:43:00 +02:00
John Tracy 22f03bd5d5
Update README for bigquery module (#569)
* Update README

Some variables that no longer exist. Table format is broken for view/table access.

* Remove floating role

* Fix README.md

* Fix tfdoc

* Fix tfdoc

Co-authored-by: Julio Castillo <jccb@google.com>
Co-authored-by: lcaggio <lorenzo.caggioni@gmail.com>
Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2022-03-23 16:34:45 +01:00
Simone Ruffilli fb90500adc
FAST: add 02-networking-peering stage. (#561)
* Networking with peering - initial commit

* README and tests

* Linter fixes

* Linter fixes

* Linter fixes

* Update README.md

* split out vpn/peering in separate files so the rest is identical

* fix dns for vpn and peering

* update tfdoc to support multiple outputs/variables files

* add peering variables

* update tfdoc for modules

* make it easier to spot errored docs

* fix doc

* yapf

* fix permadiff in firewall module source ranges

* FAST: Networking: DNS Fixes

* FAST: ability to toggle on-prem vpn creation

* FAST: fix outputs

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2022-02-28 12:27:27 +01:00
Ludovico Magnocavallo 38518ec7d5 skip links in tfdoc file list 2022-02-27 10:36:03 +01:00
Julio Castillo e7dd12fa29 Reformat all code using yapf 2022-02-14 12:27:42 +01:00
Ludovico Magnocavallo fbf4d29c88
Fix FAST names (#539)
* name checker, initial names fix

* fix name parser

* sdbx -> sbox

* explicit names

* show total name length

* check names in linting action

* validate prefix length

* fix typo
2022-02-12 13:29:22 +01:00
Ludovico Magnocavallo c8cbf9b7d7
Add utility to extract IAM bindings from state (#503)
* add utility to extract IAM bindings from state

* TODO

* TODO

* output one row per role/member pairs

* sorting

* rename, delete spurious file

* generalize output functions

* generate IAM.md

* fix org id

* add link to role in markdown

* fix org id

* replace organization ids
2022-02-05 17:34:43 +01:00
Ludovico Magnocavallo 02e8a3927d
CI link checker (#494)
* CI link checker

* fix link errors

* fix ci

* wildcard on *.md
2022-02-04 13:28:07 +01:00
Julio Castillo b11367e970 Support nullable variables in tfdoc 2022-01-28 19:09:40 +01:00
Ludovico Magnocavallo cb9340fd4f exclude in check docs 2022-01-27 17:31:08 +01:00
Julio Castillo 31f625f149 Link vars and outputs from README 2022-01-22 14:15:16 +01:00
Ludovico Magnocavallo 9a533180a0
Refactor project module, support per-file tags in tfdoc (#450)
* add support for in-doc tfdoc options overrides

* clean up project module

* add file description tags

* only output module and resource columns in tfdoc file table if they exist

* update fast READMEs

* fix check docs
2022-01-22 11:34:18 +01:00
Ludovico Magnocavallo 56877421ff
fix etra newlines in tfdoc, fixes #445 (#446) 2022-01-21 16:16:28 +01:00
Ludovico Magnocavallo e866db2c88 Merge Fabric FAST (#435)
Co-authored-by: Simone Ruffilli <sruffilli@google.com>
Co-authored-by: Julio Castillo <jccb@google.com>
2022-01-19 14:19:39 +01:00
Simone Ruffilli ee25965c89
Copyright bump (#410) 2022-01-01 15:52:31 +01:00
Ludovico Magnocavallo 2bc5f7d33c
update tfdoc (#404) 2021-12-30 10:56:19 +01:00
Ludovico Magnocavallo bf1e2e3bed backport tfdoc and check doc fixes 2021-12-22 15:44:43 +01:00
Ludovico Magnocavallo a09485201a update documentation check 2021-12-22 15:35:12 +01:00
Ludovico Magnocavallo 173def118f
fix tfdoc replacement (#398) 2021-12-21 11:51:15 +01:00
Ludovico Magnocavallo 1ac3fe4460
New tfdoc version (#396)
* update tfdoc

* rewrite check docs, refactor tfdoc replace, regenerate modules READMEs

* remove dead code from check docs

* do not fail on missing variable files in check docs

* fix typos
2021-12-21 08:51:51 +01:00
Julio Castillo 1d13e3e624 Add more validations to linter
- Ensure all variables and outputs are sorted
- Ensure all variables and outputs have a description
- Add data-solutions/data-platform-foundations to linter

Fix all modules to follow these new conventions.
2021-10-08 18:26:04 +02:00
Julio Castillo 49a1999bac Allow escaped quotes inside variable and output descriptions 2021-10-08 18:25:23 +02:00
Julio Castillo cb7c65135e
Update CI processes (#296)
- Upgrade to latest terraform version (1.0.4)
- Remove tflint from linting pipeline (was not doing anything)
- Add terraform fmt check to linting pipeline
- Pass all code through terraform fmt
2021-08-12 17:30:53 +02:00
Julio Castillo 1e11c670f5 Update copyright to 2021 2021-02-15 09:38:10 +01:00
Julio Castillo c31764fa7e
Add lint step to check module READMEs (#154)
* Add documentation check to lint process

* Split get_variables_outputs

* Install tfdoc dependencies in linting pipeline

* doc lint variant

* use new script args in ci build config

* exit early if doc does not exist

* align README docs with variables

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2020-11-07 09:12:41 +01:00
Luca Prete f2c80e17f1
[#138] Update copyright headers to 2020 (#139)
Co-authored-by: Luca Prete <lucaprete@google.com>
2020-09-23 11:07:03 +02:00
Ludovico Magnocavallo c486bfc66f
Merge development branch (#44)
* VPN-HA module initial commit

* Added readme for net-vpn-ha module

* Update readme, add simple description

* Merge new modules list and environments foundation example (#30)

* gke-cluster

* net-vpc module and tests

* add TODO to net-vpc module

* add minimal README files with input/output variables to gke and net-vpc modules

* BigQuery Module (#24)

* Bigquery Module

* Added README file

* Added type hints

* gke-cluster

* net-vpc module and tests

* add TODO to net-vpc module

* add minimal README files with input/output variables to gke and net-vpc modules

* BigQuery Module (#24)

* Bigquery Module

* Added README file

* Added type hints

* GCS module

* net vpc module: improve secondary range outputs

* net vpc module: add serve project registration

* project module

* move bigquery module to not-ready folder

* folders module

* rename project module's iam variables

* slight tweak to folder module outputs

* gcs module

* simplify net-vpc module variables

* fix module tests configurations, fix net-vpc module tests

* add pydoc utility

* add/update module READMEs

* add/update module READMEs

* add/update module READMEs

* improve variable type summary generation in tfdoc

* tfdoc: add support for replacing doc in README.md files

* improve module READMEs

* net-vpc-firewall module

* add support for sensitive output attribute in tfdoc

* remove empty function from tfdoc

* render variable type as code in tfdoc

* update module READMEs

* net address module

* net cloudnat module

* remove redundant variable from net-cloudnat module

* vpc module: add support for peering, use network name as subnet name prefix

* net-vpn-static module

* net-vpn-static module README

* net-vpn-static module README

* tfdoc: fix error on undeclared variable type

* dns module

* set version for all modules

* kms module (untested)

* change kms key self links output to map, fix gcs and kms iam variable descriptions

* fix kms module

* update kms module readme

* simplify local iam pairs in modules

* service accounts module (unfinished)

* work on service accounts module

* project module: add gcr service account

* project module: update outputs in README

* first working version of the iam service accounts module

* iam service accounts module: extra checks in locals

* modules/net-cloudnat: reorder variables

* modules/net-vpn-dynamic: initial import (untested)

* modules/net-vpn-dynamic: first working version

* modules/net-vpn-dynamic: add outputs for auto-created router

* modules/net-vpn-dynamic: update README

* modules/net-[vpn,cloudnat]: clean up variable,s remove prefix

* modules/net-vpn-dynamic: add advertisement configuration to tunnel bgp peer, refactor variables

* tfdoc: add tooltips for variable types and defaults

* modules: update README variables and outputs

* tfdoc: improve variable default rendering

* modules: update README variables and outputs

* modules/net-vpc: minimal output refactoring

* modules/vm-cos: initial import, base resources working, no outputs

* modules/vm-cos: add variable descriptions

* tfdoc: fix parsing in type and default blocks

* modules/vm-cos: fix README

* tfdoc: fix parsing in type and default blocks

* modules/vm-cos: fix README

* modules/compute-vm: initial working import (not fully tested)

* modules/vm-cos: move to not-ready

* tfdoc: fix variable defaults formatting

* modules: update README files with tfdoc fixes

* modules: add initial examples

* gke-nodepool: initial import, untested

* gke nodepool: add README, fix location variable, set node count default to 1

* gke cluster: fix private cluster variables

* gke nodepool: fix README title

* gke cluster: add output for cluster location

* gke nodepool: add missing variables for project id and cluster name, remove default from location variable, fix gke version assignment

* gke nodepool: update README

* net-cloudnat: fix router name when creating default router

* fix variables used for address and router optional creation

* vpn dynamic: fix README

* modules/net-vpn-dynamic: fix router name output

* modules/compute-vm: remove unused variable

* modules/compute-vm-cos-coredns: initial import

* Update foundations modules versions (#26)

* update foundations modules versions

* update Terraform version to v0.12.19 in CI test configuration

* backport tfdoc from Ludo's branch (#27)

* Update docs using tfdoc format (#28)

* update README files

* set all types on variables

* foundations/environments: move log filter to a variable, use org for xpn by default

* foundations/environments: do not use liens by default

* modules/ntp-vpc: better shared_vpc_host variable description

* modules/logging-sinks: initial version

* modules/logging-sinks: streamline options in sinks variable

* modules/compute-vm-cos-coredns: add support for additional files

* modules/folders: rename from 'folder'

* modules/logging-sinks: fix circular dependencies and improve variables

* modules/project: remove extra variable

* modules/bigquery: new module with dataset support only

* foundations/environments: refactor using local modules

* modules/bigquery: better variables, README description and example

* modules: fix a few READMEs

Co-authored-by: Julio Castillo <juliocc@gmail.com>

* modules/net-vpc: README description and examples

* modules/net-vpc: tweak README description and examples

* modules/net-vpc: tweak README description and examples

* modules/net-vpc-firewall: change tag-based rule default ranges, improve README examples and description

* modules/compute-vm: README changes

* modules/compute-vm: use an object for the service account variable, update README

* modules/compute-vm: update README variables table

* modules/compute-vm: add TODO list to README

* modules/compute-vm: add TODO list to README

* modules/compute-vm: add outputs for service account

* modules/net-cloudnat: README

* modules/net-cloudnat: README

* modules/net-cloudnat: add router_create variable

* modules/compute-vm: simplify service account variables

* modules/net-vpn-dynamic: fix README example, use local secret for both empty string and null

* modules/net-vpn-dynamic: improve README example

* modules/gke-cluster: minimal README tweaks

* modules/kms: fix ephemeral keys resource name

* modules/iam-service-accounts: add storage roles

* modules/gke-nodepool: fix node default scopes

* New project variable to prevent deletion of default network (#32)

* New project variable to prevent deletion of default network

This is a workaround to fix
terraform-google-modules/cloud-foundation-fabric#31 while the GCP
terraform provider is fixed

* Add TODOs to remove workarounds in the project module

* Fix Cloud Build files

* modules/gke-nodepool: add monitoring scope to defaults

* modules/iam-service-accounts: add support for IAM bindings onthe service accounts

* playground module in sandbox, remove not ready modules

* Fix ci configurations in development branch (#33)

* try fixing ci confgurations

* add exclusion match to ci boilerplate check

* add skip boilerplate comment to compute-vm-cos-coredns template fragment

* modules/gke-cluster: fix boilerplate in outputs

* Simplify tests, re-enable CI

* add instance group support to compute-vm, start tests refactoring

* modules/compute-vm: group fixes, tests

* modules/compute-vm: minimal test beautification

* simplify top-level pytest fixture

* modules/dns: tests and minor tweaks

* fix missing boilerplate in tests

* re-add requirements file to tests folder

* re-enable tests in ci build configuration

* Folder module tests and fixes (#38)

* folder tests wip

* modules/folders: tests and tweaks

* update folders and compute-vm README files

* modules/gcs: tests and minor tweaks

* Create README.md

* Update README.md

* Update README.md

* Update README.md

* Added docker image for strongSwan

* Add support for routes and tests to net-vpc module (#39)

* modules/net-vpc: add routes (untested)

* initial tests

* modules/net-vpc: add test for flow logs

* modules/net-vpc: split tests into two separate files

* modules/net-vpc: routes test

* modules/net-vpc: test routes

* Add support for Terraform plugin cache in ci test build file (#40)

* add Terraform plugin caching to test ci build configuration

* fix mkdir in test build configuration

* trigger test check

* Refactor dynamic vpn configuration for on-prem-in-a-box module

* Fix dynamic vpn for onprem-in-a-box module

* Migrate Shared VPC example to local modules (#41)

* wip

* wip

* validated, untested

* modules/compute-vm: make service account email in locals resilient to destroy

* modules/project: make project id output depend on iam roles

* fixes

* shared-vpc tweaks

* update diagram

* update README input output tables

* modules/compute-vm: add service account IAM email output

* move GKE service account roles at the project level, add GCE service account roles

* update diagram and README

* modules/project: add extra output for IAM-dependent project id

* update modules READMEs

* minor tweaks

* modules/compute-vm: fix service account output

* remove static address from NAT

* fix container service agent binding dependency

* rename shared vpc

* Update README.md

* Update README.md

* Add static vpn gw to on-prem-in-a-box module

* Refactor hub and spoke to use new modules (#42)

* modules/compute-vm: saner defaults for service account scopes

* hub and spoke refactor, docs still missing

* complete hub and spoke

* Update README.md

* Add toolbox docker container, fix gw routing to the internet

* Add DNS Hybrid connectivity parameters

* Fix onprem dns zone for the static vpn configuration

* Added readme.md for on-prem module

* Add new line at the end of the files

* Add boilerplate for cloudbuild config files

* fix boilerplate in strongswan shell script

* Update README.md

* include missing file to fix merge conflict

* remove missing file to fix merge conflict

* include missing file to fix merge conflict (again)

* remove content from spurious file used to avoid merge conflicts

* Add net-vpc-peering module

* Initial commit for hub-and-spoke-peering infrastructure example

* Fix typos in infrastructure/ READMEs

* remove stale file

* use larger resolution version of hub and spoke diagram

* Update README.md

* Update hub-and-spoke-peerings example to use internal modules

* Add initial project tests (#46)

* modules/project: make prefix optional

* initial project module tests

* modules/project: use null for unset parent

* modules/dns: backport PR6 from the CFT dns module

* Add testing resources including on-prem-in-a-box to hub-and-spoke-peerings example

* Fix firewall rules to allow connectivity, switch to custom route advertisement for onprem -> spokes connectivity

* Move locals out of main.tf

* remove ssh tag from compute-vm variable default

* Add ssh tag to the test vms

* Update README.md

* Update README.md

* Update README.md

* Hub and spoke peering changes (#48)

* rename hub-and-spoke-vpn

* add ssh tag to shared-vpc-gke instance

* rename and rework hub and spoke peering

* fix test requirements

* align hub and spoke peering with module contents

* diagram

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* minimal fixes to onprem examples variable files

* onprem example stub, missing DNS zones and private.googleapis records onprem

* add missing boilerplate

* Update README.md

* Update README.md

* infra/onprem: add test instance and minimal outputs

* add DNS modules and resource

* infra/onprem: diagram and initial README

* minor changes to onprem module and example (#49)

* update toolbox image

* infra/onprem: add zone for private access, add metadata domain to onprem dns

* infra/onprem: onnprem service account, add testing procedure in README

* Update README.md

* infra/onprem: remove extra variable

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* infra/onprem: rename forwarder address variable

* Update README:

Added explicit --tunnel-through-iap for gcloud compute ssh commands

* Update top-level and section READMEs (#50)

* top-level README WIP

* rewrite top-level README

* change top-level README title

* remove initial quote in top-level README

* Update README.md

* Update README.md

* Update README.md

* foundations README

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* add experimental scheduled cloud function module

* scheduled cloud function module: allow disabling schedule

* business-units foundation example (#52)

* Added folder-units module.

* Business units example update (WIP)

* Update all BU modules to internal ones

* Refactoring business-units example, add billing and org IAM handling

* update projects tests for new iam additive naming

* update project README for new iam additive naming

* streamline bu example and module (#53)

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>

* align net-vpn-ha interface with the other vpn modules

* update module README files

* Update README.md

* Update README.md

* Create CHANGELOG.md

* Refactor COS module to be generic (#51)

* Create generic COS module and update CoreDNS module to use it

* Update compute-vm-cos README

* Fix COS README

* Update COS example

* Skip boilerplate check for COS file template

* Make COS module more generic and provide preset configurations

* Update COS module documentation

* tfdoc: add support for multiple variables files

* compute-vm: split boot disk in separate variable file for cos module support

* Streamline cos modules (#54)

* tfdoc: fix bug in last commit

* compute-vm: add support for user-data

* compute-vm: restore noncos variable split

* remove compute-vm-cos-coredns

* compute-vm: revert to original state

* cos-container/coredns

* fix variables mess

* cos/coredns fixes

* cos/mysql

* remove stale compute-vm-cos module

* add test instance to cos modules

* tfdoc: add support for multiple output files

* cos: add initial READMEs

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* add test apply fixture

* cos-coredns: tested

* Update README.md

* Fix typo

* cos-coredns: refactor README

* Update README.md

* test yaml validity in cos modules tests

* cos mysql tests

* cos mysql: refactor and test (disk tests missing)

* onprem: fix Coredns

* cos mysql: additional disk working

* cos modules: fix instance disks for no instance

* update some modules READMEs

* update some modules READMEs

* Update README.md

* Update README.md

* add simple tests for foundations/environments

* change default for org id in foundations/environments to avoid errors when none is specified

* fix null/empty organization id in foundations/environments

* fix errors when destroying on empty state in foundations/environments

* fundations/bu: fix errors when destroying with empty state

* modules/gcs: make outputs resilient on destroy with empty state

* modules/folders: make outputs resilient on destroy with empty state

* switch organization_id variable to long form in foundations/bu and modules/folders-unit

* Update README.md

* infra/shared-vpc: remove duplicate tag attribute from bastion

Co-authored-by: Aleksandr Averbukh <averbukh@google.com>
Co-authored-by: Julio Castillo <juliocc@gmail.com>
Co-authored-by: Julio Castillo <jccb@google.com>
2020-04-03 14:06:48 +02:00
Ludovico Magnocavallo 253c51d07c
Update docs using tfdoc format (#28)
* update README files

* set all types on variables
2020-01-10 13:44:54 +01:00
Ludovico Magnocavallo 0a63efa5ea
backport tfdoc from Ludo's branch (#27) 2020-01-10 13:31:19 +01:00