Merge pull request #1538 from GoogleCloudPlatform/jccb/toc

Extend tfdoc to generate TOCs
This commit is contained in:
Julio Castillo 2023-07-28 17:45:11 +02:00 committed by GitHub
commit fc1373b85c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
11 changed files with 285 additions and 172 deletions

View File

@ -2,6 +2,14 @@
This module simplifies the creation of repositories using Google Cloud Artifact Registry.
<!-- BEGIN TOC -->
- [Standard Repository](#standard-repository)
- [Remote and Virtual Repositories](#remote-and-virtual-repositories)
- [Additional Docker and Maven Options](#additional-docker-and-maven-options)
- [Variables](#variables)
- [Outputs](#outputs)
<!-- END TOC -->
## Standard Repository
```hcl

View File

@ -4,13 +4,21 @@ Cloud Run management, with support for IAM roles, revision annotations and optio
## Examples
- [IAM and environment variables](#iam-and-environment-variables)
- [Mounting secrets as volumes](#mounting-secrets-as-volumes)
- [Revision annotations](#revision-annotations)
- [VPC Access Connector creation](#vpc-access-connector-creation)
- [Traffic split](#traffic-split)
- [Eventarc triggers](#eventarc-triggers)
- [Service account](#service-account)
<!-- BEGIN TOC -->
- [Examples](#examples)
- [IAM and environment variables](#iam-and-environment-variables)
- [Mounting secrets as volumes](#mounting-secrets-as-volumes)
- [Revision annotations](#revision-annotations)
- [VPC Access Connector creation](#vpc-access-connector-creation)
- [Traffic split](#traffic-split)
- [Eventarc triggers](#eventarc-triggers)
- [PubSub](#pubsub)
- [Audit logs](#audit-logs)
- [Using custom service accounts for triggers](#using-custom-service-accounts-for-triggers)
- [Service account](#service-account)
- [Variables](#variables)
- [Outputs](#outputs)
<!-- END TOC -->
### IAM and environment variables

View File

@ -9,25 +9,32 @@ In both modes, an optional service account can be created and assigned to either
## Examples
- [Instance using defaults](#instance-using-defaults)
- [Service account management](#service-account-management)
- [Disk management](#disk-management)
- [Disk sources](#disk-sources)
- [Disk types and options](#disk-types-and-options)
- [Boot disk as an independent resource](#boot-disk-as-an-independent-resource)
- [Network interfaces](#network-interfaces)
- [Internal and external IPs](#internal-and-external-ips)
- [Using Alias IPs](#using-alias-ips)
- [Using gVNIC](#using-gvnic)
- [Metadata](#metadata)
- [IAM](#iam)
- [Spot VM](#spot-vm)
- [Confidential compute](#confidential-compute)
- [Disk encryption with Cloud KMS](#disk-encryption-with-cloud-kms)
- [Instance template](#instance-template)
- [Instance group](#instance-group)
- [Instance Schedule](#instance-schedule)
- [Snapshot Schedules](#snapshot-schedules)
<!-- BEGIN TOC -->
- [Examples](#examples)
- [Instance using defaults](#instance-using-defaults)
- [Service account management](#service-account-management)
- [Disk management](#disk-management)
- [Disk sources](#disk-sources)
- [Disk types and options](#disk-types-and-options)
- [Boot disk as an independent resource](#boot-disk-as-an-independent-resource)
- [Network interfaces](#network-interfaces)
- [Internal and external IPs](#internal-and-external-ips)
- [Using Alias IPs](#using-alias-ips)
- [Using gVNIC](#using-gvnic)
- [Metadata](#metadata)
- [IAM](#iam)
- [Spot VM](#spot-vm)
- [Confidential compute](#confidential-compute)
- [Disk encryption with Cloud KMS](#disk-encryption-with-cloud-kms)
- [Instance template](#instance-template)
- [Instance group](#instance-group)
- [Instance Schedule](#instance-schedule)
- [Snapshot Schedules](#snapshot-schedules)
- [Variables](#variables)
- [Outputs](#outputs)
- [TODO](#todo)
<!-- END TOC -->
### Instance using defaults

View File

@ -2,17 +2,22 @@
This module allows the creation and management of folders, including support for IAM bindings, organization policies, and hierarchical firewall rules.
## Features
<!-- BEGIN TOC -->
- [Basic example with IAM bindings](#basic-example-with-iam-bindings)
- [IAM](#iam)
- [Organization Policies](#organization-policies)
- [Factory](#organization-policy-factory)
- [Organization policies](#organization-policies)
- [Organization Policy Factory](#organization-policy-factory)
- [Hierarchical Firewall Policies](#hierarchical-firewall-policies)
- [Directly Defined](#directly-defined-firewall-policies)
- [Factory](#firewall-policy-factory)
- [Directly Defined Firewall Policies](#directly-defined-firewall-policies)
- [Firewall Policy Factory](#firewall-policy-factory)
- [Log Sinks](#log-sinks)
- [Data Access Logs](#data-access-logs)
- [Tags](#tags)
- [Files](#files)
- [Variables](#variables)
- [Outputs](#outputs)
<!-- END TOC -->
## Basic example with IAM bindings

View File

@ -6,20 +6,32 @@ Due to the complexity of the underlying resources, changes to the configuration
## Examples
- [Minimal HTTP Example](#minimal-http-example)
- [Minimal HTTPS Examples](#minimal-https-examples)
- [Health Checks](#health-checks)
- [Backend Types and Management](#backend-types-and-management)
- [Instance Groups](#instance-groups)
- [Storage Buckets](#storage-buckets)
- [Network Endpoint Groups](#network-endpoint-groups-negs)
- [Zonal NEGs](#zonal-neg-creation)
- [Hybrid NEGs](#hybrid-neg-creation)
- [Internet NEGs](#internet-neg-creation)
- [Serverless NEGs](#serverless-neg-creation)
- [URL Map](#url-map)
- [SSL Certificates](#ssl-certificates)
- [Complex Example](#complex-example)
<!-- BEGIN TOC -->
- [Examples](#examples)
- [Minimal HTTP Example](#minimal-http-example)
- [Minimal HTTPS examples](#minimal-https-examples)
- [HTTP backends](#http-backends)
- [HTTPS backends](#https-backends)
- [Classic vs Non-classic](#classic-vs-non-classic)
- [Health Checks](#health-checks)
- [Backend Types and Management](#backend-types-and-management)
- [Instance Groups](#instance-groups)
- [Managed Instance Groups](#managed-instance-groups)
- [Storage Buckets](#storage-buckets)
- [Network Endpoint Groups (NEGs)](#network-endpoint-groups-negs)
- [Zonal NEG creation](#zonal-neg-creation)
- [Hybrid NEG creation](#hybrid-neg-creation)
- [Internet NEG creation](#internet-neg-creation)
- [Private Service Connect NEG creation](#private-service-connect-neg-creation)
- [Serverless NEG creation](#serverless-neg-creation)
- [URL Map](#url-map)
- [SSL Certificates](#ssl-certificates)
- [Complex example](#complex-example)
- [Files](#files)
- [Variables](#variables)
- [Outputs](#outputs)
<!-- END TOC -->
### Minimal HTTP Example

View File

@ -6,14 +6,24 @@ Due to the complexity of the underlying resources, changes to the configuration
## Examples
- [Minimal Example](#minimal-example)
- [Cross-project Backend Services](#cross-project-backend-services)
- [Health Checks](#health-checks)
- [Instance Groups](#instance-groups)
- [Network Endpoint Groups](#network-endpoint-groups-negs)
- [URL Map](#url-map)
- [SSL Certificates](#ssl-certificates)
- [Complex Example](#complex-example)
<!-- BEGIN TOC -->
- [Examples](#examples)
- [Minimal Example](#minimal-example)
- [Cross-project backend services](#cross-project-backend-services)
- [Health Checks](#health-checks)
- [Instance Groups](#instance-groups)
- [Network Endpoint Groups (NEGs)](#network-endpoint-groups-negs)
- [Zonal NEG creation](#zonal-neg-creation)
- [Hybrid NEG creation](#hybrid-neg-creation)
- [Serverless NEG creation](#serverless-neg-creation)
- [Private Service Connect NEG creation](#private-service-connect-neg-creation)
- [URL Map](#url-map)
- [SSL Certificates](#ssl-certificates)
- [Complex example](#complex-example)
- [Files](#files)
- [Variables](#variables)
- [Outputs](#outputs)
<!-- END TOC -->
### Minimal Example

View File

@ -4,23 +4,24 @@ This module allows creation and management of VPC networks including subnetworks
## Examples
- [VPC module](#vpc-module)
- [Examples](#examples)
- [Simple VPC](#simple-vpc)
- [Subnet Options](#subnet-options)
- [Subnet IAM](#subnet-iam)
- [Peering](#peering)
- [Shared VPC](#shared-vpc)
- [Private Service Networking](#private-service-networking)
- [Private Service Networking with peering routes](#private-service-networking-with-peering-routes)
- [Subnets for Private Service Connect, Proxy-only subnets](#subnets-for-private-service-connect-proxy-only-subnets)
- [DNS Policies](#dns-policies)
- [Subnet Factory](#subnet-factory)
- [Custom Routes](#custom-routes)
- [Private Google Access routes](#private-google-access-routes)
- [Allow Firewall Policy to be evaluated before Firewall Rules](#allow-firewall-policy-to-be-evaluated-before-firewall-rules)
- [Variables](#variables)
- [Outputs](#outputs)
<!-- BEGIN TOC -->
- [Examples](#examples)
- [Simple VPC](#simple-vpc)
- [Subnet Options](#subnet-options)
- [Subnet IAM](#subnet-iam)
- [Peering](#peering)
- [Shared VPC](#shared-vpc)
- [Private Service Networking](#private-service-networking)
- [Private Service Networking with peering routes](#private-service-networking-with-peering-routes)
- [Subnets for Private Service Connect, Proxy-only subnets](#subnets-for-private-service-connect-proxy-only-subnets)
- [DNS Policies](#dns-policies)
- [Subnet Factory](#subnet-factory)
- [Custom Routes](#custom-routes)
- [Private Google Access routes](#private-google-access-routes)
- [Allow Firewall Policy to be evaluated before Firewall Rules](#allow-firewall-policy-to-be-evaluated-before-firewall-rules)
- [Variables](#variables)
- [Outputs](#outputs)
<!-- END TOC -->
### Simple VPC

View File

@ -10,20 +10,26 @@ This module allows managing several organization properties:
To manage organization policies, the `orgpolicy.googleapis.com` service should be enabled in the quota project.
## Features
## TOC
<!-- BEGIN TOC -->
- [TOC](#toc)
- [Example](#example)
- [IAM](#iam)
- [Organization Policies](#organization-policies)
- [Factory](#organization-policy-factory)
- [Custom Constraints](#organization-policy-custom-constraints)
- [Custom Constraints Factory](#organization-policy-custom-constraints-factory)
- [Organization Policy Factory](#organization-policy-factory)
- [Organization Policy Custom Constraints](#organization-policy-custom-constraints)
- [Organization Policy Custom Constraints Factory](#organization-policy-custom-constraints-factory)
- [Hierarchical Firewall Policies](#hierarchical-firewall-policies)
- [Directly Defined](#directly-defined-firewall-policies)
- [Factory](#firewall-policy-factory)
- [Directly Defined Firewall Policies](#directly-defined-firewall-policies)
- [Firewall Policy Factory](#firewall-policy-factory)
- [Log Sinks](#log-sinks)
- [Data Access Logs](#data-access-logs)
- [Custom Roles](#custom-roles)
- [Tags](#tags)
- [Files](#files)
- [Variables](#variables)
- [Outputs](#outputs)
<!-- END TOC -->
## Example
@ -524,6 +530,8 @@ module "org" {
```
<!-- TFDOC OPTS files:1 -->
<!-- BEGIN TFDOC -->
## Files
@ -583,5 +591,4 @@ module "org" {
| [sink_writer_identities](outputs.tf#L103) | Writer identities created for each sink. | |
| [tag_keys](outputs.tf#L111) | Tag key resources. | |
| [tag_values](outputs.tf#L120) | Tag value resources. | |
<!-- END TFDOC -->

View File

@ -2,23 +2,30 @@
This module implements the creation and management of one GCP project including IAM, organization policies, Shared VPC host or service attachment, service API activation, and tag attachment. It also offers a convenient way to refer to managed service identities (aka robot service accounts) for APIs.
## Features
## TOC
<!-- BEGIN TOC -->
- [TOC](#toc)
- [Basic Project Creation](#basic-project-creation)
- [IAM](#iam)
- [Authoritative](#authoritative-iam)
- [Additive](#additive-iam)
- [Additive By Member](#additive-iam-by-member)
- [Authoritative IAM](#authoritative-iam)
- [Additive IAM](#additive-iam)
- [Additive IAM by Member](#additive-iam-by-member)
- [Service Identities and Authoritative IAM](#service-identities-and-authoritative-iam)
- [Using Shortcodes for Service Identities](#using-shortcodes-for-service-identities-in-additive-iam)
- [Service Identities and Manual IAM Grants](#service-identities-requiring-manual-iam-grants)
- [Using Shortcodes for Service Identities in Additive Iam](#using-shortcodes-for-service-identities-in-additive-iam)
- [Service Identities Requiring Manual Iam Grants](#service-identities-requiring-manual-iam-grants)
- [Shared VPC](#shared-vpc)
- [Organization Policies](#organization-policies)
- [Factory](#organization-policy-factory)
- [Organization Policy Factory](#organization-policy-factory)
- [Log Sinks](#log-sinks)
- [Data Access Logs](#data-access-logs)
- [Cloud KMS Encryption Keys](#cloud-kms-encryption-keys)
- [Cloud Kms Encryption Keys](#cloud-kms-encryption-keys)
- [Tags](#tags)
- [Outputs](#outputs)
- [Files](#files)
- [Variables](#variables)
- [Outputs](#outputs)
<!-- END TOC -->
## Basic Project Creation
@ -570,8 +577,8 @@ output "compute_robot" {
```
<!-- TFDOC OPTS files:1 -->
<!-- BEGIN TFDOC -->
<!-- BEGIN TFDOC -->
## Files
| name | description | resources |
@ -639,5 +646,5 @@ output "compute_robot" {
| [project_id](outputs.tf#L75) | Project id. | |
| [service_accounts](outputs.tf#L94) | Product robot service accounts in project. | |
| [sink_writer_identities](outputs.tf#L110) | Writer identities created for each sink. | |
<!-- END TFDOC -->

View File

@ -35,6 +35,7 @@ class State(enum.IntEnum):
SKIP = enum.auto()
OK = enum.auto()
FAIL_STALE_README = enum.auto()
FAIL_STALE_TOC = enum.auto()
FAIL_UNSORTED_VARS = enum.auto()
FAIL_UNSORTED_OUTPUTS = enum.auto()
FAIL_VARIABLE_PERIOD = enum.auto()
@ -52,6 +53,7 @@ class State(enum.IntEnum):
State.SKIP: ' ',
State.OK: '',
State.FAIL_STALE_README: '✗R',
State.FAIL_STALE_TOC: '✗T',
State.FAIL_UNSORTED_VARS: 'SV',
State.FAIL_UNSORTED_OUTPUTS: 'SO',
State.FAIL_VARIABLE_PERIOD: '.V',
@ -71,74 +73,78 @@ def _check_dir(dir_name, exclude_files=None, files=False, show_extra=False):
diff = None
readme = readme_path.read_text()
mod_name = str(readme_path.relative_to(dir_path).parent)
result = tfdoc.get_doc(readme)
if not result:
state = State.SKIP
else:
try:
new_doc = tfdoc.create_doc(readme_path.parent, files, show_extra,
exclude_files, readme)
newvars = new_doc.variables
newouts = new_doc.outputs
variables = [v.name for v in newvars if v.file.endswith('variables.tf')]
outputs = [o.name for o in newouts if o.file.endswith('outputs.tf')]
except SystemExit:
state = state.SKIP
else:
state = State.OK
current_doc = tfdoc.get_doc(readme)
current_toc = tfdoc.get_toc(readme)
if current_doc or current_toc:
new_doc = tfdoc.create_doc(readme_path.parent, files, show_extra,
exclude_files, readme)
new_toc = tfdoc.create_toc(readme)
newvars = new_doc.variables
newouts = new_doc.outputs
variables = [v.name for v in newvars if v.file.endswith('variables.tf')]
outputs = [o.name for o in newouts if o.file.endswith('outputs.tf')]
if new_doc.content != result['doc']:
state = State.FAIL_STALE_README
header = f'----- {mod_name} diff -----\n'
ndiff = difflib.ndiff(result['doc'].split('\n'),
new_doc.content.split('\n'))
diff = '\n'.join([header] + list(ndiff))
state = State.OK
elif empty := [v.name for v in newvars if not v.description]:
state = state.FAIL_VARIABLE_DESCRIPTION
diff = "\n".join([
f'----- {mod_name} variables missing description -----',
', '.join(empty),
])
if current_doc and new_doc.content != current_doc['doc']:
state = State.FAIL_STALE_README
header = f'----- {mod_name} diff -----\n'
ndiff = difflib.ndiff(current_doc['doc'].splitlines(keepends=True),
new_doc.content.splitlines(keepends=True))
diff = ''.join([header] + [x for x in ndiff if x[0] != ' '])
elif empty := [o.name for o in newouts if not o.description]:
state = state.FAIL_VARIABLE_DESCRIPTION
diff = "\n".join([
f'----- {mod_name} outputs missing description -----',
', '.join(empty),
])
elif current_toc and new_toc != current_toc['toc']:
state = State.FAIL_STALE_TOC
header = f'----- {mod_name} diff -----\n'
ndiff = difflib.ndiff(current_toc['toc'].splitlines(keepends=True),
new_toc.splitlines(keepends=True))
diff = ''.join([header] + [x for x in ndiff if x[0] != ' '])
elif variables != sorted(variables):
state = state.FAIL_UNSORTED_VARS
diff = "\n".join([
f'----- {mod_name} variables -----',
f'variables should be in this order: ',
', '.join(sorted(variables)),
])
elif empty := [v.name for v in newvars if not v.description]:
state = state.FAIL_VARIABLE_DESCRIPTION
diff = "\n".join([
f'----- {mod_name} variables missing description -----',
', '.join(empty),
])
elif outputs != sorted(outputs):
state = state.FAIL_UNSORTED_OUTPUTS
diff = "\n".join([
f'----- {mod_name} outputs -----',
f'outputs should be in this order: ',
', '.join(sorted(outputs)),
])
elif empty := [o.name for o in newouts if not o.description]:
state = state.FAIL_VARIABLE_DESCRIPTION
diff = "\n".join([
f'----- {mod_name} outputs missing description -----',
', '.join(empty),
])
elif nc := [v.name for v in newvars if not v.description.endswith('.')]:
state = state.FAIL_VARIABLE_PERIOD
diff = "\n".join([
f'----- {mod_name} variable descriptions missing ending period -----',
', '.join(nc),
])
elif variables != sorted(variables):
state = state.FAIL_UNSORTED_VARS
diff = "\n".join([
f'----- {mod_name} variables -----',
f'variables should be in this order: ',
', '.join(sorted(variables)),
])
elif nc := [o.name for o in newouts if not o.description.endswith('.')]:
state = state.FAIL_OUTPUT_PERIOD
diff = "\n".join([
f'----- {mod_name} output descriptions missing ending period -----',
', '.join(nc),
])
elif outputs != sorted(outputs):
state = state.FAIL_UNSORTED_OUTPUTS
diff = "\n".join([
f'----- {mod_name} outputs -----',
f'outputs should be in this order: ',
', '.join(sorted(outputs)),
])
yield mod_name, state, diff
elif nc := [v.name for v in newvars if not v.description.endswith('.')]:
state = state.FAIL_VARIABLE_PERIOD
diff = "\n".join([
f'----- {mod_name} variable descriptions missing ending period -----',
', '.join(nc),
])
elif nc := [o.name for o in newouts if not o.description.endswith('.')]:
state = state.FAIL_OUTPUT_PERIOD
diff = "\n".join([
f'----- {mod_name} output descriptions missing ending period -----',
', '.join(nc),
])
yield mod_name, state, diff
@click.command()

View File

@ -1,6 +1,6 @@
#!/usr/bin/env python3
# Copyright 2022 Google LLC
# Copyright 2023 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -46,6 +46,7 @@ import string
import urllib.parse
import click
import marko
__version__ = '2.1.0'
@ -80,6 +81,8 @@ OUT_RE = re.compile(r'''(?smx)
''')
OUT_TEMPLATE = ('description', 'value', 'sensitive')
TAG_RE = re.compile(r'(?sm)^\s*#\stfdoc:([^:]+:\S+)\s+(.*?)\s*$')
TOC_BEGIN = '<!-- BEGIN TOC -->'
TOC_END = '<!-- END TOC -->'
UNESCAPED = string.digits + string.ascii_letters + ' .,;:_-'
VAR_ENUM = enum.Enum('V', 'OPEN ATTR ATTR_DATA SKIP CLOSE COMMENT TXT')
VAR_RE = re.compile(r'''(?smx)
@ -244,9 +247,7 @@ def format_doc(outputs, variables, files, show_extra=False):
if outputs:
buffer += ['', '## Outputs', '']
buffer += list(format_outputs(outputs, show_extra))
if buffer:
buffer.append('')
return '\n'.join(buffer)
return '\n'.join(buffer).strip()
def format_files(items):
@ -322,15 +323,39 @@ def format_variables(items, show_extra=True):
yield format
def create_toc(readme):
'Create a Markdown table of contents a for README.'
doc = marko.parse(readme)
lines = []
headings = [x for x in doc.children if x.get_type() == 'Heading']
for h in headings[1:]:
title = h.children[0].children
slug = title.lower().strip()
slug = re.sub('[^\w\s-]', '', slug)
slug = re.sub('[-\s]+', '-', slug)
link = f'- [{title}](#{slug})'
indent = ' ' * (h.level - 2)
lines.append(f'{indent}{link}')
return "\n".join(lines)
# replace functions
def get_doc(readme):
'Check if README file is marked, and return current doc.'
m = re.search('(?sm)%s\n(.*)\n%s' % (MARK_BEGIN, MARK_END), readme)
m = re.search('(?sm)%s(.*)%s' % (MARK_BEGIN, MARK_END), readme)
if not m:
return
return {'doc': m.group(1), 'start': m.start(), 'end': m.end()}
return {'doc': m.group(1).strip(), 'start': m.start(), 'end': m.end()}
def get_toc(readme):
'Check if README file is marked, and return current toc.'
t = re.search('(?sm)%s(.*)%s' % (TOC_BEGIN, TOC_END), readme)
if not t:
return
return {'toc': t.group(1).strip(), 'start': t.start(), 'end': t.end()}
def get_doc_opts(readme):
@ -373,24 +398,34 @@ def get_readme(readme_path):
raise SystemExit(f'Error opening README {readme_path}: {e}')
def replace_doc(readme_path, doc, readme=None):
def render_doc(readme, doc):
'Replace document in module\'s README.md file.'
readme = readme or get_readme(readme_path)
result = get_doc(readme)
if not result:
raise SystemExit(f'Mark not found in README {readme_path}')
if doc == result['doc']:
return
try:
open(readme_path, 'w').write('\n'.join([
readme[:result['start']].rstrip(),
MARK_BEGIN,
doc,
MARK_END,
readme[result['end']:].lstrip(),
]))
except (IOError, OSError) as e:
raise SystemExit(f'Error replacing README {readme_path}: {e}')
if not result or doc == result['doc']:
return readme
return '\n'.join([
readme[:result['start']].rstrip(),
MARK_BEGIN,
doc,
MARK_END,
readme[result['end']:].lstrip(),
])
def render_toc(readme, toc):
'Replace toc in module\'s README.md file.'
result = get_toc(readme)
if not result or toc == result['toc']:
return readme
return '\n'.join([
readme[:result['start']].rstrip(),
'',
TOC_BEGIN,
toc,
TOC_END,
'',
readme[result['end']:].lstrip(),
])
@click.command()
@ -405,10 +440,17 @@ def main(module_path=None, exclude_file=None, files=False, replace=True,
readme_path = os.path.join(module_path, 'README.md')
readme = get_readme(readme_path)
doc = create_doc(module_path, files, show_extra, exclude_file, readme)
toc = create_toc(readme)
tmp = render_doc(readme, doc.content)
final = render_toc(tmp, toc)
if replace:
replace_doc(readme_path, doc.content, readme)
try:
with open(readme_path, 'w') as f:
f.write(final)
except (IOError, OSError) as e:
raise SystemExit(f'Error replacing README {readme_path}: {e}')
else:
print(doc)
print(final)
if __name__ == '__main__':