zecfoundation
/
cloud-foundation-fabric
mirror of https://github.com/ZcashFoundation/cloud-foundation-fabric.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1538 from GoogleCloudPlatform/jccb/toc 

Extend tfdoc to generate TOCs
This commit is contained in:
Julio Castillo 2023-07-28 17:45:11 +02:00 committed by GitHub
parent 6ff556755b 32bb69caa5
commit fc1373b85c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
11 changed files with 285 additions and 172 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
modules/artifact-registry/README.md
View File

@ -2,6 +2,14 @@
This module simplifies the creation of repositories using Google Cloud Artifact Registry. This module simplifies the creation of repositories using Google Cloud Artifact Registry.
<!-- BEGIN TOC -->
- [Standard Repository](#standard-repository)
- [Remote and Virtual Repositories](#remote-and-virtual-repositories)
- [Additional Docker and Maven Options](#additional-docker-and-maven-options)
- [Variables](#variables)
- [Outputs](#outputs)
<!-- END TOC -->
## Standard Repository ## Standard Repository
```hcl ```hcl

22
modules/cloud-run/README.md
View File

@ -4,13 +4,21 @@ Cloud Run management, with support for IAM roles, revision annotations and optio
## Examples ## Examples
- [IAM and environment variables](#iam-and-environment-variables) <!-- BEGIN TOC -->
- [Mounting secrets as volumes](#mounting-secrets-as-volumes) - [Examples](#examples)
- [Revision annotations](#revision-annotations) - [IAM and environment variables](#iam-and-environment-variables)
- [VPC Access Connector creation](#vpc-access-connector-creation) - [Mounting secrets as volumes](#mounting-secrets-as-volumes)
- [Traffic split](#traffic-split) - [Revision annotations](#revision-annotations)
- [Eventarc triggers](#eventarc-triggers) - [VPC Access Connector creation](#vpc-access-connector-creation)
- [Service account](#service-account) - [Traffic split](#traffic-split)
- [Eventarc triggers](#eventarc-triggers)
- [PubSub](#pubsub)
- [Audit logs](#audit-logs)
- [Using custom service accounts for triggers](#using-custom-service-accounts-for-triggers)
- [Service account](#service-account)
- [Variables](#variables)
- [Outputs](#outputs)
<!-- END TOC -->
### IAM and environment variables ### IAM and environment variables

45
modules/compute-vm/README.md
View File

@ -9,25 +9,32 @@ In both modes, an optional service account can be created and assigned to either
## Examples ## Examples
- [Instance using defaults](#instance-using-defaults)
- [Service account management](#service-account-management) <!-- BEGIN TOC -->
- [Disk management](#disk-management) - [Examples](#examples)
- [Disk sources](#disk-sources) - [Instance using defaults](#instance-using-defaults)
- [Disk types and options](#disk-types-and-options) - [Service account management](#service-account-management)
- [Boot disk as an independent resource](#boot-disk-as-an-independent-resource) - [Disk management](#disk-management)
- [Network interfaces](#network-interfaces) - [Disk sources](#disk-sources)
- [Internal and external IPs](#internal-and-external-ips) - [Disk types and options](#disk-types-and-options)
- [Using Alias IPs](#using-alias-ips) - [Boot disk as an independent resource](#boot-disk-as-an-independent-resource)
- [Using gVNIC](#using-gvnic) - [Network interfaces](#network-interfaces)
- [Metadata](#metadata) - [Internal and external IPs](#internal-and-external-ips)
- [IAM](#iam) - [Using Alias IPs](#using-alias-ips)
- [Spot VM](#spot-vm) - [Using gVNIC](#using-gvnic)
- [Confidential compute](#confidential-compute) - [Metadata](#metadata)
- [Disk encryption with Cloud KMS](#disk-encryption-with-cloud-kms) - [IAM](#iam)
- [Instance template](#instance-template) - [Spot VM](#spot-vm)
- [Instance group](#instance-group) - [Confidential compute](#confidential-compute)
- [Instance Schedule](#instance-schedule) - [Disk encryption with Cloud KMS](#disk-encryption-with-cloud-kms)
- [Snapshot Schedules](#snapshot-schedules) - [Instance template](#instance-template)
- [Instance group](#instance-group)
- [Instance Schedule](#instance-schedule)
- [Snapshot Schedules](#snapshot-schedules)
- [Variables](#variables)
- [Outputs](#outputs)
- [TODO](#todo)
<!-- END TOC -->
### Instance using defaults ### Instance using defaults

15
modules/folder/README.md
View File

@ -2,17 +2,22 @@
This module allows the creation and management of folders, including support for IAM bindings, organization policies, and hierarchical firewall rules. This module allows the creation and management of folders, including support for IAM bindings, organization policies, and hierarchical firewall rules.
## Features
<!-- BEGIN TOC -->
- [Basic example with IAM bindings](#basic-example-with-iam-bindings)
- [IAM](#iam) - [IAM](#iam)
- [Organization Policies](#organization-policies) - [Organization policies](#organization-policies)
- [Factory](#organization-policy-factory) - [Organization Policy Factory](#organization-policy-factory)
- [Hierarchical Firewall Policies](#hierarchical-firewall-policies) - [Hierarchical Firewall Policies](#hierarchical-firewall-policies)
- [Directly Defined](#directly-defined-firewall-policies) - [Directly Defined Firewall Policies](#directly-defined-firewall-policies)
- [Factory](#firewall-policy-factory) - [Firewall Policy Factory](#firewall-policy-factory)
- [Log Sinks](#log-sinks) - [Log Sinks](#log-sinks)
- [Data Access Logs](#data-access-logs) - [Data Access Logs](#data-access-logs)
- [Tags](#tags) - [Tags](#tags)
- [Files](#files)
- [Variables](#variables)
- [Outputs](#outputs)
<!-- END TOC -->
## Basic example with IAM bindings ## Basic example with IAM bindings

40
modules/net-lb-app-ext/README.md
View File

@ -6,20 +6,32 @@ Due to the complexity of the underlying resources, changes to the configuration
## Examples ## Examples
- [Minimal HTTP Example](#minimal-http-example) <!-- BEGIN TOC -->
- [Minimal HTTPS Examples](#minimal-https-examples) - [Examples](#examples)
- [Health Checks](#health-checks) - [Minimal HTTP Example](#minimal-http-example)
- [Backend Types and Management](#backend-types-and-management) - [Minimal HTTPS examples](#minimal-https-examples)
- [Instance Groups](#instance-groups) - [HTTP backends](#http-backends)
- [Storage Buckets](#storage-buckets) - [HTTPS backends](#https-backends)
- [Network Endpoint Groups](#network-endpoint-groups-negs) - [Classic vs Non-classic](#classic-vs-non-classic)
- [Zonal NEGs](#zonal-neg-creation) - [Health Checks](#health-checks)
- [Hybrid NEGs](#hybrid-neg-creation) - [Backend Types and Management](#backend-types-and-management)
- [Internet NEGs](#internet-neg-creation) - [Instance Groups](#instance-groups)
- [Serverless NEGs](#serverless-neg-creation) - [Managed Instance Groups](#managed-instance-groups)
- [URL Map](#url-map) - [Storage Buckets](#storage-buckets)
- [SSL Certificates](#ssl-certificates) - [Network Endpoint Groups (NEGs)](#network-endpoint-groups-negs)
- [Complex Example](#complex-example) - [Zonal NEG creation](#zonal-neg-creation)
- [Hybrid NEG creation](#hybrid-neg-creation)
- [Internet NEG creation](#internet-neg-creation)
- [Private Service Connect NEG creation](#private-service-connect-neg-creation)
- [Serverless NEG creation](#serverless-neg-creation)
- [URL Map](#url-map)
- [SSL Certificates](#ssl-certificates)
- [Complex example](#complex-example)
- [Files](#files)
- [Variables](#variables)
- [Outputs](#outputs)
<!-- END TOC -->
### Minimal HTTP Example ### Minimal HTTP Example

26
modules/net-lb-app-int/README.md
View File

@ -6,14 +6,24 @@ Due to the complexity of the underlying resources, changes to the configuration
## Examples ## Examples
- [Minimal Example](#minimal-example) <!-- BEGIN TOC -->
- [Cross-project Backend Services](#cross-project-backend-services) - [Examples](#examples)
- [Health Checks](#health-checks) - [Minimal Example](#minimal-example)
- [Instance Groups](#instance-groups) - [Cross-project backend services](#cross-project-backend-services)
- [Network Endpoint Groups](#network-endpoint-groups-negs) - [Health Checks](#health-checks)
- [URL Map](#url-map) - [Instance Groups](#instance-groups)
- [SSL Certificates](#ssl-certificates) - [Network Endpoint Groups (NEGs)](#network-endpoint-groups-negs)
- [Complex Example](#complex-example) - [Zonal NEG creation](#zonal-neg-creation)
- [Hybrid NEG creation](#hybrid-neg-creation)
- [Serverless NEG creation](#serverless-neg-creation)
- [Private Service Connect NEG creation](#private-service-connect-neg-creation)
- [URL Map](#url-map)
- [SSL Certificates](#ssl-certificates)
- [Complex example](#complex-example)
- [Files](#files)
- [Variables](#variables)
- [Outputs](#outputs)
<!-- END TOC -->
### Minimal Example ### Minimal Example

35
modules/net-vpc/README.md
View File

@ -4,23 +4,24 @@ This module allows creation and management of VPC networks including subnetworks
## Examples ## Examples
- [VPC module](#vpc-module) <!-- BEGIN TOC -->
- [Examples](#examples) - [Examples](#examples)
- [Simple VPC](#simple-vpc) - [Simple VPC](#simple-vpc)
- [Subnet Options](#subnet-options) - [Subnet Options](#subnet-options)
- [Subnet IAM](#subnet-iam) - [Subnet IAM](#subnet-iam)
- [Peering](#peering) - [Peering](#peering)
- [Shared VPC](#shared-vpc) - [Shared VPC](#shared-vpc)
- [Private Service Networking](#private-service-networking) - [Private Service Networking](#private-service-networking)
- [Private Service Networking with peering routes](#private-service-networking-with-peering-routes) - [Private Service Networking with peering routes](#private-service-networking-with-peering-routes)
- [Subnets for Private Service Connect, Proxy-only subnets](#subnets-for-private-service-connect-proxy-only-subnets) - [Subnets for Private Service Connect, Proxy-only subnets](#subnets-for-private-service-connect-proxy-only-subnets)
- [DNS Policies](#dns-policies) - [DNS Policies](#dns-policies)
- [Subnet Factory](#subnet-factory) - [Subnet Factory](#subnet-factory)
- [Custom Routes](#custom-routes) - [Custom Routes](#custom-routes)
- [Private Google Access routes](#private-google-access-routes) - [Private Google Access routes](#private-google-access-routes)
- [Allow Firewall Policy to be evaluated before Firewall Rules](#allow-firewall-policy-to-be-evaluated-before-firewall-rules) - [Allow Firewall Policy to be evaluated before Firewall Rules](#allow-firewall-policy-to-be-evaluated-before-firewall-rules)
- [Variables](#variables) - [Variables](#variables)
- [Outputs](#outputs) - [Outputs](#outputs)
<!-- END TOC -->
### Simple VPC ### Simple VPC

23
modules/organization/README.md
View File

@ -10,20 +10,26 @@ This module allows managing several organization properties:
To manage organization policies, the `orgpolicy.googleapis.com` service should be enabled in the quota project. To manage organization policies, the `orgpolicy.googleapis.com` service should be enabled in the quota project.
## Features ## TOC
<!-- BEGIN TOC -->
- [TOC](#toc)
- [Example](#example)
- [IAM](#iam) - [IAM](#iam)
- [Organization Policies](#organization-policies) - [Organization Policies](#organization-policies)
- [Factory](#organization-policy-factory) - [Organization Policy Factory](#organization-policy-factory)
- [Custom Constraints](#organization-policy-custom-constraints) - [Organization Policy Custom Constraints](#organization-policy-custom-constraints)
- [Custom Constraints Factory](#organization-policy-custom-constraints-factory) - [Organization Policy Custom Constraints Factory](#organization-policy-custom-constraints-factory)
- [Hierarchical Firewall Policies](#hierarchical-firewall-policies) - [Hierarchical Firewall Policies](#hierarchical-firewall-policies)
- [Directly Defined](#directly-defined-firewall-policies) - [Directly Defined Firewall Policies](#directly-defined-firewall-policies)
- [Factory](#firewall-policy-factory) - [Firewall Policy Factory](#firewall-policy-factory)
- [Log Sinks](#log-sinks) - [Log Sinks](#log-sinks)
- [Data Access Logs](#data-access-logs) - [Data Access Logs](#data-access-logs)
- [Custom Roles](#custom-roles) - [Custom Roles](#custom-roles)
- [Tags](#tags) - [Tags](#tags)
- [Files](#files)
- [Variables](#variables)
- [Outputs](#outputs)
<!-- END TOC -->
## Example ## Example
@ -524,6 +530,8 @@ module "org" {
``` ```
<!-- TFDOC OPTS files:1 --> <!-- TFDOC OPTS files:1 -->
<!-- BEGIN TFDOC --> <!-- BEGIN TFDOC -->
## Files ## Files
@ -583,5 +591,4 @@ module "org" {
| [sink_writer_identities](outputs.tf#L103) | Writer identities created for each sink. | | | [sink_writer_identities](outputs.tf#L103) | Writer identities created for each sink. | |
| [tag_keys](outputs.tf#L111) | Tag key resources. | | | [tag_keys](outputs.tf#L111) | Tag key resources. | |
| [tag_values](outputs.tf#L120) | Tag value resources. | | | [tag_values](outputs.tf#L120) | Tag value resources. | |
<!-- END TFDOC --> <!-- END TFDOC -->

27
modules/project/README.md
View File

@ -2,23 +2,30 @@
This module implements the creation and management of one GCP project including IAM, organization policies, Shared VPC host or service attachment, service API activation, and tag attachment. It also offers a convenient way to refer to managed service identities (aka robot service accounts) for APIs. This module implements the creation and management of one GCP project including IAM, organization policies, Shared VPC host or service attachment, service API activation, and tag attachment. It also offers a convenient way to refer to managed service identities (aka robot service accounts) for APIs.
## Features ## TOC
<!-- BEGIN TOC -->
- [TOC](#toc)
- [Basic Project Creation](#basic-project-creation) - [Basic Project Creation](#basic-project-creation)
- [IAM](#iam) - [IAM](#iam)
- [Authoritative](#authoritative-iam) - [Authoritative IAM](#authoritative-iam)
- [Additive](#additive-iam) - [Additive IAM](#additive-iam)
- [Additive By Member](#additive-iam-by-member) - [Additive IAM by Member](#additive-iam-by-member)
- [Service Identities and Authoritative IAM](#service-identities-and-authoritative-iam) - [Service Identities and Authoritative IAM](#service-identities-and-authoritative-iam)
- [Using Shortcodes for Service Identities](#using-shortcodes-for-service-identities-in-additive-iam) - [Using Shortcodes for Service Identities in Additive Iam](#using-shortcodes-for-service-identities-in-additive-iam)
- [Service Identities and Manual IAM Grants](#service-identities-requiring-manual-iam-grants) - [Service Identities Requiring Manual Iam Grants](#service-identities-requiring-manual-iam-grants)
- [Shared VPC](#shared-vpc) - [Shared VPC](#shared-vpc)
- [Organization Policies](#organization-policies) - [Organization Policies](#organization-policies)
- [Factory](#organization-policy-factory) - [Organization Policy Factory](#organization-policy-factory)
- [Log Sinks](#log-sinks) - [Log Sinks](#log-sinks)
- [Data Access Logs](#data-access-logs) - [Data Access Logs](#data-access-logs)
- [Cloud KMS Encryption Keys](#cloud-kms-encryption-keys) - [Cloud Kms Encryption Keys](#cloud-kms-encryption-keys)
- [Tags](#tags) - [Tags](#tags)
- [Outputs](#outputs)
- [Files](#files)
- [Variables](#variables)
- [Outputs](#outputs)
<!-- END TOC -->
## Basic Project Creation ## Basic Project Creation
@ -570,8 +577,8 @@ output "compute_robot" {
``` ```
<!-- TFDOC OPTS files:1 --> <!-- TFDOC OPTS files:1 -->
<!-- BEGIN TFDOC -->
<!-- BEGIN TFDOC -->
## Files ## Files
| name | description | resources | | name | description | resources |
@ -639,5 +646,5 @@ output "compute_robot" {
| [project_id](outputs.tf#L75) | Project id. | | | [project_id](outputs.tf#L75) | Project id. | |
| [service_accounts](outputs.tf#L94) | Product robot service accounts in project. | | | [service_accounts](outputs.tf#L94) | Product robot service accounts in project. | |
| [sink_writer_identities](outputs.tf#L110) | Writer identities created for each sink. | | | [sink_writer_identities](outputs.tf#L110) | Writer identities created for each sink. | |
<!-- END TFDOC --> <!-- END TFDOC -->

126
tools/check_documentation.py
View File

@ -35,6 +35,7 @@ class State(enum.IntEnum):
SKIP = enum.auto() SKIP = enum.auto()
OK = enum.auto() OK = enum.auto()
FAIL_STALE_README = enum.auto() FAIL_STALE_README = enum.auto()
FAIL_STALE_TOC = enum.auto()
FAIL_UNSORTED_VARS = enum.auto() FAIL_UNSORTED_VARS = enum.auto()
FAIL_UNSORTED_OUTPUTS = enum.auto() FAIL_UNSORTED_OUTPUTS = enum.auto()
FAIL_VARIABLE_PERIOD = enum.auto() FAIL_VARIABLE_PERIOD = enum.auto()
@ -52,6 +53,7 @@ class State(enum.IntEnum):
State.SKIP: ' ', State.SKIP: ' ',
State.OK: '✓ ', State.OK: '✓ ',
State.FAIL_STALE_README: '✗R', State.FAIL_STALE_README: '✗R',
State.FAIL_STALE_TOC: '✗T',
State.FAIL_UNSORTED_VARS: 'SV', State.FAIL_UNSORTED_VARS: 'SV',
State.FAIL_UNSORTED_OUTPUTS: 'SO', State.FAIL_UNSORTED_OUTPUTS: 'SO',
State.FAIL_VARIABLE_PERIOD: '.V', State.FAIL_VARIABLE_PERIOD: '.V',
@ -71,74 +73,78 @@ def _check_dir(dir_name, exclude_files=None, files=False, show_extra=False):
diff = None diff = None
readme = readme_path.read_text() readme = readme_path.read_text()
mod_name = str(readme_path.relative_to(dir_path).parent) mod_name = str(readme_path.relative_to(dir_path).parent)
result = tfdoc.get_doc(readme) current_doc = tfdoc.get_doc(readme)
if not result: current_toc = tfdoc.get_toc(readme)
state = State.SKIP if current_doc or current_toc:
else: new_doc = tfdoc.create_doc(readme_path.parent, files, show_extra,
try: exclude_files, readme)
new_doc = tfdoc.create_doc(readme_path.parent, files, show_extra, new_toc = tfdoc.create_toc(readme)
exclude_files, readme) newvars = new_doc.variables
newvars = new_doc.variables newouts = new_doc.outputs
newouts = new_doc.outputs variables = [v.name for v in newvars if v.file.endswith('variables.tf')]
variables = [v.name for v in newvars if v.file.endswith('variables.tf')] outputs = [o.name for o in newouts if o.file.endswith('outputs.tf')]
outputs = [o.name for o in newouts if o.file.endswith('outputs.tf')]
except SystemExit:
state = state.SKIP
else:
state = State.OK
if new_doc.content != result['doc']: state = State.OK
state = State.FAIL_STALE_README
header = f'----- {mod_name} diff -----\n'
ndiff = difflib.ndiff(result['doc'].split('\n'),
new_doc.content.split('\n'))
diff = '\n'.join([header] + list(ndiff))
elif empty := [v.name for v in newvars if not v.description]: if current_doc and new_doc.content != current_doc['doc']:
state = state.FAIL_VARIABLE_DESCRIPTION state = State.FAIL_STALE_README
diff = "\n".join([ header = f'----- {mod_name} diff -----\n'
f'----- {mod_name} variables missing description -----', ndiff = difflib.ndiff(current_doc['doc'].splitlines(keepends=True),
', '.join(empty), new_doc.content.splitlines(keepends=True))
]) diff = ''.join([header] + [x for x in ndiff if x[0] != ' '])
elif empty := [o.name for o in newouts if not o.description]: elif current_toc and new_toc != current_toc['toc']:
state = state.FAIL_VARIABLE_DESCRIPTION state = State.FAIL_STALE_TOC
diff = "\n".join([ header = f'----- {mod_name} diff -----\n'
f'----- {mod_name} outputs missing description -----', ndiff = difflib.ndiff(current_toc['toc'].splitlines(keepends=True),
', '.join(empty), new_toc.splitlines(keepends=True))
]) diff = ''.join([header] + [x for x in ndiff if x[0] != ' '])
elif variables != sorted(variables): elif empty := [v.name for v in newvars if not v.description]:
state = state.FAIL_UNSORTED_VARS state = state.FAIL_VARIABLE_DESCRIPTION
diff = "\n".join([ diff = "\n".join([
f'----- {mod_name} variables -----', f'----- {mod_name} variables missing description -----',
f'variables should be in this order: ', ', '.join(empty),
', '.join(sorted(variables)), ])
])
elif outputs != sorted(outputs): elif empty := [o.name for o in newouts if not o.description]:
state = state.FAIL_UNSORTED_OUTPUTS state = state.FAIL_VARIABLE_DESCRIPTION
diff = "\n".join([ diff = "\n".join([
f'----- {mod_name} outputs -----', f'----- {mod_name} outputs missing description -----',
f'outputs should be in this order: ', ', '.join(empty),
', '.join(sorted(outputs)), ])
])
elif nc := [v.name for v in newvars if not v.description.endswith('.')]: elif variables != sorted(variables):
state = state.FAIL_VARIABLE_PERIOD state = state.FAIL_UNSORTED_VARS
diff = "\n".join([ diff = "\n".join([
f'----- {mod_name} variable descriptions missing ending period -----', f'----- {mod_name} variables -----',
', '.join(nc), f'variables should be in this order: ',
]) ', '.join(sorted(variables)),
])
elif nc := [o.name for o in newouts if not o.description.endswith('.')]: elif outputs != sorted(outputs):
state = state.FAIL_OUTPUT_PERIOD state = state.FAIL_UNSORTED_OUTPUTS
diff = "\n".join([ diff = "\n".join([
f'----- {mod_name} output descriptions missing ending period -----', f'----- {mod_name} outputs -----',
', '.join(nc), f'outputs should be in this order: ',
]) ', '.join(sorted(outputs)),
])
yield mod_name, state, diff elif nc := [v.name for v in newvars if not v.description.endswith('.')]:
state = state.FAIL_VARIABLE_PERIOD
diff = "\n".join([
f'----- {mod_name} variable descriptions missing ending period -----',
', '.join(nc),
])
elif nc := [o.name for o in newouts if not o.description.endswith('.')]:
state = state.FAIL_OUTPUT_PERIOD
diff = "\n".join([
f'----- {mod_name} output descriptions missing ending period -----',
', '.join(nc),
])
yield mod_name, state, diff
@click.command() @click.command()

90
tools/tfdoc.py
View File

@ -1,6 +1,6 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
# Copyright 2022 Google LLC # Copyright 2023 Google LLC
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. # you may not use this file except in compliance with the License.
@ -46,6 +46,7 @@ import string
import urllib.parse import urllib.parse
import click import click
import marko
__version__ = '2.1.0' __version__ = '2.1.0'
@ -80,6 +81,8 @@ OUT_RE = re.compile(r'''(?smx)
''') ''')
OUT_TEMPLATE = ('description', 'value', 'sensitive') OUT_TEMPLATE = ('description', 'value', 'sensitive')
TAG_RE = re.compile(r'(?sm)^\s*#\stfdoc:([^:]+:\S+)\s+(.*?)\s*$') TAG_RE = re.compile(r'(?sm)^\s*#\stfdoc:([^:]+:\S+)\s+(.*?)\s*$')
TOC_BEGIN = '<!-- BEGIN TOC -->'
TOC_END = '<!-- END TOC -->'
UNESCAPED = string.digits + string.ascii_letters + ' .,;:_-' UNESCAPED = string.digits + string.ascii_letters + ' .,;:_-'
VAR_ENUM = enum.Enum('V', 'OPEN ATTR ATTR_DATA SKIP CLOSE COMMENT TXT') VAR_ENUM = enum.Enum('V', 'OPEN ATTR ATTR_DATA SKIP CLOSE COMMENT TXT')
VAR_RE = re.compile(r'''(?smx) VAR_RE = re.compile(r'''(?smx)
@ -244,9 +247,7 @@ def format_doc(outputs, variables, files, show_extra=False):
if outputs: if outputs:
buffer += ['', '## Outputs', ''] buffer += ['', '## Outputs', '']
buffer += list(format_outputs(outputs, show_extra)) buffer += list(format_outputs(outputs, show_extra))
if buffer: return '\n'.join(buffer).strip()
buffer.append('')
return '\n'.join(buffer)
def format_files(items): def format_files(items):
@ -322,15 +323,39 @@ def format_variables(items, show_extra=True):
yield format yield format
def create_toc(readme):
'Create a Markdown table of contents a for README.'
doc = marko.parse(readme)
lines = []
headings = [x for x in doc.children if x.get_type() == 'Heading']
for h in headings[1:]:
title = h.children[0].children
slug = title.lower().strip()
slug = re.sub('[^\w\s-]', '', slug)
slug = re.sub('[-\s]+', '-', slug)
link = f'- [{title}](#{slug})'
indent = ' ' * (h.level - 2)
lines.append(f'{indent}{link}')
return "\n".join(lines)
# replace functions # replace functions
def get_doc(readme): def get_doc(readme):
'Check if README file is marked, and return current doc.' 'Check if README file is marked, and return current doc.'
m = re.search('(?sm)%s\n(.*)\n%s' % (MARK_BEGIN, MARK_END), readme) m = re.search('(?sm)%s(.*)%s' % (MARK_BEGIN, MARK_END), readme)
if not m: if not m:
return return
return {'doc': m.group(1), 'start': m.start(), 'end': m.end()} return {'doc': m.group(1).strip(), 'start': m.start(), 'end': m.end()}
def get_toc(readme):
'Check if README file is marked, and return current toc.'
t = re.search('(?sm)%s(.*)%s' % (TOC_BEGIN, TOC_END), readme)
if not t:
return
return {'toc': t.group(1).strip(), 'start': t.start(), 'end': t.end()}
def get_doc_opts(readme): def get_doc_opts(readme):
@ -373,24 +398,34 @@ def get_readme(readme_path):
raise SystemExit(f'Error opening README {readme_path}: {e}') raise SystemExit(f'Error opening README {readme_path}: {e}')
def replace_doc(readme_path, doc, readme=None): def render_doc(readme, doc):
'Replace document in module\'s README.md file.' 'Replace document in module\'s README.md file.'
readme = readme or get_readme(readme_path)
result = get_doc(readme) result = get_doc(readme)
if not result: if not result or doc == result['doc']:
raise SystemExit(f'Mark not found in README {readme_path}') return readme
if doc == result['doc']: return '\n'.join([
return readme[:result['start']].rstrip(),
try: MARK_BEGIN,
open(readme_path, 'w').write('\n'.join([ doc,
readme[:result['start']].rstrip(), MARK_END,
MARK_BEGIN, readme[result['end']:].lstrip(),
doc, ])
MARK_END,
readme[result['end']:].lstrip(),
])) def render_toc(readme, toc):
except (IOError, OSError) as e: 'Replace toc in module\'s README.md file.'
raise SystemExit(f'Error replacing README {readme_path}: {e}') result = get_toc(readme)
if not result or toc == result['toc']:
return readme
return '\n'.join([
readme[:result['start']].rstrip(),
'',
TOC_BEGIN,
toc,
TOC_END,
'',
readme[result['end']:].lstrip(),
])
@click.command() @click.command()
@ -405,10 +440,17 @@ def main(module_path=None, exclude_file=None, files=False, replace=True,
readme_path = os.path.join(module_path, 'README.md') readme_path = os.path.join(module_path, 'README.md')
readme = get_readme(readme_path) readme = get_readme(readme_path)
doc = create_doc(module_path, files, show_extra, exclude_file, readme) doc = create_doc(module_path, files, show_extra, exclude_file, readme)
toc = create_toc(readme)
tmp = render_doc(readme, doc.content)
final = render_toc(tmp, toc)
if replace: if replace:
replace_doc(readme_path, doc.content, readme) try:
with open(readme_path, 'w') as f:
f.write(final)
except (IOError, OSError) as e:
raise SystemExit(f'Error replacing README {readme_path}: {e}')
else: else:
print(doc) print(final)
if __name__ == '__main__': if __name__ == '__main__':