Merge pull request #598 from GoogleCloudPlatform/597-fix-link-checker
[#597] Fix link_checker
This commit is contained in:
Julio Castillo
GitHub
commit
be644c6f29
|
|
@ -29,7 +29,7 @@ All notable changes to this project will be documented in this file.
|
|||
- **incompatible change** the variable for service identities IAM has changed in the project factory
|
||||
- add `data-catalog-policy-tag` module
|
||||
- new [workload identity federetion example](examples/cloud-operations/workload-identity-federation)
|
||||
- new `api-gateway` [module](/modules/api-gateway) and [example](examples/serverless/api-gateway).
|
||||
- new `api-gateway` [module](./modules/api-gateway) and [example](examples/serverless/api-gateway).
|
||||
- **incompatible change** the `psn_ranges` variable has been renamed to `psa_ranges` in the `net-vpc` module and its type changed from `list(string)` to `map(string)`
|
||||
- **incompatible change** removed `iam` flag for organization and folder level sinks
|
||||
- **incompatible change** removed `ingress_settings` configuration option in the `cloud-functions` module.
|
||||
|
|
|
|||
|
|
@ -300,7 +300,7 @@ Subnets created using the `net-vpc` module are PGA-enabled by default.
|
|||
- 199.36.153.4/30 (`restricted.googleapis.com`) and 199.36.153.8/30 (`private.googleapis.com`) should be routed from on-premises to the trusted landing VPC, and from there to the `default-internet-gateway`. \
|
||||
The `vpn_onprem_configs` variable contains the ranges advertised from GCP to on-premises. Furthermore, the trusted landing VPC (e.g. see `landing-trusted-vpc` in [`landing.tf`](./landing.tf)) has explicit routes to send traffic destined to restricted and private - googleapis.com to the Internet gateway (which works for Google APIs only, and not for the whole Internet, since Cloud NAT is not configured in the trusted landing VPC).
|
||||
|
||||
- On-premises, a private DNS zone for `googleapis.com` should be created and configured per [this article](https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid#config-domain). Its configuration can be copied from the module `googleapis-private-zone` in [`dns.tf`](./dns.tf)
|
||||
- On-premises, a private DNS zone for `googleapis.com` should be created and configured per [this article](https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid#config-domain). Its configuration can be copied from the module `googleapis-private-zone` in [`dns-landing.tf`](./dns-landing.tf)
|
||||
|
||||
### Preliminar activities
|
||||
|
||||
|
|
@ -336,9 +336,7 @@ Copy `vpc-peering-prod.tf` to `vpc-peering-staging.tf` and replace "prod" with "
|
|||
|
||||
Configure the NVAs deployed or update the sample [NVA config file](data/nva-startup-script.tftpl) making sure they support the new subnets.
|
||||
|
||||
DNS configurations are managed in the `dns-*.tf` files.
|
||||
Copy the `dns-prod.tf` to `dns-staging.tf` and replace within the files "prod" with "staging", where relevant.
|
||||
Don't forget to add a peering zone in the landing project and point it to the newly created environment private zone.
|
||||
DNS configurations are centralised in the `dns-*.tf` files. Spokes delegate DNS resolution to Landing through DNS peering, and optionally define a private zone (e.g. `dev.gcp.example.com`) which the landing peers to. To configure DNS for a new environment, copy one of the other environments DNS files (e.g. (dns-dev.tf)[dns-dev.tf]) into a new `dns-*.tf` file suffixed with the environment name (e.g. `dns-staging.tf`), and update its content accordingly. Don't forget to add a peering zone from the landing to the newly created environment private zone.
|
||||
|
||||
<!-- TFDOC OPTS files:1 show_extra:1 -->
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
|
|
|||
|
|
@ -228,7 +228,7 @@ Subnets created by the `net-vpc` module are PGA-enabled by default.
|
|||
- 199.36.153.4/30 (`restricted.googleapis.com`) and 199.36.153.8/30 (`private.googleapis.com`) should be routed from on-prem to VPC, and from there to the `default-internet-gateway`. \
|
||||
Per variable `vpn_onprem_configs` such ranges are advertised to onprem - furthermore every VPC (e.g. see `landing-vpc` in [`landing.tf`](./landing.tf)) has explicit routes set in case the `0.0.0.0/0` route is changed.
|
||||
|
||||
- A private DNS zone for `googleapis.com` should be created and configured per [this article](https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid#config-domain), as implemented in module `googleapis-private-zone` in [`dns.tf`](./dns.tf)
|
||||
- A private DNS zone for `googleapis.com` should be created and configured per [this article](https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid#config-domain), as implemented in module `googleapis-private-zone` in [`dns-landing.tf`](./dns-landing.tf)
|
||||
|
||||
### Preliminar activities
|
||||
|
||||
|
|
@ -259,7 +259,7 @@ The new VPC requires a set of dedicated CIDRs, one per region, added to variable
|
|||
>
|
||||
Variables managing L7 Interal Load Balancers (`l7ilb_subnets`) and Private Service Access (`psa_ranges`) should also be adapted, and subnets and firewall rules for the new spoke should be added as described above.
|
||||
|
||||
DNS configurations are centralised in the `dns.tf` file. Spokes delegate DNS resolution to Landing through DNS peering, and optionally define a private zone (e.g. `staging.gcp.example.com`) which the landing peers to. To configure DNS for a new environment, copy all the `prod-*` modules in the `dns.tf` file to `staging-*`, and update their content accordingly. Don't forget to add a peering zone from Landing to the newly created environment private zone.
|
||||
DNS configurations are centralised in the `dns-*.tf` files. Spokes delegate DNS resolution to Landing through DNS peering, and optionally define a private zone (e.g. `dev.gcp.example.com`) which the landing peers to. To configure DNS for a new environment, copy one of the other environments DNS files (e.g. (dns-dev.tf)[dns-dev.tf]) into a new `dns-*.tf` file suffixed with the environment name (e.g. `dns-staging.tf`), and update its content accordingly. Don't forget to add a peering zone from the landing to the newly created environment private zone.
|
||||
|
||||
<!-- TFDOC OPTS files:1 show_extra:1 -->
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
|
|
|||
|
|
@ -242,7 +242,7 @@ Subnets created by the `net-vpc` module are PGA-enabled by default.
|
|||
- 199.36.153.4/30 (`restricted.googleapis.com`) and 199.36.153.8/30 (`private.googleapis.com`) should be routed from on-prem to VPC, and from there to the `default-internet-gateway`. \
|
||||
Per variable `vpn_onprem_configs` such ranges are advertised to onprem - furthermore every VPC (e.g. see `landing-vpc` in [`landing.tf`](./landing.tf)) has explicit routes set in case the `0.0.0.0/0` route is changed.
|
||||
|
||||
- A private DNS zone for `googleapis.com` should be created and configured per [this article](https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid#config-domain), as implemented in module `googleapis-private-zone` in [`dns.tf`](./dns.tf)
|
||||
- A private DNS zone for `googleapis.com` should be created and configured per [this article](https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid#config-domain), as implemented in module `googleapis-private-zone` in [dns-landing.tf](./dns-landing.tf)
|
||||
|
||||
### Preliminar activities
|
||||
|
||||
|
|
@ -282,7 +282,7 @@ VPN configuration also controls BGP advertisements, which requires the following
|
|||
- `vpn_onprem_configs` to configure the new advertisments to on-premises for the new CIDRs
|
||||
- `vpn_spoke_configs` to configure the new advertisements to `landing` for the new VPC - new keys (one per region) should be added, such as e.g. `staging-ew1` and `staging-ew4`
|
||||
|
||||
DNS configurations are centralised in the `dns.tf` file. Spokes delegate DNS resolution to Landing through DNS peering, and optionally define a private zone (e.g. `staging.gcp.example.com`) which the landing peers to. To configure DNS for a new environment, copy all the `prod-*` modules in the `dns.tf` file to `staging-*`, and update their content accordingly. Don't forget to add a peering zone from Landing to the newly created environment private zone.
|
||||
DNS configurations are centralised in the `dns-*.tf` files. Spokes delegate DNS resolution to Landing through DNS peering, and optionally define a private zone (e.g. `dev.gcp.example.com`) which the landing peers to. To configure DNS for a new environment, copy one of the other environments DNS files (e.g. (dns-dev.tf)[dns-dev.tf]) into a new `dns-*.tf` file suffixed with the environment name (e.g. `dns-staging.tf`), and update its content accordingly. Don't forget to add a peering zone from the landing to the newly created environment private zone.
|
||||
|
||||
<!-- TFDOC OPTS files:1 show_extra:1 -->
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
|
|
|||
|
|
@ -84,7 +84,7 @@ If you're not using Fast, refer to the [Variables](#variables) table at the bott
|
|||
Besides the values above, a project factory takes 2 additional inputs:
|
||||
|
||||
- `data/defaults.yaml`, manually configured by adapting the [`data/defaults.yaml`](./data/defaults.yaml), which defines per-environment default values e.g., for billing alerts and labels.
|
||||
- `data/projects/*.yaml`, one file per project (optionally grouped in folders), which configures each project. A [`data/projects/project.yaml`](./data/projects/project.yaml) is provided as reference and documentation for the schema. Projects will be named after the filename, e.g., `fast-dev-lab0.yaml` will create project `fast-dev-lab0`.
|
||||
- `data/projects/*.yaml`, one file per project (optionally grouped in folders), which configures each project. A [`data/projects/project.yaml`](./data/projects/project.yaml.sample) is provided as reference and documentation for the schema. Projects will be named after the filename, e.g., `fast-dev-lab0.yaml` will create project `fast-dev-lab0`.
|
||||
|
||||
Once the configuration is complete, run the project factory by running
|
||||
|
||||
|
|
|
|||
|
|
@ -87,13 +87,13 @@ module "iot-platform" {
|
|||
```
|
||||
|
||||
## Example integrated with Data Foundation Platform
|
||||
In this example, we will show how to extend the **[Data Foundations Platform](../../data-solutions/data-platform-foundations/)** to include IoT Platform as a new source of data.
|
||||
In this example, we will show how to extend the **[Data Foundations Platform](../../examples/data-solutions/data-platform-foundations/)** to include IoT Platform as a new source of data.
|
||||
|
||||
|
||||
|
||||
1. First, we will setup Environment following instructions in **[Environment Setup](../../data-solutions/data-platform-foundations/01-environment/)** to setup projects and SAs required. Get output variable project_ids.landing as will be used later
|
||||
1. First, we will setup Environment following instructions in **[Environment Setup](../../examples/data-solutions/data-platform-foundations/)** to setup projects and SAs required. Get output variable project_ids.landing as will be used later
|
||||
|
||||
1. Second, execute instructions in **[Environment Setup](../../data-solutions/data-platform-foundations/02-resources/)** to provision PubSub, DataFlow, BQ,... Get variable landing-pubsub as will be used later to create IoT Registry
|
||||
1. Second, execute instructions in **[Environment Setup](../../examples/data-solutions/data-platform-foundations/)** to provision PubSub, DataFlow, BQ,... Get variable landing-pubsub as will be used later to create IoT Registry
|
||||
|
||||
1. Now it is time to provision IoT Platform. Modify landing-project-id and landing_pubsub_topic_id with output variables obtained before. Create device certificates as shown in the Simple Example and register them in devices.yaml file together with deviceids.
|
||||
|
||||
|
|
@ -112,7 +112,7 @@ module "iot-platform" {
|
|||
}
|
||||
# tftest:skip
|
||||
```
|
||||
1. After that, we can setup the pipeline "PubSub to BigQuery" shown at **[Pipeline Setup](../../data-solutions/data-platform-foundations/03-pipeline/pubsub_to_bigquery.md)**
|
||||
1. After that, we can setup the pipeline "PubSub to BigQuery" shown at **[Pipeline Setup](../../examples/data-solutions/data-platform-foundations/)**
|
||||
|
||||
1. Finally, instead of testing the pipeline by sending messages to PubSub, we can now test sending telemetry messages from simulated IoT devices to our IoT Platform, for example using the MQTT demo client at https://github.com/googleapis/nodejs-iot/tree/main/samples/mqtt_example . We shall edit the client script cloudiot_mqtt_example_nodejs.js to send messages following the pipeline message format, so they are processed by DataFlow job and inserted in the BigQuery table.
|
||||
```
|
||||
|
|
|
|||
|
|
@ -486,11 +486,11 @@ resource "tls_self_signed_cert" "self_signed_cert" {
|
|||
|
||||
An External Global Load Balancer is made of multiple components, that change depending on the configurations. Sometimes, it may be tricky to understand what they are, and how they relate to each other. Following, we provide a very brief overview to become more familiar with them.
|
||||
|
||||
- The global load balancer [forwarding rule](global_forwarding_rule.tf) binds a frontend public Virtual IP (VIP) to an HTTP(S) [target proxy](target_proxy.tf).
|
||||
- If the target proxy is HTTPS, it requires one or more managed or unmanaged [SSL certificates](ssl_certificates.tf).
|
||||
- Target proxies leverage [url-maps](url_map.tf): set of L7 rules, which create a mapping between specific hostnames, URIs (and more) to one or more [backends services](backend_services.tf).
|
||||
- [Backend services](backend_services.tf) can either link to a bucket or one or multiple groups, which can be GCE instance groups or NEGs. It is assumed in this module that buckets and groups are previously created through other modules, and passed in as input variables.
|
||||
- Backend services support one or more [health checks](health_checks.tf), used to verify that the backend is indeed healthy, so that traffic can be forwarded to it. Health checks currently supported in this module are HTTP, HTTPS, HTTP2, SSL, TCP.
|
||||
- The global load balancer [forwarding rule](global-forwarding-rule.tf) binds a frontend public Virtual IP (VIP) to an HTTP(S) [target proxy](target-proxy.tf).
|
||||
- If the target proxy is HTTPS, it requires one or more managed or unmanaged [SSL certificates](ssl-certificates.tf).
|
||||
Target proxies leverage [url-maps](url-map.tf): set of L7 rules, which create a mapping between specific hostnames, URIs (and more) to one or more [backends services](backend-services.tf).
|
||||
- [Backend services](backend-services.tf) can either link to a bucket or one or multiple groups, which can be GCE instance groups or NEGs. It is assumed in this module that buckets and groups are previously created through other modules, and passed in as input variables.
|
||||
- Backend services support one or more [health checks](health-checks.tf), used to verify that the backend is indeed healthy, so that traffic can be forwarded to it. Health checks currently supported in this module are HTTP, HTTPS, HTTP2, SSL, TCP.
|
||||
|
||||
<!-- TFDOC OPTS files:1 -->
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ destinations. Its main use is in CI pipelines triggered by pull requests.
|
|||
|
||||
import collections
|
||||
import pathlib
|
||||
import requests
|
||||
import urllib.parse
|
||||
|
||||
import click
|
||||
|
|
@ -31,37 +32,57 @@ DOC = collections.namedtuple('DOC', 'path relpath links')
|
|||
LINK = collections.namedtuple('LINK', 'dest valid')
|
||||
|
||||
|
||||
def check_docs(dir_name):
|
||||
'Traverse dir_name and check links in Markdown files.'
|
||||
def check_link(link, readme_path, external):
|
||||
'Checks if a link element has a valid destination.'
|
||||
link_valid = None
|
||||
url = urllib.parse.urlparse(link.dest)
|
||||
# If the link is public, say the link is anyway valid
|
||||
# if --external is not set; check the link otherwise
|
||||
if url.scheme:
|
||||
link_valid = True
|
||||
if external:
|
||||
try:
|
||||
response = requests.get(link.dest)
|
||||
link_valid = response.ok
|
||||
except requests.exceptions.RequestException:
|
||||
link_valid = False
|
||||
# The link is private
|
||||
else:
|
||||
link_valid = (readme_path.parent / url.path).exists()
|
||||
return LINK(link.dest, link_valid)
|
||||
|
||||
|
||||
def check_docs(dir_name, external=False):
|
||||
'Traverses dir_name and checks for all Markdown files.'
|
||||
dir_path = BASEDIR / dir_name
|
||||
parser = marko.parser.Parser()
|
||||
for readme_path in sorted(dir_path.glob('**/*.md')):
|
||||
if '.terraform' in str(readme_path) or '.pytest' in str(readme_path):
|
||||
continue
|
||||
|
||||
root = parser.parse(readme_path.read_text())
|
||||
elements = collections.deque([root])
|
||||
links = []
|
||||
for el in marko.parser.Parser().parse(readme_path.read_text()).children:
|
||||
if not isinstance(el, marko.block.Paragraph):
|
||||
continue
|
||||
for subel in el.children:
|
||||
if not isinstance(subel, marko.inline.Link):
|
||||
continue
|
||||
link_valid = None
|
||||
url = urllib.parse.urlparse(subel.dest)
|
||||
if url.scheme:
|
||||
link_valid = True
|
||||
else:
|
||||
link_valid = (readme_path.parent / url.path).exists()
|
||||
links.append(LINK(subel.dest, link_valid))
|
||||
while elements:
|
||||
el = elements.popleft()
|
||||
if isinstance(el, marko.inline.Link):
|
||||
links.append(check_link(el, readme_path, external))
|
||||
elif hasattr(el, 'children'):
|
||||
elements.extend(el.children)
|
||||
|
||||
yield DOC(readme_path, str(readme_path.relative_to(dir_path)), links)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.argument('dirs', type=str, nargs=-1)
|
||||
def main(dirs):
|
||||
'Check links in Markdown files contained in dirs.'
|
||||
@click.option('-e', '--external', is_flag=True, default=False,
|
||||
help='Whether to test external links.')
|
||||
def main(dirs, external):
|
||||
'Checks links in Markdown files contained in dirs.'
|
||||
errors = 0
|
||||
for dir_name in dirs:
|
||||
print(f'----- {dir_name} -----')
|
||||
for doc in check_docs(dir_name):
|
||||
for doc in check_docs(dir_name, external):
|
||||
state = '✓' if all(l.valid for l in doc.links) else '✗'
|
||||
print(f'[{state}] {doc.relpath} ({len(doc.links)})')
|
||||
if state == '✗':
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
click
|
||||
marko
|
||||
requests
|
||||
yamale
|
||||
yapf
|
||||
|
|
|
|||
Loading…
Reference in New Issue