diff --git a/.github/workflows/linting.yml b/.github/workflows/linting.yml index 1c63b78d..6d773f9b 100644 --- a/.github/workflows/linting.yml +++ b/.github/workflows/linting.yml @@ -16,8 +16,6 @@ name: "Linting" on: pull_request: branches: - - fast-dev - - fast-dev-gke - master tags: - ci diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 3fc3fe56..3e452275 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -14,12 +14,10 @@ name: "Tests" on: - schedule: - - cron: "45 2 * * *" + # schedule: + # - cron: "45 2 * * *" pull_request: branches: - - fast-dev - - fast-dev-gke - master tags: - ci diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d82b1ac7..18a42f59 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -72,11 +72,13 @@ pytest tests/examples Once everything looks good, add/commit any pending changes then push and open a PR on GitHub. We typically enforce a set of design and style conventions, so please make sure you have familiarized yourself with the following sections and implemented them in your code, to avoid lengthy review cycles. HINT: if you work on high-latency or low-bandwidth network use `TF_PLUGIN_CACHE_DIR` environment variable to dramatically speed up the tests, for example: + ```bash TF_PLUGIN_CACHE_DIR=/tmp/tfcache pytest tests ``` Or just add into your [terraformrc](https://developer.hashicorp.com/terraform/cli/config/config-file): + ``` plugin_cache_dir = "$HOME/.terraform.d/plugin-cache" ``` @@ -544,6 +546,7 @@ locals { #### The `prefix` variable If you would like to use a "prefix" variable for resource names, please keep its definition consistent across all modules: + ```hcl # variables.tf variable "prefix" { @@ -563,6 +566,7 @@ locals { ``` For blueprints the prefix is mandatory: + ```hcl variable "prefix" { description = "Prefix used for resource names." @@ -608,6 +612,7 @@ The linting workflow tests: - that the correct copyright boilerplate is present in all files, using `tools/check_boilerplate.py` - that all Terraform code is linted via `terraform fmt` +- that Terraform variables and outputs are sorted alphabetically - that all README files have up to date outputs, variables, and files (where relevant) tables, via `tools/check_documentation.py` - that all links in README files are syntactically correct and valid if internal, via `tools/check_links.py` - that resource names used in FAST stages stay within a length limit, via `tools/check_names.py` @@ -639,90 +644,123 @@ The test workflow runs test suites in parallel. Refer to the next section for mo #### Using and writing tests -Our testing approach follows a simple philosophy: we mainly test to ensure code works, and it does not break due to changes to dependencies (modules) or provider resources. +Our testing approach follows a simple philosophy: we mainly test to ensure code works, and that it does not break due to changes to dependencies (modules) or provider resources. This makes testing very simple, as a successful `terraform plan` run in a test case is often enough. We only write more specialized tests when we need to check the output of complex transformations in `for` loops. -As our testing needs are very simple, we also wanted to reduce the friction required to write new tests as much as possible: our tests are written in Python, and use `pytest` which is the standard for the language. We adopted this approach instead of others (Inspec/Kitchen, Terratest) as it allows writing simple functions as test units using Python which is simple and widely known. +As our testing needs are very simple, we also wanted to reduce the friction required to write new tests as much as possible: our tests are written in Python and use `pytest` which is the standard for the language, leveraging our [`tftest`](https://pypi.org/project/tftest/) library, which wraps the Terraform executable and returns familiar data structures for most commands. -The last piece of our testing framework is our [`tftest`](https://pypi.org/project/tftest/) library, which wraps the Terraform executable and returns familiar data structures for most commands. +Writing `pytest` unit tests to check plan results is really easy, but since wrapping modules and examples in dedicated fixtures and hand-coding checks gets annoying after a while, we developed a thin layer that allows us to use `tfvars` files to run tests, and `yaml` results to check results. In some specific situations you might still want to interact directly with `tftest` via Python, if that's the case skip to the legacy approach below. -##### Testing end-to-end examples +##### Testing end-to-end examples via `tfvars` and `yaml` -Putting it all together, here is how an end-to-end blueprint test works. +Our new approach to testing requires you to: -Each example is a Python module in its own directory, and a Terraform fixture that calls the example as a module: +- create a folder in the right `tests` hierarchy where specific test files will be hosted +- define `tfvars` files each with a specific variable configuration to test +- define `yaml` "inventory" files with the plan and output results you want to test +- declare which of these files need to be run as tests in a `tftest.yaml` file + +Let's go through each step in succession, assuming you are testing the new `net-glb` module. + +First create a new folder under `tests/modules` replacing any dash in the module name with underscores. You also need to create an empty `__init__.py` file in it, since the folder represents a package from the point of view of `pytest`. Note that if you were testing a blueprint the folder would go in `tests/blueprints`. ```bash -tests/blueprints/cloud_operations/iam_delegated_role_grants/ -├── fixture -│   ├── main.tf -│   └── variables.tf -├── __init__.py -└── test_plan.py +mkdir tests/modules/net_glb +touch tests/modules/net_glb/__init__.py ``` -One point of note is that the folder contains a Python module, so any dash needs to be replaced with underscores to make it importable. The actual test in the `test_plan.py` file looks like this: - -```python -def test_resources(e2e_plan_runner): - "Test that plan works and the numbers of resources is as expected." - modules, resources = e2e_plan_runner() - assert len(modules) == 6 - assert len(resources) == 18 -``` - -It uses our pytest `e2e_plan_runner` fixture, which assumes a Terraform test setup is present in the `fixture` folder alongside the test file, runs `plan` on it, and returns the number of modules and resources. - -The Terraform fixture is a single block that runs the whole example as a module, and a handful of variables that can be used to test different configurations (not used above so they could be replaced with static strings). +Then define a `tfvars` file with one of the module configurations you want to test. If you have a lot of variables which are shared across different tests, you can group all the common variables in a single `tfvars` file and associate it with each test's specific `tfvars` file (check the [organization module test](./tests/modules/organization/tftest.yaml) for an example). ```hcl -module "test" { - source = "../../../../../blueprints/cloud-operations/asset-inventory-feed-remediation" - project_create = var.project_create - project_id = var.project_id +# file: tests/modules/net_glb/test-simple.tfvars +name = "glb-test-0" +project_id = "my-project" +backend_buckets_config = { + default = { + bucket_name = "my-bucket" + } } ``` -You can run this test as part of or entire suite of tests, the blueprints suite, or individually: +Next define the corresponding inventory `yaml` file which will be used to assert values from the plan that uses the `tfvars` file above. In the inventory file you have three sections available: + +- `values` is a map of resource indexes (the same ones used by Terraform state) and their attribute name and values; you can define just the attributes you are interested in and the other will be ignored +- `counts` is a map of resource types (eg `google_compute_engine`) and the number of times each type occurs in the plan; here too just define the ones the need checking +- `outputs` is a map of outputs and their values; where a value is unknown at plan time use the special `__missing__` token + +```yaml +# file: tests/modules/net_glb/test-simple.yaml +values: + google_compute_global_forwarding_rule.default: + description: Terraform managed. + load_balancing_scheme: EXTERNAL + google_compute_target_http_proxy.default[0]: + name: glb-test-1 +counts: + google_compute_backend_bucket: 1 + google_compute_global_forwarding_rule: 1 + google_compute_health_check: 1 + google_compute_target_http_proxy: 1 + google_compute_url_map: 1 +outputs: + address: __missing__ + backend_service_ids: __missing__ + forwarding_rule: __missing__ + group_ids: __missing__ + health_check_ids: __missing__ + neg_ids: __missing__ +``` + +Create as many pairs of `tfvars`/`yaml` files as you need to test every scenario and feature, then create the file that triggers our fixture and converts them into `pytest` tests. + +```yaml +# file: tests/modules/net_glb/tftest.yaml +module: modules/net-glb +# if there are variables shared among all tests you can define a common file +# common_tfvars: +# - defaults.tfvars +tests: + test-plan: + tfvars: + - test-plan.tfvars + - test-plan-extra.tfvars +``` + +A good example of tests showing different ways of leveraging our framework is in the [`tests/modules/organization`](./tests/modules/organization) folder. + +##### Writing tests in Python (legacy approach) + +Where possible, we recommend using the testing framework described in the previous section. However, if you need it, you can still write tests using Python directly. + +In general, you should try to use the `plan_summary` fixture, which runs a a terraform plan and returns a `PlanSummary` object. The most important arguments to `plan_summary` are: +- the path of the Terraform module you want to test, relative to the root of the repository +- a list of paths representing the tfvars file to pass in to terraform. These paths are relative to the python file defining the test. + +If successful, `plan_summary` will return a `PlanSummary` object with the `values`, `counts` and `outputs` attributes following the same semantics described in the previous section. You can use this fields to write your custom tests. + +Like before let's imagine we're writing a (python) test for `net-glb` module. First create a new folder under `tests/modules` replacing any dash in the module name with underscores. You also need to create an empty `__init__.py` file in it, to ensure `pytest` discovers you new tests automatically. ```bash -# run all tests -pytest -# only run example tests -pytest tests/blueprints -# only run this example tests -pytest tests/blueprints/cloud_operations/iam_delegated_role_grants/ -# only run a single unit -pytest tests/blueprints/cloud_operations/iam_delegated_role_grants/test_plan.py::test_resources +mkdir tests/modules/net_glb +touch tests/modules/net_glb/__init__.py ``` -##### Testing modules - -The same approach used above can also be used for testing modules when a simple plan is enough to validate code. When specific features need to be tested though, the `plan_runner` pytest fixture can be used so that plan resources are returned for inspection. - -The following example from the `project` module leverages variables in the Terraform fixture to define which module resources are returned from plan. - +Now create a file containing your tests, e.g. `test_plan.py`: ```python -def test_iam(plan_runner): - "Test IAM bindings." - iam = ( - '{"roles/owner" = ["user:one@example.org"],' - '"roles/viewer" = ["user:two@example.org", "user:three@example.org"]}' - ) - _, resources = plan_runner(iam=iam) - roles = dict((r['values']['role'], r['values']['members']) - for r in resources if r['type'] == 'google_project_iam_binding') - assert roles == { - 'roles/owner': ['user:one@example.org'], - 'roles/viewer': ['user:three@example.org', 'user:two@example.org']} +def test_name(plan_summary, tfvars_to_yaml, tmp_path): + s = plan_summary('modules/net-glb', tf_var_files=['test-plan.tfvars']) + address = 'google_compute_url_map.default' + assert s.values[address]['project'] == 'my-project' ``` +For more examples on how to write python tests, the tests for [`organization`](./tests/modules/organization/test_plan_org_policies.py) and [`net-vpc`](./tests/modules/net_vpc/test_routes.py) modules. + #### Testing documentation examples -Most of our documentation examples are also tested via the `examples` test suite. To enable an example for testing just use the special `tftest` comment as the last line in the example, listing the number of modules and resources tested. +Most of our documentation examples are also tested via the `examples` test suite. To enable an example for testing just use the special `tftest` comment as the last line in the example, listing the number of modules and resources expected. -A few preset variables are available for use, as shown in this example from the `dns` module documentation. +A [few preset variables](./tests/examples/variables.tf) are available for use, as shown in this example from the `dns` module documentation. ```hcl module "private-dns" { @@ -739,6 +777,25 @@ module "private-dns" { # tftest modules=1 resources=2 ``` +Note that all HCL code examples in READMEs are automatically tested. To prevent this behavior, include `tftest skip` somewhere in the code. + +#### Running tests from a temporary directory + + Most of the time you can run tests using the `pytest` command as described in previous. However, the `plan_summary` fixture allows copying the root module and running the test from a temporary directory. + +To enable this option, just define the environment variable `TFTEST_COPY` and any tests using the `plan_summary` fixture will automatically run from a temporary directory. + +Running tests from temporary directories is useful if: +- you're running tests in parallel using `pytest-xdist`. In this case, just run you tests as follows: + ```bash + TFTEST_COPY=1 pytest -n 4 + ``` +- you're running tests for the `fast/` directory which contain tfvars and auto.tfvars files (which are read by terraform automatically) making your tests fail. In this case, you can run + ``` + TFTEST_COPY=1 pytest fast/ + ``` + + #### Fabric tools The main tool you will interact with in development is `tfdoc`, used to generate file, output and variable tables in README documents.