cloud-foundation-fabric/fast/stages/3-project-factory/dev/README.md

92 lines
5.7 KiB
Markdown
Raw Normal View History

# Project factory
The Project Factory (or PF) builds on top of your foundations to create and set up projects (and related resources) to be used for your workloads.
It is organized in folders representing environments (e.g., "dev", "prod"), each implemented by a stand-alone terraform [process factory](../../../../blueprints/factories/README.md).
## Design overview and choices
<p align="center">
<img src="diagram.svg" alt="Project factory diagram">
</p>
A single factory creates projects in a well-defined context, according to your resource management structure. For example, in the diagram above, each Team is structured to have specific folders projects for a given environment, such as Production and Development, per the resource management structure configured in stage `01-resman`.
Projects for each environment across different teams are created by dedicated service accounts, as exemplified in the diagram above. While there's no intrinsic limitation regarding where the project factory can create a projects, the IAM bindings for the service account effectively enforce boundaries (e.g., the production service account shouldn't be able to create or have any access to the development projects, and vice versa).
The project factory stage lightly wraps the underlying [project-factory module](../../../../modules/project-factory/), including Shared VPC service project attachment, VPC SC perimeter membership, etc.
## How to run this stage
This stage is meant to be executed after "foundational stages" (i.e., stages [`00-bootstrap`](../../0-bootstrap), [`01-resman`](../../1-resman), 02-networking (either [VPN](../../2-networking-b-vpn), [NVA](../../2-networking-c-nva), [NVA with BGP support](../../2-networking-e-nva-bgp)) and [`02-security`](../../2-security)) have been run.
IAM interface refactor (#1595) * IAM modules refactor proposal * policy * subheading * Update 20230816-iam-refactor.md * log Julio's +1 * data-catalog-policy-tag * dataproc * dataproc * folder * folder * folder * folder * project * better filtering in test examples * project * folder * folder * organization * fix variable descriptions * kms * net-vpc * dataplex-datascan * modules/iam-service-account * modules/source-repository/ * blueprints/cloud-operations/vm-migration/ * blueprints/third-party-solutions/wordpress * dataplex-datascan * blueprints/cloud-operations/workload-identity-federation * blueprints/data-solutions/cloudsql-multiregion/ * blueprints/data-solutions/composer-2 * Update 20230816-iam-refactor.md * Update 20230816-iam-refactor.md * capture discussion in architectural doc * update variable names and refactor proposal * project * blueprints first round * folder * organization * data-catalog-policy-tag * re-enable folder inventory * project module style fix * dataproc * source-repository * source-repository tests * dataplex-datascan * dataplex-datascan tests * net-vpc * net-vpc test examples * iam-service-account * iam-service-account test examples * kms * boilerplate * tfdoc * fix module tests * more blueprint fixes * fix typo in data blueprints * incomplete refactor of data platform foundations * tfdoc * data platform foundation * refactor data platform foundation iam locals * remove redundant example test * shielded folder fix * fix typo * project factory * project factory outputs * tfdoc * test workflow: less verbose tests, fix tf version * re-enable -vv, shorter traceback, fix action version * ignore github extension warning, re-enable action version * fast bootstrap IAM, untested * bootstrap stage IAM fixes * stage 0 tests * fast stage 1 * tenant stage 1 * minor changes to fast stage 0 and 1 * fast security stage * fast mt stage 0 * fast mt stage 0 * fast pf
2023-08-20 00:44:20 -07:00
It's of course possible to run this stage in isolation, by making sure the architectural prerequisites are satisfied (e.g., networking), and that the Service Account running the stage is granted the appropriate roles.
### Provider and Terraform variables
As all other FAST stages, the [mechanism used to pass variable values and pre-built provider files from one stage to the next](../../0-bootstrap/README.md#output-files-and-cross-stage-variables) is also leveraged here.
The commands to link or copy the provider and terraform variable files can be easily derived from the `stage-links.sh` script in the FAST root folder, passing it a single argument with the local output files folder (if configured) or the GCS output bucket in the automation project (derived from stage 0 outputs). The following examples demonstrate both cases, and the resulting commands that then need to be copy/pasted and run.
```bash
../../../stage-links.sh ~/fast-config
# copy and paste the following commands for '3-project-factory'
2023-05-15 03:55:43 -07:00
ln -s ~/fast-config/providers/3-project-factory-providers.tf ./
ln -s ~/fast-config/tfvars/0-globals.auto.tfvars.json ./
2023-05-15 03:55:43 -07:00
ln -s ~/fast-config/tfvars/0-bootstrap.auto.tfvars.json ./
ln -s ~/fast-config/tfvars/1-resman.auto.tfvars.json ./
ln -s ~/fast-config/tfvars/2-networking.auto.tfvars.json ./
ln -s ~/fast-config/tfvars/2-security.auto.tfvars.json ./
```
```bash
../../../stage-links.sh gs://xxx-prod-iac-core-outputs-0
# copy and paste the following commands for '3-project-factory'
gcloud alpha storage cp gs://xxx-prod-iac-core-outputs-0/providers/3-project-factory-providers.tf ./
gcloud alpha storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/0-globals.auto.tfvars.json ./
gcloud alpha storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/0-bootstrap.auto.tfvars.json ./
gcloud alpha storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/1-resman.auto.tfvars.json ./
gcloud alpha storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/2-networking.auto.tfvars.json ./
gcloud alpha storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/2-security.auto.tfvars.json ./
```
IAM interface refactor (#1595) * IAM modules refactor proposal * policy * subheading * Update 20230816-iam-refactor.md * log Julio's +1 * data-catalog-policy-tag * dataproc * dataproc * folder * folder * folder * folder * project * better filtering in test examples * project * folder * folder * organization * fix variable descriptions * kms * net-vpc * dataplex-datascan * modules/iam-service-account * modules/source-repository/ * blueprints/cloud-operations/vm-migration/ * blueprints/third-party-solutions/wordpress * dataplex-datascan * blueprints/cloud-operations/workload-identity-federation * blueprints/data-solutions/cloudsql-multiregion/ * blueprints/data-solutions/composer-2 * Update 20230816-iam-refactor.md * Update 20230816-iam-refactor.md * capture discussion in architectural doc * update variable names and refactor proposal * project * blueprints first round * folder * organization * data-catalog-policy-tag * re-enable folder inventory * project module style fix * dataproc * source-repository * source-repository tests * dataplex-datascan * dataplex-datascan tests * net-vpc * net-vpc test examples * iam-service-account * iam-service-account test examples * kms * boilerplate * tfdoc * fix module tests * more blueprint fixes * fix typo in data blueprints * incomplete refactor of data platform foundations * tfdoc * data platform foundation * refactor data platform foundation iam locals * remove redundant example test * shielded folder fix * fix typo * project factory * project factory outputs * tfdoc * test workflow: less verbose tests, fix tf version * re-enable -vv, shorter traceback, fix action version * ignore github extension warning, re-enable action version * fast bootstrap IAM, untested * bootstrap stage IAM fixes * stage 0 tests * fast stage 1 * tenant stage 1 * minor changes to fast stage 0 and 1 * fast security stage * fast mt stage 0 * fast mt stage 0 * fast pf
2023-08-20 00:44:20 -07:00
If you're not using FAST, refer to the [Variables](#variables) table at the bottom of this document for a full list of variables, their origin (e.g., a stage or specific to this one), and descriptions explaining their meaning.
Besides the values above, the project factory is driven by YAML data files, with one file per project. Please refer to the underlying [project factory module](../../../../modules/project-factory/) documentation for details on the format.
IAM interface refactor (#1595) * IAM modules refactor proposal * policy * subheading * Update 20230816-iam-refactor.md * log Julio's +1 * data-catalog-policy-tag * dataproc * dataproc * folder * folder * folder * folder * project * better filtering in test examples * project * folder * folder * organization * fix variable descriptions * kms * net-vpc * dataplex-datascan * modules/iam-service-account * modules/source-repository/ * blueprints/cloud-operations/vm-migration/ * blueprints/third-party-solutions/wordpress * dataplex-datascan * blueprints/cloud-operations/workload-identity-federation * blueprints/data-solutions/cloudsql-multiregion/ * blueprints/data-solutions/composer-2 * Update 20230816-iam-refactor.md * Update 20230816-iam-refactor.md * capture discussion in architectural doc * update variable names and refactor proposal * project * blueprints first round * folder * organization * data-catalog-policy-tag * re-enable folder inventory * project module style fix * dataproc * source-repository * source-repository tests * dataplex-datascan * dataplex-datascan tests * net-vpc * net-vpc test examples * iam-service-account * iam-service-account test examples * kms * boilerplate * tfdoc * fix module tests * more blueprint fixes * fix typo in data blueprints * incomplete refactor of data platform foundations * tfdoc * data platform foundation * refactor data platform foundation iam locals * remove redundant example test * shielded folder fix * fix typo * project factory * project factory outputs * tfdoc * test workflow: less verbose tests, fix tf version * re-enable -vv, shorter traceback, fix action version * ignore github extension warning, re-enable action version * fast bootstrap IAM, untested * bootstrap stage IAM fixes * stage 0 tests * fast stage 1 * tenant stage 1 * minor changes to fast stage 0 and 1 * fast security stage * fast mt stage 0 * fast mt stage 0 * fast pf
2023-08-20 00:44:20 -07:00
Once the configuration is complete, run the project factory with:
```bash
terraform init
terraform apply
```
<!-- TFDOC OPTS files:1 show_extra:1 -->
<!-- BEGIN TFDOC -->
## Files
| name | description | modules |
|---|---|---|
| [main.tf](./main.tf) | Project factory. | <code>project-factory</code> |
| [outputs.tf](./outputs.tf) | Module outputs. | |
| [variables.tf](./variables.tf) | Module variables. | |
## Variables
| name | description | type | required | default | producer |
|---|---|:---:|:---:|:---:|:---:|
FAST multitenant bootstrap and resource management, rename org-level FAST stages (#1052) * rename stages * remove support for external org billing, rename output files * resman: make groups optional, align on new billing account variable * bootstrap: multitenant outputs * tenant bootstrap stage, untested * fix folder name * fix stage 0 output names * optional creation for tag keys in organization module * single tenant bootstrap minus tag * rename output files, add tenant tag key * fix organization module tag values output * test skipping creation for tags in organization module * single tenant bootstrap plan working * multitenant bootstrap * tfdoc * fix check links error messages * fix links * tfdoc * fix links * rename fast tests, fix bootstrap tests * multitenant stages have their own folder, simplify stage numbering * stage renumbering * wip * rename tests * exclude fast providers in fixture * stage 0 tests * stage 1 tests * network stages tests * stage tests * tfdoc * fix links * tfdoc * multitenant tests * remove local files * stage links command * fix links script, TODO * wip * wip single tenant bootstrap * working tenant bootstrap * update gitignore * remove local files * tfdoc * remove local files * allow tests for tenant bootstrap stage * tenant bootstrap proxies stage 1 tfvars * stage 2 and 3 service accounts and IAM in tenant bootstrap * wip * wip * wip * drop multitenant bootstrap * tfdoc * add missing stage 2 SAs, fix org-level IAM condition * wip * wip * optional tag value creation in organization module * stage 1 working * linting * linting * READMEs * wip * Make stage-links script work in old macos bash * stage links command help * fix output file names * diagrams * fix svg * stage 0 skeleton and diagram * test svg * test svg * test diagram * diagram * readme * fix stage links script * stage 0 readme * README changes * stage readmes * fix outputs order * fix link * fix tests * stage 1 test * skip stage example * boilerplate * fix tftest skip * default bootstrap stage log sinks to log buckets * add logging to tenant bootstrap * move iam variables out of tenant config * fix cicd, reintroduce missing variable * use optional in stage 1 cicd variable * rename extras stage * rename and move identity providers local, use optional for cicd variable * tfdoc * add support for wif pool and providers, ci/cd * tfdoc * fix links * better handling of modules repository * add missing role on logging project * fix cicd pools in locals, test cicd * fix workflow extension * fix module source replacement * allow tenant bootstrap cicd sa to impersonate resman sa * tenant workflow templates fix for no providers file * fix output files, push github workflow template to new repository * remove try from outpout files * align stage 1 cicd internals to stage 0 * tfdoc * tests * fix tests * tests * improve variable descriptions * use optional in fast features * actually create tenant log sinks, and allow the resman sa to do it * test * tests * aaaand tests again * fast features tenant override * fast features tenant override * fix wording * add missing comment * configure pf service accounts * add missing comment * tfdoc * tests * IAM docs * update copyright --------- Co-authored-by: Julio Castillo <jccb@google.com>
2023-02-04 06:00:45 -08:00
| [billing_account](variables.tf#L19) | Billing account id. If billing account is not part of the same org set `is_org_level` to false. | <code title="object&#40;&#123;&#10; id &#61; string&#10; is_org_level &#61; optional&#40;bool, true&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ | | <code>0-bootstrap</code> |
| [prefix](variables.tf#L39) | Prefix used for resources that need unique names. Use 9 characters or less. | <code>string</code> | ✓ | | <code>0-bootstrap</code> |
| [factory_data_path](variables.tf#L32) | Path to folder containing YAML project data files. | <code>string</code> | | <code>&#34;data&#47;projects&#34;</code> | |
## Outputs
| name | description | sensitive | consumers |
|---|---|:---:|---|
IAM interface refactor (#1595) * IAM modules refactor proposal * policy * subheading * Update 20230816-iam-refactor.md * log Julio's +1 * data-catalog-policy-tag * dataproc * dataproc * folder * folder * folder * folder * project * better filtering in test examples * project * folder * folder * organization * fix variable descriptions * kms * net-vpc * dataplex-datascan * modules/iam-service-account * modules/source-repository/ * blueprints/cloud-operations/vm-migration/ * blueprints/third-party-solutions/wordpress * dataplex-datascan * blueprints/cloud-operations/workload-identity-federation * blueprints/data-solutions/cloudsql-multiregion/ * blueprints/data-solutions/composer-2 * Update 20230816-iam-refactor.md * Update 20230816-iam-refactor.md * capture discussion in architectural doc * update variable names and refactor proposal * project * blueprints first round * folder * organization * data-catalog-policy-tag * re-enable folder inventory * project module style fix * dataproc * source-repository * source-repository tests * dataplex-datascan * dataplex-datascan tests * net-vpc * net-vpc test examples * iam-service-account * iam-service-account test examples * kms * boilerplate * tfdoc * fix module tests * more blueprint fixes * fix typo in data blueprints * incomplete refactor of data platform foundations * tfdoc * data platform foundation * refactor data platform foundation iam locals * remove redundant example test * shielded folder fix * fix typo * project factory * project factory outputs * tfdoc * test workflow: less verbose tests, fix tf version * re-enable -vv, shorter traceback, fix action version * ignore github extension warning, re-enable action version * fast bootstrap IAM, untested * bootstrap stage IAM fixes * stage 0 tests * fast stage 1 * tenant stage 1 * minor changes to fast stage 0 and 1 * fast security stage * fast mt stage 0 * fast mt stage 0 * fast pf
2023-08-20 00:44:20 -07:00
| [projects](outputs.tf#L17) | Created projects. | | |
| [service_accounts](outputs.tf#L27) | Created service accounts. | | |
<!-- END TFDOC -->