|
||
---|---|---|
.. | ||
functions | ||
templates | ||
README.md | ||
diagram1.png | ||
diagram2.png | ||
main.tf | ||
outputs.tf | ||
package-lock.json | ||
send-requests.sh | ||
terraform.tfvars.sample | ||
variables.tf | ||
versions.tf |
README.md
Apigee X Analytics in Bigquery
The following blueprint shows to how to create an Apigee X trial organization, with an environment group, an environment attached to that environment group and an instance attached to that environment. It creates a NEG that exposes Apigee service attachment. The NEG is added as a backend to a GLB. API proxy requests will pass through the GLB.
In addition to this it also creates the setup depicted in the diagram below to export the Apigee analytics of an organization daily to a BigQuery table.
Find below a description on how the analytics export to BigQuery works:
- A Cloud Scheduler Job runs daily at a selected time, publishing a message to a Pub/Sub topic.
- The message published triggers the execution of a function that makes a call to the Apigee Analytics Export API to export the analytical data available for the previous day.
- The export function is passed the Apigee organization, environments, datastore name as environment variables. The service account used to run the function needs to be granted the Apigee Admin role on the project. The Apigee Analytics engine asynchronously exports the analytical data to a GCS bucket. This requires the Apigee Service Agent service account to be granted the Storage Admin role on the project.
- A notification of the files created on GCS is received in a Pub/Sub topic that triggers the execution of the cloud function in charge of loading the data from GCS to the right BigQuery table partition. This function is passed the name of the BigQuery dataset, its location and the name of the table inside that dataset as environment variables. The service account used to run the function needs to be granted the Storage Object Viewer role on the GCS bucket, the BigQuery Job User role on the project and the BigQuery Data Editor role on the table.
Note: This setup only works if you are not using custom analytics.
Running the blueprint
-
Clone this repository or open it in cloud shell, then go through the following steps to create resources:
-
Copy the file terraform.tfvars.sample to a file called
terraform.tfvars
and update the values if required. -
Initialize the terraform configuration
terraform init
-
Apply the terraform configuration
terraform apply
Once the resources have been created, do the following:
Create an A record in your DNS registrar to point the environment group hostname to the public IP address returned after the terraform configuration was applied. You might need to wait some time until the certificate is provisioned.
Testing the blueprint
Do the following to verify that everything works as expected.
-
Create an Apigee datastore
./create-datastore.sh
-
Deploy an api proxy
./deploy-apiproxy.sh test
-
Send some traffic to the proxy
./send-requests.sh test.my-domain.com 1000
-
At 4am (UTC) every day the Cloud Scheduler will run and will export the analytics to the BigQuery table. Double-check they are there.
Variables
name | description | type | required | default |
---|---|---|---|---|
envgroups | Environment groups (NAME => [HOSTNAMES]). | map(list(string)) |
✓ | |
environments | Environments. | map(object({…})) |
✓ | |
instances | Instance. | map(object({…})) |
✓ | |
project_id | Project ID. | string |
✓ | |
psc_config | PSC configuration. | map(string) |
✓ | |
datastore_name | Datastore. | string |
"gcs" |
|
organization | Apigee organization. | object({…}) |
{…} |
|
path | Bucket path. | string |
"/analytics" |
|
project_create | Parameters for the creation of the new project. | object({…}) |
null |
|
vpc_create | Boolean flag indicating whether the VPC should be created or not. | bool |
true |
Outputs
name | description | sensitive |
---|---|---|
ip_address | IP address. |