# Data ingestion Demo

In this folder, you can find an example to ingest data on the `data platform` instantiated [here](../). 

The example is not intended to be a production-ready code. 

## Demo use case
The demo imports purchase data generated by a store. 

## Input files 
Data are uploaded to the `drop off` GCS bucket. File structure:
 - `customers.csv`: Comma separate value with customer information in the following format: Customer ID, Name, Surname, Registration Timestamp
 - `purchases.csv`: Comma separate value with customer information in the following format: Item ID, Customer ID, Item, Item price, Purchase Timestamp

## Data processing pipelines
Different data pipelines are provided to highlight different features and patterns. For the purpose of the example, a single pipeline handle all data lifecycles. When adapting them to your real use case, you may want to evaluate the option to handle each functional step on a separate pipeline or a dedicated tool. For example, you may want to use `Dataform` to handle data schemas lifecycle.

Below you can find a description of each example:
 - Simple import data: [`datapipeline.py`](./datapipeline.py) is a simple pipeline to import provided data from the `drop off` Google Cloud Storage bucket to the Data Hub Confidential layer joining `customers` and `purchases` tables into `customerpurchase` table.
 - Import data with Policy Tags: [`datapipeline_dc_tags.py`](./datapipeline.py) imports provided data from `drop off` bucket to the Data Hub Confidential layer protecting sensitive data using Data Catalog policy Tags.
 - Delete tables: [`delete_table.py`](./delete_table.py) deletes BigQuery tables created by import pipelines. 

## Running the demo
To run demo examples, please follow the following steps:

- 01: Copy sample data to the `drop off` Cloud Storage bucket impersonating the `load` service account.
- 02: Copy sample data structure definition in the `orchestration` Cloud Storage bucket impersonating the `orchestration` service account.
- 03: Copy the Cloud Composer DAG to the Cloud Composer Storage bucket impersonating the `orchestration` service account.
- 04: Build the Dataflow Flex template and image via a Cloud Build pipeline
- 05: Open the Cloud Composer Airflow UI and run the imported DAG.
- 06: Run the BigQuery query to see results.

You can find pre-computed commands in the `demo_commands` output variable of the deployed terraform [data pipeline](../).