CGH DEV Docker Data Pipeline Template
- Cassandra DB (with 3 nodes) - pipeline data store
- Maria DB (single node) - airflow metadata instance
- Rabbit MQ - broker to handle Airflow worker requests
- Airflow - Web and Scheduler (with celery and flower) + example based on Papermill
- Airflow Worker (with 2 nodes)
- Jupyter Notebook