This is a recipe for Composing ingredients based on airflow, kafka,spark and cooking it in docker containers using docker-compose.

This is Like Story Telling ,

To Stream data , consuming it to tables and writing queries to get some insights. However, it often comes with additional overhead. Luckily, there are tools that allow you to automate many parts of this process. 

This repository takes on the topic of Streaming Data , Capturing it and getting insights. It mainly leans on three nifty tools, being Kafka, Airflow, and Spark.

The corresponding walkthrough/post on Medium lays out the workings of this repo step-by-step.