Skip to content

A simple docker-compose ready repository for using Apache Airflow.

License

Notifications You must be signed in to change notification settings

frankcash/airflow_compose

Repository files navigation

airflow_compose

A project for quickly spinning up Airflow in docker-compose.

Airflow is a tool for maintaining ETLs and workflows. If you do not know a lot about it please check out the official docs.

Running with Docker Compose

docker-compose up --build

Running Locally

Getting dependencies installed

cd /path/to/my/airflow_compose/
virtualenv -p `which python3` venv
source venv/bin/activate

From inside the virtualenv:

export SLUGIFY_USES_TEXT_UNIDECODE=yes
pip install -r requirements.txt
airflow initdb

Running the web server

cd /path/to/my/airflow/workspace
source venv/bin/activate

export AIRFLOW_HOME=`pwd`/airflow_home
airflow webserver

Running the scheduler

cd /path/to/my/airflow/workspace
export AIRFLOW_HOME=`pwd`/airflow_home

source venv/bin/activate
airflow scheduler

Adding database

Go to the configuration tab underneath admin to add a database connection.

Resources for Learning Apache Airflow

Technical Reads

Podcasts