This repository contains a example environment that can be used by anyone who wants how to code pipelines in Airflow

• Python 3.6 or higher https://www.python.org/
• Docker https://docs.docker.com/get-docker/
• Docker Compose https://docs.docker.com/compose/install/
• MySql client (ubuntu: sudo apt-get install libmysqlclient-dev, mac: brew install mysql)
• ONlY on Mac - Developer tools xcode-select --install

To start using repository just go in your terminal to folder when project was cloned and type
python3 teach_aiflow.py
What gonna happen:

• Docker will build all needed images
• By Docker Compose a local cluster will be raised
• Example data wil be loaded into databases
• Virtual environment will be created
• All dependencies needed for Airflow will be installed inside virtual environment

Pre check
If for command docker ps you get error that cannot connect to docker engine please try to run it with sudo. If that work all commands here should be run with sudo as well.

all should be run as python3 teach_airflow.py <argument>

  --reload_data  Reload data in all databases
  --start        Start airflow docker containers
  --stop         Stop airflow docker containers
  --restart      Restart airflow docker containers
  --remove       Delete all docker images, containers
  --rebuild      Rebuild airflow env
  --add_vir_env  Add virtual env
  --rm_vir_env   Remove virtual env

http://localhost:8080

http://localhost:5555

* login: airflow
* password: airflow
* database: airflow
* host: localhost
* port: 5433

* login: airflow
* password: airflow
* database: public
* host: localhost
* port: 3307

* username: airflow
* password: airflow
* host: localhost
* port: 21

* login: airflow
* password: airflow
* database: airflow
* host: postgres
* port: 5432

* login: airflow
* password: airflow
* database: public
* host: mysql
* port: 3306

* username: airflow
* password: airflow
* host: ftp
* port: 21