Dagster is an opinionated system and programming model for data pipelines. This process goes by many names -- ETL (extract-transform-load), ELT (extract-load-transform), model production, data integration, and so on -- but in essence they all describe the same activity: performing a set of computations structured as a DAG (directed, acyclic graph) that end up producing data assets, whether those assets be tables, files, machine-learning models, etc.

This repository contains a number of distinct subprojects:

• dagster: The core programming model and abstraction stack; a stateless single-node, single-process execution engine; and a CLI tool for driving that engine.
• dagit: Dagit is a rich viewer for Dagster assets, including a DAG browser, a type-aware config editor, and a streaming execution interface.
• dagster-ge: A Dagster integration with Great Expectations. (see https://github.com/great-expectations/great_expectations)
• dagster-pandas: A Dagster integration with Pandas.
• dagster-sqlalchemy: A Dagster integration with SQLAlchemy.
• dagstermill: An experimental prototype for integrating productionized notebooks into dagster pipelines. Built on the papermill library (https://github.com/nteract/papermill).
• airline-demo: A substantial demo project illustrating how these tools can be used together to manage a realistic data pipeline.
• js_modules/dagit - a web app that is a ui for dagit
• dagma - An experimental execution engine for Dagster built on top of AWS Lambda.

Go to https://dagster.readthedocs.io/ for complete documentation, including a step-by-step tutorial and notes on the demo project.

For details on contributing or running the project for development, see https://dagster.readthedocs.io/en/latest/contributing.html.