DBT Airflow Factory
Introduction
The factory is a library for parsing DBT manifest files and building Airflow DAG.
The library is expected to be used inside an Airflow environment with a Kubernetes image referencing dbt.
dbt-airflow-factory’s main task is to parse manifest.json
and create Airflow DAG out of it. It also reads config
YAML files from config
directory and therefore is highly customizable (e.g., user can set path to manifest.json
).
DAG building is an on-the-fly process without materialization. Also, the process may use Airflow Variables as a way of configuration.
Community
Although the tools was created by GetInData and used in their project it is open-sourced and every one is welcome to use and contribute to make it better and even more usefull.