---
title: "Airflow"
canonical: "https://docs.vaultspeed.com/space/VPD/3012624432/Airflow"
format: markdown
---
One of the VaultSpeed FMC solutions uses Apache Airflow. More information on Airflow can be found here: [https://airflow.apache.org/](https://airflow.apache.org/) .


> ❌ Linked to the VaultSpeed 7.0 release, even when remaining on Airflow 2.11 you need to run the migration explained in the [https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/5213749341](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/5213749341) before you can install and run new DAGs.

## How does it work?

VaultSpeed generates code that defines a workflow for Airflow. To run this code we have built our own airflow provider with extra features used for data warehouse loads. The provider can be downloaded through the cloud application.  
After installing airflow, the provider has to be installed (pip), and the generated code should be placed in the Airflow folder. The Airflow scheduler and web server will load this code and start scheduling and executing it. The workflows can be monitored and managed through the Airflow web application.

## Why Airflow?

We use Airflow because it is open source (part of the Apache Software Foundation) and very flexible and robust.   
In Airflow, a workflow is defined by writing a python script, such that the workflows can be built dynamically. Airflow is written in Python so it is easy to add custom code and write additional workflows for other tasks.

Airflow also has a nice frontend that allows you to manage your workflows and monitor them e.g. monitoring runtimes, logs, etc. When a task in a workflow fails, you can check the logs and rerun the entire load or only the tasks that depend on the failed task.

Airflow can backfill task i.e. given a start date and an interval it will schedule the runs starting from that date, and execute all the tasks that are scheduled in the past as if they actually ran in the past.

Ease of deployment of workflow changes for continuous integration, and the ability to easily run the same code on different environments (just change the connections to databases, etc).

Flexible job execution can be locally in parallel or by using Celery ([http://www.celeryproject.org/](http://www.celeryproject.org/)) to distribute the tasks over multiple workers.

There is a clear difference between the installation instructions on the Airflow 2 and Airflow 3, therefore, we have the 2 separated.

> Macro (children)