Apache Airflow agent

Apache Airflow is a widely used open source platform for managing workflows with a robust ecosystem. Flyte provides an Airflow plugin that allows you to run Airflow tasks as Flyte tasks. This allows you to use the Airflow plugin ecosystem in conjunction with Flyte's powerful task execution and orchestration capabilities.

Note

The Airflow agent does not support all Airflow operators. We have tested many, but if you run into issues, please file a bug report or reach out to the Union team.

Installation

To install the Airflow agent, run the following command:

pip install flytekitplugins-airflow

This integration has two components:

• Airflow compiler: This component compiles Airflow tasks to Flyte tasks, so Airflow tasks can be directly used inside the Flyte workflow.
• Airflow agent: This component allows you to execute Airflow tasks either locally or on a Flyte cluster.

Example usage

Note

You don't need an Airflow cluster to run Airflow tasks, since flytekit will automatically compile Airflow tasks to Flyte tasks and execute them on the Airflow agent.

For a usage example, see Airflow agent example.

Local testing

Airflow doesn't support local execution natively. However, Flyte compiles Airflow tasks to Flyte tasks, which enables you to test Airflow tasks locally in flytekit's local execution mode.

Note

In some cases, you will need to store credentials in your local environment when testing locally.

Union cluster deployment

After you have finished testing the agent locally, contact the Union team to enable it in your cluster.