Workflow-Orchestration on Ryan Orban

The Unbundling of Airflow

me@ryanorban.com (Ryan Orban) — Tue, 15 Feb 2022 00:00:00 +0000

Summary

Apache Airflow became the dominant workflow orchestration platform for data engineering over the 2017–2021 period — the default choice for scheduling ETL pipelines, ML training jobs, and data processing DAGs. But by 2022, fal.ai was diagnosing its unbundling: specialized tools were taking over specific parts of what Airflow does, often doing them better.

Airflow and XCom: Inter-Task Communication Use Cases

me@ryanorban.com (Ryan Orban) — Wed, 05 Aug 2020 00:00:00 +0000

Summary

Apache Airflow tasks in a DAG are designed to be independent — they run in separate processes, often on different machines. XCom (cross-communication) is the mechanism for passing small amounts of data between tasks. A task can “push” an XCom value to Airflow’s metadata database, and a downstream task can “pull” that value using the task ID and key. This enables passing results like file paths, record counts, or status codes between tasks in the same DAG run.

We're All Using Airflow Wrong and How to Fix It

me@ryanorban.com (Ryan Orban) — Fri, 24 Jul 2020 00:00:00 +0000

Summary

Bluecore Engineering makes a pointed argument in this post: the standard way most teams use Apache Airflow — with PythonOperator, BashOperator, and other operators that run code directly in the Airflow worker processes — is architecturally wrong. The reason: these operators mix the execution environment of your task code with the Airflow worker infrastructure. This creates dependency conflicts (task A needs library version X, task B needs version Y), resource contention (a memory-heavy task starves the Airflow scheduler), and fragile deployments (task code must be deployed to every Airflow worker).