Subject
3 entries
Workflow Orchestration
Bookmarks
The Unbundling of Airflow
fal.ai's analysis of why Apache Airflow is being unbundled by specialized tools — Prefect and Dagster for orchestration, dbt for transformation, Temporal for long-running workflows. The monolithic DAG orchestrator is fracturing along functional lines.
Airflow and XCom: Inter-Task Communication Use Cases
A guide to Airflow's XCom (cross-communication) mechanism for passing data between tasks in a DAG — covering when to use it, when to avoid it, and practical use cases. XCom is one of the most misused Airflow features.
We're All Using Airflow Wrong and How to Fix It
Bluecore Engineering's argument that most Airflow deployments misuse operators — using Python, Bash, and Spark operators that run code inside the Airflow worker, which breaks isolation and causes resource contention. The fix: use KubernetesPodOperator for everything.