Subject
4 entries
Airflow
Bookmarks
The Unbundling of Airflow
fal.ai's analysis of why Apache Airflow is being unbundled by specialized tools — Prefect and Dagster for orchestration, dbt for transformation, Temporal for long-running workflows. The monolithic DAG orchestrator is fracturing along functional lines.
How Affirm Uses AWS Fargate and Apache Airflow for Batch Jobs
Affirm's engineering blog post on running Apache Airflow DAGs on AWS Fargate for batch job management — serverless task execution without persistent workers. A practical architecture case study for ML and data pipeline orchestration at a fintech scale.
Airflow and XCom: Inter-Task Communication Use Cases
A guide to Airflow's XCom (cross-communication) mechanism for passing data between tasks in a DAG — covering when to use it, when to avoid it, and practical use cases. XCom is one of the most misused Airflow features.
We're All Using Airflow Wrong and How to Fix It
Bluecore Engineering's argument that most Airflow deployments misuse operators — using Python, Bash, and Spark operators that run code inside the Airflow worker, which breaks isolation and causes resource contention. The fix: use KubernetesPodOperator for everything.