https://ryanorban.com/categories/batch-processing/Recent content in Batch-Processing on Ryan OrbanHugoen-usme@ryanorban.com (Ryan Orban)me@ryanorban.com (Ryan Orban)Ryan OrbanSat, 16 Jul 2022 00:00:00 +0000https://ryanorban.com/notes/gazette-core/Sat, 16 Jul 2022 00:00:00 +0000me@ryanorban.com (Ryan Orban)https://ryanorban.com/notes/gazette-core/<h3 id="summary" class="scroll-mt-8 group"> Summary <a href="#summary" class="no-underline hidden opacity-50 hover:opacity-100 !text-inherit group-hover:inline-block" aria-hidden="true" title="Link to this heading" tabindex="-1"> <svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" class="lucide lucide-link w-4 h-4 block" viewBox="0 0 24 24" > <path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71" /> <path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71" /> </svg> </a> </h3> <p>Gazette is an open-source data platform built around the concept of journals — append-only, durable log primitives similar to Kafka partitions but with a different operational model. The core pitch is flexibility: you can run SQL queries, batch jobs, and stream processing against the same underlying data store without choosing a single paradigm upfront.</p>https://ryanorban.com/notes/affirm-fargate-airflow/Thu, 07 Jan 2021 00:00:00 +0000me@ryanorban.com (Ryan Orban)https://ryanorban.com/notes/affirm-fargate-airflow/<h3 id="summary" class="scroll-mt-8 group"> Summary <a href="#summary" class="no-underline hidden opacity-50 hover:opacity-100 !text-inherit group-hover:inline-block" aria-hidden="true" title="Link to this heading" tabindex="-1"> <svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" class="lucide lucide-link w-4 h-4 block" viewBox="0 0 24 24" > <path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71" /> <path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71" /> </svg> </a> </h3> <p>Affirm&rsquo;s engineering team describes their batch job infrastructure built on Apache Airflow and AWS Fargate. The core architectural choice: instead of running persistent Celery workers that idle between tasks, Fargate spins up containers on demand per Airflow task and terminates them when done. This eliminates the cost and operational overhead of managing worker fleets — a meaningful win for a batch workload where jobs are bursty rather than continuous.</p>