Subject
69 entries
Distributed Systems
Bookmarks
Distributed Consensus Reading List
Heidi Howard's curated list of papers on distributed consensus — organized from foundational Paxos through Multi-Paxos, Raft, flexible quorums, and Byzantine fault tolerance. The definitive paper trail for understanding how consensus algorithms actually work.
A Distributed Systems Reading List
Fred Hebert's curated distributed systems reading list — covering foundational papers, books, and blog posts from the Fallacies of Distributed Computing through CAP theorem, consensus, CRDTs, and failure handling. One of the more practical and opinionated guides to the field.
Rakis Stability Test 1: Results from a Month of Running an AI Network
Rakis ran Stability Test 1: a month-long production test of a fully in-browser peer-to-peer AI inference network. 10 million tokens across ~2,000 nodes with embedding-based consensus for verification — early evidence that browser-native decentralized AI inference is viable.
Advent of Distributed Systems
Advent of Distributed Systems is a coding challenge series in the style of Advent of Code but focused on distributed systems problems — consensus, replication, fault tolerance, and network partitions. Hands-on learning for distributed concepts that are hard to study from papers alone.
RAG at Planet Scale
Arcus describes their multi-tiered RAG approach for handling massive external data corpora at planet scale — one of the largest RAG deployments of 2023. The key innovation is tiered retrieval that narrows the candidate pool progressively rather than searching the full index directly.
Automerge: CRDTs for Collaborative Applications
Automerge is a CRDT (Conflict-free Replicated Data Type) library for building collaborative applications that work offline and sync automatically without conflicts. The technical foundation of the local-first software movement.
CRDT: Fractional Indexing
Evan Wallace's visual explainer of fractional indexing as a CRDT technique for ordered lists — items get floating-point positions between existing neighbors. A clean conceptual foundation for understanding how collaborative list ordering works without central coordination.
Monolith: Real Time Recommendation System With Collisionless Embedding Table
Monolith is ByteDance's production recommendation system purpose-built for online training with dynamic sparse features, featuring a collisionless embedding table with expirable embeddings and frequency filtering. It demonstrates that general-purpose ML frameworks like TensorFlow are ill-suited for industrial recommendation with non-stationary data distributions.
Notes on Theory of Distributed Systems
James Aspnes's freely-distributed lecture notes on the theory of distributed systems, covering fault tolerance, consensus, synchrony models, and randomized algorithms. A rigorous but accessible graduate reference that grounds distributed computing in formal models.
Architecture Notes
Architecture Notes is a Substack by Mahdi Yusuf covering system design and software architecture — weekly deep dives on distributed systems, engineering leadership, and large-scale system patterns. 29,000+ subscribers; known for thorough, practitioner-level explanations.
How Amazon Web Services Uses Formal Methods
Newcombe et al. (CACM 2015) describe how Amazon Web Services engineers use TLA+ to specify and verify distributed systems protocols, finding real bugs in S3, DynamoDB, and EBS before deployment. One of the few industrial accounts of formal methods working in production at scale.
Pathways: Asynchronous Distributed Dataflow for ML
Google's Pathways is a single-controller orchestration layer for ML accelerators that runs sharded asynchronous dataflow across thousands of TPUs while matching SPMD performance. Designed to break the MPI-style lockstep model so heterogeneous workloads like MoE, pipelining, and foundation-model multiplexing become first-class.
Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing
EPFL researchers show that existing Byzantine-robust aggregation rules (Krum, coordinate-wise median, RFA) fail catastrophically on non-iid data, then fix the problem with a one-step bucketing scheme that randomly groups worker updates before aggregation. The first result with provable convergence guarantees for Byzantine robustness under realistic data heterogeneity.
Data Mesh: Delivering Data-Driven Value at Scale
Zhamak Dehghani's 2022 O'Reilly book defines data mesh — a sociotechnical approach to data architecture that treats data as a product owned by domain teams, distributed across a federated data platform, and governed by global standards without centralized control. The book is the canonical reference for moving beyond monolithic data lakes and warehouses.
CRDTs: Conflict-free Replicated Data Types
crdt.tech is the canonical reference hub for Conflict-free Replicated Data Types — the data structures that enable real-time collaborative editing without central coordination. The math guarantees eventual consistency even when network partitions split collaborators.
Testing Distributed Systems
A curated list of resources on testing distributed systems — covering Jepsen, TLA+, chaos engineering, simulation testing, and formal verification. The most comprehensive reference starting point for this notoriously hard problem.
High Scalability
High Scalability is a long-running blog covering distributed systems architecture and how companies like Google, Uber, and Meta build systems at scale. Deep technical case studies from production systems — essential reading for engineers designing for scale.
Ray: A Distributed Framework for Emerging AI Applications
Micah Lerner's paper summary of Ray — UC Berkeley's distributed computing framework for AI workloads. Ray unified task-parallel and actor-based distributed computing in a Python-native API, becoming the foundation for libraries like RLlib, Tune, and later Ray Serve.
MIT 6.033: Computer System Engineering
MIT 6.033 Computer System Engineering covers the design of large, complex software systems — reliability, fault tolerance, operating systems, networking, and distributed systems. One of MIT's most comprehensive systems courses, available free via OpenCourseWare.
System Design Primer
Donnemartin's system-design-primer is the most starred GitHub repository for learning how to design large-scale systems — covers scalability, caching, databases, load balancing, and common interview patterns. The Anki flashcard deck makes it unusually retention-friendly.
Running Elasticsearch: Fun & Profit
A free online book on running Elasticsearch in production — cluster sizing, index design, mapping, query optimization, and operational concerns like snapshots and upgrades. Practitioner-focused with real-world configuration guidance rather than API documentation.
Deterministic Aperture: Twitter's Load Balancing Algorithm
Twitter's Deterministic Aperture load balancing algorithm assigns each client a deterministic subset ('aperture') of backends, reducing connection fan-out while maintaining even load distribution. A principled alternative to round-robin and power-of-two-choices that scales better with horizontal expansion.
What Does It Take to Make Google Work at Scale?
A slide deck on what it takes to make Google's infrastructure work at scale — covering the distributed systems challenges, data storage, and engineering decisions behind running at internet scale. A useful systems design reference from before Designing Data-Intensive Applications became the canonical text.
Principles of Distributed Computing — ETH Zurich All-Stars
ETH Zurich's Principles of Distributed Computing lecture series — foundational theory covering consensus, fault tolerance, and distributed algorithms. The "all-stars" edition collects contributions from leading researchers in the field.
Model for Massively Parallel Computation — MapReduce Theory
Grigory Yaroslavtsev's theoretical treatment of MapReduce as a model for massively parallel computation — covering the MRC complexity class and what it tells us about which problems can be solved efficiently at scale.
Distributed Systems and the End of the API
Prismatic's provocative argument that the traditional synchronous REST API is the wrong primitive for distributed systems — proposing message-passing and event streams as the better foundation. Anticipates the 2014-era shift toward event-driven architectures and the explosion of streaming systems.
Why Apache Spark is a Crossover Hit for Data Scientists
Cloudera's post on why Apache Spark resonated with data scientists in ways Hadoop MapReduce never did — the interactive REPL, Python support, and in-memory computation made it feel like a supercharged pandas rather than a distributed systems project.
Introduction to Deep Learning on Hadoop (Hadoop Summit 2014)
A Hadoop Summit 2014 session proposal on deep learning at Hadoop scale — from the team behind DL4J (DeepLearning4J), a Java-native deep learning framework designed to run on Hadoop/Spark clusters. A snapshot of the moment distributed deep learning was being invented.
Pig Not-So-Foreign Language: Paper Notes
Bugra Akyildiz's notes on the 'Pig Latin: A Not-So-Foreign Language for Data Processing' paper by Yahoo! Research — explaining how Pig Latin compiles high-level data flow operations to MapReduce jobs. Captures why Pig was a meaningful step up from raw MapReduce for ETL work.
Deploying Storm on GCE
Tutorial on deploying Apache Storm on Google Compute Engine — a setup guide for real-time stream processing at a time when cloud deployments of Storm were uncommon. GCE was a relatively new platform and Storm was the dominant real-time processing framework before Flink/Spark Streaming.
Hadoop Creator: Google Is Living a Few Years in the Future
Doug Cutting (Hadoop creator) on Google living years ahead in infrastructure — the observation that Google's internal systems consistently anticipate what the rest of the industry will need, and then the open-source community builds it later.
How Companies Are Using Spark
Strata/O'Reilly coverage of how companies were adopting Apache Spark in 2013, early in the engine's rise to ubiquity. A snapshot of early enterprise Spark use cases before it displaced Hadoop MapReduce as the default.
Presto: Interacting with Petabytes of Data at Facebook
Hacker News discussion on Facebook's newly open-sourced Presto SQL engine, capable of querying petabytes of data interactively. A watershed moment — before Presto, interactive SQL at Facebook scale wasn't possible.
Apache Hadoop 2 Is Now GA
Hortonworks announcement that Apache Hadoop 2 reached general availability in October 2013 — introducing YARN as the cluster resource manager. A landmark release that decoupled compute from MapReduce and made Hadoop a general-purpose cluster platform.
Timely Dataflow: An Introduction
An introduction to Timely Dataflow, the distributed computation model developed at Microsoft Research that unified batch and streaming processing through a novel timestamp-based progress tracking system. A technically significant but underappreciated alternative to the Spark/Storm paradigm.
Distributed Systems for Fun and Profit
Mixu's free online book on distributed systems fundamentals — covering consistency models, CAP theorem, replication, and consensus — written for practitioners who want theoretical grounding without the academic overhead. One of the clearest introductions to the field that exists.
Apache Helix: Distributed Systems Get Simpler
Cloudera's post on Apache Helix, LinkedIn's cluster management framework that abstracts distributed state machine management over ZooKeeper. Helix made it practical to build distributed systems with complex partition assignment and rebalancing without reimplementing the coordination logic from scratch.
The Datacenter as a Computer: Warehouse-Scale Machines
High Scalability's coverage of Google's 'The Datacenter as a Computer' second edition — Barroso and Hölzle's canonical text on warehouse-scale machine design. Defined the vocabulary and engineering tradeoffs for operating entire buildings as programmable compute platforms.
Introduction to Apache Kafka (TriHUG, July 2013)
TriHUG July 2013 talk introducing Apache Kafka — the distributed log system LinkedIn built and open-sourced. Caught at the moment Kafka was still an unfamiliar tool to most data engineers, before it became the de-facto streaming backbone of the modern data stack.
High Performance at Massive Scale: Lessons Learned at Facebook
Summary of a 2009 Facebook engineering talk on high-performance systems at massive scale — covering their memcached deployment, MySQL sharding, and the operational realities of running at hundreds of millions of users. An early public window into big-company distributed systems practice.
Yahoo! Spinning Continuous Computing with YARN
Yahoo's 2013 exploration of using YARN as a substrate for continuous/streaming computation beyond batch MapReduce. An early signal that the Hadoop ecosystem was trying to absorb real-time processing use cases before Apache Spark and Flink fully took over.
Netflix Genie: Hadoop Platform-as-a-Service
Netflix open-sourced Genie in mid-2013 — a REST-based Hadoop Platform-as-a-Service that abstracted job submission across multiple Hadoop clusters. A key piece of Netflix's data platform that became an influential pattern for multi-cluster job routing.
Facebook Unveils Presto for 250 PB Data Warehouse
GigaOm's coverage of Facebook unveiling Presto, their distributed SQL query engine for interactive queries against a 250 petabyte data warehouse. Presto addressed the core limitation of Hive — batch latency — by using a pipelined execution model that avoided writing intermediate results to disk.
Hadoop 2.0 & YARN: The Big Data Breakthrough
ReadWrite's accessible overview of Hadoop 2.0 and YARN, explaining why the resource manager redesign was a bigger deal than an incremental release — it turned Hadoop from a MapReduce platform into a general-purpose cluster resource manager.
Twitter Powers of Ten
Rob Weir's 2011 post using powers-of-ten framing to characterize Twitter's data and scale properties — from individual tweets to the full firehose. A snapshot of the social media scale conversation before big data tooling became mainstream.
Simple Explanation of How Nutanix Works
A YouTube explainer of how Nutanix's distributed file system works — covering the Curator distributed metadata service, erasure coding, data placement, and how Nutanix collocates compute and storage in a hyper-converged node. Simple enough for sales conversations, technically accurate.
CAP Confusion: Problems with 'Partition Tolerance'
Cloudera's clarification of the most common CAP theorem misreading: partition tolerance isn't a feature you choose — it's a property you must accept because network partitions happen. The real CAP choice is between consistency and availability when partitions occur.
MapReduce Patterns, Algorithms, and Use Cases
Ilya Katsov's comprehensive taxonomy of MapReduce design patterns — from basic counting and filtering through complex join strategies and graph algorithms. The field guide for wringing correct and efficient computation out of the MapReduce model.
Welcome to Berkeley: Where Hadoop Isn't Nearly Fast Enough
GigaOM's April 2013 profile of UC Berkeley's AMPLab making the case that Hadoop is too slow for interactive and iterative workloads — the academic origin story of Apache Spark, Shark (early Spark SQL), and Mesos. A snapshot of the moment Spark was about to go mainstream.
Hadoop Illuminated — Free Open-Source Hadoop Book
Hadoop Illuminated is a free, open-source book on Apache Hadoop — a community-maintained guide covering HDFS, MapReduce, Hive, Pig, and the broader Hadoop ecosystem. One of the better free learning resources at a time when the Hadoop ecosystem was evolving faster than formal textbooks.
Apache Incubator Giraph — Distributed Graph Processing on Hadoop
Apache Giraph is a graph processing framework built on Hadoop — an open-source implementation of Google's Pregel model for iterative graph algorithms at scale. The Apache answer to graph-scale problems like PageRank, community detection, and shortest paths on billion-node graphs.
YARN: Yet Another Resource Negotiator
Apache YARN (Yet Another Resource Negotiator) is the cluster resource management layer introduced in Hadoop 2 — the architectural change that turned Hadoop from a MapReduce system into a general-purpose distributed compute platform.
Airbnb Open Sources Chronos: A Distributed Cron Replacement
Airbnb's open-sourcing of Chronos, a distributed cron replacement built on Apache Mesos. A notable moment in the early days of the Mesos ecosystem, showing how Twitter-era infrastructure patterns were spreading across Silicon Valley companies.
Beyond Hadoop: Next-Generation Big Data Architectures
GigaOM's 2010 survey of Google's next-generation big data architectures — MPI, Pregel, Dremel, and Percolator — that were being developed as alternatives or complements to MapReduce. The paper that first systematically articulated why MapReduce wasn't sufficient for everything.
Shark: Real-time Queries and Analytics for Big Data
O'Reilly Strata article on Shark — the precursor to Spark SQL that brought real-time interactive queries to Hadoop/Spark in 2012. Part of the wave of tools (Impala, Shark, Drill) that challenged Hive's batch-query dominance.
HDFS Has Won: De Facto Standard for Centralized Data Storage
A 2012 claim that HDFS had emerged as the de facto standard for centralized big data storage — a snapshot of a moment when Hadoop's dominance seemed settled. Written just before the data lake era that HDFS would define, and before object storage (S3) would eventually displace it.
Big Ideas: Demystifying Hadoop
A 'Big Ideas: Demystifying Hadoop' YouTube explainer from 2012 — one of many educational resources that emerged as Hadoop moved from niche to mainstream. Aimed at explaining the MapReduce paradigm and HDFS to practitioners who hadn't yet had to deal with data at scale.
Google Spanner: Globally Distributed Transactions (OSDI 2012)
Google's Spanner paper from OSDI 2012 — the design of Google's globally distributed SQL database with externally consistent transactions. TrueTime, the GPS/atomic-clock-based approach to distributed timestamps, is the paper's most memorable technical contribution.
HBase vs Cassandra: Why We Moved
A firsthand account of migrating from HBase to Cassandra, written in 2010 but widely read through 2012. The core finding: Cassandra was operationally simpler and more resilient to node failures, while HBase required careful HDFS management and had more complex failure modes.
HyperDex: NoSQL Database Based in Hyperspace
The HyperDex academic paper introducing a NoSQL key-value store that used hyperspace hashing to support efficient secondary attribute searches — the main weakness of most key-value stores at the time. An elegant theoretical approach that didn't achieve mainstream adoption.
Benchmarking High Performance I/O with SSD for Cassandra on AWS
Netflix's 2012 benchmark of SSD-backed Cassandra on AWS EC2 — showing 3-4x latency improvements over spinning disk for their use case. An early real-world data point on SSD economics in cloud distributed databases.
YARN: Hadoop NextGen MapReduce
YARN's official documentation from 2012 — the architecture that decoupled Hadoop cluster resource management from MapReduce. By separating resource negotiation into its own layer, YARN turned Hadoop from a MapReduce platform into a general-purpose cluster OS.
Why the Days Are Numbered for Hadoop As We Know It
A 2012 GigaOM piece arguing that Hadoop's architecture had fundamental limitations that would force it to evolve or be displaced — written at the peak of Hadoop hype. Prescient in identifying YARN and the multi-framework future, but underestimated how long it would take for cloud-native alternatives to win.
Twitter to Open Source Hadoop-Like Tool (Storm)
GigaOM coverage of Twitter's plans to open-source Storm — their real-time stream processing system. The moment the Hadoop-for-streaming gap became a major industry conversation, and Nathan Marz's Storm became the answer.
Large-Scale Graph Computing at Google (Pregel)
Google Research's 2009 blog post introducing Pregel — their internal system for large-scale graph computation using a bulk-synchronous-parallel model. The post that launched the graph processing systems category and eventually spawned Apache Giraph, GraphX, and the whole vertex-centric computing tradition.
Dremel: Interactive Analysis of Web-Scale Datasets
Google's Dremel paper — the system that enabled sub-second SQL queries over petabyte datasets via columnar storage and a multi-level serving tree. The direct precursor to BigQuery, and the inspiration behind Apache Parquet's nested record encoding.
Apache Incubator Giraph
Apache Giraph's incubator homepage from 2012 — the open-source implementation of Google's Pregel bulk-synchronous-parallel graph processing model. Bookmarked during an early phase of the Hadoop ecosystem expansion into graph workloads.
Apache HBase
Apache HBase's homepage from 2012 — the open-source implementation of Google Bigtable that added random-read/write access to Hadoop's otherwise write-once HDFS. HBase filled the gap MapReduce couldn't: low-latency lookups on data stored across a distributed cluster.
Netflix to Open Source Army of Cloud Monkeys
Wired's report on Netflix open-sourcing the Simian Army — a suite of tools including Chaos Monkey that deliberately broke production systems to test resilience. The foundational document of chaos engineering as a discipline.