Apache on Ryan Orban

Apache Incubator Giraph — Distributed Graph Processing on Hadoop

me@ryanorban.com (Ryan Orban) — Sun, 31 Mar 2013 00:00:00 +0000

Summary

Apache Giraph was an open-source implementation of Google Pregel — Google’s internal bulk synchronous parallel (BSP) graph processing system described in a 2010 paper. Pregel solved the problem of running iterative graph algorithms (like PageRank, shortest paths, or community detection) on graphs with billions of nodes and edges, where the data doesn’t fit in memory on a single machine and the iterative nature of graph algorithms makes MapReduce extremely inefficient (each iteration requires a new MapReduce job with full disk I/O).

YARN: Yet Another Resource Negotiator

me@ryanorban.com (Ryan Orban) — Tue, 19 Mar 2013 00:00:00 +0000

Summary

YARN is the resource management framework introduced in Apache Hadoop 2.0 that fundamentally changed what Hadoop clusters could do. Before YARN, Hadoop clusters ran exactly one type of workload: MapReduce jobs. YARN decoupled the cluster resource management layer from the MapReduce programming model, making the cluster a general-purpose compute platform that multiple frameworks could share.

Apache HBase

me@ryanorban.com (Ryan Orban) — Sat, 07 Jul 2012 00:00:00 +0000

Summary

Apache HBase is a distributed, column-oriented NoSQL database built on top of HDFS — the open-source implementation of Google Bigtable. The fundamental gap it fills: HDFS is optimized for sequential writes (append-only) and full-scan reads. MapReduce batch jobs work well with this, but any use case requiring low-latency random reads or updates — looking up a single user’s data, incrementing a counter, serving a recommendation in real-time — is impossible with HDFS alone. HBase adds a key-value store layer over HDFS that enables random reads and writes at millisecond latency.

Apache Hive

me@ryanorban.com (Ryan Orban) — Sat, 07 Jul 2012 00:00:00 +0000

Summary

Apache Hive is a data warehouse layer built on top of Apache Hadoop that provides SQL-like query capability via HiveQL — a dialect of SQL that compiles to MapReduce jobs. The core problem Hive solved was accessibility: writing MapReduce programs in Java required significant expertise, but most data analysts already knew SQL. Hive translated the familiar SQL mental model into the underlying distributed execution without requiring analysts to understand MapReduce internals.

Apache Incubator Giraph

me@ryanorban.com (Ryan Orban) — Sat, 07 Jul 2012 00:00:00 +0000

Summary

Apache Giraph entered the Apache Incubator as the open-source community’s answer to Google Pregel — Google’s internal system for running iterative graph algorithms at massive scale. The basic problem: MapReduce handles batch processing well but is a poor fit for graph computation, where algorithms like PageRank and shortest paths require many iterations, each needing to pass state between neighboring vertices. Running each iteration as a separate MapReduce job means repeated disk I/O between every step — prohibitively slow for large graphs.