Graph-Processing on Ryan Orban

Apache Incubator Giraph — Distributed Graph Processing on Hadoop

me@ryanorban.com (Ryan Orban) — Sun, 31 Mar 2013 00:00:00 +0000

Summary

Apache Giraph was an open-source implementation of Google Pregel — Google’s internal bulk synchronous parallel (BSP) graph processing system described in a 2010 paper. Pregel solved the problem of running iterative graph algorithms (like PageRank, shortest paths, or community detection) on graphs with billions of nodes and edges, where the data doesn’t fit in memory on a single machine and the iterative nature of graph algorithms makes MapReduce extremely inefficient (each iteration requires a new MapReduce job with full disk I/O).

Apache Incubator Giraph

me@ryanorban.com (Ryan Orban) — Sat, 07 Jul 2012 00:00:00 +0000

Summary

Apache Giraph entered the Apache Incubator as the open-source community’s answer to Google Pregel — Google’s internal system for running iterative graph algorithms at massive scale. The basic problem: MapReduce handles batch processing well but is a poor fit for graph computation, where algorithms like PageRank and shortest paths require many iterations, each needing to pass state between neighboring vertices. Running each iteration as a separate MapReduce job means repeated disk I/O between every step — prohibitively slow for large graphs.

Large-Scale Graph Computing at Google (Pregel)

me@ryanorban.com (Ryan Orban) — Sat, 07 Jul 2012 00:00:00 +0000

Summary

This 2009 Google Research blog post introduced Google Pregel to the public — a system Google built internally for large-scale graph computation that established the vertex-centric programming model now used throughout distributed graph processing. MapReduce was Google’s workhorse for batch data processing, but graph algorithms like PageRank, shortest paths, and community detection are inherently iterative: each round of computation depends on the results of the previous round. Forcing this into MapReduce meant a full job per iteration — expensive disk I/O and job startup costs that made it impractical for serious graph workloads.