Cloudera Rebuilding Machine Learning for Hadoop with Oryx

me@ryanorban.com (Ryan Orban) — Sat, 01 Mar 2014 00:00:00 +0000

Summary

Cloudera launched Oryx as an open-source framework for large-scale, real-time machine learning on Hadoop. The design follows the Lambda Architecture pattern (coined by Nathan Marz): a batch layer periodically retrains models on all historical data stored in HDFS, while a speed layer serves the current model and handles real-time updates. The framework was built primarily around collaborative filtering for recommendation systems — predicting what content users would engage with based on historical behavior.

Streaming MapReduce with Summingbird

me@ryanorban.com (Ryan Orban) — Wed, 04 Sep 2013 00:00:00 +0000

Summary

Summingbird is a Scala library Twitter open-sourced in 2013 that lets you write a single MapReduce computation and run it on both batch (Hadoop/Scalding) and streaming (Storm) backends. The core idea: computations that aggregate data (count, sum, top-K, approximate counts via HyperLogLog) can be expressed abstractly using algebraic structures (monoids, semigroups), and those abstractions can be automatically parallelized and run either offline over historical data or online over live streams.

Lambda-Architecture on Ryan Orban

Cloudera Rebuilding Machine Learning for Hadoop with Oryx

Summary

Streaming MapReduce with Summingbird

Summary