HyperLogLog in Pure SQL

me@ryanorban.com (Ryan Orban) — Sat, 31 May 2014 00:00:00 +0000

Summary

HyperLogLog is a probabilistic data structure for cardinality estimation: it approximates COUNT(DISTINCT …) over arbitrarily large datasets using a small, fixed amount of memory. The algorithm was introduced by Philippe Flajolet et al. in 2007 and achieves typical error rates under 2% using only a few kilobytes of state — regardless of whether you’re counting thousands or billions of distinct values.

Probabilistic Data Structures for Web Analytics and Data Mining

me@ryanorban.com (Ryan Orban) — Sun, 17 Mar 2013 00:00:00 +0000

Summary

This Highly Scalable Blog post is the canonical reference for understanding which probabilistic data structure to reach for when you’re building web analytics pipelines. The central insight is that most analytics use cases don’t need exact answers — they need fast answers at scale. Trading a small, bounded error rate for dramatic reductions in memory and computation is almost always the right engineering call when you’re processing billions of events.

Probabilistic-Data-Structures on Ryan Orban

HyperLogLog in Pure SQL

Summary

Probabilistic Data Structures for Web Analytics and Data Mining

Summary