Ab-Testing on Ryan Orban

How Optimizely (Almost) Got Me Fired

me@ryanorban.com (Ryan Orban) — Mon, 29 Sep 2014 00:00:00 +0000

Summary

This SumAll blog post became one of the most-cited criticisms of naive A/B testing tools when it was published. The author ran experiments through Optimizely, declared winners based on the platform’s significance indicators, shipped changes — and later discovered the winning variants were not actually better. The culprit: optional stopping, also called the peeking problem.

Fun with Stats: How Big of a Sample Size Do I Need?

me@ryanorban.com (Ryan Orban) — Sat, 12 Jul 2014 00:00:00 +0000

Summary

Julia Evans writes characteristically direct posts about technical concepts, and this one tackles sample size calculation for statistical hypothesis testing. The core question she answers is practical: if I want to detect a 5% improvement in my metric, how many users do I need in my experiment? The answer requires understanding statistical power — the probability that your test will detect a real effect if one exists.

So You Think You Can Test?

me@ryanorban.com (Ryan Orban) — Wed, 02 Jul 2014 00:00:00 +0000

Summary

Lukas Vermeer’s hackathon project is a deceptively simple interactive quiz: you’re shown a series of test results and asked to judge whether each shows a real effect (A/B test) or no effect (A/A test). The exercise is humbling — human intuition for statistical patterns is poor, and most people perform near chance. It’s a visceral demonstration of why A/B testing requires statistical significance testing rather than visual inspection.

Pelican + PlanOut: A/B Testing on a Static Site

me@ryanorban.com (Ryan Orban) — Tue, 27 May 2014 00:00:00 +0000

Summary

Trent Hauck’s post demonstrates how to integrate PlanOut — Facebook’s experiment framework — with Pelican, a Python-based static site generator. The combination is technically interesting: static sites don’t have server-side request handling, so running A/B tests normally requires either a CDN with edge logic or client-side assignment. This post works around the limitation by having PlanOut generate the variant assignments at build time.

Statistical Formulas for Programmers

me@ryanorban.com (Ryan Orban) — Mon, 20 May 2013 00:00:00 +0000

Summary

Evan Miller wrote a post translating the most commonly needed statistics formulas into programmer-friendly notation — avoiding the Greek letter soup of academic statistics textbooks and presenting formulas in a way that maps to implementation. The target audience: software engineers who need to implement A/B testing, analyze experiment results, or build product analytics without a statistics background.