Subject
5 entries
Ab Testing
Bookmarks
How Optimizely (Almost) Got Me Fired
SumAll's account of how Optimizely's 'optional stopping' in A/B tests produced false positives that led to bad product decisions — a landmark post in the backlash against naive A/B testing tools. The core issue: peeking at results and stopping when p<0.05 inflates false positive rates dramatically.
Fun with Stats: How Big of a Sample Size Do I Need?
Julia Evans walks through sample size calculation for experiments, grounding statistical power and significance in a concrete worked example. A good entry point for engineers who run A/B tests but haven't internalized what sample size actually buys you.
So You Think You Can Test?
An interactive tool by Lukas Vermeer that lets you distinguish A/A tests from A/B tests visually — a demonstration that human intuition about statistical significance is unreliable. Forces the realization that we can't eyeball whether a difference is real.
Pelican + PlanOut: A/B Testing on a Static Site
Trent Hauck's post combining Facebook's PlanOut experiment framework with the Pelican static site generator — a creative integration showing how to run A/B tests on a static site without server-side logic. An early example of bringing rigorous experimentation tooling to lightweight web stacks.
Statistical Formulas for Programmers
Evan Miller's reference sheet of statistical formulas presented as code-friendly pseudocode rather than academic notation. A practical bridge between statistical theory and implementation, covering the formulas programmers actually need for A/B testing and product analytics.