Subject
12 entries
Experimentation
Bookmarks
Multi-Armed Bandits and the Stitch Fix Experimentation Platform
Stitch Fix's blog on multi-armed bandits as an alternative to A/B testing — Thompson Sampling routes traffic toward better-performing arms dynamically, reducing wasted exposure. Strong motivation for when bandits beat traditional experimentation.
How Optimizely (Almost) Got Me Fired
SumAll's account of how Optimizely's 'optional stopping' in A/B tests produced false positives that led to bad product decisions — a landmark post in the backlash against naive A/B testing tools. The core issue: peeking at results and stopping when p<0.05 inflates false positive rates dramatically.
Scalable A/B Experiments at Pinterest
Pinterest's engineering blog on how they built scalable A/B testing infrastructure — covering experiment assignment, metric computation, and statistical significance at Pinterest's scale. A practitioner's account of the gap between textbook A/B testing and production experimentation.
So You Think You Can Test?
An interactive tool by Lukas Vermeer that lets you distinguish A/A tests from A/B tests visually — a demonstration that human intuition about statistical significance is unreliable. Forces the realization that we can't eyeball whether a difference is real.
Pelican + PlanOut: A/B Testing on a Static Site
Trent Hauck's post combining Facebook's PlanOut experiment framework with the Pelican static site generator — a creative integration showing how to run A/B tests on a static site without server-side logic. An early example of bringing rigorous experimentation tooling to lightweight web stacks.
A/B Test Calculator — ABBA (Thumbtack)
Thumbtack's ABBA (A/B Analysis) tool — a Bayesian A/B test calculator that reports the probability one variant beats another, rather than traditional p-values. Practically more useful than frequentist tests for the decisions product teams actually make.
Why You Should Be Spot-Checking Algorithms on Your Machine Learning Problems
Jason Brownlee's Machine Learning Mastery argument for spot-checking — trying 10-15 algorithms quickly on a new dataset before committing to tuning any single one. Saves time by letting the data tell you which algorithm families are worth investing in.
Multi-Armed Bandit Experiments
Analytics blog post on multi-armed bandit experiments as a replacement for static A/B testing in website optimization — covers epsilon-greedy, UCB, and Thompson Sampling with practical framing for product teams.
A Book About Bandit Algorithms
John Myles White's free book on bandit algorithms for website optimization — covers epsilon-greedy, softmax, UCB, and Thompson Sampling with practical web application examples. An accessible bridge from theory to product experimentation.
Understanding Multi-Armed Bandit Algorithms
DataBozo's conceptual explanation of multi-armed bandit algorithms — epsilon-greedy, UCB, and Thompson Sampling compared from first principles. Aimed at practitioners who want to understand the tradeoffs before implementing.
Why Wikipedia's A/B Testing Is All Wrong (And How Contextual Bandits Can Fix It)
Synference blog post critiquing Wikipedia's A/B testing approach — argues that testing average effects ignores user heterogeneity, and contextual bandits could deliver personalized treatments rather than one-size-fits-all decisions.
The Importance of Sequential Testing
Austin Rochford's introduction to sequential testing — the SPRT and Bayesian alternatives to fixed-horizon A/B tests that let you stop early when results are clear without inflating false positive rates.