Subject
8 entries
A B Testing
Bookmarks
Scalable A/B Experiments at Pinterest
Pinterest's engineering blog on how they built scalable A/B testing infrastructure — covering experiment assignment, metric computation, and statistical significance at Pinterest's scale. A practitioner's account of the gap between textbook A/B testing and production experimentation.
Multiple A/B/n Tests in Marketing with ANOVA and R
Marketing Distillery's practical guide to running multiple A/B/n tests using ANOVA in R — extending the standard two-group t-test to handle multiple variants simultaneously without inflating false positive rates.
A/B Test Calculator — ABBA (Thumbtack)
Thumbtack's ABBA (A/B Analysis) tool — a Bayesian A/B test calculator that reports the probability one variant beats another, rather than traditional p-values. Practically more useful than frequentist tests for the decisions product teams actually make.
Multi-Armed Bandit Experiments
Analytics blog post on multi-armed bandit experiments as a replacement for static A/B testing in website optimization — covers epsilon-greedy, UCB, and Thompson Sampling with practical framing for product teams.
Why Wikipedia's A/B Testing Is All Wrong (And How Contextual Bandits Can Fix It)
Synference blog post critiquing Wikipedia's A/B testing approach — argues that testing average effects ignores user heterogeneity, and contextual bandits could deliver personalized treatments rather than one-size-fits-all decisions.
The Importance of Sequential Testing
Austin Rochford's introduction to sequential testing — the SPRT and Bayesian alternatives to fixed-horizon A/B tests that let you stop early when results are clear without inflating false positive rates.
Deriving Formulas for Sample Size in A/B Tests
Mathematical derivation of the expected sample size needed in A/B tests using frequentist power analysis. Derives the N formula from effect size, significance level (α), and power (1-β) — useful for anyone designing experiments who wants to understand where 'you need at least X users' comes from.
A/B Testing with Bayesian Bandits in Google Analytics
FastML walkthrough of Google Analytics Content Experiments — which use a multi-armed bandit algorithm instead of fixed 50/50 splits to dynamically allocate traffic toward better-performing variants. A practical introduction to bandit-based online optimization.