Pitfalls

How Optimizely (Almost) Got Me Fired

SumAll's account of how Optimizely's 'optional stopping' in A/B tests produced false positives that led to bad product decisions — a landmark post in the backlash against naive A/B testing tools. The core issue: peeking at results and stopping when p<0.05 inflates false positive rates dramatically.