Significance, and the question that matters more
Statistical significance tells you whether a difference is likely real rather than noise. The convention is 95% confidence, which means roughly a one-in-twenty chance of calling a winner that is not one.
The more useful question for an early-stage company is different: how long would this take? A startup with 600 visitors a day trying to detect a 5% lift needs months of runtime. Knowing that up front tells you whether A/B testing is a sensible use of your time at all, or whether you should be making bigger changes and measuring them more crudely.
Three ways tests go wrong
- Stopping when it looks good. Checking daily and stopping the moment confidence crosses 95% inflates false positives badly. Decide the sample size first.
- Too few conversions. Under about 100 conversions per variant, single events move the result. This tool warns below 30 because that is where the number becomes actively misleading.
- Running less than a week. Tuesday traffic does not behave like Saturday traffic. Always cover whole weeks.
The honest answer for most seed-stage companies
If you need two months to detect a 10% lift, testing button colours is not your growth strategy. Reserve testing for high-traffic surfaces and make bigger, judgement-led changes everywhere else.
Questions
What confidence should I require?
95% for anything you intend to ship and keep. 90% is defensible for a low-risk reversible change where the cost of being wrong is small.
Can I stop as soon as it hits 95%?
No. Peeking repeatedly and stopping at the first significant reading substantially inflates your false-positive rate. Fix the sample size before you start and run to it.
What is the confidence interval telling me?
The range the true absolute difference is likely to sit in. If that range includes zero, you cannot rule out that there is no difference at all.
My traffic is tiny. What should I do instead?
Test bigger changes — whole page rewrites, different offers, different channels — where effects are large enough to detect quickly. Save incremental testing for when you have the volume.