I hate A/B tests that are recommended to startups when they don’t have the volume to run them. I hate when they are used to avoid being courageous. I hate the false lessons they teach. I hate the illusion of certainty they give. In short I hate nearly everything anyone says about them and nearly every application of them. So yes, I hate A/B tests.
They Provide No Insights and Generate False Narratives
A well executed A/B test will tell you which side performed better but it won’t tell you why. At the end of the day you may know that “Sign Up” works better than “Create Account”. Is it because the word “Account” has a very specific meaning for your target audience? Did “Sign Up” simply fit the button better? There are any number of possible explanations and you have no idea what the real answer is.
The worst thing is that humans need narratives and when presented with a fact that is unsupported by a narrative we invent one. That invented narrative will spread further and be remembered longer than the original ‘fact’. But you have no idea if that narrative is actually true – someone just made it up. Try it yourself, go along to Optimizely and try not to create a narrative in your own mind.
They require a LOT of traffic
Split tests require a lot more traffic than most people who are running them have. If you have a page converting at 5% then on average you need about 30,000 visitors to that page to run an A/B test with 95% confidence. With the random walk inherent in your results you might sometimes need triple that. How long is that test going to take to run? Are you happy standing still during that time?
Even then, the test is going to be wrong 5% of the time – statistically significant does not equal true.
The moment anything changes it invalidates your result
An A/B test requires both sides of the test to be identical except for the item being tested. So what happens when something changes after the test? A soon as anything changes then the test needs to be rerun because the change invalidates the test. NO-ONE does this.
- If you change a word on the page, a colour, a button, then the test is invalid
- If your audience changes then your test is invalid
- If the damn time of the year changes then your test is invalid
- Even if significant time passes, your test is invalid as consumers tastes and views evolve over time
Most things aren’t worth split testing
Most of the time the cost of testing is going to far outweigh the potential benefit of the change. Your page probably has thousands of variables. The vast majority aren’t going to be worth the cost of testing. Don’t expect minor changes to have major results.
On what basis are you choosing your test and what is your rationale? Fear of being wrong, fear of being caught out, fear of being seen as rash or just because you think you should? Stop covering your ass and have the courage to plough forward and trust your own judgement.
P.S. A/B tests have their place and that place is where you are making fundamental changes that could radically impact performance and where you have large scale.