Stop Making These 7 A/B Testing Mistakes Today

 

‘Perfection is a process’

 

Replace ‘perfection’ in the above quote with ‘A/B testing’ and it would still make eminent sense. Achieving the perfect website, mobile application or marketing campaign requires you to tinker around with several facets. This helps you understand which changes find approval with your users, thus allowing you to incorporate those changes permanently. A/B testing is a great example of this as it lets you test out different variations and hypotheses while also collecting valuable data in the process. When done right, it can pay rich dividends. However, too many conversion rate optimization services and individuals fail to harness the full benefit of A/B testing due to certain mistakes. What are these A/B testing mistakes and how to avoid them? Read on for the answers.

Not testing long enough

We get it, time is of the essence these days and spending too much of it on testing out a few designs or text changes can be damaging. But the opposite can be true as well. Not running A/B testing long enough is the most common mistake made by overeager testers. The science is simple – the longer you run the test, the more precise your A/B testing results will be.

So, what’s the minimum duration to run an A/B test? The answer depends on three things,

• the size of your audience
• the number of variants you are testing

• the number of users allotted per variant

A general rule of thumb is to run an A/B test for around three to six weeks. If you are still not sure on what’s the right duration for your experiment, use an A/B testing calculator to arrive at a ballpark figure.

Testing one variation too many

It’s common to come up with a flood of ideas when you first sit down to improve the conversion rate of your website. While it’s good practice to note them all down, you must resist the temptation to create and test variations for all of them. That’s because it could easily lead to situations where,

• the number of users per variation might not be large enough to derive reliable results
• there might be too many winners because the variations aren’t distinct enough

So, what’s the ideal number of A/B testing variations to run? 3 to 5 is considered ideal. If you end up with more than that, simply consolidate them and begin testing.

Being too cautious with user buckets

Yes, A/B testing is a type of experiment. And like all experiments, it has the potential to backfire. That’s why only a small part of your users is exposed to the test. But there is a downside to this preventive measure. Marketers can get too cautious when earmarking user buckets. This can introduce noise into your data and impact it negatively.

In that sense, problems caused by allocating too few users per variation are similar to the ones caused by not running an A/B test long enough. And while there’s no ideal blanket figure that holds true in all use cases, A/B testing best practices advise allocating at least 5% of your users to a variation. You can also use an A/B testing sample size calculator for the purpose.

Running too many A/B tests simultaneously

This is another mistake born out of the time-intensive nature of A/B testing. Many conversion rate optimization services run several tests at the same time to obtain results quicker. But there’s an unseen effect at play here. Many of those tests are likely to hinder each other and produce flawed results as a consequence. Decisions you might make on the basis of such results may do more harm than good.

So, take your time with A/B testing and let each test run its course before introducing another one. If you have to run them at once, ensure the user buckets are allocated appropriately. For instance, if ABC and 123 are the tests you’ve chosen to run together, put variations of ABC in control group of 123 and vice versa.

Using dark patterns to inflate numbers

Dark patterns are defined as user interface tactics that nudge a user into taking an action they didn’t mean to take. This might sound like a CRO (conversion rate optimization) technique, except that it’s subversive and unethical. Invisible unsubscribe is the most common of these tactics as it simply hides the unsubscribe link from your emails. Sure, that will cause fewer people to unsubscribe from your emails. But data derived from this tactic is worth its weight in trash. There are plenty of similar dark pattern examples out there that might help in the short-term, but will hurt your business in the long-run.

The effectiveness of A/B testing lies in the quality of data it generates. If the data itself is suspect, actions based on it can act as roadblocks, instead of catalysts, towards reaching your goals. If you see a parallel here with the detriments of black-hat SEO, you are absolutely spot on.

Ignoring the impact of seasonal/periodic changes

Cycles and patterns are inherent to any business. When running an A/B test, it helps if you’re clued into these factors and how they affect the business itself. For instance, a gift shop might see more buyer activity in a particular season than others. The same holds true for gyms, electronic stores, travel agencies or any other business you can think of.

Let’s consider another example of a tax filing website. The site is likely to see more visitor activity at the end of the financial year compared to other periods. Converting a visitor is easier in this period than in, say, November. This naturally means you can’t test the same variations for both periods. When you’re aware of these seasonal/periodic fluctuations and the behavioral change they cause in visitors, you’re better prepared to plan and execute A/B testing.

Not running tests regularly

The quote at the beginning should’ve probably read ‘perfection is a continuous process’. That’s because A/B testing is a never-ending process. Every time you conduct a test, you learn more about your audience. You learn what works and why does it work. All these are insights that can be used in every aspect of your marketing. For one, they could help you derive greater bang for the buck from your PPC campaign.

After scoring well with your initial A/B tests, chances are high that you could hit a plateau. This is not a sign that you should stop experimenting. In fact, you need to double up and invest more time and money into A/B testing. Of course, it goes without saying that these resources must be backed up with thorough research and data-backed hypotheses.

Conclusion

A well-planned program can get you some amazing A/B testing statistics and put your business on the path to supersize profits. But that path is full of pitfalls and we’ve talked about seven of the most common ones that conversion rate optimization services and agencies should avoid. Even if you’re marketing your business on your own, learning how to avoid these mistakes will help you save a lot of time and money.