Split testing is one of the best techniques for optimizing the conversion rate of your site. When properly done, it’s funny and attractive, because you can get visible results in a very short period of time. If you’re one of our customers, you’ve probably felt the excitement of running new tests and learning something new about your customers. But A/B testing is not magical—it won’t always help you improve your revenues.
There’s no doubt we like A/B Testing, isn’t it? We offer an A/B Testing solution for WordPress, and you’re on an A/B testing blog so… what were you expecting? 😉 Nonetheless, we know there’s a lot of myths about A/B testing and many unanswered questions. Today I share the 15 most-common myths and, for each of them, what the truth really is.
Myth #1. Conversion Rate Optimization is About Guessing
You’ve been in your business for so long that you know your customers perfectly. You know what they like, what they don’t. You know how to sell your product. Therefore, you don’t need a tool to tell you what works and what doesn’t.
The Truth. A/B Testing will not replace your guesswork, because guesswork is actually the first step for A/B Testing. As we’ve already discussed in this blog, A/B Testing is a methodology for making informed decisions. If you have a new idea on what to improve in your site, that’s great! A/B Testing is no more than a tool that will collect some information about your website and will determine whether that new idea is actually better (as you thought) or not. A/B Testing is a marketing tool that will help you make your decisions based on real data, not only opinions.
Myth #2. Marketers’ Knowledge Works Better than A/B Testing
OK, in the previous myth you were the one making the guesses so, yeah, we may agree your guesses could have been wrong. But if we’re talking about marketers, that’s a completely different story, right?
Marketing is about communicating the value of a product or service to its potential customers. Its goal is to promote and ultimately sell that product or service. Marketers are experts at marketing—they know how to use social media to boost your sales, they know what content is most appropriate for your blog, they know how to design your landing pages… there’s no need for A/B Testing! Their experience is all you need to run a campaign.
The Truth. Even the most talented marketers can be wrong. After all, they’re people like you and me. Now, I’m not saying marketers are not helpful or necessary, or that their knowledge is not valuable. In fact, having a marketing expert in your business will make a huge difference in your marketing campaigns—they do know a lot about your customers and how to target them! But that doesn’t mean A/B testing is not necessary.
A/B Testing doesn’t aim to replace a marketing expert, but to complement her. Let your marketers design your marketing campaigns based on their experience and, if they’re as good as they claim to be, they’ll use split testing techniques for validating their instincts.
Myth #3. Any Change to Your Website Should Be A/B Tested
When you start reading about A/B testing, the same examples pop up over and over again: modify the color of your action buttons, use a different image for promoting your product, test a different copy. So, apparently, any change that could be tested in your website should be tested. It doesn’t really matter if it’s small or big, or the total impact it might have in our conversion rates… if somebody did that before, why shouldn’t you?
The Truth. It’s easy to find case studies demonstrating the success of certain layouts, buttons, color schemes… And once you find them, testing them in your own website is very tempting, even though you haven’t given much of a thought about what you’re testing and why.
Each test is unique—the audience that participates in your experiment is not the same as the audience that participated in that case study you’re thinking of, the product you’re selling is different too, the promotions, the marketing funnels… So having case studies of what works and what doesn’t is not the ultimate solution.
Case studies and other websites might be a great source of inspiration, a source of new ideas to test. But remember A/B Testing is not about testing everything at all times. If you follow our testing strategy, you first need to guess what might be the bottleneck in your sales funnel and, then, hypothesize possible solutions to fix that pain area.
Nelio A/B Testing
Native Tests for WordPress
Use your WordPress page editor to create variants and run powerful tests with just a few clicks. No coding skills required.
Myth #4. Duplicate Your Competitors’ Design and You’ll Succeed
Again, this myth is very related to the previous one. So, you don’t want to test “everything” that’s on the web, but you want to make sure you test “everything” your competitors are doing. After all, your competitors are targeting a similar audience with a similar product. That makes them the perfect source of information!
The Truth. Yes, looking at your competitors and trying to better understand how they operate is totally legit and a great starting point to create your own optimization plan. But, again, that’s only one more source of hundreds you can use to get new ideas to test.
Duplicating other people’s layouts, color schemes, and so on won’t “automagically” improve your conversion rates. There’s a lot of things that might be affecting your conversion rates that are more important. Are you using appropriate keywords? Do you speak the same language your customers do? Do your customers have some concerns regarding the reliability and security of your platform?
Myth #5. You Should Focus on Converting Visitors Solely
Conversion optimization is about improving the conversions of your website. No more, no less. Therefore, once you find an alternative that converts more visitors into customers, you’re done. It doesn’t really matter what you do to increase the conversions!
The Truth. There’s a lot of things you could do to improve your conversion rate, such as misleading your prospects and make them believe you offer something you actually don’t. Sure, that might increase your conversions (for instance, you’re getting more subscriptions), but as soon as the customers realize you don’t offer what you claim to offer, they’ll unsubscribe and won’t ever come back. Do you really want that? I don’t think so.
Your goal should not be to manipulate the customers into converting, but to get more and better customers. You want your customers to be engaged with your product or service because, in the end, you want your business to grow.
Myth #6. A/B Testing is Expensive
Split Testing tools collect a lot of information about your visitors and process it to give you some useful insights. This makes A/B Testing services too expensive for a small company, and you can’t afford it.
The Truth. Nelio A/B Testing offers different plans based on your needs. Our Basic Plan is only 29$ per month (less than one dollar per day). I honestly think this is a very affordable and powerful solution! But if, for some reason, you’re on a near-zero-dollar budget, there are other solutions that might help you. For instance, Google Analytics’ Content Experiments will let you run split tests on your site (even though you’ll need to have some tech background to use it).
Myth #7. Only Tech Guys Can Setup Split Tests
In order to run an A/B Test, you need to create the alternative content you’ll test against your current website design and you’ll have to define the goals of your experiments. Apparently, in order to do that you need to add some weird scripts in your theme and/or use complicated shortcodes in your pages. Clearly, you can’t do any of those things easily! You need some help, don’t you?
The Truth. Yes, there are some complicated tools out there that will require some complex work on your end. But if you’re running WordPress, then you can use Nelio A/B Testing! Our tool is a native conversion optimization plugin for WordPress, so there’s no need to learn how to use an external tool. Just log into your WordPress dashboard, create a new experiment, and start editing alternative content using your favorite WordPress editor!
Myth #8. A/B Testing is for Sites with Tons of Traffic
Your website doesn’t have tons of traffic, just a few hundreds per day (if lucky). You know that you need “a lot” of visitors to get accurate results from your A/B Testing solution, so, unfortunately, A/B Testing simply makes no sense in a site like yours.
The Truth. Sure, running a lot of tests with a lot of variants each at the same time on a site with low traffic isn’t a very good idea. But if you’re running only one experiment at a time, you don’t need a ton of visitors for getting insightful results. For one single experiment with two variants, I recommend waiting until it reaches 1,500 to 2,000 page views and, after that, checking whether the results are statistically significant. If you have 100 visitors per day and you’re testing a page that gets a lot of traffic (typically, the landing page), you could get useful results in less than a month.
Myth #9. The Results are Immediate
Right after you start an A/B Experiment, your A/B Testing tool will collect information about your visitors and will present you the curated results. In just a few hours (almost in real time), you’ll be able to see which version performs the best. After only 100 page views, your original page has a 10% conversion rate and your alternative a 15%, you already know that the alternative version is much better!
The Truth. Not so fast! Sure, A/B Testing tools show you results right after the experiment has been started. If you’re like me, you’ll probably like to check how the experiment is progressing as time goes by. But don’t be fooled! We’ve already mentioned this in Myth #8, but before calling a winner your results have to be statistically significant. If you stop an experiment too soon, you might take the wrong decision. That’s why almost all A/B Testing tools have a Confidence value: they want to help you make the right choice.
Myth #10. Best Alternatives Are Always Prettier
The idea behind split testing is to test new designs, layouts, copies… and compare their performance to the current version of your website. It’s obvious that the new designs and alternatives will work better if they look prettier. It’s impossible for an ugly alternative to win, so there’s no point on testing them!
The Truth. The first reason for using A/B Testing is to base your decisions on actual data, not your opinion. As the saying goes, “Beauty is in the eye of the beholder”, so it doesn’t really matter what you find appealing or beautiful, you have to let your visitors decide what converts the best. After all, sometimes the best alternative is clearly uglier, isn’t it?
Myth #11. A/B Testing Negatively Affects Your SEO
By definition, A/B Tests compare two versions of the same page. Usually the two versions differ only on a few, tiny details, which means that almost all the relevant content is duplicated. As you already know, duplicated content is penalized by Search Engines, so it’s clear that running A/B tests will negatively affect your SEO.
The Truth. This myth is completely false. In Myth #6 we’ve seen that Google offers its own A/B Testing tool, which should serve as an indicator that there will be no penalties in your SEO. In fact, Google gives you a few guidelines on how to properly test your content to improve your conversions, all of which are followed by Nelio A/B Testing.
Myth #12. Once Your Test is Over, You’re Done
So you started an A/B Experiment to convert more visitors into customers. After setting it up, you started it and waited until the minimum sample was reached, the results were statistically significant, and one alternative was clearly better than the others. That’s awesome! You now have what you wanted. Time to move on to the next thing in your TO-DO list which, by the way, is not anything related to A/B tests.
The Truth. A/B Testing is a continuous process. You should try to improve your site always, tweaking different details in each experiment (ideally, one at a time). For instance, if you tested the color of your call-to-action button and found out the best color, it’s time to test the button’s label. Or simplify the subscription form. Or rearrange your subscription packages. Or… There’s plenty of things that could be addressed in a website, so don’t stop after the first experiment and keep going!
Myth #13. You Need a Large Team to Run A/B Tests
Marketing, web design, copywriting, knowledge about your business and your customers… there’s a lot of things you should know about to run proper A/B tests. If you had a larger team with experts on all those different areas, you’d be able to easily design and run your A/B Testing campaigns. But alone? No way!
The Truth. It’s true that, the bigger your team is, the easiest it is to design effective marketing campaigns. The combined knowledge and expertise of your team will certainly boost the impact of your tests. But that doesn’t mean you can’t do anything by yourself. If you’re running a small business or blog all by yourself, you’ve already demonstrated what you’re capable of. A/B Testing is no more than a simple tool that will help you to evolve your website, making sure that each step you take is a step towards excellence. Besides, running Nelio A/B Testing on top of WordPress will feel like second nature!
Myth #14. Your Conversion Rates Are so Good, There’s Simply no Room for Improvements
If you already have a high conversion rate, you probably don’t see the point of running split tests. Clearly, there’s a limit to how much you can improve your conversion rate and, apparently, you’ve already reached yours!
The Truth. Even if your conversion rate is high, there are always certain pages or areas that could be improved. On the other hand, web evolves rapidly; if you want to be up-to-date you’ll have to tweak your design and the way you share your message every now and then. Whenever you’re about to change your website, A/B Testing is the best tool for that!
Myth #15. An Inconclusive Test is A Useless Test
Sometimes you’ll run a test and wait for the data to tell you which alternative is better. The days and weeks go by but the results are inconclusive—that is, there’s no clear winner because both alternatives offer a similar performance. You just wasted your time with that test!
The Truth. You’re probably focusing on the nitty-gritty details of your website, such as “what if I replace this word with that word”. When you reach a point where all your tests are inconclusive, it’s time to take a step back, look again at your site as a whole, and start from scratch. You need to ask yourself what’s preventing your visitors from converting and address that issue. Look at the tickets your customers open and try answering those in your website. Or run usability tests and get useful insights on how your visitors feel while they’re browsing our website. I’m pretty sure you’ll discover new things that should be tested!
That’s all folks! 😀 I hope you enjoyed this post. And, as always, if you have any questions… do not hesitate and ask in the comments below!
Featured image by Christophe Verdier.
Leave a Reply