Lego's fortune teller

Internet is full of A/B testing examples that worked (or didn’t) for some companies. As far as I know (and please correct me if I’m wrong) the largest collection of (vendor-independent) examples is the 536 case studies by WhichTestWon (the “unbiased resource for people who love testing“). Even if not all of them are available to non-members, quite a few are, and you can also subscribe to their newsletter to get one new case study a week directly into your inbox. You can even submit your own and, if done with Nelio, we’d love to know as well!

So, with this or other similar pages, it’s very easy to start checking how good you are at guessing the design variants that convert better. You may turn out to be quite good at that and guess right most of the time, but most likely you’ll be surprised by many of the results that will probably seem a little bit counter-intuitive.

Either case, the question is: Is playing A/B testing games useful at all for your business? I’m afraid the answer is, for the most part, “no”. The good point of browsing A/B testing libraries is that you can get some ideas that you could test on your own site. It’s always good to see what others have tried before and get some inspiration from them. But, some words of caution before adopting any kind of, let’s say, example-driven approach for your own A/B testing strategy:

  • Most online examples only give partial information on the test conditions and results, which may mislead you (even if not on purpose) when interpreting the tests.
  • The data is provided by the companies running the tests in-house, not by independent third parties. Again, this doesn’t mean companies try to lie when providing the results but they may make mistakes since they are not likely to be testing experts.
  • Results of those tests cannot be extrapolated to your company. For every example of a winner A/B Test you’ll find somebody claiming the opposite results. Don’t take them for granted. If you see a test that you think it could work for you, make sure you test that it really does before implementing it.
  • Tests are done with a specific purpose. A test that works to make more people click on a specific area of your site may cause undesired effects if your goal is increasing the time-on-site of your visitors. Keep this in mind when choosing possible tests to replicate.
  • Don’t become overconfident. Guessing correctly a few online examples may give a false sensation of security. Do never assume you know your visitors well enough to believe that you can improve the conversion rate of your site based only on your assumptions.

Key point to take is: play as many A/B testing quizzes as you like, but don’t gamble on them!

Featured image by Pascal

Leave a Reply

Your email address will not be published. Required fields are marked *

I have read and agree to the Nelio Software Privacy Policy

Your personal data will be located on SiteGround and will be treated by Nelio Software with the sole purpose of publishing this comment here. The legitimation is carried out through your express consent. Contact us to access, rectify, limit, or delete your data.