A/B Testing Checklist: from Good to Great Tests

Just think about your latest test—you had a hunch that a certain change in your website would lead to more conversions. Where did you get that hunch from? Maybe you asked your customers, maybe you read about that specific test somewhere. But, in the end, when you were about to test that change in your own site you didn’t know whether it would actually work better than what you already had. That is, you were walking blindly, fingers-crossed, hoping for the best.

A/B Testing is 50% science, 50% luck, and there’s no way we can change this reality. If you want to maximize the results of your testing efforts, you have to focus on the scientific facet of A/B Testing. You have to be systematic. You have to be patient. You need an optimization plan. Do you want to be on the right path? Take a look at the following checklist and make sure you never again miss anything!

The Warm-up Checklist

The first stage in A/B Testing is planning. In particular, you’ll have to ask yourself what you want test and how this test will impact your website. Oftentimes, you’ll focus your efforts on the top of the funnel where there are more users (and, therefore, where the impact can be, apparently, greater), but any other section or page might also be relevant. Consider the following list before starting new tests and make sure you double-checked each point:

1. Measure your website performance. Before moving forward with split tests, you need to have some insights on your site. For example, you should know how many unique visitors come to your site, what are your top pages, how much time they spend on each, which channels are really contributing to your conversions, or which pages are the “exit” pages of your website. All this information can be retrieved from Google Analytics, so I recommend you take a look at it.
2. Understand your visitors’ behavior. Once you have the overall picture of your website, you need to figure out why things are the way they are. Heatmaps are one of the easiest ways to understand your visitors. A heatmap shows you what captures your visitors’ attention on a certain page, and it’s very, very helpful for detecting design flaws (such as, for instance, the fact that your visitors try to click a non-clickable element).
3. Define your website goals. This one is obvious, but many people forget about it. What’s the goal of your website? Do you want to sell more products? Do you want more subscribers? You should have a clear vision of what goals you pursue, or you won’t be able to reach them.
4. Think about how you can track this goals. Once you’ve identified “the goals”, you need to determine how a visitor fulfills them. For instance, if you want to increase the subscribers of your newsletter, then you want your visitors to submit a certain form. If, on the other hand, you want to increase your revenue, you may want your visitors to subscribe to the Professional Package of your services (instead of the Basic Plan). In other words, in this step you need to translate the abstract goals into concrete actions.
5. Create a hypothesis. That is, decide what to test and how. Will you change your subscription form? Will you test a different layout? A hypothesis is no more than a claim (“if I change this into that, I’ll get more whatever”) that has to be supported or disproved by evidence (i.e. the results of an A/B test).

The Creative Checklist

During the past months I’ve had the opportunity to talk to many of our customers and there’s one thing that struggles almost all of them—the creative facet of A/B Testing. Coming up with useful hypothesis and designing experiments to actually test them is not easy. But it’s important, because knowing what to test and how to do it will be the difference between successful or failed tests.

Here you have a list of things that could be tested and, for each of them, a set of general guidelines you should consider. Remember, though, none of this is written in stone. Take what you need and leave the rest behind.

Nelio A/B Testing

Native Tests for WordPress

Use your WordPress page editor to create variants and run powerful tests with just a few clicks. No coding skills required.

More Info

The Basics of a (Landing) Page

Landing Pages are one of the most important components of your website. It’s the first thing your visitors will probably see, so you want to make sure that you engage them from the very first moment. Getting people to act, though, is far from easy. If you want to avoid some common mistakes, just follow these recommendations:

1. Use clear headlines. The headline of a Landing Page is the first message your visitor will see, so make sure it’s clear and focuses on value (not features).
2. Visible Call to Action Buttons. Whenever you’re looking at your pages, make sure:
   • they have a call to action above the fold,
   • it stands out (in color and size),
   • it’s clickable and it gets your visitors to the appropriate page,
   • has a clear message that tells your visitors what will happen if they click on it, and
   • it only uses one or two words (the fewer the better).
3. Create succinct content. 40% of Internet users abandon a website page in the first three seconds. Don’t overwhelm them with too much information—focus on what matters and nothing more.
4. Don’t use more than two main colors on your pages. Not only the content has to be clear, but also the design. You should be able to make relevant things stand out with just a couple of colors (for instance, our website uses blue for the general design and orange for the relevant things).
5. Back up your claims. Testimonials, social shares, number of registered customers… if you can show to your prospects what your customers think of your service or product, it can have a huge impact in your conversion rates.

Testing Ideas

If there are pages on your website that don’t adhere to the previous recommendations, then you have an idea of what to test. There’s a lot of things you could test to improve the compliance of these items:

• One VS Multiple call to actions. If you have a long page, nicely organized and chunked in sections, you could add a call to action button in each section. You don’t know which section will lead to a conversion, but whichever it is, the button for converting will be there.
• Single Headline VS Primary and Secondary supporting copies. Sometimes, it’s difficult to come up with a catchy, self-explanatory headline. If you can, great for you! But it might be worth to test an alternative version where you use a secondary small statement that supports and further explains the first claim.
• Long VS Short Page Copy. Even though the general guideline is “be succinct”, it’s not always the best idea. Some items need a more detailed explanation than others. Therefore, try a shorter version that goes straight to the point against a longer one that describes every single feature of your product or service.

Images and Videos

A couple of days ago we discussed why you should use videos in your WordPress site and we reviewed different types of video you could produce. As it turns out, they have greater share growth and engagement rates than images. So, why don’t you use them?

Here are some recommendations you should follow when embedding videos in your website:

1. Check the Call to Action. Services like YouTube allow you to define clickable areas in your videos that can serve as CTA buttons. It’s obvious, but make sure they point to the proper page in your site.
2. Test it on Multiple Devices. If you upload your videos to your WordPress server, make sure they work on all devices. Jenni McKinnon shared a great guide for using video in WordPress, and warned that «browsers and mobile devices often have different requirements when it comes to the video compression methods that are able to be displayed».

Testing Ideas

Let’s take a look at a couple of ideas for testing videos:

• Image VS Video. If you’ve never used videos before, then this is clearly the first test you should try out. For instance, if you’re offering a WordPress plugin (or an online service), use a screenshot of the plugin in one version and a video demoing its functionalities on the other, and see what converts better.
• Autoplay VS Manual Play. I’m one of those users that don’t like when things start to play automatically. But a video that autoplays will surely grab my attention (in my case, so that I can stop it). Autoplaying content may have a clear impact on your conversion rate, so why don’t you go ahead and test what works better for you?

Forms

Forms are one of the best ways to get information from our prospects. Once you “know” who a prospect is (that is, once you know, at least, her name and e-mail), you can establish a relation with her and you can promote your service or product. Creating better forms is easy if you apply the following checklist:

1. Minimum required fields. It’s always very tempting to ask for as many information from our prospects as possible. If you want to do that, that’s fine; but make sure that the amount of required fields is as small as possible.
2. Required fields are easier to spot. There’s nothing more annoying than taking the time to fill-in a form, submit it, and discover that it’s not working because I missed a required field. Make sure they stand out!
3. There’s a useful validation layer. If a form cannot be submitted, tell your visitor why. She took the time to fill-in the form and now she’s stuck because some field does not contain an expected value. Tell her so and help her fix it quickly.
4. Feedback. After a form has been successfully submitted, let the user know. Show her a confirmation message so that she can rest assured.
5. Confirmation e-mails. If a user subscribes to a newsletter or signs-up to your service, then send her a welcome message too. Start the relationship with your customers as soon as possible! Make them feel special 🙂

Testing Ideas

The previous checklist already tells us multiple possible tests:

• Number of fields. If you want to gather links, ask for the e-mail only. Sure, there’s a lot of information you’d like to get from your visitors, but there’s always time to get that information later.
• Required VS Standard fields. Check if less required fields converts better. Remember we visitors are lazy… the easier it is to submit a form, the better 😉
• Long form VS Multi-step form. Sometimes, we do need to ask for a lot of information. In those cases, you could try to show all form’s fields all at once (which might be discouraging), or you can design a multi-step form, where the visitor completes a section at a time (which looks shorter and, hence, faster).
• Field order. Try reordering your form. For instance, if you ask for the “difficult” field first, and they fill it, then it’s more likely that they’ll fill in the rest of the form, because they already spent some time answering the complex question. Even WordPress developers are considering this!

The Cool-down Checklist

Finally, here’s what you have to remember when the test is (about to be) over:

1. The sample of the test is big enough. Never stop an experiment before it had, at least, 700 or 1,000 page views per alternative. After that, you can probably stop the experiment and look at the next recommendation.
2. Seek statistical significance. Once you stop an experiment with a winning variant, look at the significance level. Only if it’s over 90% (I’d even say 95%) should you “believe” what the test says and make the winning alternative permanent.
3. Look for insights. Regardless of the results you obtained, you should be able to learn something from the tests you run. Every single test you run will teach you something about your customers and your website, and this is valuable information you can use in your next experiment.

What Now?

Now it’s time for you to start testing! You already have all the ingredients you’ll need for running successful tests, so don’t waste any more time. And, remember, if you have other testing ideas, or if you think our checklist(s) are missing something… let us know in the comments!

Featured image by Jeffery Wong.