WordPress A/B Testing Plugin Comparison: What’s New for 2021

In web design, knowing what works and what doesn’t is crucial when making decisions aimed at generating revenue. However, in the marketing world there are a lot of variables to take into account. Relying on our experience alone is too risky. For that reason, A/B testing is the best tool to help us make decisions based on proven information.

The advantage of an A/B test is that it is a type of experiment in which you can isolate one or a set of variables and analyze if the conversion of your website improves. That is, you can compare two different versions of, for example, the title and an image of a page, and then analyze their performance without having to modify the rest of the elements.

Basically, an A/B test works as follows: the traffic coming to the site is randomly divided into two groups, so that 50% of the visitors see design A while the other 50% see design B. By tracking how the users in each group react, we can calculate the conversion rate (the percentage of visitors who “convert,” where conversion refers to the completion of a desired action such as clicking a button, making a purchase, requesting information, etc). And if there is a statistically significant difference between the two, declare a winning design.

In short, A/B tests are the best tool to delve into each specific element of your website and determine which one best serves your visitors and converts more traffic into sales. And, beware, it can also happen that two variants of an A/B test are so similar that there is no clear winner.

Key Features of an A/B Testing Plugin for WordPress

Before listing WordPress plugins for A/B testing, I would like to discuss the features to consider when selecting the A/B testing plugin that best suits your needs. Some of them are features that apply to the selection of any type of plugin and others are unique to an A/B testing tool.

Native WordPress Plugin

Before installing any type of plugin on your website, it is important to understand what type of plugin it is. Is it a native WordPress plugin, designed for WordPress and offering functionalities directly in the WordPress environment? Is it a framework that is installed in WordPress to create an environment for you as if it were another tool (similar to a page builder)? Or is it just a service to install a tracking code on your website to use it in an external tool? As you can see, although you can find many A/B test plugins in the WordPress directory, first you must understand what type of plugin it is. The advantages of a native A/B testing solution for WordPress are as follows:

• Easier learning curve. No need to learn yet another tool. Everything (from creating tests to viewing the results) is done from the WordPress Dashboard using the same interface you’re already familiar with.
• Automatic WordPress site improvement. A native solution for WordPress can automatically evolve your WordPress site. For example, as I discuss below, you can update the site to reflect the winning variant as soon as the results become statistically significant.
• Precision tests. The “test unit” in a generic tool is the web page since the HTML on the page is the only thing those tools have access to. Instead, a native solution for WordPress can access all WordPress components, including pages, posts, themes… so you may choose to test a combination of these components (for example, test two different themes throughout the site) rather than specific page.
• More control. Can we show the tests only to (un)registered users, or to users with a certain role? This type of control in the testing process can only be done when the A/B testing tool has access to the internal parts of your WordPress installation.

Types of Tests

According to the Number of Variants

Some A/B test plugins only allow you to create A/B tests with two versions: the original version and an alternative variant. Other plugins also allow you to create multivariate tests. That is, in addition to the original version you can create more than one alternative variant. The advantage of a tool that allows you to create multivariate tests is that you can compare several variables at the same time. These types of tests are especially recommended on pages with a lot of traffic.

According to Their Scope

The vast majority of A/B testing plugins only allow you to create A/B tests of elements that are on a specific page or post. That is, perform an A/B test in which you compare two or more versions of a certain page. But other plugins allow you to create tests on a global scope that can affect more than one page or post on a website. For example, some plugins allow you to create A/B tests of navigation menus that obviously have an impact on the entire web since the menu is accessed from any page. And the same happens with A/B tests of widgets, templates, themes or CSS styles, among others.

If you create a test that can affect all pages of your website, make sure that the plugin is able to display the test variant consistently on all pages. For example, if a plugin allows you to A/B test the price of a product on your home page, the price should be the same on any other page in which it is displayed.

Also keep in mind that the scope of an A/B test can condition the tests that you can run concurrently on a website. It doesn’t make much sense to run a theme test and a menu test that have a global impact on all pages simultaneously. What variable will you attribute a conversion to: a theme or a menu variant?

On the other hand, if a plugin only allows you to create a single A/B test on your website concurrently, you will have missed many opportunities to improve unrelated pages. For example, why not try optimizing different independent product pages?

The plugin you use should know how to correctly handle all these points related to the scope of a test.

WordPress Elements that You Can Test

A good A/B testing tool should provide you with the ability to test a large set of items including:

• Menus (both for the main menu, as the footer, or any other secondary menu that you may have on your website),
• Titles of pages or posts,
• Headlines (the title, excerpt, and featured image of a post),
• Text fragments from any page,
• Images from any page,
• Buttons and other calls to action,
• CSS styles of layouts,
• Templates,
• Widgets,
• Forms,
• Custom post types,
• Themes

Precisely, if the tool is a native plugin for WordPress, it will be much easier for you to perform this wide range of types of tests. Even so, an A/B test plugin whose way of generating a variant is through a shortcode that is added to the original page and whose content is dynamically replaced, will only allow A/B tests of the elements included in the content of a page or post.

Nelio A/B Testing

Native Tests for WordPress

Use your WordPress page editor to create variants and run powerful tests with just a few clicks. No coding skills required.

More Info

Editor for Creating and Modifying Variants

When you create a variant of an A/B test, many times you will want that variant to be a version very similar to the original, making small changes. What process should you follow to create that variant? What environment does the plugin provide you with to create and modify it as you wish? Do you have to learn a new tool? Do you need to know how to code? How flexible is it?

This point is closely related to the type of WordPress plugin you choose. If it is native to WordPress and you can use the same editor that you are used to to create your pages, posts, menus, widgets, etc., it’ll be much more convenient.

Defining Goals and Conversion Actions

A/B tests are performed with one or more goals in mind such as “getting more subscribers” or “showing interest in our products.” These objectives are met when the visitor performs certain actions such as “fill out the subscription form,” “visit the price page,” or “visit the page of a specific product,” to name a few examples.

Conversion actions can include: clicking on a button, clicking on any other type of content on a page, visiting any other page or post on the web, visiting external links, submitting forms, purchasing products, watching a video, and so on. And the more options the plugin gives you to define conversion goals and actions, the more complete and accurate the results you can get.

That said, a plugin should not only allow you to define a large set of conversion actions, ideally you should be able to do it easily and simply, without the need for programming knowledge.

Segmentation

A very interesting feature that some A/B test plugins offer is the ability to segment visitors. That is, an A/B test can only be seen by a certain group of visitors to your website with common characteristics: they are from a specific country, speak a certain language, come from another website, etc.

Being able to filter the visitors and analyze the results by predefined segments will allow you to be much more rigorous with the analysis of the results obtained in your tests. The more details and variables you can isolate in the analysis, the better you will be able to optimize the conversion of your website.

Test Completion Options

The duration for which you need to run your A/B tests depends on several factors, such as the complexity of the test (the more variants, the more time you will need) and the number of visitors participating in the test (the more visitors, the more reliable the results).

If a test is stopped too early, the results you get may not be meaningful. And if you keep it running for too long, the results may no longer be valid because the circumstances have changed. Remember that in an A/B test there may not be one variant that is better than another.

The A/B test plugins should allow you to finish the tests manually, or automatically following certain reliability criteria of the results obtained.

Information Provided in the Results

Basically, an A/B test plugin has to answer the following question: what is the probability that variant B is better than the original. An A/B test plugin should be transparent and inform you which statistical test it uses but, at the same time, it should do all the statistical work for you. It should give you the results that help you make decisions without having to look up the definition of conversion rate or statistical significance. It should help you not to go crazy interpreting the information it gives you so that, with a simple glance, you can know if there is a winning version and what is the reliability of that result.

Application of the Winner

A functionality that is rarely taken into account in an A/B test plugin that can save you a lot of time is to be able to apply the winning variant easily. There are tools that after completing the A/B test and win variant B, then you must go to the website and make the changes to apply as final that variant B.

In some cases, this may mean that you have to change a few buttons, images or text fragments on a page and the effort involved is small. But in some cases it may involve redoing the menus, widgets, forms, etc., which can take a lot more work.

To reduce the risk of error to zero and save time making these changes, you will appreciate that the A/B test tool itself automatically allows you to apply the winning variant at the end of a test.

Overload of Your Server

Each A/B test plugin implements different components for creating, tracking, and calculating test results. If all the components are running on your server, you should keep the following points in mind:

• Is your server responsible for assigning the variant that corresponds to your visitor?
• How is a variant generated? To what extent does it affect page loading?
• Is all the tracking information stored and calculated on your server or on an external service?

As you can see, depending on the solution provided by the plugin, it can have an impact on the price of your hosting service and SEO. If you don’t want problems, do not select an A/B test plugin without being clear about the previous points.

CDN or Cache Plugin Compatibility

In simplified form, a cache or CDN makes it easier to keep copies of pages (on the client’s computer in the case of a cache, or on content distribution networks for the CDN) so that each time a client makes a request from that page to a web server, instead of having to load it from scratch, it can provide the copy in question much faster.

Caching mechanisms are a real challenge for A/B tests and it could happen that if your website is behind a cache, all your visitors will always end up seeing the original version, making A/B testing impossible. For example, an A/B test plugin whose way of generating a variant is through a shortcode that is added to the original page and the content is replaced dynamically, may suffer from caching or redirection problems.

Make sure your A/B testing plugin is compatible with caches and understand well what solution they offer to find a balance between conversion optimization and performance.

Compatibility With Third-Party Plugins

One of the things you soon discover when you dive into the world of WordPress is that no matter how standard WordPress is, no two website configurations are the same. Each one has its own theme and plugins installed and customized, so you never know if any of them may or may not be compatible with your A/B test plugin.

Our first recommendation is that before selecting a plugin (whether A/B testing or any other type) make sure that there is a team behind it that is willing to follow good programming practices and is clear about the mistakes they should not make when developing a plugin.

And then, of course, before installing an A/B testing plugin, check its documentation to better understand to what extent it will be compatible with everything you have on your site.

Criteria that Every WordPress Plugin Should Meet

Finally, before selecting an A/B test plugin, make sure it meets the selection criteria that we recommend :

• Look at the reputation of the developer,
• Make sure the plugin is under active development, and
• Check its support and community

A/B Testing Plugins for WordPress

Here are the the A/B testing plugins for WordPress that you can download from the WordPress directory. The 4 plugins listed below have been tested with at least WordPress version 5.0.

1. Nelio A/B Testing
2. Split Test for Elementor
3. Marketing Optimizer
4. A/B Testing for WordPress

Nelio A/B Testing

Nelio A/B Testing is a native WordPress plugin developed by us that offers you the most complete solution to perform A/B tests and multivariate tests on any WordPress website. It offers you a fully integrated solution with the Gutenberg block editor and is also compatible with page builders.

Features

• Easy to use and create tests.
• Unlimited users.
• A/B tests, multivariant tests, heatmaps, clickmaps and scrollmaps.
• Consistency of the test scope throughout the site.
• Elements you can test:
  • page
  • post
  • custom post type
  • headlines
  • templates
  • themes
  • widgets
  • menus
  • CSS
  • WooCommerce product summaries
• You use WordPress editors (including menus, widgets, and CSS editors) to create tests.
• Definition of multiple conversion goals and actions:
  • visit a page, post or custom post type of your WordPress
  • access an external page (based on URL)
  • submit a form
  • click an element on the page
  • buy a WooCommerce product
  • custom JavaScript event
• It allows segmenting the traffic that participates in a test:
  • by parameter in url
  • referrer
  • language
  • location
  • user login
  • weekday
  • time frame
  • browser
  • type of device
  • operating system
  • window width
  • cookie
  • IP address
• Completion of a test (manual, automatic by volume of visits or by date).
• Very easy to interpret reports with graphs of the evolution of the daily conversion rate, improvement or worsening of each variant, test summary, etc.
• Automatic application of the winning variant as final.
• Minimal client server overhead: storage and processing of test tracking data is performed on Nelio’s servers.
• Compatible with cache plugins or CDN.
• It is compatible with third-party plugins.
• Compatible with WooCommerce.
• More than 1,000 active installations, version compatible with the latest WordPress update and good rating in the WordPress plugin directory.
• Support service in the free version and premium plans.

Plans and Pricing

Free version accessible from the WordPress.org plugin directory : it includes the basic features allowing the creation and execution of page tests.

Premium plans: plans and prices depend on the number of sites on which you want to install the plugin, the volume of page views with tests running, and various functionalities.

Split Test For Elementor

Split Test For Elementor is a native plugin for WordPress that allows you to create A/B tests of pages or posts using the functionalities of Elementor’s page builder. It is a fully integrated solution with said page builder.

Features

• Easy to use and create tests with Elementor.
• Unlimited users.
• Test A/B.
• Elements that you can test (only if they have been created with Elementor):
  • page
  • post
  • Elementor widgets
  • Elementor menu
• You use the WordPress Elementor page builder to create tests.
• A single conversion goal:
  • visit a page of your WordPress (or external in which a tracking code must be added).
• It does not allow traffic segmentation.
• Completion of a test manually.
• Very easy to interpret reports with graphs of the evolution of the daily conversion rate.
• Automatic application of the winning variant as final.
• Client’s server overload: the storage and processing of the test is done on the client’s server.
• It may not be compatible with cache or CDN plugins.
• It may not be compatible with third-party plugins.
• It is not compatible with WooCommerce.
• More than 2,000 active installations, version compatible with the latest WordPress update and very few ratings in the WordPress plugin directory.
• Support service in premium plans.

Plans and Pricing

Free version accessible from the WordPress.org plugin directory : it includes the basic features allowing the creation and execution of page tests.

Premium plans: premium plans and pricing depend on the number of sites where you want to install the plugin.

Marketing Optimizer

The Marketing Optimizer plugin creates a framework in your WordPress that allows you to create tests of entire pages, or of other elements that you can insert into your pages using shortcodes. In order to run the tests, you must integrate the plugin into its premium platform, which also includes a CRM and other marketing automation features.

Features

• Creation of tests in the plugin framework.
• Unlimited users.
• Test A/B.
• Elements you can test:
  • popup forms (via shortcode)
  • action calls (via shortcode)
  • landing pages
• You use a framework to create the elements to test.
• The conversion goals and actions:
  • page visit
  • click on call to action
  • click on the popup form
• Completion of a test manually.
• Reports only available on the premium platform.
• Manual application of the winning variant as final.
• Minimal client server overhead: storage and processing is done on the premium platform.
• It may not be compatible with cache plugins or CDN.
• It may not be compatible with third-party plugins.
• Not compatible with WooCommerce.
• More than 80 active installations, version compatible with the latest WordPress update and very few ratings in the WordPress directory.
• Support service in premium plans.

Plans and Pricing

Free version accessible at the WordPress.org plugin directory which includes test creation features.

Premium plans: the premium plans and prices depend on the functionalities that its platform offers you.

A/B Testing for WordPress

The A/B Testing for WordPress plugin allows you to create two page section variants that can be integrated as an A/B test using a shortcode.

Features

• Easy to use and create tests.
• Unlimited users and with access according to WordPress roles.
• Test A/B.
• No consistency of testing scope across the site.
• Elements you can test:
  • sections that are integrated into pages using shortcodes
• You can use a section editor similar to WordPress’ own to create tests.
• Definition of conversion goals:
  • visit a page or post of your WordPress
  • access an external page (based on URL)
• No traffic segmentation.
• Completion of a test manually.
• Very simple report showing the total conversion ratio for each variant.
• Manual application of the winning variant as final.
• Client server overload: the storage and processing of the test is done on the client servers but little information is saved.
• You may have compatibility issues with cache plugins or CDN.
• It is compatible with third-party plugins.
• It allows compatibility with WooCommerce.
• More than 300 active installations, version compatible with the latest WordPress update and very few reviews in the WordPress plugin directory.
• Support service in the WordPress plugin directory.

Plans and Pricing

Free version accessible at the WordPress.org plugin directory which includes all the features mentioned above. It does not have premium versions.

Other A/B Testing Tools

In the WordPress plugin directory you will also find other plugins to create A/B tests. I have omitted all those that were not tested at least with WordPress version 5.0. Some of the plugins I have omitted in this post basically serve as a connection to A/B test creation platforms external to WordPress. However, I do think they deserve a mention nonetheless: Convertize (premium version starting at $49/month), Visual Website Optimizer (premium version starting at $199/month), Convert Experiences (premium version starting at $699/month), and Optimizely (personalized prices but at starting approx. $800/month), among others.

What is The Best A/B Testing Plugin?

If you’re still not sure which A/B testing plugin is right for you, here is a quick summary of some things to keep in mind:

• Nelio A/B Testing is the most complete and versatile native plugin for WordPress to be able to create A/B tests without having to leave your WordPress, and it’s been developed by us,
• Split Test For Elementor is a plugin that you should consider if you are an Elementor fan,
• Marketing Optimizer is used to create tests that are integrated into a tool with additional digital marketing functionalities,
• A/B Testing for WordPress is a free plugin that allows you to create quick and easy tests.
• You have other very complete external WordPress platforms with which you can also create A/B tests.

You may also be interested in knowing our experience and the A/B tests that we are carrying out on our own website to improve it. And of course, don’t hesitate to ask us about any topic related to the creation of A/B tests.