What’s New in Nelio A/B Testing 6.0

We’re extremely proud to announce that earlier this week we launched a new version of Nelio’s split testing plugin for WordPress and, dare we say, it’s pretty awesome. Nelio A/B Testing 6.0 comes with a complete redesign of its front-end scripts that make alternative loading and visitor tracking faster and more reliable. It also implements a completely new type of test (one you’ve all been asking for a long time) and several improvements to already-existing tests.

Today I’d like to share with you all the ins and outs of this new version. Hopefully, by the end of this article, you’ll want to upgrade and try it out right away. So, without further ado, let’s go!

Simplified Alternative-Loading and Tracking Scripts

A/B Testing, at its core, is a pretty simple process: given a page on your website, create an alternative version B and show each variant to half of your visitors. Track what they do in each variant and use the collected data to establish which variant, if any, is “the best.” This means that any split testing plugin needs three things:

1. An editor to create alternative variants
2. A method to assign and load alternative content
3. A method for tracking data

And our plugin is no exception. As a WordPress plugin, Nelio A/B Testing gets (1) for free: you can easily create test variants using the page editor you’re already using in WordPress. Nice! What about the other two?

When it comes to alternative loading (2), our plugin combines PHP and JavaScript code. Our JavaScript scripts are responsible of assigning a variant to each visitor and redirecting them to the appropriate URL. Our PHP components, on the other hand, are responsible of rendering the appropriate variant based on the current URL requested.

Finally, regarding (3), our plugin also relies on JavaScript. Our tracking script is responsible for listening to and tracking all views, clicks, events, and what not that occur on your website.

But how do (2) and (3) actually work?

Previously, on Nelio A/B Testing…

Previous versions of Nelio A/B Testing used to include several front-end scripts, each responsible of different aspects of an A/B test. In particular, these were:

• an alternative-loader.js script, which was responsible of determining if the current page was under test and, if it was, assigning a variant to the visitor and redirect them to said variant when needed
• a main.js script that, once it knew the correct variant had already been loaded, tracked all the relevant events (like test views, conversions, or any data required to build a heatmap)

Moreover, both JavaScript files relied on an inline script each: alternative-loader.js needed information on your testing settings and the active tests, whereas main.js needed information on the conversion goals, actions, and segments of all your running tests.

Originally, we split our testing scripts in two because we noticed that alternative-loader.js should only be loaded on the pages that were under test (which are usually only a subset of your website). main.js, however, was needed on all pages, because conversions can occur anywhere on your website.

Unfortunately, as time went by the plugin grew more and more complex. Cache and optimization plugins (which, let’s be honest, most of you use on your websites) could delay and defer our scripts, potentially breaking their execution order and, thus, our testing process. We fixed this by adding yet another inline script (internally known as kickoff.js) that was responsible of guaranteeing our scripts run in the correct order at the right time, but we were never satisfied with this solution.

Nelio A/B Testing

Native Tests for WordPress

Use your WordPress page editor to create variants and run powerful tests with just a few clicks. No coding skills required.

More Info

Simpler Solution

Nelio A/B Testing 6.0 untangles this mess by implementing a simpler solution that, as you’ll see in the next section, unlocks some new opportunities. This new version only uses one script, public.js, that integrates the responsibilities of all our previous scripts.

Being it a single script, we no longer need helper functions to coordinate and orchestrate the execution of multiple scripts. As soon as public.js runs, it’s guaranteed that everything will run smoothly and correctly.

Moreover, not only have we combined the previous two scripts into a single one, but we’ve also taken the opportunity to review the source code and optimize it, so that the resulting script is smaller than ever.

In previous versions, alternative-loader.js was 44Kb, main.js was 52Kb, and kickoff.js was 4Kb (i.e. 100Kb total). Today, public.js is only 76Kb – that’s almost 25% less than before! And, mind you, the script now comes with an extra algorithm for loading alternative content, as we shall see in a minute.

Alternative Loading Behind the Scenes

One caveat with split testing is content flickering, a phenomenon where the content on a webpage briefly appears and disappears repeatedly. This might occur, for example, when the visitor lands on a tested page and they briefly see variant A before they’re redirected to the variant they’re actually supposed to see.

To overcome this issue, Nelio A/B Testing implements a simple, yet clever solution: it adds a white overlay on top of your page’s content. This way, our plugin has time to determine whether the page is under test and, if it is, which variant should be presented to the visitor. If the plugin concludes that the content should be replaced, it can safely do so behind the scenes (i.e. under the overlay), without the visitor ever noticing any content flickering.

The white overlay is added using the following style:

<style id="nelio-ab-testing-overlay" type="text/css">
@keyframes nelio-ab-testing-overlay {
  to { width: 0; height: 0; }
}
html::before, body::before {
  animation: 1ms {$time}ms linear nelio-ab-testing-overlay forwards;
  background: #fff !important;
  display: block;
  content: "";
  position: fixed;
  top: 0;
  left: 0;
  width: 100vw;
  height: 120vh;
  mouse-events: none;
  z-index: 9999999999;
}
</style>

Once alternative content has been properly loaded, our plugin will remove the style tag from the HTML and the overlay will disappear.

But what if something goes wrong and our script fails? Or what if it takes too long to run? Does this mean the screen will remain white indefinitely? No, not at all. The previous style will automatically remove the overlay after $time milliseconds – a value you can tweak using this filter.

Inline Tests are Faster Tests

Nelio A/B Testing loads alternative content by redirecting the visitor to an alternative URL. For example, if the visitor lands on https://example.com/some-page/ and this page is under test, the plugin will randomly assign a variant to the visitor and redirect the visitor to a different URL when needed:

• https://example.com/some-page/?nab=0 is variant A
• https://example.com/some-page/?nab=1 is variant B
• etc

By adding the nab query arg in the URL, our plugin makes sure that each variant has its own unique URL and, thus, it’s possible to run split tests behind cached and CDN websites. All we have to do is redirect the visitor to the correct URL. Unfortunately, JavaScript redirections have one major drawback: speed. Or, should I say, lack thereof.

Think about it for a moment: when a visitor lands on https://example.com/some-page/, the browser requests the page to the server and renders its content as it receives the HTML source code, as well as downloading all related assets (images, styles, etc) in parallel. At some point during this process, though, our script kicks in and may decide the visitor should be redirected to a different URL, which means all we’ve been doing so far is useless.

To minimize the impact of a redirection, Nelio A/B Testing adds its public script at the very top of a page’s source code. This way, one of the first things the browser will download and run is Nelio’s script and, therefore, the redirection will occur as soon as possible, minimizing the amount of potentially useless work.

But we wanted to do better.

Nelio A/B Testing 6.0 implements inline tests – a test in which all the variants are already included in the page and loading alternative content requires no redirection whatsoever.

CSS Inline Tests

Let’s consider CSS tests as an example. As you may already know, a CSS test lets you define alternative CSS stylesheets that will only be loaded when the corresponding variant is loaded.

In previous versions of our plugin, when a visitor accesses https://example.com/some-page/?nab=1, Nelio A/B Testing used to tweak the page’s head tag and include the alternative style (unless, of course, it was the control version, in which no alternative style was loaded at all):

<!DOCTYPE html>
<html lang="en-US">
<head>
  <script id="nelio-ab-testing-main-js" ...></script> 
  ...
  <style id="nab-alternative-css" type="text/css">
    /* alternative CSS here */
  </style>
</head>
<body ...

Unfortunately, this approach meant that JavaScript redirections were always required – if the visitor was supposed to see a variant, our plugin had to rewrite the URL to let your WordPress server generate the page with the appropriate style tag in it.

Nelio A/B Testing 6.0 completely reinvents this type of test and makes the redirection no longer necessary. Now all alternative styles are included in the original request. Here’s how https://example.com/some-page/ looks like:

<!DOCTYPE html>
<html lang="en-US">
<head>
  <script id="nelio-ab-testing-main-js" ...></script>
  ...
  <noscript class="nab-exp-ID nab-alt-1">
    <style type="text/css">/* variant B */</style>
  </noscript>
  <noscript class="nab-exp-ID nab-alt-2">
    <style type="text/css">/* variant C */</style>
  </noscript>
  <noscript class="nab-exp-ID nab-alt-3">
    <style type="text/css">/* variant D */</style>
  </noscript>
  ...
</head>
<body ...

which means that, when our alternative loading script runs, all it has to do is apply the correct style tag while removing the other ones. Simple and effective!

Supported Inline Tests

At the moment, the only tests that support inline testing are:

• CSS Tests (always)
• JavaScript Tests (always)
• Page/Post/Custom Post Type Tests (optional)

In page, post, and custom post type tests, inline testing is optional. That is, when you create the test, it’s up to you if you want to enable inline testing or not:

As you can see in the previous screenshot, inline testing in this type of tests is limited to the page’s title and content only – you won’t be able to test, for example, alternative page templates.

JavaScript Tests Unlock New Testing Opportunities

Finally, Nelio A/B Testing 6.0 has a new type of test: the long-awaited JavaScript tests! As you can probably imagine, a JavaScript test is basically like a CSS test, the only difference being that you write JS code instead of CSS stylesheets.

When your alternative snippet is included in the page, it ends up being embedded in a function like this one:

function( done: () => void, utils: Utils ): void {
  /* Your code */
}
type Utils = {
  readonly domReady: ( fn: () => void ) => void;
}

which means there’s a few things you should be aware of:

1. There’s a couple of variables you have access to: utils and done.
2. When you’re done loading alternative content, you have to call done() to let our plugin know it’s now safe to remove the white overlay and show the content.
3. Currently, utils only contains one property: domReady. You can use utils.domReady to run an anonymous function with your code once the DOM is ready.
4. Your code will not pollute the global scope. If you want to define a global variable x or a global function f, you’ll have to explicitly do so by typing window.x or window.f.

Pretty neat, huh? We hope you like this version. Upgrade today and get the most out of your WordPress site!

Featured Image by Saffu on Unsplash.