Documentation

All the information you need in one place

Is Nelio A/B Testing compatible with cache plugins?

Absolutely! Nelio A/B Testing will work properly even if your WordPress site is behind a cache system or a CDN. Here’s all you need to know about it.

How do Caches and CDNs Work?

Whenever a visitor requests a page from your website, WordPress runs to generate the requested page “on the fly.” As you can imagine, this takes some time and might result in slow response times, especially if you have a complex website.

A cache is a system designed to speed up this process by preventing WordPress from running for every request. To do so, all it has to do is keep a copy of the page WordPress has generated. This way, when new visitors come and request the same page, the cache can simply return the cached copy.

A CDN (Content Distribution Network) is a slightly more complicated cache system. They also keep a copies of your pages in order to prevent your server/WordPress from running. However, what makes CDNs great is that their cached copies are stored in multiple servers across the globe so that, when a visitor requests a certain page, the response can be served by the server closest to them and, thus, faster.​

Keeping static copies of your website around poses a clear problem: if one of your pages is cached and you change it, visitors will probably get cached copy instead of the new content. Cache and CDN systems implement several mechanisms to cope with this, like keeping cached copies around for a limited amount of time or offering mechanisms to let WordPress warn them when they need to invalidate a cached copy.

A/B Testing and Caching

Cache mechanisms are a real challenge for A/B testing. When your website is behind a cache or CDN, it’s possible that all your visitors end up seeing the same (cached) variant, even though some of them are supposed to see the other variant.

Some A/B Testing plugins for WordPress are simply not compatible with caches and make you choose between performance and conversion optimization. If you deactivate the cache, WordPress can serve all requests individually and, thus, generate the appropriate variant for each visitor.

Nelio A/B Testing, on the other hand, is not only compatible with caches, but it even makes use of them, either by using query arguments or by salting your cache with a cookie. Let’s take a closer look at each solution.

The Importance of Purging Your Cache

After starting or stopping a test, the cache and CDN networks should be flushed, just to make sure that there aren’t any cached versions of your site in which the tested page and the list of running experiments are out of sync.

Our plugin comes with several hooks to automatically flush the cache of some common cache-plugins for WordPress, but we strongly recommend to manually flush them anyway.

Method 1. Query Arguments with JavaScript Redirections

As you may already know, when a visitor first lands on your website, Nelio A/B Testing randomly assigns them a number and stores it into a cookie named nabAlternative. Then, our plugin checks if the current page is under test and, if it is, it rewrites the current URL by appending the special query argument nab:

  1. http://example.com/tested-page/?nab=0 is variant A
  2. http://example.com/tested-page/?nab=1 is variant B
  3. http://example.com/tested-page/?nab=2 is variant C
  4. etc

which instructs our plugin the variant to load. The specific value in nab is generated using the nabAlternative cookie. For example, in a classic A/B test with only two variants, even values in nabAlternative produce nab=0, and odd values result in nab=1.

With this approach, each variant has its own unique URL, allowing almost every cache system to cache them independently.

Plugins that Need Customization

Here’s a non-exhaustive list of plugins that need customization to cache query arguments:

  • Breeze. Go to your WordPress Dashboard » Settings » Breeze and click on the Advanced Options settings tab. Then look for the Cache Query Strings section and add nab (source).
  • WP Rocket. Go to your WordPress Dashboard » Settings » WP Rocket and click on the Advanced Rules settings tab. Then look for the Cache Query String(s) section and add nab (source).

Drawbacks

Using query arguments has been Nelio’s default approach to loading alternative content for over a decade, proving to be an extremely effective and reliable solution for dealing with cache systems in WordPress. Unfortunately, it has its drawbacks.

Basically, every time a visitor lands on a tested page (https://example.com/tested-page/), our plugin has to perform a JavaScript redirection to send them to the appropriate URL, which is slower than serving the appropriate content right away.

The plugin has several mechanisms to reduce the number of redirections it has to perform, like preventing redirections if the visitor is supposed to see the control version, implementing CSS and JavaScript tests as inline test that require no redirection, or preloading query args in your links so that browsing from one page to another is as fast as possible, to name a few.

Nelio A/B Testing 7.0 (June 2024) added support for cache plugins that support dynamic caching. In other words, it’s now possible to load alternative content using the value in the nabAlternative cookie, making JavaScript redirections unnecessary.

In this section we share the recommended approach for the most common caching systems. But, before you use Dynamic Caching on your website, please read the following disclaimer:

  • All the information offered is provided as a reference and may have to be adapted to your specific site. We do not offer free support in relation to them. If you need assistance implementing them, you can avail of our paid support services.
  • Unless otherwise specified, if a caching plugin is described as “compatible,” such compatibility is based on a claim from the caching plugin’s authors. We verify and publish their claims when the plugin’s authors contact us, but things might have changed since then. If you need assistance with a caching plugin that is supposed to be compatible with Nelio A/B Testing and isn’t, please contact the authors of the caching plugin directly.
  • We strongly recommend to test the settings thoroughly before implementing them on a production site, as their use will be at your own risk.

If you do want to use cookie-based testing with dynamic caching, go to Nelio A/B Testing’s settings and check the option Load variants using nabAlternative cookie:

Screenshot of Nelio A/B Testing's settings
Look for the Variant Loading section and enable alternative content loading with cookies.

WordPress Caching Plugins

Breeze

Cookie-testing in Breeze is currently not available (we’ve contacted its authors).

W3 Total Cache

W3 Total Cache supports dynamic caching based on the value of a cookie. To enable it for Nelio A/B Testing, go to your WordPress Dashboard » Performance » Cache Groups, and scroll down to Manage Cookie Groups.

Create as many groups as possible values in the nabAlternative cookie (by default, it’s 24) using the Create Group button. Name each group as nabalternative_x and set its Cookies field to nabAlternative=x, as follows:

Screenshot of W3 Total Cache’s settings with two Cookie Groups for Nelio A/B Testing.

where x is, by default, a number from 0 to 23. Finally, add one additional group named nabalternative_none and set its Cookies field to nabAlternative=none:

Example of a cookie group to cache pages for visitors that don’t participate in any test (i.e. its participation cookie is set to "none").

Then go to Nelio A/B Testing » Settings and make sure that the following options in Variant Loading are checked:

Load variants using nabAlternative cookie
Run JavaScript redirection if cookie is missing

Finally, go back to Performance » Dashboard and click on Empty All Caches.

Important. If you were already using Cookie Groups in your setup, you may need to tweak the steps described above to suit your particular needs.

Rollback. If you want to go back to using Nelio’s default variant loading strategy, simply remove the groups you’ve created in W3 Total Cache, deactivate the Load variants using nabAlternative cookie setting in Nelio A/B Testing, and Empty All Caches in W3 Total Cache.

LiteSpeed Cache

LiteSpeed Cache also supports dynamic caching. To enable it in Nelio A/B Testing, go to our plugin’s Settings screen and enable the following two optins in the Variant Loading section:

Load variants using nabAlternative cookie
Run JavaScript redirection if cookie is missing

Then go to your WordPress Dashboard » LiteSpeed Cache » Cache and scroll down to find a setting named Drop Query String and include nab in it:

LiteSpeed Cache’s setting to drop certain query args
LiteSpeed Cache’s setting to drop certain query args.

Finally, Save Changes in LiteSpeed Cache’s settings screen and Purge All its caches using the menu in the top admin bar.

Rollback. If you want to go back to using Nelio’s default variant loading strategy, simply deactivate the Load variants using nabAlternative cookie setting in Nelio A/B Testing, remove the nab query arg from the list of droppable query strings in LiteSpeed Cache settings, and purge all caches.

WP Rocket

WP Rocket supports dynamic caching from the get-go. If you want to use it in Nelio A/B Testing, simply go to our plugin’s Settings screen and make sure the following option in Variant Loading is checked:

Load variants using nabAlternative cookie

Then go to your WordPress Dashboard » Settings » WP Rocket » Advanced Rules and make sure nab is not included in the option named Cache Query String(s).

Finally, all you gotta do is purge your cache: go to WP Rocket’s Dashboard and scroll down to find the Quick Actions section and click on the Clear and Preload Cache button.

Rollback. If you want to go back to using Nelio’s default variant loading strategy, simply deactivate the Load variants using nabAlternative cookie setting in Nelio A/B Testing, tweak WP Rocket’s settings to make nab a cacheable query arg again, and then purge your cache.

WP Super Cache

WP Super Cache can support dynamic caching if the appropriate setting is properly toggled. To enable dynamic caching in WP Super Cache and Nelio A/B Testing, please go to our plugin’s Settings screen and make sure the following options in Variant Loading are checked:

Load variants using nabAlternative cookie
Run JavaScript redirection if cookie is missing

Then, go to your WordPress Dashboard » Settings » WP Super Cache and click on the Advanced tab. Make sure Dynamic Caching is enabled:

WP Super Cache Advanced Settings Screen
Enable Dyanmic Caching in WP Super Cache’s Advanced Settings Screen.

Finally, go to the Contents tab and click on Delete Cache.

Rollback. If you want to go back to using Nelio’s default variant loading strategy, simply deactivate the Load variants using nabAlternative cookie setting in Nelio A/B Testing, deactivate Dynamic Caching in WP Super Cache’s settings, and delete its cache.

Caching Servers, Services, and CDNs

At the time of launching Nelio A/B Testing 7.0, we haven’t reviewed nor tested any caching mechanisms other than the WordPress plugins listed in the section above. If you’re using any of these, please contact us using the form below and we’ll add it in our development queue.