Old radio with the camera focused on the on/off button

We have recently launched Nelio Session Recordings, a new plugin for WordPress that allows you to collect the sessions of your website visitors, so that you can play them back later. This way, you can see exactly what they do (and what they don’t) as if you were sitting next to them when they browse your website.

Each session shows their mouse movements, clicks, how they scroll up and down your pages, whether they fill out the fields of a form, etc. That is, all the actions that visitors perform with their mouses and keyboards on your website.

Regarding the business model, Nelio Session Recordings can be purchased as a separate plugin or as an add-on for Nelio A/B Testing, our flagship plugin for A/B testing in WordPress. So today we are going to focus on the add-on version and how we sell it.

What we have done is to add a selector on the Nelio A/B Testing pricing page where the visitor can select whether or not they want to include the Nelio Session Recordings add-on when purchasing Nelio A/B Testing.

And then the question arises: should this selector be selected by default, or not? Do we want visitors to have to explicitly select whether they want Nelio Session Recordings, or do we prefer that they have to opt-out from this option? As you can imagine, this scenario is perfect to create an A/B test that helps us choose one of the two available solutions.

Creation of the A/B test

Having the option to buy an add-on checked by default may (or may not) increase sales of both the base product and the add-on. But we have to try both options to know if one of these is better than the other.

To create the A/B test we only have to go to the Nelio A/B Testing menu on our WordPress dashboard an select a new A/B test of pages. Next we will have a view like the one we can see in the following screenshot, in which we set up the test:

View of set-up for the A/B test where we test opt-in vs opt-out add-on selection.
View of set-up for the A/B test where we test opt-in vs opt-out add-on selection.

We select the pricing page that we are going to test. In our case, by default, we have decided that the pricing page has the add-on selector unchecked (opt-in). Then, we create a variant of this same pricing page and we can edit it to modify the behavior of the selector according to our previous hypothesis. That is, in the variant the selector will be checked by default, which means that the add-on is selected (opt-out).

You can see the differences between both versions in the following comparison:

Original version (opt-in behavior).
Alternative version (opt-out behavior).
Original (opt-in) vs alternative (opt-out) version.

Regarding the goals to be measured in the test, we define five:

  1. Clicks on the purchase button of any plan.
  2. Clicks on the purchase button of the basic plan.
  3. Clicks on the purchase button of the professional plan.
  4. Clicks on the purchase button of the enterprise plan.
  5. Actual purchases of any plan.

With all this, we are ready to start the test and see the results. Remember that you don’t have to do anything else on your end. The tool you use to do A/B testing (in our case, Nelio A/B Testing ) will be responsible for dividing the traffic that comes to your page so that one half sees the original version and the other sees the variation with the changes.

We now have everything ready so that the test can help us choose whether it is better (or not) to have the opt-in or opt-out behavior by default.

Nelio A/B Testing

Native Tests for WordPress

Use your WordPress page editor to create variants and run powerful tests with just a few clicks. No coding skills required.

Analysis of the results

We have left the test running for a month and a half. Let’s see the results for each of the goals in detail.

First of all, if we focus on the clicks that visitors make on the purchase buttons of any plan that appears on the pricing page, we see that the alternative version, which has the opt-out behavior, gets 5.6% fewer clicks.

Results of the A/B test from the point of view of clicks on any plan.
Results of the A/B test from the point of view of clicks on any plan.

However, the statistical confidence level does not reach any relevant value, so the difference between both versions is not statistically significant. That is, it looks like no one is better than the other. We will have to see the results of the other goals to make a decision on this matter.

If we go to look at the second goal, which only takes into account clicks to the cheapest plan, we see that the opposite happens here. The opt-in behavior gets 3% more clicks. However, the result isnot statistically significant either. There are no notable differences between both options.

Results of the A/B test from the point of view of clicks in the basic plan.
Results of the A/B test from the point of view of clicks in the basic plan.

If we look at the results in terms of clicks for the intermediate plan, the results are not conclusive either. Although the variation in the numbers here is greater, it is not enough to opt for one of the two analyzed samples of our visitors.

Of course, it seems that it is 33% worse to have the opt-out behavior if we want visitors to click on the buttons of the professional plan.

Results of the A/B test from the point of view of clicks in the professional plan.
Results of the A/B test from the point of view of clicks in the professional plan.

And this is something that happens again for the clicks to the most expensive plan of the three. In this case, the results are not significant either, although we get 18% fewer clicks to this plan if we have the opt-out behavior by default. But, as we’ve said, the differences in the data are too small to draw any strong conclusions.

Results of the A/B test from the point of view of clicks in the enterprise plan.
Results of the A/B test from the point of view of clicks in the enterprise plan.

Finally, if we look at the final sales we have had on this page, we can see that it seems that having the opt-out behavior has achieved 6% more sales.

But if we go to the overall numbers, we see that this represents only two more sales than the version without the add-on selected. Therefore, as we can see from the test results, there is no version that performs better than the other from a statistical point of view.

Results of the A/B test from the sales point of view.
Results of the A/B test from the sales point of view.

At this point, we cannot draw strong conclusions about either option. Apparently, for our audience it is not clearly better to have to opt-in or opt-out by default. This tells us that either of the two options behaves in a similar way.

With this A/B test we have determined that the add-on selector, which we thought would be quite important on our pricing page, is really not that relevant to visitors.

You may think that we have wasted time with it, but the truth is that thanks to this test we have been able to settle the discussion about whether we should do it one way or another, since the way in which we do it does not matter.

Furthermore, if we had not performed the test we would not be able to determine this, nor the opposite. And if one of the two options had been better than the other, we would have detected it thanks to the test and not personal opinions.

We will continue to analyze the data on this page in the coming months to verify that the trend continues and to test other changes and study them in a timely manner. And to you, dear reader, we recommend that if you have a website that you manage, you test it with an A/B test. Only then will you be able to find its best version or justify the changes you make with real data.

Featured Image by Gabriel on Unsplash.

2 responses to “Opt-in vs Opt-out Addons”

  1. Filipe Rocha Avatar
    Filipe Rocha

    It’s a bit confusing as in your text you say that the alternative version is the opt-out behavior but in the images it says it’s the optin behavior, you might want to look into that.

    1. Antonio Villegas Avatar

      Nice catch! It is a problem with the labels in the images. It should say “opt-out”. Let me fix the images…

Leave a Reply

Your email address will not be published. Required fields are marked *

I have read and agree to the Nelio Software Privacy Policy

Your personal data will be located on SiteGround and will be treated by Nelio Software with the sole purpose of publishing this comment here. The legitimation is carried out through your express consent. Contact us to access, rectify, limit, or delete your data.