One of the most interesting features of the premium version of Nelio A/B Testing is that you can segment the visitors who participate in an A/B test.
Your website visitors are not all the same. They come from different countries, speak different languages, use different browsers, access the site from different websites, use different devices, and have many other subtle and big differences.
So why A/B testing all visitors at once? The results that are derived from non-segmented A/B tests can provide illusory results that lead to biased actions.
Segmentation allows us to further refine the audience we want to understand. This is especially useful when you want only a part of your visitors to participate in an A/B test because you want to see how the changes work only for that specific target. You may be interested in testing the changes of an A/B test on your website only with visitors from a certain country, or only with those visitors who access at a certain specific time.
What is a segment?
A segment is a set of visitors with a series of common characteristics. We define these characteristics through segmentation rules.
In Nelio A/B Testing, when we create an A/B test we have the Segmentation section in which we can specify the visitor segments that will participate in the A/B test.
We can create as many segments as we want. To do this, we click on the left part of the segmentation section to create a new segment. This will allow us to have a new segment in which to add segmentation rules:
When you are in a segment, you can give it an explanatory name that defines it and start adding segmentation rules to it. Segmentation rules are added by clicking on the icon bar just below the segment. Each of these icons represents a category of segmentation rules.
By clicking on a category we see the different segmentation rules of that category. We select the one we want to add and this rule will appear within the segment in which we have added it.
In the previous image we see that within the visitor category we have several segmentation rules, which allow us, for example, to select visitors who are in a certain location. In the next section of this post you will see the details of all the segmentation rules included in Nelio A/B Testing.
When you add segmentation rules, you must take into account the following restrictions:
- You can only add one segmentation rule of each type within the same segment. This is a simplification that allows us to make the segments easier to understand.
- Given a segment with several segmentation rules, only those visitors who meet all the rules will be in the segment.
Keep the second point in mind when defining your segments. The combination of rules is mandatory for visitors belonging to the segment. If you create segments with rules that are too strict, it may take a long time to get results as the number of visitors that meet them all can be very small.
Available segmentation rules
Segmentation rules are conditions that visitors must meet to belong to the segment in which these rules are defined. Nelio A/B Testing includes a very complete set of different types of segmentation rules.
We will now list them and see how they can be used.
URL parameter
If we want to include in a segment the visitors who access our website with a specific parameter in the URL, we can do so using this rule.
When selecting the rule we must indicate the name of the parameter and the condition that we want to be fulfilled. Among the available conditions we have the following:
- Is equal to: we must type a value and the rule is fulfilled if the parameter has this value.
- Is not equal to: we must type a value and the rule is fulfilled if the parameter does not have that value.
- Contains: we must type a string of characters and the rule is met if the parameter contains that string in its value.
- Does not contain: we must type a string of characters and the rule is met if the parameter does not contain that string in its value.
- Exists: the rule is satisfied if the parameter exists, regardless of its value.
- Does not exist: the rule is satisfied if the parameter does not exist.
You can see the conditions in the following screenshot:
For example, if the URL of your page under test is https://mipage.com and you want only visitors who come to your page with the following URL https://mipage.com/?source=my-campaign, you should configure the rule as follows:
This way, only visitors who have the source parameter with the value my-campaign participate in the A/B test.
Referrer
The referrer is a value that tells us the URL from which the visitor came to our page when they first landed on it. Through this segmentation rule we can achieve that only visitors whose referrer meets certain conditions participate in the A/B test.
We need to select one of the conditions that we want the referrer to meet. In this case, the conditions are that the referrer value is equal or not to a value, or that the value contains or does not contain a string of characters.
If we want the visitors who participate in the A/B test to be those who come from Facebook, we can indicate that the condition is that the referrer contains the word “facebook”:
Taking into account that the referrer is a URL, we can use the condition we want and the value can be any URL from which we want visitors to come to our website (or a string of characters, as we have seen in the example above for “contains” or “does not contain” conditions).
Language
If you want only visitors whose browser is configured in a specific language participate in the A/B test, you have to use the language segmentation rule:
Among the conditions it includes, we can define to have the language equal to or different from the value indicated by us:
Note that this rule allows you to select more than one language in the language selector. Therefore, if you want only visitors who understand English and Spanish to participate in the A/B test, you will have to configure the rule like this:
Location
If you want only visitors who are in a specific geographic location to participate in an A/B test, you must use the location segmentation rule:
The location of the visitor is guessed from the IP that the visitor has. In this case, the rule condition may be that the location is the same or different from one or more specific locations:
If we want to target only visitors from California and New York, we will establish the rule as follows:
Keep in mind that the location selector allows multiple selection, so only visitors that belong to one of the locations in the selected list will fulfill the condition. If we have selected the opposite condition (visitors that do not belong to a location on the list), only visitors whose location is not on the list will participate in the A/B test.
User login
If we want to target only those visitors who are (or are not) logged in our website with a WordPress user, we will use this rule:
This way, if we select the condition that the visitor is not logged into our website, we get only not-logged-in visitors to participate in the A/B test. This is especially useful to prevent our team (authors, editors, administrators and others) from participating in the test.
Day of visit
We use this segmentation rule when we want only the visitors who visit the page under test on a specific day or set of days of the week to participate in the test.
We can set the condition to select visitors that land on a specific day or visitors that land on a day that is not in the list of days.
For example, if we want to target the visitors who land on our web on Friday or Saturday, we will establish the rule as follows:
If, on the other hand, we want the visitors who do not participate in the test to be those who come on Friday or Saturday, we would change the condition to indicate that the visiting day is neither of these two. Or we could leave the condition as is and select every day in the selector except Friday and Saturday.
Time of visit
We use this segmentation rule when we want only the visitors who visit the page under test in specific hours to participate in the test.
We can specify that the condition is that visitors arrive at specific times or that visitors arrive at different times than those that we have indicated in the list.
For example, if we want to get visitors who land on the web from 2 to 4 in the afternoon or from 4 to 6 in the afternoon, we establish the rule as follows:
If, on the contrary, we want the visitors who do not participate in the test to be those who come during those hours, we would change the condition to indicate that the visit time is not in either of these two periods.
Browser
We can target visitors who use a certain specific browser using this segmentation rule.
If we want the visitors who participate in the test to be those who use a specific browser, we will use the condition stating that “the browser is equal to”. On the contrary, if we want to target visitors who do not use a certain browser, we will use the opposite condition:
For example, if we want to target only visitors who use Chrome or Firefox, we will establish the rule as follows:
If we want to keep all the other visitors, we would change the condition so that only those visitors who do not have a Chrome or Firefox browser participate in the test.
Device type
We can target visitors who use a certain device type using this segmentation rule.
If we want the visitors who participate in the test to be those who use a specific device type, we will use the condition stating that “the type of device is equal to”. On the contrary, if we want to keep visitors who do not use a certain device type, we will use the opposite condition:
For example, if we want to target only visitors who use tablets or desktop computers, we will establish the rule as follows:
If we wanted to target all the other visitors, we would change the condition so that only those visitors who do not have a tablet or desktop device participate in the test.
Operating system
We can target visitors using a certain type of operating system using this segmentation rule.
If we want the visitors who participate in the test to be those who use a specific operating system, we will use the condition stating that “the operating system is equal to”. On the contrary, if we want to target visitors who do not use a certain type of operating system, we will use the opposite condition:
For example, if we want to target only visitors who use Mac OS X, we will establish the rule as follows:
If we wanted to keep all the other visitors, we would change the condition so that only those visitors who do not use a Mac OS X operating system participate in the test.
Window width
If we’re only interested in visitors who have a certain width in their specific browser window to participate in the A/B test, we can make use of this segmentation rule.
The rule has three conditions on the width of the browser window:
- that is greater than a certain number of pixels,
- that is less than a certain number of pixels,
- that the number of pixels is within a certain range.
For example, if we want to get visitors whose browser has a window width between 800 and 1200 pixels, we establish the rule as follows:
This rule is especially useful for testing changes that only affect a specific screen resolution and thus target those visitors that satisfy the width rule we want.
Cookie
If we want to include in a segment the visitors who access our website having a specific cookie, we can do so using this rule.
When selecting the rule we must indicate the name of the cookie and the condition that we want to be fulfilled. Among the available conditions we have the following:
- It is equal to: we must add a value and the rule is fulfilled if the cookie has this value.
- It is not equal to: we must add a value and the rule is satisfied if the cookie does not have that value.
- Contains: we must add a string of characters and the rule is satisfied if the cookie contains that string in its value.
- Does not contain: we must add a string of characters and the rule is satisfied if the cookie does not contain that string in its value.
- Exists: the rule is satisfied if the cookie exists, regardless of its value.
- Does not exist: the rule is satisfied if the cookie does not exist.
You can see the conditions in the following screenshot:
For example, if you want only visitors who come to your page and have the mycookie cookie to participate in the A/B test, you must configure the rule as follows:
Similarly, if we also want the value of the cookie to be a specific one or contain a certain string of characters, we can also do it by choosing the appropriate condition from those described above.
IP address
Finally, if we want only visitors who have a certain IP to be those who participate in the A/B test, we use the IP address segmentation rule.
First, we must indicate if the condition is that visitors have (or don’t have) that IP:
Once the condition is selected, we have a field available to add the IPs that we want to be in the list of IPs allowed or not allowed (depending on the chosen condition). Write the IP that you want to include and add it to the list by clicking the corresponding button.
In addition, we can remove IPs from the list by clicking on the icon that appears to the right of the specific IP in the list of IPs.
Results of an A/B test with segmentation
Once you have started an A/B test that includes segments, you will have the test results page available. In this view Nelio A/B Testing includes the segmentation section with the details of the defined segments.
Keep in mind that if there is more than one defined segment, Nelio A/B Testing will also show you an additional segment, called the Default Segment. This special segment includes the results of the union of all segments. That is, the results data from all the visitors who have participated in the test, either contributing to the results of one segment or another.
If you only have one segment defined, this will be the only one that appears in the segmentation section.
If you defined multiple segments, selecting one will change the results data shown on the results screen with the results of the visitors who have been part of that selected segment. This way, the results of the A/B test can be seen from this new perspective that segmentation provides, leaving you only with the results of a specific segment.