A/A Test Example where both variants are the same

Which version of the above pricing page you prefer? Can’t see the difference? Well, you’re right, there is none.

What you see is a good example of A/A Testing. In short, A/A Testing is an A/B Testing experiment where the two alternatives are in fact exactly the same.

Yes, I know, this looks kinda stupid. But we believe there are two good reasons to use A/A tests from time to time:

  • Checking your Split Testing tool. You’re relying on the results provided by your A/B testing tool (hopefully ours if you’re on WordPress) to make critical decisions about the look & feel, structure, navigation flow,… of your website. If the tool is not working as it should, it means that all these changes could be killing your conversion rate instead of improving it and make you lose lots of money. An A/A Test would help you to detect this. If the tool tells you that one variant in an A/A Test is significantly better than the other, well, it’s time to change
  • Reminder of the importance of a careful interpretation of the results. It may surprise you the results in an A/A test are not going to be exactly the same for both alternatives. Even if you’re showing exactly two clones of the same page to all your visitors, random effects will make that both clones have a slightly different conversion rate. Again, they can’t be significantly different but by no means they will be exactly equal. We’ve found this useful as a way to make our customers realize that they should not rush to select an alternative. As soon as a variant is slightly better we all feel the urge to stop the test (we don’t want to lose customers by showing the not-so-good alternative to them). Seeing that these rate differences also happen in an A/A test helps them to hold their horses in future tests. Remember, we may not like it but the underlying math in A/B Testing is there for a reason!

So, do you want to A/A Test us?

One response to “The importance of A/A Testing (no, not a typo!)”

  1. Georgi Avatar
    Georgi

    This is one of over 20 articles I’ve read on the topic and not understanding what the deal was with A/A testing (A/A/B, A/A/B/B etc.) I decided to share my thoughts on the issue from a statistical/hypothesis testing perspective. In general, I don’t think there is any merit in doing such tests. The results aren’t interesting cause they don’t tell us anything we don’t already know. My detailed response is here: http://blog.analytics-toolkit.com/2014/aa-aab-aabb-tests-cro/

    I see your point as far as “Reminder of the importance of a careful interpretation of the results” goes, but I would much prefer those people to read a bit more on sampling theory and statistical inference instead of relying on something shaky as an A/A test for that.

    Kind Regards,
    Georgi

Leave a Reply

Your email address will not be published. Required fields are marked *

I have read and agree to the Nelio Software Privacy Policy

Your personal data will be located on SiteGround and will be treated by Nelio Software with the sole purpose of publishing this comment here. The legitimation is carried out through your express consent. Contact us to access, rectify, limit, or delete your data.