Which version of the above pricing page you prefer? Can’t see the difference? Well, you’re right, there is none.
What you see is a good example of A/A Testing. In short, A/A Testing is an A/B Testing experiment where the two alternatives are in fact exactly the same.
Yes, I know, this looks kinda stupid. But we believe there are two good reasons to use A/A tests from time to time:
- Checking your Split Testing tool. You’re relying on the results provided by your A/B testing tool (hopefully ours if you’re on WordPress) to make critical decisions about the look & feel, structure, navigation flow,… of your website. If the tool is not working as it should, it means that all these changes could be killing your conversion rate instead of improving it and make you lose lots of money. An A/A Test would help you to detect this. If the tool tells you that one variant in an A/A Test is significantly better than the other, well, it’s time to change
- Reminder of the importance of a careful interpretation of the results. It may surprise you the results in an A/A test are not going to be exactly the same for both alternatives. Even if you’re showing exactly two clones of the same page to all your visitors, random effects will make that both clones have a slightly different conversion rate. Again, they can’t be significantly different but by no means they will be exactly equal. We’ve found this useful as a way to make our customers realize that they should not rush to select an alternative. As soon as a variant is slightly better we all feel the urge to stop the test (we don’t want to lose customers by showing the not-so-good alternative to them). Seeing that these rate differences also happen in an A/A test helps them to hold their horses in future tests. Remember, we may not like it but the underlying math in A/B Testing is there for a reason!
So, do you want to A/A Test us?
Leave a Reply