Image of some medicine

I could have written this headline myself but I didn’t, I’ve taken it from this piece from Wired about treatments to prevent deaths from opioid overdose.

So overdose in the Wired article refers to the literal meaning of the word but I think we can agree to be living on a continuous an A/B Testing overdose. A/B testing is all around the media (together with all the “siblings” and name variants: split testing, multi-variate testing, multi-armed bandit testing algorithms, greedy testing algorithms,…). All kinds of websites, people and companies are talking about A/B Testing this, A/B testing that and the best A/B Testing strategies for you.

This is, in principle good, the more the people know what A/B Testing is and how they can use it for their own benefit the better.

The problem is that many of these information sources have no idea what they are talking about or just use the fact that A/B testing is trendy to attract visitors and “intoxicate” them. Take as an example the article from Wired. Regardless how much I love the magazine (I’m even a subscriber), the title is misleading. Whether on purpose (to make you pay more attention, i.e. something like “Alternative overdose treatments” is clearly less catchy than the original “A/B Testing overdose treatments“) or in blissful ignorance I don’t know, but if you read the article it says:

Two personal devices can deliver the lifesaving antidote, but users’ access depends on whether their overdose is illicit

As you can read, users are not assigned one of the two treatments at random but depending on whether they were taking legal or illega opioids. This violates a basic principle of A/B Testing making it impossible to compare the effectiveness of both treatments (one can work better than the other not because the treatment itself is better but because the users that are eligible to follow that treatment have another common characteristic that accounts for this) and we fall in the correlation vs causality trap.

So, yes please, join the A/B Testing movement. Just make sure you do it with the right partners!

Leave a Reply

Your email address will not be published. Required fields are marked *