A/B Testing: Everything That Glitters Is Not Gold

The variable often missing from ad effectiveness measurement surveys.

by Joel Rubinson
President at Rubinson Partners Inc

Ask any academic – they will tell you that controlled testing is the gold standard for determining incrementality (such as an increase in sales caused by an advertising treatment).

I’d respond, “Yes, except when it’s not done properly, which is most of the time!”

Let’s talk about ad effectiveness measurement via surveys. Usually, for testing digital ad serving, a pixel is in the ad, it fires when the ad is served and pings the survey panel operator’s servers. Then, a survey invitation is sent. A control cell of respondents is created post-hoc by sending invitations to those matched demographically.

Sounds fairly straightforward, but here’s a problem I see all too often: the most important variable to control for is often missing from the study plan. What variable is this? Brand predisposition.

Start to control for brand predisposition

Consumers who are predisposed to a brand pay more attention to its advertising, click its links, consider it, buy it, are loyal to it, etc. Frankly, this characteristic, consumer predisposition, is the most predictive variable of whether or not a given consumer will convert in a campaign time window – more than whether or not they were exposed to an ad. Hence, if you don’t control for prior predisposition, you can get seriously inflated answers.

The reason the bias tends to go in one direction (overstatement of effects) is that marketers target ads to those who are more likely to be interested in their offering. As a result, the exposed cell will have a higher proportion of those predisposed to the brand versus the control cell, producing artificially higher conversion rates among the exposed (in addition to true incrementality) unless you explicitly control for this variable.

The problem doesn’t stop there. My white paper research proves that those pre-disposed to choosing a given brand are also five times more likely to respond to its advertising! There’s the double whammy: brand predisposition not only affects the baseline of conversions, but it also affects the responsiveness to advertising. For both reasons, if your exposed and control cells are not balanced on brand predisposition, you are likely to think your advertising campaign is working much better than it really is.

Adjust your results to reasonable reach levels

Now here’s a second adjustment you probably have to make that you might not have thought of. When you trigger survey invitations based on ad serving, your exposed cell is equivalent to 100% reach which never occurs in real life. Hence, the absolute magnitude of the lift is not reflective of what was spent on the ad campaign.