Advertising platforms are not the only ones susceptible to this flawed way of thinking. Advertisers make the same error. They’re targeting personalised ads at an audience that is already very likely to buy their product. You watch a Renault commercial, and then your screen is taken over by Twingos. You put a dress in an online shopping basket, then it’s stalking you across the internet. You liked World of Warcraft , and now your timeline is full of Larp events (“OrcFest 2019: bring your battle axe!”).
But who knows, maybe you really would have bought that dress anyway, maybe you’ve had your eye on a Twingo for months, and perhaps you’ve just ordered a battle axe.
The majority of advertising companies feed their complex algorithms silos full of data even though the practice never delivers the desired result.
I had never really thought about this. Algorithmic targeting may be technologically ingenious, but if you’re targeting the wrong thing then it’s of no use to advertisers. Most advertising platforms can’t tell clients whether their algorithms are just putting fully-automated teenagers in the waiting area (increasing the selection effect) or whether they’re bringing in people who wouldn’t have come in otherwise (increasing the advertising effect).
"We are setting ourselves up for failure," Lewis explained, "because we are optimising for the wrong thing."
We currently assume that advertising companies always benefit from more data. And certainly, live-gaming is more likely to appeal to gamers. But the majority of advertising companies feed their complex algorithms silos full of data even though the practice never delivers the desired result. In the worst case, all that invasion of privacy can even lead to targeting the wrong group of people.
This insight is conspicuously absent from the debate about online privacy. At the moment, we don’t even know whether all this privacy violation works as advertised.
Run an experiment!
Luckily there is a way to measure the unadulterated effect of ads: do an experiment. Divide the target group into two random cohorts in advance: one group sees the ad, the other does not. Designing the experiment thus excludes the effects of selection.
Economists at Facebook conducted 15 experiments that showed the enormous impact of selection effects. A large retailer launched a Facebook campaign. Initially it was assumed that the retailer’s ad would only have to be shown 1,490 times before one person actually bought something. But the experiment revealed that many of those people would have shopped there anyway; only one in 14,300 found the webshop because of the ad. In other words, the selection effects were almost 10 times stronger than the advertising effect alone!
And this was no exception. Selection effects substantially outweighed advertising effects in most of these Facebook experiments. At its strongest, the selection bias was even 50 (!) times more influential.
In seven of the 15 Facebook experiments, advertising effects without selection effects were so small as to be statistically indistinguishable from zero.
Now we arrive at perhaps the most fundamental question: what, in the end, is there really to know in advertising? Can advertisers ever know exactly what their ad brings in?
Google CEO Eric Schmidt told his TV colleague Mel Karmazin that when it comes to online advertising, that question was easy to answer. Lewis went on to work for Schmidt, but research he conducted for Yahoo! in 2011 puts the lie to that claim. The title of his paper: On the near impossibility of measuring the returns to advertising.
Disappointment had been the study’s driving force. At Yahoo!, Lewis had run 25 gigantic ad-experiments. And still, he was left with a lot of uncertainty about the actual effects of advertising.
"People thought that after a one-million-person experiment, we could walk away, and know exactly how advertising works," Johnson recalled. He added, "if you’ve got a million you should be able to count angels dancing on pins."
So what went wrong? If you want to measure something small, you have to go big. Let’s say I want to know how many people have the rare disease cystic fibrosis. Cystic fibrosis affects one in 3,400 people (0.03%). But let’s say I don’t know that.
So I open the phone book and I call 10,000 people. Plus another 10,000. And another 10,000. Then another 10,000.
So you see, the results of my poll are all over the place. 10,000 is simply too small a sample to get reliable estimates. We’d better call a million people. And another million. And another million. Now we’re getting somewhere.
Imagine, then, that I had wanted to know how many people had contracted the flu last year (one in 20). Ten thousand calls would have been enough to get reliable estimates. More people get the flu, so a flu study can have smaller test groups.
The point is, advertising is like cystic fibrosis, not the flu. And even that’s extremely unfair to cystic fibrosis, since people buying things because they saw an ad is even rarer than cystic fibrosis.
0.0003% has Cystic fibrosis00.030.0800.050.11
Johnson sighs: "It’s very hard to change behaviour by showing people pictures and movies that they don’t want to look at."
To illustrate, consider Steve Tadelis’s eBay research. Ebay lost 63 cents on every dollar they put into Google search advertising, but that’s actually an imprecise estimate. If the experiment were to be replicated infinitely (and another ad stop, and another ad stop, and another ad stop...), in 95% of all ad stops the loss would fall in the range of negative $1.24 and negative $0.03. This is what statisticians call the confidence interval. In advertising research, the confidence interval tends to be huge.
EBay’s performance was so shoddy that the only logical conclusion would have been: stop buying search ads! But if eBay’s marketing had been just a tiny bit more effective – say they only lost 10 cents on every dollar they invested – then their experiment would have shown that the marketing department had delivered something between a 70 cent loss and a 50 cent profit.
Advertising does far less than most advertisers believe
What good is information like this? Such experiments tend to have an either-or conclusion: the campaign was either profitable or it wasn’t. This may give you a sense of direction, but it cannot provide certainty. "Simply rejecting that a campaign was a total waste of money is not an ambitious goal," Randall Lewis wrote in his study. Still, in practice, that proved ‘nearly impossible’.
Advertising rationally, the way it’s described in economic textbooks, is unattainable. Then how do advertisers know what they ought to pay for ads?
"Yeah, basically they don’t know," Lewis said in one of those throw-away clauses that kept running through my head for days after.
Keep that in mind the next time you read one of those calamity stories about Google, Facebook or Cambridge Analytica. If people were easier to manipulate with images and videos they don’t really want to see, economists would have a much easier task. Realistically, advertising does something, but only a small something – and at any rate it does far less than most advertisers believe.
"What frustrates me is there’s a bit of magical thinking here," Johnson says. "As if Cambridge Analytica has hacked our brains so that we’re going to be like lemmings and jump off cliffs. As if we are powerless.”
So we arrive at our final question: who wants to know the truth?
It’s a question that has long fascinated the economist Justin Rao (who’s worked for Yahoo!, Microsoft and others). Before he worked on advertising, he did field research with a cult that predicted the end of days on 21 May 2011. Rao awarded prizes to cult members. Those willing to accept their prize after Judgement Day – when the world would be annihilated and the faithful would ascend to heaven – were promised more money. Their belief in the apocalypse proved uncompromising. Even an extra 500 dollars couldn’t seduce the cultists.
"Beliefs formed on insufficient evidence seem tough to move," Rao wrote.
When Rao joined Microsoft, the eBay studies by Steve Tadelis had just been published and were all over the media: the Harvard Business Review, The Economist, The Atlantic and the BBC all covered the story. Marketing blogs couldn’t stop talking about it.
According to Rao, "Probably even Steve’s mother e-mailed him."
But did it matter? At Microsoft, Rao had a search engine at his disposal: Bing. Following the news about the millions of dollars eBay had wasted, brand keyword advertising only declined by 10%. The vast majority of businesses proved hell-bent on throwing away their money.
The fact that the eBay news did not even encourage advertisers to experiment more was perhaps the most striking.
Rao did observe the occasional ad stop at Bing. Rao was able to use ad stops like these, just as Tadelis had at eBay, to assess the effects on search traffic.
When these experiments showed that ads were utterly pointless, advertisers were not bothered in the slightest. They charged gaily ahead, buying ad after ad. Even when they knew, or could have known, that their ad campaigns were not very profitable, it had no impact on how they behaved.
"Beliefs formed on insufficient evidence seem tough to move."
#FitGirl analytics
Steve Tadelis saw this first-hand too. The financial director of eBay asked Tadelis to look into the second item on the list of so-called success campaigns: affiliate marketing. An example of this type of advertising could be eBay paying some influencer #fitgirl to embed a link to a particular brand of yoga pants in an Instagram post.
The affiliate marketing boss was okay with Tadelis experimenting, but he did issue a caveat. "Let me tell you something Steve," he had said. "If we run this experiment, and the results look like what you showed us with search advertising, I’m not going to believe you."
"It was clear to me that he meant it," Tadelis recalled. "So I told him: ‘Well, if this is about religion, I can’t help you. I have nothing against religion, I just don’t think it has a place in marketing analytics.’”
It might sound crazy, but companies are not equipped to assess whether their ad spending actually makes money. It is in the best interest of a firm like eBay to know whether its campaigns are profitable, but not so for eBay’s marketing department.
Its own interest is in securing the largest possible budget, which is much easier if you can demonstrate that what you do actually works. Within the marketing department, TV, print and digital compete with each other to show who’s more important, a dynamic that hardly promotes honest reporting.
The fact that management often has no idea how to interpret the numbers is not helpful either. The highest numbers win.
Randall Lewis told me about a meeting with the man responsible for evaluating Yahoo’s marketing strategy. The man had apparently done everything Lewis had advised against – and worse. He graciously admitted that he either added or omitted data to his model if it led to the ‘wrong’ results. Lewis: "I was like: oh man. All of that is bad scientific practice, but it’s actually great job preservation practice."
"Bad methodology makes everyone happy,” said David Reiley, who used to head Yahoo’s economics team and is now working for streaming service Pandora. "It will make the publisher happy. It will make the person who bought the media happy. It will make the boss of the person who bought the media happy. It will make the ad agency happy. Everybody can brag that they had a very successful campaign."
Marketers are often most successful at marketing their own marketing.
Is online advertising working? We simply don’t know
Perhaps what’s driving this phenomenon is something much more profound. Something that applies not just to advertising. "There is a fear that saying ‘I don’t know’ amounts to an admission of incompetence," Tadelis observed. "But ignorance is not incompetence, curiosity is not incompetence."
We want certainty. We used to find it in the Don Drapers of the world, the ones with the best one-liners up their sleeves. Today we look for certainty from data analysts who are supposed to just show us the numbers.
Lewis admitted that it’s not all bad. Decisions have to be made, somebody has to lay out a strategy, doubt must stop at some point. For that reason, companies hire overconfident people who act like they know what they cannot possibly know.
Lewis could never do the sort of work they do. "I would feel like it’s a random coin toss for most decisions," he said. But somebody has to toss the coin. And a company full of Randalls only leads to analysis paralysis. Nothing happens.
Randall Lewis had left Google and was working for Netflix when he attended the Datalead Conference in Paris in November 2015.
His time at Yahoo! and Google had taught him how difficult it is to advertise better. But Lewis wasn’t out to blind with science, he didn’t want to burn it all down and turn his back. He wanted to make the near impossible just a little bit more possible. And let’s be honest: advertising a bit better is actually quite a lot compared to stumbling about in the dark. It can prevent blunders of the eBay sort.
"Marketeers actually believe that their marketing works, even if it doesn’t."
Lewis had come to Paris to present one of his improvements. At Google, he had built a platform that gives advertisers a cheap and simple way to experiment with banner ads. "I think this is a revolution in advertising," he said proudly. Buyers could finally optimise for the right thing.
About a quarter of the way through his presentation, an audience member stood up and asked: do advertising companies actually want to know this? Aren’t they primarily interested in research that reassures?
"That’s actually endemic to the entire industry," Lewis replied. He started on one of his brilliantly inaccessible Lewisian responses. "The moral hazard problem is a series of cognitive dissonance biases …"
Halfway through his impenetrable answer, another audience member interruption. This time it came from Steve Tadelis. "What Randall is trying to say," the former eBay economist interjected, "is that marketeers actually believe that their marketing works, even if it doesn’t. Just like we believe our research is important, even if it isn’t."
Lewis laughed. "Thank you, Steve."
Jesse and Maurits worked together on this piece, but the story is told by Jesse. This article was first published in Dutch on De Correspondent. It was translated by Alana Gillespie.
