We Don’t Know As Much As We Think We Do About the Economy, Part 3: Don’t Ever Say “A Study Shows” To Prove Anything
I have been discussing what I learned in a podcast in which Jim Manzi cautions people to be very skeptical about the authority of conclusions reached by social scientists, including economists. The point he consistently makes is that, to have reliability, any phenomenon should be repeatable in different situations in different contexts, allowing people to make consistent and reliable non-obvious predictions. One important corollary is not to make too much of a single study — no matter how interesting the result might be, or how much it seems to confirm your pre-existing biases.
We all do this. Here’s a fun real-world example showing why it’s a bad idea.
Economic theory tends to hold that greater choice leads to greater demand and consumption — thus, if you want to sell more, offer more choices to your consumers. But Manzi tells the story of researchers who set out to test this. They ran an experiment in which they set up a table at a grocery store on two successive Saturdays. At the table, they had a selection of jams and jellies. One Saturday they had a selection of six different jams and jellies, and the other Saturday they had 24 varieties. On each day, they asked shoppers to taste their wares, and if they liked them, the shopper would be given a dollar coupon to redeem at the checkout counter to purchase one of the jams or jellies.
Conventional economic theory would hold that the day where the greater choice was available, a higher percentage of coupons would be redeemed. But the researchers found something counterintuitive and interesting. On the day where they had six different jams or jellies for purchase, fully 30% of the shoppers used a coupon to buy jams or jellies. On the day when they had 24 different varieties of jams or jellies, only 3% of shoppers redeemed the coupon.
Fascinating, huh? Reducing choice actually increased sales. You can easily see journalists writing an article that wows the public. Call the Freakonomics guys. Can’t you envision a section of a chapter talking about this surprising result, and discussing the likely reasons for it? Perhaps the shoppers were paralyzed by indecision when presented with so many choices. Valuable information, certainly, for any marketer to know.
Valuable — and almost certainly wrong.
Manzi says it was a mistake to draw sweeping conclusions from this single experiment. After all, look at how extreme the results are.
Can it really be true that all you have to do to increase sales by a factor of 10 is to remove 75% of your supply from your shelves? If this is the case, Manzi says, retailers everywhere are leaving “suitcases of money on the ground.” That is a HUGE effect — and frankly, so remarkable that it should raise a red flag.
The effect was so remarkable, in fact, that people have tried to replicate this experiment using other products, in other contexts, to see if the results can be repeated. And they can’t be. Manzi says that the experiments produuce a wide range of distributions — and that the general trend is that greater choice leads to greater demand.
Remember this the next time a single social science study comes to a conclusion you like. It’s tempting to cite such a study — we’re almost all guilty of this. But I am personally going to try to be aware of this in the future. Unless a phenomenon is proven through repeated observation in different contexts, allowing people to make repeatable, reliable, non-obvious predictions, the result of a social science study should always be viewed with immense skepticism.