I have been discussing what I learned in a podcast in which Jim Manzi cautions people to be very skeptical about the authority of conclusions reached by social scientists, including economists. The point he consistently makes is that, to have reliability, any phenomenon should be repeatable in different situations in different contexts, allowing people to make consistent and reliable non-obvious predictions. One important corollary is not to make too much of a single study — no matter how interesting the result might be, or how much it seems to confirm your pre-existing biases.

We all do this. Here’s a fun real-world example showing why it’s a bad idea.

Economic theory tends to hold that greater choice leads to greater demand and consumption — thus, if you want to sell more, offer more choices to your consumers. But Manzi tells the story of researchers who set out to test this. They ran an experiment in which they set up a table at a grocery store on two successive Saturdays. At the table, they had a selection of jams and jellies. One Saturday they had a selection of six different jams and jellies, and the other Saturday they had 24 varieties. On each day, they asked shoppers to taste their wares, and if they liked them, the shopper would be given a dollar coupon to redeem at the checkout counter to purchase one of the jams or jellies.

Conventional economic theory would hold that the day where the greater choice was available, a higher percentage of coupons would be redeemed. But the researchers found something counterintuitive and interesting. On the day where they had six different jams or jellies for purchase, fully 30% of the shoppers used a coupon to buy jams or jellies. On the day when they had 24 different varieties of jams or jellies, only 3% of shoppers redeemed the coupon.

Fascinating, huh? Reducing choice actually increased sales. You can easily see journalists writing an article that wows the public. Call the Freakonomics guys. Can’t you envision a section of a chapter talking about this surprising result, and discussing the likely reasons for it? Perhaps the shoppers were paralyzed by indecision when presented with so many choices. Valuable information, certainly, for any marketer to know.

Valuable — and almost certainly wrong.

Manzi says it was a mistake to draw sweeping conclusions from this single experiment. After all, look at how extreme the results are.

Can it really be true that all you have to do to increase sales by a factor of 10 is to remove 75% of your supply from your shelves? If this is the case, Manzi says, retailers everywhere are leaving “suitcases of money on the ground.” That is a HUGE effect — and frankly, so remarkable that it should raise a red flag.

The effect was so remarkable, in fact, that people have tried to replicate this experiment using other products, in other contexts, to see if the results can be repeated. And they can’t be. Manzi says that the experiments produuce a wide range of distributions — and that the general trend is that greater choice leads to greater demand.

Remember this the next time a single social science study comes to a conclusion you like. It’s tempting to cite such a study — we’re almost all guilty of this. But I am personally going to try to be aware of this in the future. Unless a phenomenon is proven through repeated observation in different contexts, allowing people to make repeatable, reliable, non-obvious predictions, the result of a social science study should always be viewed with immense skepticism.

I recommended this podcast in which Jmes Manzi argued that the confidence we have in social science predictions depends upon the ability of the “scientist” to make accurate predictions about the future that are non-obvious and can be repeated.

One phenomenon he discusses is the way people react in economics if their predictions go wrong. Manzi said that, before the the $820 stimulus was passed, he said he didn’t know whether it would work — but he did know that its supporters would say it worked regardless of the data:

[At the time I said:] I don’t believe any of the folks making these confident assertions really know what the effect will be. And the only prediction I’ll make is this: I’ll predict that, in early 2011, you know, professor, famous economist X said: unemployment will be about x%, say 10 percentage points without the bill and 8% with the bill. When it gets to be 2011, if unemployment is 10%, here’s what that professor is going to say: You know, conditions were worse than we thought they were; so without the bill unemployment would have been 12%, not 10%. Now unemployment is 10%. See, I was right all along; it lowered it by 2 points. And that’s exactly what happened, of course. That’s exactly what the economist said. And it has nothing to do with Democrats versus Republicans, by the way. If John McCain had been President, it would have been Republican advisors, too. And what I said is you cannot know the counterfactual reliably.

The other thing stimulus supporters say is: it would have worked better if it had been bigger.

In other words: if the data don’t prove your policy successful, you always say the situation was worse than you had realized, and the cure should have been more extreme.

This takes place in politics as well. If your candidate loses, one side will say it was because he wasn’t moderate enough, and he frightened off undecideds. The other side will say it was because he wasn’t principled/hardline enough, and he lost the base.

Each side will always draw the lessons they want to draw, and a plausible case can generally be made. But generalizing from specific instances is well-nigh impossible. Moderate candidate x might do great while moderate candidate y tanks; hardline candidate a might wow the electorate while hardline candidate b loses them. If winning elections were always about providing candidates closer or further from the center, winning elections would be easy.

In the economic context, our inability to predict the future with perfect accuracy leads people like me to believe we should have less government involvement, because it is often hubris to believe your particular intervention will have the desired effect on the economy. Far better to leave decisions to the collective expertise of society, which in the aggregate knows far more than any set of people in a room in Washington D.C., no matter how smart and well-informed they may be.

In the context of politics, our inability to predict the future leads people like me to suggest that candidates simply advocate what they believe. (Shocking suggestion, I know.) If you can’t be sure how to manipulate people, how about not trying to manipulate them at all?

And maybe — just maybe — your sincerity will actually win them over. Even if not, at least you don’t have to remember what your positions are supposed to be. You can just remember what they actually are.